Some documents are made available in PDF format. You will need Adobe Reader 
to view them.
PDF Version [887 KB]
Slide 1
________________________________
slide notes:
Key Message: Introductory slide
Est. Presentation Time: Less than 1 minute (Total for entire module is estimated at 60-90 minutes)
Explanation of Cues/Builds: None
Suggested Comments: Let’s begin the Work Zone Safety Performance Measures training module. Performance measurement is becoming a major focus area at both the Federal and State level. Performance measures can help guide agencies in making effective decisions on the use of limited resources and in communicating agency performance, efforts, and needs to the public, elected officials, and other stakeholders. Understanding how and why to use performance measures is important to having a successful performance measurement program.
Suggested Questions: None
Additional Information: None
Possible Problems: None
_____________________________________________________________________________
Slide 2
Module Objectives
- Define performance measures
- Understand the rationale for using performance measures for work zone safety
- Apply the process and test for developing good performance measures
- Identify when and how to measure performance and what to do with the resulting data
________________________________
slide notes:
Key Message: Introduce Module Objectives
Est. Presentation Time: 1-2 minutes
Explanation of Cues/Builds: None
Suggested Comments: We are here to discuss the elements and methodologies that result in the development of effective performance measures and why performance measurement and performance data collection are important aspects of work zone safety. Performance measures can assist with evaluation of the performance of a work zone and of work zone policies, programs, and practices.
Suggested Questions: None
Additional Information: None
Possible Problems: None
____________________________________________________________________________
Slide 3
Work Zone Safety Performance Measures
- Work Zone Safety Performance Measures Guidance Booklet
- Module is based on information in this booklet
- Available at atssa.com
________________________________
slide notes:
Key Message: Introduce WZ Safety Performance Measures Guidance Booklet
Est. Presentation Time: Less than 1 minute(s)
Explanation of Cues/Builds: None
Suggested Comments: The purpose of this guidance booklet is to introduce the concept of performance measures and performance data collection for work zone safety.
Suggested Questions: None
Additional Information: None
Possible Problems: None
_____________________________________________________________________________
Slide 4
What are Performance Measures?
- Performance measures are sets of defined, outcomes-based conditions or response times that are used to evaluate success
- Performance measures focus on what to achieve, not how to achieve it
________________________________
slide notes:
Key Message: Define performance measures
Est. Presentation Time: 2 minutes
Explanation of Cues/Builds: None
Suggested Comments: Performance measures are a tool to help an organization and its customers and other stakeholders define and evaluate success. Performance measures define what to achieve, not how to achieve it. Measures should be tied to overall objectives for a project, policy, program, or practice.
Suggested Questions: None
Additional Information: None
Possible Problems: None
____________________________________________________________________________
Slide 5
Performance Measure Types/Formats
- Measures can be:
- Objective or subjective
- Detailed or high-level
- Multi-level (multiple defined levels of performance) or pass/fail
- Applied at the project level, the region/district level, the State/agency level, or at a policy level
________________________________
slide notes:
Key Message: Discuss the fact that performance measures come in different shapes and sizes
Est. Presentation Time: 2 minutes
Explanation of Cues/Builds: None
Suggested Comments: Performance measures can be objective, such as the additional measured travel time during construction, or subjective, such as whether motorists feel safe when driving through a work zone. Performance measures in some cases may be detailed, such as defining the amount of time the contractor has to clear a non-injury incident, whereas others are high-level key performance indicators, such as the number of citizen complaints received. Some performance measures include multiple defined levels of performance, and others are pass/fail. Performance measures may be applied at the project level, the State/agency level, or at a policy level.
Suggested Questions: None
Additional Information: None
Possible Problems: None
___________________________________________________________________________
Slide 6
Measures vs. Goals
- Performance Goal = the minimum acceptable level of performance for a particular performance measure
- Pass/Fail measure: Goal is to “Pass.”
- Multi-level measure:
- Several performance levels are defined
- One of the levels is designated as the “performance goal”
- Performance Measure = the set of “levels of performance” including the designated “goal”
________________________________
slide notes:
Key Message: Discuss the difference between performance measures and performance goals
Est. Presentation Time: 2 minutes
Explanation of Cues/Builds: None
Suggested Comments: The minimum acceptable level of performance for a particular measure is often referred to as the “performance standard” or “performance goal.” For a Pass/Fail measure, the “performance goal” is simply to “Pass.” For a Multi-level measure, there are several levels of performance defined, and one of the levels is designated as the “performance goal.” The set of “levels of performance,” including the “goal,” makes up the performance measure.
Suggested Questions: None
Additional Information: None
Possible Problems: None
_____________________________________________________________________________
Slide 7
Example Performance Measure
________________________________
slide notes:
Key Message: Provides an example of a multi-level performance measure for work zone safety
Est. Presentation Time: 1-2 minutes
Explanation of Cues/Builds: None
Suggested Comments: Using work zone crash rate as a measure of effectiveness, an example of a 5-level performance measure is shown. In this example, Level 4 (Good) is defined as the performance goal or the “passing” level. The full set of levels of performance here, including the designated goal, make up the performance measure. Another example of a multi-level performance measure is traffic level of service.
If you are using pass-fail performance measures then you would only have the performance goal, you would not have the additional levels of performance listed in the chart.
Suggested Questions: None
Additional Information:
This particular measure likely would apply to a long-term work zone; however, performance measures are not solely for long-term work zones. Even short-term temporary work zones can have major safety and traffic implications, and performance measures (such as maximum allowable queue length) can be generated and used to define acceptable performance.
The actual values for the various levels of performance need to be tailored to suit the locality and the specific project.
Possible Problems: None
____________________________________________________________________________
Slide 8
Additional Examples of Performance Goals
- Pass/fail — Incident Rate (IR) for Worker injuries is less than 4.0
- Subjective — Drivers feel safe traveling through work zones
- Surrogate/Congestion — No stopped queues in the work zone longer than ¼ mile
- Policy — Respond to all customer complaints within 24 hours
- State/Agency Level — No construction-related fatalities in work zones State-wide in 2010
________________________________
slide notes:
Key Message: Discuss examples of different types and levels of performance goals
Est. Presentation Time: 1-2 minutes
Explanation of Cues/Builds: None
Suggested Comments: An example of an objective, multi-level measures for a project was provided on the previous slide. These are examples of other types of performance goals. Some goals will result in quantitative results, while others may be more qualitative.
Surrogate and safety-related congestion measures can play an important role when the work zone is short in duration or when crash data may not be readily available. A number of congestion measures have a strong relation to safety. For example, the presence of stopped queues creates traffic shock-waves that can contribute to crashes when sight distance is limited.
The Incident Rate refers to the OSHA worker injury rate.
The subjective measure could be examined using driver intercept surveys or web surveys.
Suggested Questions: What are some other examples of safety-related performance goals that you can think of? Can you name another congestion-related goal that may affect safety?
Additional Information: None
Possible Problems: None
____________________________________________________________________________
Slide 9
Why Use Performance Measures for Work Zone Safety?
- Utilize resources more effectively
- Improve the quality of service to the public
- Document and communicate agency goals, priorities, and objectives
- Assess, document, and communicate agency performance and efforts currently and over time
- Strengthen accountability and enhance decision-making
________________________________
slide notes:
Key Message: Discuss reasons to use Work Zone safety performance measures
Est. Presentation Time: 1-2 minutes
Explanation of Cues/Builds: None
Suggested Comments: There are a number of reasons to use performance measures for work zone safety. Agencies and organizations can use performance measures and goals to define what to achieve, to drive innovation, and to improve accountability and transparency. It is important as we change practices and policies to know what we are trying to achieve and whether the changes were effective. Assessing performance can also produce information on the benefits and costs of improved quality of service as well as lessons learned. These lessons can help agencies improve on practices and better mitigate traffic impacts on future projects.
Suggested Questions: If being presented in a workshop or class environment, suggest an exercise to involve participants by asking for everything the class can think of related to work zone safety that could be measured and tracked. Congestion items should be encouraged too as long as the participants can describe the safety tie.
Additional Information: None
Possible Problems: None
____________________________________________________________________________
Slide 10
Why Use Performance Measures for Work Zone Safety?
- Assess the effectiveness of new and existing policies, practices, or procedures, allowing comparisons to agency benchmarks
- Identify unacceptable or unsafe situations that need to be corrected
- Justify incentive/disincentive payments or penalties
- Validate recognition and awards for employees
________________________________
slide notes:
Key Message: Discuss reasons to use Work Zone safety performance measures
Est. Presentation Time: 1-2 minutes
Explanation of Cues/Builds: None
Suggested Comments: Assessing performance can also provide quantitative data that can be used to help drive improved employee performance that can be recognized through awards. The performance information can also be used as a fair, quantitative source of data for assigning incentive and disincentive payments to contractors to encourage excellent performance and discourage poor performance.
Suggested Questions: How many people in this course are familiar with FHWA’s Work Zone Self Assessment? The WZSA is a tool to assess agency performance and improve on many areas of work zone policy, planning, design, and operations through self-rated questions. The tool measures performance over a baseline and can be used as an analogy to performance measurement for work zone safety.
Additional Information: None
Possible Problems: None
____________________________________________________________________________
Slide 11
Sources for Performance Measures and Goals
- Existing agency goals
- Brainstorming/working sessions
- Goals/measures from other agencies/contracts
- Common industry standards
- Subject matter experts
- Research (such as NCHRP Report 627)
________________________________
slide notes:
Key Message: Provide additional sources for work zone performance goals and measures
Est. Presentation Time: 1-2 minutes
Explanation of Cues/Builds: None
Suggested Comments: Some suggested considerations for sources of performance measure ideas are listed here. Review of literature, discussions with other stakeholders, and results from research reports can be beneficial in determining performance measures and goals for your situation.
Suggested Questions: Does your agency have internal/ external subject matter experts?
Additional Information: None
Possible Problems: None
____________________________________________________________________________
Slide 12
To Develop Performance Measures…
- Identify stakeholders and invite them to participate in the goal-setting exercise
- Identify any existing agency performance measures
________________________________
slide notes:
Key Message: Discuss steps to follow in developing performance measures
Est. Presentation Time: 2-3 minutes
Explanation of Cues/Builds: None
Suggested Comments: Developing performance measures and performance goals is a group exercise and an iterative process. For the group exercise, stakeholders may include both traditional and non-traditional partners, and will vary depending on whether you are setting policy, region/district, State/agency-level, or project level goals. The agency may already have goals or policies in place (such as no work zone delays greater than 20 minutes) that you will want to be aware of and incorporate as appropriate into the performance measures.
Suggested Questions: Who would you invite to a goal-setting exercise for a project-level goal? (Possible answers: project engineer, upper management, regional TMC, contracts/procurement office, fiscal/accounting office, FHWA Division Office (if Federal-Aid), emergency service providers (fire, police, EMS), transit providers, freight movers, utilities, local Chamber of Commerce, schools, a representative from the contracting industry, consultants.)
Additional Information: None
Possible Problems: None
_____________________________________________________________________________
Slide 13
To Develop Performance Measures…
- Hold initial brainstorming session to define draft performance goals:
- What are the stakeholders trying to achieve?
- Focus on what to achieve not how to achieve it
________________________________
slide notes:
Key Message: Discuss the role of brainstorming sessions
Est. Presentation Time: 1-2 minutes
Explanation of Cues/Builds: None
Suggested Comments: Initial brainstorming sessions are an effective way to draft initial performance goals. During brainstorming, the focus should be on what the results look like, not what the process looks like.
Suggested Questions: If being presented in a workshop or class environment, take the list of measurable, trackable items related to safety in work zones that was developed in slide 9 and invite participants to suggest a set of goals for that list of items.
Additional Information: None
Possible Problems: None
___________________________________________________________________________
Slide 14
To Develop Performance Measures…
- Craft each idea into a clear, well-written performance goal
- Consider surrogate measures when what you’re really trying to measure is not available
- Make decisions on format–pass/fail or multi-level, detailed or high-level, subjective or objective–as you go
- Organize/categorize your measures
- Test, refine, and finalize your measures
- Keep asking, “What have we missed?”
________________________________
slide notes:
Key Message: Discuss the process to follow in developing performance measures
Est. Presentation Time: 1-2 minutes
Explanation of Cues/Builds: None
Suggested Comments: Develop draft performance goals during the brainstorming session(s). If using a pass/fail approach, then focus on refining the goals. If using a multi-level approach, then the group will need to define the multiple levels of performance that will make up the performance measure. Subsequently, determine appropriate data sources and refine the goals and measures based on the feasibility of data and how well you think you will be able to measure performance given the data sources and their accuracy. By asking yourselves, “What have we missed?” several times throughout the process, you are more likely to capture items that may have otherwise been forgotten.
Suggested Questions: None
Additional Information: More information on this process can be found in FHWA’s Performance Contracting Framework document. The relevant section is available online at: http://www.fhwa.dot.gov/hfl/framework/06.cfm
Possible Problems: None
____________________________________________________________________________
Slide 15
What Makes a Good Performance Goal? — “Am I SMART”?
- I — Is the goal under the Influence of the party responsible for meeting the performance goal?
- S — Is the performance goal Specific?
- M — Is the performance goal Measurable?
- A — Is the performance goal Achievable?
- R — Is the performance goal Results-oriented?
- T — Does the performance goal have a Time element as to when it needs to be met?
________________________________
slide notes:
Key Message: Discuss the “Am I SMART” Test for determining if a performance goal is a good one
Est. Presentation Time: 1-2 minutes
Explanation of Cues/Builds: None
Suggested Comments: There are a number of characteristics that make a performance goal good. A simple test is to ask the following question for each draft performance goal — “Am I SMART?” If the answer is yes to each of these questions for a draft performance goal, then the performance goal is likely a good one. If the answer is no, consider revising or deleting it. By applying this simple test, the probability of ending up with usable, successful, and reasonable goals improves greatly.
Suggested Questions: What are some other factors that make a good goal?
Additional Information: In addition to the ISMART criteria, there are a number of additional considerations to determining if a performance goal is a good one. Other considerations include determining if the performance measure/performance goal is:
Consistent with agency needs and focused on what really matters to the agency, to the workers, and to the motorists/public.
Practical to apply and readily observed.
Currently being measured.
Reasonable to measure from a cost/resources standpoint.
Possible Problems: None
____________________________________________________________________________
Slide 16
Example 1 — Project Level Goal
- You have assembled your stakeholders and have held your initial brainstorming session
- The project is a reconstruction of a 5.56 mile segment of rural highway
- One of the goals that the stakeholder group identified was, “Improve safety”
- The group also decided that they want to make their goals pass/fail, detailed, and objective
________________________________
slide notes:
Key Message: Introduce Example 1 — a project level example
Est. Presentation Time: 1-2 minutes
Explanation of Cues/Builds: None
Suggested Comments: This example comes from an actual brainstorming exercise and project in Michigan. It provides an example of going from a good general concept to a defined goal that was included in a real contract. The brainstorming group was made up of a broad group of functional areas and levels within the agency.
Suggested Questions: Is “improve safety” a good goal?
Additional Information: None
Possible Problems: None
____________________________________________________________________________
Slide 17
Example 1 — Testing the Goal
- You lead the group through the ISMART test on “Improve safety”
- I (Influence) — Maybe. It depends on to whom the goal is assigned
- S (Specific) — No. More details are needed
- M (Measurable) — Maybe. More details are needed
- A (Achievable) — Maybe. More details are needed
- R (Results-Oriented) — Yes
- T (Time Element) — No
- It appears that more work is needed on this draft goal
________________________________
slide notes:
Key Message: Discuss testing the “improve safety” goal to see if it is a good goal
Est. Presentation Time: 1-2 minutes
Explanation of Cues/Builds: None
Suggested Comments: The lack of detail in the initial goal would make it challenging to implement and to determine if it was achieved. There also needs to be a time element defined so that we will know how long we have to meet the goal.
Suggested Questions: What details could we add to the goal to make it more specific?
Additional Information: None
Possible Problems: None
____________________________________________________________________________
Slide 18
Example 1 — Refining the Goal
- You lead the group through a discussion to refine the goal to make it pass “ISMART”
- The resulting pass-fail, detailed, objective, SMART goal is:
- Maintain the pre-construction average crash rate of no more than 1.0 crashes per month on the 5.56 mile section of roadway for the duration for the project
- The group feels that this goal is under the influence of the contractor and assigns it to the contractor through a performance contract
________________________________
slide notes:
Key Message: Discuss refining the draft goal
Est. Presentation Time: 1-2 minutes
Explanation of Cues/Builds: None
Suggested Comments: In this case, the group added more specifics to the goal, defined the zone that the goal covers and added a time element. The agency decided that this goal was under the influence of the contractor and included it in the construction contract.
Suggested Questions: None
Additional Information: None
Possible Problems: None
_____________________________________________________________________________
Slide 19
Example 2 — Agency Level Goal
- At a press-conference following a tragic work zone fatal crash, the Governor states, “We will be examining our processes and policies, and our goal will be to have zero work zone fatalities in 2011”
- As Chief Engineer, it has now become your responsibility to implement and achieve this goal
________________________________
slide notes:
Key Message: Introduce Example 2 — an agency-level example
Est. Presentation Time: 1-2 minutes
Explanation of Cues/Builds: None
Suggested Comments: Not all goals come from brainstorming exercises, and not all goals are coordinated before being released or stated.
Suggested Questions: Have you ever had a goal generated by an elected official?
Additional Information: None
Possible Problems: None
____________________________________________________________________________
Slide 20
Example 2 — Testing the Goal
- You run through the ISMART test on “Have zero work zone fatalities in 2011”
- I (Influence) — Maybe. Some fatalities are not preventable
- S (Specific) — Yes
- M (Measurable) — Yes
- A (Achievable) — Maybe. Some fatalities are not preventable
- R (Results-Oriented) — Yes
- T (Time Element) — Yes
________________________________
slide notes:
Key Message: Discuss testing the “Have zero work zone fatalities in 2011” goal to see if it is a good goal
Est. Presentation Time: 1-2 minutes
Explanation of Cues/Builds: None
Suggested Comments: Overall the goal is fairly good. The main issue is whether the agency has true influence over the goal and whether the goal is achievable given that some crashes may be unrelated to the work zone (for example — impaired drivers, etc.).
Suggested Questions: How could we change the goal to improve the influence and achievability factors?
Additional Information: None
Possible Problems: None
____________________________________________________________________________
Slide 21
Example 2 — Refining the Goal
- Based on the “ISMART” test, you recommend to the Secretary that the goal be changed to “Have zero construction-related work zone fatalities in 2011″ to improve the ability to influence the outcome
- The Secretary agrees, and proposes the change to the Governor, but the Governor rejects the change
- Thus, the goal stands as “Have zero work zone fatalities in 2011“
________________________________
slide notes:
Key Message: Discuss refining the draft goal
Est. Presentation Time: 1-2 minutes
Explanation of Cues/Builds: None
Suggested Comments: The unfortunate reality is that we can’t always have the opportunity to change/alter the goals that we are assigned.
Suggested Questions: None
Additional Information: None
Possible Problems: None
____________________________________________________________________________
Slide 22
Methodology
- Defines:
- How to measure performance
- When to measure performance
- Who will measure performance
- Develop the methodology to suit your agency, project, and situation
- Develop the methodology up front and test it in the field for practicality before finalizing it and the performance measures/goals
________________________________
slide notes:
Key Message: Discuss the need to define a methodology that will suit your agency, the contractor (if pay is tied to the measure), and the specific project and test it to make sure that it is practical
Est. Presentation Time: 1-2 minutes
Explanation of Cues/Builds: None
Suggested Comments: It is important to test your performance measurement methodology in the field to ensure that it can be performed with the planned/available resources, and that it will not endanger the evaluators. This is a good time to make any necessary adjustments before finalizing.
Suggested Questions: None
Additional Information: None
Possible Problems: None
____________________________________________________________________________
Slide 23
The Performance Measurement Methodology Includes:
| 1. Specific Measure of Effectiveness 2. Unit of Measure 3. Measurement Method |
How |
| 4. Frequency and Timing | When |
| 5. Evaluator | Who |
________________________________
slide notes:
Key Message: Introduce the elements that make up the performance measurement methodology and map them to how, when, and who
Est. Presentation Time: 1-2 minutes
Explanation of Cues/Builds: None
Suggested Comments: It is important to define how performance will be measured for each performance goal. This slide and the following slides describe 5 components that are important to defining the measurement methodology for a performance measure/goal.
It is important to define and document each of these components so that all stakeholders will have a mutual understanding of how and when performance will be measured.
Suggested Questions: None
Additional Information: These elements are key if a contractor will be responsible for meeting the performance goals, and especially so if performance will be tied to incentive/disincentive fees.
Possible Problems: None
____________________________________________________________________________
Slide 24
How To Measure Performance
- Develop a methodology for assessing each performance measure that defines:
- The specific measure of effectiveness
________________________________
slide notes:
Key Message: Discuss defining the specific measure of effectiveness
Est. Presentation Time: 2 minutes
Explanation of Cues/Builds: None
Suggested Comments: The measure of effectiveness is usually included in the performance goal itself. The boxes provide the measures of effectiveness for the two example goals introduced earlier.
Suggested Questions: None
Additional Information: None
Possible Problems: None
____________________________________________________________________________
Slide 25
How To Measure Performance
- Develop a methodology for assessing each performance measure that defines:
- The unit of measure
________________________________
slide notes:
Key Message: Discuss defining the unit of measure/size of a single sample
Est. Presentation Time: 2 minutes
Explanation of Cues/Builds: None
Suggested Comments: The boxes provide potential units of measure for the two example goals introduced earlier.
Suggested Questions: None
Additional Information: None
Possible Problems: None
_____________________________________________________________________________
Slide 26
- Develop a methodology for assessing each performance measure that defines:
- Process or test for evaluating performance (measurement method)
________________________________
slide notes:
Key Message: Discuss defining the measurement method
Est. Presentation Time: 2 minutes
Explanation of Cues/Builds: None
Suggested Comments: The boxes provide example measurement methods for the two example goals introduced earlier.
Suggested Questions: None
Additional Information: None
Possible Problems: None
___________________________________________________________________________
Slide 27
When to Measure Performance
- Number of Options:
- Continuously — Traffic Volumes and delays
- Daily — Maximum queue buildups, compliance with traffic control plans
- End of project — Crash frequencies
- Annually — Policy/State-level goals
________________________________
slide notes:
Key Message: Discuss when to measure performance
Est. Presentation Time: 1-2 minutes
Explanation of Cues/Builds: None
Suggested Comments: There are a number of options of when to measure performance. When to measure should be defined for each performance goal and depends on the measure, the situation (e.g., project characteristics) and the resources (time, equipment, personnel, funding) available for measurement .
Suggested Questions: None
Additional Information: None
Possible Problems: None
___________________________________________________________________________
Slide 28
When To Measure Performance
- Develop a methodology for assessing each performance measure that defines:
- Frequency and timing
________________________________
slide notes:
Key Message: Discuss the frequency and timing of performance measurement
Est. Presentation Time: 2 minutes
Explanation of Cues/Builds: None
Suggested Comments: The boxes define when and how often performance will be measured for the two example goals introduced earlier.
For longer frequency and timing periods, such as end of project or end of year, spot measures during the project or year may be helpful to gauge how things are going and whether any changes to the strategies that have been implemented to help achieve the goal are necessary or desired. For example #2, if you are finding that the number of WZ fatalities are already beyond your target at the mid-year point, you will likely want to develop strategies and a revised action plan to keep it from getting worse. You may not want to wait until the end of the year only to discover that your strategies have not been effective.
Suggested Questions: None
Additional Information: None
Possible Problems: None
____________________________________________________________________________
Slide 29
Who Will Measure Performance
- Number of Options:
- Agency — Inspector, Project Engineer, Central Office
- Contractor
- Third Independent Party
________________________________
slide notes:
Key Message: Discuss who will measure performance
Est. Presentation Time: 1-2 minutes
Explanation of Cues/Builds: None
Suggested Comments: There are a number of options when it comes to defining who will measure performance against the goal. Each option has its advantages and challenges in terms of cost, bias, and efficiency.
Suggested Questions: What would some of the advantages and challenges be of having a contractor evaluate their own performance?
Additional Information: None
Possible Problems: None
___________________________________________________________________________
Slide 30
Who Will Measure Performance
- Develop a methodology for assessing each performance measure that defines:
- Responsible party for measuring performance (Evaluator)
________________________________
slide notes:
Key Message: Discuss defining who will measure performance
Est. Presentation Time: 2 minutes
Explanation of Cues/Builds: None
Suggested Comments: The boxes define who would evaluate performance for the two example goals introduced earlier.
Suggested Questions: None
Additional Information: None
Possible Problems: None
___________________________________________________________________________
Slide 31
Summary — Example 1
- Draft Goal: Improve Safety
- Final Goal: Maintain the pre-construction average crash rate of no more than 1.0 crashes per month on the 5.56 mile section of roadway for the duration for the project
- Measurement Methodology:
| Measure of Effectiveness | Unit of Measure | Measurement Method | Frequency and Timing | Evaluator |
|---|---|---|---|---|
| Crash Rate on the 5.56 Mile Section | Average crashes per month from actual construction start date to project final acceptance date | Obtain number of crashes from the State crash database. Compute average monthly crash rate by dividing number of crashes by the construction duration in months | Once at end of project | State DOT Project Engineer |
________________________________
slide notes:
Key Message: Summarize Example 1 — project level goal
Est. Presentation Time: 1-2 minutes
Explanation of Cues/Builds: None
Suggested Comments: This is what is looks like when the components are put together.
Suggested Questions: None
Additional Information: None
Possible Problems: None
___________________________________________________________________________
Slide 32
Summary — Example 2
- Draft Goal: Have zero work zone fatalities in 2011
- Final Goal: Have zero work zone fatalities in 2011
- Measurement Methodology:
| Measure of Effectiveness | Unit of Measure | Measurement Method | Frequency and Timing | Evaluator |
|---|---|---|---|---|
| Number of work zone fatalities | Work zone fatalities statewide in 2011 | Obtain number of work zone fatalities from the State crash database | Once at end of project | State DOT Chief Engineer or Designee |
________________________________
slide notes:
Key Message: Summarize Example 2 — agency level goal
Est. Presentation Time: 1-2 minutes
Explanation of Cues/Builds: None
Suggested Comments: This is what is looks like when the components are put together.
Suggested Questions: None
Additional Information: None
Possible Problems: None
__________________________________________________________________________
Slide 33
Data Considerations
________________________________
slide notes:
Key Message: Transition to Data Considerations Est. Presentation Time: 1-2 minutes Explanation of Cues/Builds: None Suggested Comments: Now, we’ll talk about some of the data considerations for work zone safety performance measures and goals3333. Suggested Questions: None Additional Information: None Possible Problems: None
__________________________________________________________________________
Slide 34
What Kind of Data to Collect
- Safety — work zone crash rates, worker injury rates, subjective ratings, surrogate measures
- Construction congestion — queue length, incident clearance time, travel time/delay, subjective ratings
- Customer satisfaction — number/frequency of complaints, subjective ratings
________________________________
slide notes:
Key Message: Discuss types of data by category/area
Est. Presentation Time: 1-2 minutes
Explanation of Cues/Builds: None
Suggested Comments: Different types of data are used to measure performance in different categories. You will want to collect data that directly feeds the measures of effectiveness in your performance goals and associated measurement methodology.
This slide shows a list of categories and examples of the types of data that can be used. It should be noted that work zone crash data can take some time to obtain. If this data is collected in the project log, or if surrogate measures are used, that time can be reduced.
Construction congestion data can provide good information for safety evaluations.
Suggested Questions: None
Additional Information: None
Possible Problems: None
___________________________________________________________________________
Slide 35
Data Collection Techniques
- Scale data collection efforts to relative size/impact of the project
- Use objective national standard methods and tests where appropriate, applicable, and available
- Use on-site inspectors/inspections as appropriate
________________________________
slide notes:
Key Message: Discuss considerations for data collection
Est. Presentation Time: 1-2 minutes
Explanation of Cues/Builds: None
Suggested Comments: The first consideration must be safety of the data collectors. There are a number of other data collection issues and techniques to consider as well, including those on this and the following slides. Using personnel who are already on-site (such as construction inspectors) can be a cost effective and safe option.
Suggested Questions: None
Additional Information: None
Possible Problems: None
___________________________________________________________________________
Slide 36
Data Collection Techniques
- When possible, use data that are already being collected and are readily available
- Permanent count stations
- Logs and observations from on-site personnel
- Use automated data collection where available
________________________________
slide notes:
Key Message: Discuss considerations for data collection
Est. Presentation Time: 1-2 minutes
Explanation of Cues/Builds: None
Suggested Comments: Data sources that are readily available for use may provide the simplest and least resource-intensive information to measure performance. But measures should not be developed simply because data already exist to monitor the measure. It needs to be a good measure.
For example, traffic data may be available from count stations. Practitioners should sample this data to understand its format, complexity, and accuracy to determine how well it will support evaluation of the measures.
When using automated equipment, remember to check and calibrate the equipment regularly.
Suggested Questions: None
Additional Information: Minnesota DOT has a website that provides public access to volume, speed, occupancy, and density information on freeways throughout the Minneapolis area. Practitioners can download data from the site to assist with determining the types of measures that can be studied. Other locations may benefit from the metrics that MnDOT uses and how the data relate.
Possible Problems: None
__________________________________________________________________________
Slide 37
Data Collection Techniques
- Understand any potential issues with data reporting that might impact your results
- Ensure all data collected are tied to the measurement methodology
- Use statistical power analysis to determine the amount of data needed
- Investigate the availability of published statistics for baseline data
- Consider using surrogate measures if crash data is not readily available
________________________________
slide notes:
Key Message: Discuss considerations for data collection
Est. Presentation Time: 1-2 minutes
Explanation of Cues/Builds: None
Suggested Comments: An example of a potential issue is the timeliness of crash data from police reports. Since there is often a lag time in obtaining this data, an agency may have to develop a special way to get the reports sooner, get crash information from project log books, or use a surrogate measure.
Statisticians, in-house data analysts, consultants, and university researchers might be resources to assist with understanding the types of statistical needs and associated considerations when measuring performance. Power analysis will allow you to estimate the number of data samples needed for statistical confidence that your sample data will adequately represent the population of data. Published statistics on work zone crash rates can be found in several sources. A good example is http://www.workzonesafety.org/crash_data/state_data.
Suggested Questions: None
Additional Information: Use surrogate measures (that you are confident reflect safety), or drop the measure altogether, if the primary measure is not practical to use, or is otherwise not measurable, unsafe to measure, too costly to measure, etc. Examples of surrogate measures may include damaged traffic control devices or the number of traffic control devices that have to be replaced.
Possible Problems: None
___________________________________________________________________________
Slide 38
Data Collection Considerations
- Minimize interference with the construction contractor
- Work zones are dynamic environments; data collection techniques also may need to be dynamic
________________________________
slide notes:
Key Message: Discuss considerations for data collection
Est. Presentation Time: 1-2 minutes
Explanation of Cues/Builds: None
Suggested Comments: Focus resources on fewer measures and fewer data sources if needed to get better results as opposed to a larger pool of measures with limited resources for each. Before finalizing the set of performance measures for use, it is important to test them against existing/typical conditions and outcomes to ensure that they are reasonable and achievable. Whatever you decide to do, be sure to be fair and reasonable.
Suggested Questions: None
Additional Information: None
Possible Problems: None
____________________________________________________________________________
Slide 39
How the Data Can Be Used
- Project-level results may require immediate action by the responsible party
- For example, if work is causing excessive delays on a particular day, the work may need to be shut down or altered immediately
- Agency-level results may demonstrate a need for proactive adjustments for future projects
- For example, if construction-related work zone fatalities are occurring, a new policy may be needed for application on all future projects
________________________________
slide notes:
Key Message: Discuss short and long-term uses for the data collected and analyzed
Est. Presentation Time: 2 minutes
Explanation of Cues/Builds: None
Suggested Comments: When you measure performance you end up with large amounts of data and scoring results. The challenge is to use these results for making decisions or taking action. The performance measurement team should carefully plan a strategy by interested audience level for summarizing the results and presenting them in a way that will be well received by that level. Different audiences will be interested in different levels of information. For example, project-level personnel will need detailed results for each performance measure (and possibly each sample), whereas upper level management may only be interested in seeing summary scores or receiving a quick briefing on how things are going.
Suggested Questions: None
Additional Information: It is recommended that this strategy include periodic reports that summarize performance. To help identify trends, the reports should compare the results against the results for previous periods or projects. These reports should be actively discussed with action plans generated as needed.
Possible Problems: None
____________________________________________________________________________
Slide 40
Lessons Learned and Review
________________________________
slide notes:
Key Message: Transition to lessons learned
Est. Presentation Time: 1-2 minutes
Explanation of Cues/Builds: None
Suggested Comments: Let’s discuss some lessons learned from real-world applications of performance measures and goals.
Suggested Questions: None
Additional Information: None
Possible Problems: None
____________________________________________________________________________
Slide 41
Lessons Learned
- The performance measures/goals MUST be under the influence of the party responsible for meeting them
- Don’t start from scratch. Listen to others’ experiences
- Make sure that you cover everything that you’re trying to achieve
- Focus first on what to achieve, not how to achieve it
________________________________
slide notes:
Key Message: Discuss lessons and tips for developing performance measures and goals
Est. Presentation Time: 1-2 minutes
Explanation of Cues/Builds: None
Suggested Comments: These lessons have been learned from real-world applications of performance measures. If the measure/goal is not under the influence of the responsible party, then it will strictly be by chance whether the goal is met. Taking advantage of others’ experiences and materials can give you a head start on developing and implementing your measures and can help you to avoid obstacles and pitfalls. Try not to leave any “holes” in your performance measures — if they don’t cover everything that you want to achieve, then you won’t likely be fully satisfied with the outcomes. Engineers have a tendency to focus on “how”. It is a harder exercise to focus on the desired outcome (or what to achieve), but this is what is necessary in developing and implementing performance measures. Keep the group focused on “what”.
Suggested Questions: None
Additional Information: None
Possible Problems: None
___________________________________________________________________________
Slide 42
Lessons Learned
- Include time-response performance goals as appropriate
- Multi-level measures are more informative than pass/fail measures (Did we just fail, or did we fail badly?)
- When nearing completion ask, “What have we missed?”
- Test your goals in the field
________________________________
slide notes:
Key Message: Discuss lessons and tips for developing performance measures and goals
Est. Presentation Time: 1-2 minutes
Explanation of Cues/Builds: None
Suggested Comments: These lessons have been learned from real-world applications of performance measures. Including time-response performance goals will help to ensure that deficiencies are taken care of in a timely manner and won’t linger. Multi-level measures tell you where in the spectrum of performance you fell, not just whether you passed or failed. While multi-level measures can be more time-intensive to develop, they provide useful information from a management perspective. Covering areas that you have missed before the performance measures are finalized prevents frustration and change orders/modifications further down the road and should result in a higher level of satisfaction with the end product.
Testing your goals and measurement methodologies in the field before finalizing them can help to identify issues early on so that you can make any necessary adjustments.
Suggested Questions: None
Additional Information: None
Possible Problems: None
__________________________________________________________________________
Slide 43
Lessons Learned
- Remember, this is an iterative process
- Get reviews/approval/buy-in from affected offices and stakeholders
- Clearly define what it means to meet the performance goals
- Consider how a performance goal will be measured or evaluated — define it up front
- Focus resources on fewer measures and data sources for better results
________________________________
slide notes:
Key Message: Discuss lessons and tips for developing performance measures and goals
Est. Presentation Time: 1-2 minutes
Explanation of Cues/Builds: None
Suggested Comments: These lessons have been learned from real-world applications of performance measures. Stakeholder groups rarely come up with the final set of performance measures on the first try. It usually takes several iterations to get all of the ideas down and to get them formulated into good performance measures and goals. Getting buy-in as you go along will help to ensure a smooth implementation and should reduce resistance. Defining whether the performance goals need to be met on average, absolutely, or for a certain percentage of the time is important and needs to be defined along with how and when performance will be measured. This can greatly reduce frustration and confusion.
Suggested Questions: None
Additional Information: None
Possible Problems: None
_____________________________________________________________________________
Slide 44
Review of Module Objectives
- Define performance measures
- Understand the rationale for using performance measures for work zone safety
- Apply the process and test for developing good performance measures
- Identify when and how to measure performance and what to do with the resulting data
________________________________
slide notes:
Key Message: Review and discuss Module Objectives
Est. Presentation Time: Less than 1 minute
Explanation of Cues/Builds: None
Suggested Comments: Let’s review the module and discuss any questions that you might have.
Suggested Questions: None
Additional Information: None
Possible Problems: None
____________________________________________________________________________
Slide 45
Additional Resources
- Performance Contracting Framework
- The FHWA Work Zone Safety and Mobility Performance Measurement Website
- Implementing the Rule on Work Zone Safety and Mobility
- The National Work Zone Safety Information Clearinghouse
________________________________
slide notes:
Key Message: Discuss resources for additional information
Est. Presentation Time: Less than 1 minute
Explanation of Cues/Builds: None
Suggested Comments: Here are some additional resources on work zone performance measures.
Suggested Questions: None
Additional Information: None
Possible Problems: None
____________________________________________________________________________
Slide 46
Questions?
________________________________
slide notes:
Key Message: Closing Slide
Est. Presentation Time: Less than 1 minute
Explanation of Cues/Builds: None
Suggested Comments: Let’s discuss any questions that you might have.
Suggested Questions: What are some examples of performance measures/goals that you have used in your agency/organization?
Additional Information: None
Possible Problems: None
Posted: 3/2/2011