Work Zone Safety Performance Measures Guidance Booklet
Some documents are made available in PDF format. You will need Adobe Reader 
to view them.
PDF Version [467 KB]
Purpose
The purpose of this guidance booklet is to introduce the concept of performance measures and performance data collection for work zone safety. The booklet:
- Defines performance measures.
- Discusses the rationale for using performance measures for work zone safety.
- Provides a process and test for developing good performance measures.
- Provides sample performance measures for work zones.
- Discusses when and how to measure performance, including types of data and data collection techniques, and what to do with the resulting data.
- Provides real-world lessons learned.
- Provides guidance on where to go for more information.
What are Performance Measures?
Performance measures are sets of defined, outcome-based conditions or response times that are used to evaluate success. Performance measures focus on what to achieve, not how to achieve it. Measures come in all different shapes and sizes. Measures can be:
- Objective (e.g., additional measured travel time during construction) or subjective (e.g., motorists feel safe driving through a work zone).
- Detailed (e.g., the amount of time the contractor has to clear a non-injury incident) or high-level key performance indicators (e.g., the number of customer complaints received).
- Multiple defined levels of performance or pass/fail.
- Applied at the Project level, the region/district level, the State/agency level, or at a policy level.
The minimum acceptable level of performance for a particular measure is often referred to as the "performance standard" or "performance goal." For a pass/fail measure, the "performance goal" is simply to "pass." For a multi-level measure, there are several performance levels defined and one of the levels is designated as the "performance goal." The set of "levels of performance," including the "goal," makes up the performance measure.
Using work zone crash rate as a measure of effectiveness, an example of a five-level performance measure1 is shown below. In this example, level 4 (good) is defined as the performance goal or the "passing" level. For this example, the levels of performance are set up as threshold values that specify the different levels of performance. Each level of performance could also be set up as a range of values.
The values shown in the example are for sample purposes only. The actual values for the various levels of performance need to be tailored to suit the specific situation to which they are being applied. Situations vary based on:
- The scope of the measure:
- Whether the measure is being applied at the project level, region/district level, the state/agency level, or at a policy level.
- Whether the measure is for a short-term or long-term situation.
- The units for the measure.
- The traffic and physical environment in which the measure is being applied.
- How the measure is being used.
- Who the audience for the measure is (e.g., agency, elected officials, the public).
In the example, this particular measure likely would apply to a long-term work zone. However, performance measures are not solely for long-term work zones. Even short-term temporary work zones can have major safety and traffic implications, and performance measures (such as maximum allowable queue length) can be generated and used to define acceptable performance. While the sample measure would most likely be applied at the project level, it could be aggregated to the region/district or State/agency level by doing a frequency plot of how many project crash rates fell within each of the five levels.
With regard to the units for the measure, it should be noted that crash rates can be measured and reported using a variety of measures of exposure. Examples of measures of exposure include million vehicle miles traveled, the number of vehicles traveling through the work zone, time-based measures, etc. To use a measure of crash rate, the agency needs to agree on which measure of exposure will be used. It may also be necessary or desirable to convert crashes to crash costs to account for severity and use a measure such as crash costs per hour of work zone per mile.
Why Use Performance Measures for Work Zone Safety?
Agencies and organizations use performance measures and goals to define what to achieve, to drive innovation, and to improve accountability and transparency. Performance measures and their associated performance goals can help agencies:
- Utilize resources more effectively.
- Improve the quality of service provided to the public.
- Document and communicate agency goals, priorities, and objectives.
- Assess, document, and communicate agency performance and efforts currently and over time.
- Strengthen accountability and enhance decision-making.
- Assess the effectiveness of new and existing policies, practices, or procedures, allowing comparisons to agency benchmarks.
- Identify unacceptable or unsafe situations that need correction at the project level.
- Justify incentive/disincentive payment or penalties.
- Validate recognition and awards for employees.
How Do I Develop Performance Measures?
Developing performance measures and performance goals is a group exercise and an iterative process. Steps to developing performance measures include:
1. Identify the stakeholders and invite them to participate in the goal-setting exercise. These stakeholders may include both traditional and non-traditional partners, and will vary depending on whether you are setting policy or establishing State/agency-level, region/district-level, or project level goals. Non-traditional partners are those stakeholders who are not typically part of the discussion about work zone safety, but who may have perspectives or inputs that should be considered. For example, if an agency is developing project-level measures for work zones near schools, areas of heavy transit use, or central business districts, it would be beneficial to invite a representative from the school administration, transit provider, or chamber of commerce, respectively.
2. Identify any currently existing agency performance measures. These may be in the form of goals or policies set by the current administration (e.g., maximum work zone delay of 20 minutes), acceptance criteria in the standard specs, etc.
3. Hold initial brainstorming sessions to define draft performance goals. Once the first two steps have been taken, it will be time for the group to identify a set of draft performance goals. As can be seen in What Makes a Good Performance Goal (see text box), there are a number of characteristics that make a performance goal good, and the group should keep these in mind as work proceeds. A brainstorming session, led by a facilitator, is a good way to approach this step. This process for the stakeholder group should be broken down into a series of steps:
- Ask the group of stakeholders to define what they are trying to achieve. The initial conversation-sparking questions will vary depending on whether you are setting project-level, region/district-level, state/agency-level goals, or policy goals. The first run-through will typically be rough ideas. It is important to capture these ideas on a flip chart or presentation slide without worrying too much initially about format. The group's focus should be on what to achieve rather than how to achieve it.
- After each idea has been identified, work with the stakeholders to craft the ideas into good, well-written performance goals. This will be an iterative process and may take several sessions. This process may also result in several of the initial ideas being combined or discarded. The facilitator can encourage participation with pre-planned questions.
- Make decisions on format as you go — pass/fail or multilevel, detailed or high-level, subjective or objective. The decisions usually become apparent through the brainstorming discussions on the goals.
- If the group decides on multi-level performance measures, work to establish the other levels of performance to go along with the performance goal. The combined levels of performance will need to cover any situation that the evaluation team might see in the field.
- Keep asking, "What have we missed?" During and between subsequent sessions, group members will often think of areas that have not been covered. It is important to capture these ideas and develop goals for them as appropriate.
4. Organize and categorize your performance measures. Once the group has a draft set of measures, consider categorizing them by benefit category such as safety, construction congestion, quality, user satisfaction, etc. Another way is to categorize by infrastructure type. You may need to generate new categories for measures that do not fit into established categories.
5. Test, refine, and finalize your performance measures. Before finalizing the set of performance measures, it is important to test them against existing or typical conditions or outcomes to ensure that they are reasonable and achievable. This process can be broken up into a set of secondary activities that include:
- Establishing the baseline – what are we achieving now, and how does that compare to our performance measures and goals? This can be determined through an initial data collection exercise. When dealing with project-level measures, it is also important to baseline the existing traffic and safety conditions to ensure that the performance goal is reasonable. Updating the baseline after drafting the initial set of performance measures avoids unnecessary data collection and reclassification.
- Testing your performance measures in the field if applicable. This step helps to determine measurability and needed evaluation resources (personnel and equipment) in the field.
- Refining and finalizing your performance measures. If during the baseline comparison and field testing process a performance goal is found to be unreasonable or difficult to measure, the stakeholder group should consider refining the goal or eliminating that particular measure.
More information on this process can be found in FHWA's Performance Contracting Framework document. The relevant section is available online at: http://www.fhwa.dot.gov/hfl/framework/06.cfm
What Makes a Good Performance Goal?
There are a number of characteristics that make a performance goal good.2 A simple test is to ask the following question for each potential draft performance goal – "Am I SMART?" This acrostic is explained as follows:
I – Is the goal under the Influence of the party responsible for meeting the performance goal?
S – Is the performance goal Specific?
M – Is the performance goal Measurable?
A – Is the performance goal Achievable?
R – Is the performance goal Results-oriented?
T – Does the performance goal have a Time element as to when it needs to be met?
If the answer to each of these questions is yes, then the draft performance goal is likely a good one. If the answer is no, consider revising or deleting it. Other key considerations include determining if the performance measure/performance goal is:
- Consistent with agency needs and focused on what really matters to the agency, to the workers, and to the motorists/public.
- Practical to apply and readily observed.
- Currently being measured.
- Reasonable to measure from a cost/resources standpoint.
2 For more detailed information, see the "Performance Goals" chapter in FHWA's Performance Contracting, which can be viewed at: http://www.fhwa.dot.gov/hfl/framework/06.cfm
Sample Performance Measures
Sample performance measures/goals can help provide a starting point for discussion when you are trying to develop performance measures/goals for your specific project or agency. The FHWA Highways for LIFE program developed a sample set of performance measures and goals for construction work zones as part of the Performance Contracting Framework, and a sampling of these are shown in Table 1 below. For each of these performance measures, the performance goal is considered the "4-Good" level of performance. A number of these measures may also be applicable to maintenance work zones.
Note that the performance measures in Table 1 are samples only and use sample values. The actual performance measures and values used would need to be tailored to an agency's specific needs and situation in coordination with guidance from FHWA. The examples do, however, provide a good starting point for an agency's discussion on performance measures. Other sources of performance measures and goals include:
- Existing agency goals.
- Goals/measures from other agencies/contracts.
- Common industry standards.
- Brainstorming/working sessions.
- Subject matter experts.
- Research (such as NCHRP Report 627).
The samples in Table 1 are all objective, multi-level, project-level measures/goals. There are other types/styles/levels of performance goals as discussed earlier. Examples of other types of goals include:
- Pass/fail – Incident Rate (IR) for Worker injuries is less than 4.0.
- Subjective – Drivers feel safe traveling through the work zone.
- Policy – We will respond to all customer complaints within 24 hours.
- State/agency level – No construction-related fatalities in our work zones state-wide in 2010.
Each performance measure in Table 1 can be used stand-alone as a pass/fail performance goal, where the "4-Good" Level in the table is the performance goal.
| Category | Element | PM# | 5 – Excellent | 4 – Good | 3 – Fair | 2 – Poor | 1 – Very Poor | Notes |
|---|---|---|---|---|---|---|---|---|
| Safety1 | Injuries – (workers) contractor / subcontractors on site personnel, government representatives, consultant, vendors, delivery personnel | 1 | Incident Rate (IR) for worker injuries is less than 2.0 | Incident Rate (IR) for worker injuries is less than 4.0 | Incident Rate (IR) for Worker injuries is less than 5.5 | Incident Rate (IR) for Worker injuries is less than 6.5 | Incident Rate (IR) for Worker injuries is greater than or equal to 6.5 | a. Each State Agency shall establish a target Incident Rate to be used for each project based on local and Statewide available Incident Rate data analysis. The shown Incident Rates are provided as an example. |
| Safety1 | Vehicular Crashes | 2 | Site Crash Rate during construction divided by the Crash Rate prior to construction is less than 1.0 | Site Crash Rate during construction divided by the Crash Rate prior to construction is equal to 1.0 | Site Crash Rate during construction divided by the Crash Rate prior to construction is less than 1.2 | Site Crash Rate during construction divided by the Crash Rate prior to construction is less than 1.3 | Site Crash Rate during construction divided by the Crash Rate prior to construction is equal to or greater than 1.3 |
|
| Construction Congestion | Travel time/delay during construction | 3 | No motorist delay (as compared to reconstruction travel time) | Rural: Average motorist delay less than 15 minutes (as compared to preconstruction travel time) Urban: Average motorist delay less than 20 minutes (as compared to preconstruction travel time) | Rural: Average motorist delay less than 20 minutes (as compared to preconstruction travel time) Urban: Average motorist delay less than 30 minutes (as compared to preconstruction travel time) | Rural: Average motorist delay less than 30 minutes (as compared to preconstruction travel time) Urban: Average motorist delay less than 45 minutes (as compared to preconstruction travel time) | Rural: Average motorist delay ≥ 30 minutes (as compared to preconstruction travel time) Urban: Average motorist delay ≥ 45 minutes (as compared to pre-construction travel time) |
|
| Construction Congestion | Queue Length During Construction | 4 | No queue | No stopped queue (speed less than 10 mph) | Rural: < ½ mile stopped queue (speed less than 10 mph) Urban: < 1 mile stopped queue (speed less than 10 mph) | Rural: < 1 mile stopped queue (speed less than 10 mph) Urban: < 2 mile stopped queue (speed less than 10 mph) | Rural: ≥ 1 mile stopped queue (speed less than 10 mph) Urban: ≥ 2 mile stopped queue (speed less than 10 mph) |
|
| 1 See: http://www.fhwa.dot.gov/hfl/framework/06.cfm This Web site also includes sample performance measures for other categories. Note: These measures/goals/values are shown as an example starting point for consideration and would need to be tailored to the project/agency. |
||||||||
When and How Do I Measure Performance?
As previously stated, the minimum acceptable level of performance for a particular measure is often referred to as the "performance standard" or "performance goal." If you set performance goals, you need to measure performance against those goals to determine to what extent they were met. It is very important to develop a measurement methodology for each performance measure. Elements of this methodology include:
- The specific measure of effectiveness for the performance measure.
- The unit of measure.
- The process or test for measuring performance (measurement method).
- The frequency and timing for performance measurement.
- The responsible party for measuring performance (evaluator).
An example measurement methodology for the initial sample project-level performance measure presented earlier is shown below:
| Performance Goal | Measure of Effectiveness | Unit of Measure | Measurement Method | Frequency and Timing | Evaluator |
|---|---|---|---|---|---|
| Work zone crash rate equal to preconstruction crash rate | Work zone crash rate | Crashes per month (other options are crashes per 100,000 VMT, crashes per 100,000 vehicles through the work zone) | Compute the average number of crashes per month for a 3-year period prior to construction. During construction, record the number of crashes for each month and compare that to the pre-construction rate. | End of each month during the construction period | State DOT Project Engineer |
For this example, a 100 percent sample of the data would be used. However, a 100 percent data sample is not always practical or desired for all performance measures/goals. The performance evaluation team will need to determine a sampling strategy that will both provide statistical confidence in the results and fit the available budget and resources.
In terms of the frequency and timing of performance assessments, performance needs to be measured frequently enough to provide good information flow, but not so frequently as to be burdensome from a resource standpoint or to interfere with construction activities. Frequency and timing will depend on the specific performance measure, the length of the project (or time period covered), and the resources available (both personnel and equipment). It will be up to the agency to determine the optimum frequency and timing for their specific situation. Some examples include:
- For crash-related measures on shorter-term projects, measuring performance one time at the end of the project may be most appropriate. For longer term projects, once per month, or once per year may be appropriate.
- For some measures such as delay, continuous measurements are possible using advanced technologies. These measurements can provide information in a timely manner to field and supervisory personnel so that adjustments can be made quickly when needed.
- For State/agency/policy-level performance goals, summary statistics at the end of the year or just prior to the planning/scheduling process for next year may be appropriate.
Types of Data
As can be seen in Table 1, different types of data are used to measure performance in different categories. Table 2 provides a sample list of types of data (and example data sources) that can be used to determine whether performance goals are met.
| Performance Measure Category | Types of Data | Example Data Sources |
|---|---|---|
| Safety | Work zone crash rates | · Police Reports · Project Log Books |
| Safety | Worker injury rates | · OSHA reports |
| Safety | Subjective rating of safety | · Driver surveys |
| Construction Congestion | Queue length | · Sensors · Observation at site · Cameras/video |
| Construction Congestion | Incident clearance time | · Project log books · Video · Sensors |
| Construction Congestion | Travel time/delay | · Pilot vehicles · License-plate matching systems |
| Construction Congestion | Subjective rating of delay/congestion | · Driver surveys |
| Customer Satisfaction | Number/frequency of complaints | · Call center call volumes · Website/emails |
| Customer Satisfaction | Subjective ratings of satisfaction | · Public surveys |
It is important to consider data availability when considering which type of data to use. Data that are available in a timely manner and that are collectable with a reasonable amount of resources (or that you already have access to) are generally best.
Data Collection Techniques
In developing the measurement methodology, it is important to think about the data collection techniques that you will use to gather the data needed to assess the measures. The first consideration must be safety of the data collectors. Other helpful considerations include:
- Scale your data collection efforts to the relative size/impact of the project.
- Use objective national standard methods and tests where appropriate, applicable, and available.
- When possible, use data that are already being collected and are readily available (e.g., from permanent traffic counter stations).
- Use automated data collection where appropriate, and remember that this equipment needs to be checked and calibrated throughout the effort.
- Ensure that all data collected are tied to the measurement methodology.
- Understand any potential issues with data reporting (e.g., timeliness of availability of crash data) that might impact your results.
- Use statistical power analysis to determine the amount of data that you will need to obtain statistically significant results.
- Investigate the availability of published statistics, such as average crash rates or work zone fatalities, for similar projects.
- Crash data can be hard to obtain in a timely and accurate manner — consider the possibility of using surrogate measures (that you are confident reflect safety) if the primary measure is not practical to use, or is otherwise not measurable, unsafe to measure, or too costly to measure, etc. An example of a surrogate measure would be the frequency of "close calls" evaluated using video analysis.
- Design your data collection technique to minimize interference with the construction contractor's work efforts and workspace.
- Remember that work zones are dynamic environments and that your data collection techniques may need to change with different phases of the project.
The types of data collection equipment needed vary based on the performance goals, the performance measurement budget, the layout of the work zone, and whether permanent data collection infrastructure is already available in the area of interest. There is a wide variety of equipment available for measuring traffic conditions, including counters, RTMS sensors, cameras, full ITS systems, and personal observation logs. As an example, the project team could measure delay through pilot car runs at random times through the work zone or by using a license plate matching system. Crashes can be tracked using a project log as well as through police reports to ensure immediate availability of the data.
There is no "one-size-fits-all" for a measurement methodology. If it is impractical to apply performance measures at a project or single work-zone level, performance measures can be used at a program level or a State/agency level to measure performance for items such as the number and severity of work zone-related crashes over the past year.
You should develop the performance measurement methodology to suit your specific agency, project, and situation. It is recommended that you develop this methodology up front and test it in the field for practicality before finalizing it and the performance measures. Guidance on developing the measurement methodology is available in the report Performance Contracting Framework, and is available online at http://www.fhwa.dot.gov/hfl/framework/07.cfm.
What Do I Do With the Data and Results?
When you measure performance you end up with large amounts of data and scoring results. The challenge is to use these results for making decisions or taking action. Project-level results may require immediate action by the responsible party (for example, if a maximum queue length in the work zone is exceeded or a high frequency of crashes indicates a traffic control plan may not be working well); agency-level results may demonstrate a need for proactive adjustments for future projects; region/district or program-level results may include changes during the planning of future projects; and State/agency- or policy-level results may reveal whether policy adjustments are needed. The data and results will also be useful for agencies in determining whether their work zone policy goals and objectives are being met. The bottom line is that you will need to have a system in place that will get the performance data and results into the hands of the people that need them in a timely and effective manner.
The audiences for the data and results can range from project-level personnel, to District-level management, to central-office management, to elected/appointed officials and the traveling public. These different audiences will be interested in different levels of information. For example, project-level personnel will need detailed results for each performance measure (and possibly each sample), whereas upper-level management may only be interested in seeing summary scores or receiving a quick briefing on how things are going.
The performance measurement team should carefully plan a strategy, organized by interested audience, for summarizing the results and presenting them in a way that will be well received by that audience. It is recommended that this strategy include periodic reports that summarize performance and compare the results against the results for previous periods or projects to help identify trends. These reports should be actively discussed with action plans generated as needed, because it is important to examine problems and failures to try to determine the cause and possible actions to solve the problem or to avoid similar problems in the future. It is also important to be fair by recognizing and examining the successes as well as the problems/failures.
Performance data and results will provide you with valuable baseline data for impacts assessment and future project planning as well as for contractor incentives and disincentives at the project level. They can also provide a basis for employee recognition and award programs on individual projects or agency districts, etc.
Lessons Learned
The lessons below have been learned from real-world applications of performance measures:
- The performance measures/goals MUST be under the influence of the party responsible for meeting them.
- Take advantage of the experience of others – do not start from scratch.
- When developing performance goals, make sure that you cover everything that you are trying to achieve.
- Focus on what to achieve, not how to achieve it.
- Include time-response performance goals as appropriate.
- Multi-level measures are more informative than pass/fail measures (did we just fail, or did we fail badly?).
- Test the goals in the field before finalizing them.
- When nearing completion of the draft set of measures/goals, it is important to sit back and think, "What have we missed?"
- Defining performance measures/goals is an iterative process.
- Getting reviews/approval/buy-in from the offices and stakeholders that will be impacted is very important.
- Clearly define what it means to meet the performance goals.
- When developing performance goals, it is important to consider how they will be measured and define this up front.
- Focus resources on fewer measures and fewer data sources if needed to get better results as opposed to a larger pool of measures with limited resources for each.
Where Can I Go For More Information?
There are a number of existing sources for information on work zone performance measures, including:
- Performance Contracting Framework (http://www.fhwa.dot.gov/hfl/framework/)
- The FHWA Work Zone Safety and Mobility Performance Measurement Website (http://ops.fhwa.dot.gov/wz/decision_support/perf_measurement.htm)
- Implementing the Rule on Work Zone Safety and Mobility (http://ops.fhwa.dot.gov/wz/rule_guide/rule_guide.pdf)
- The National Work Zone Safety Information Clearinghouse (http://www.workzonesafety.org/)
This material is based upon work supported by the Federal Highway Administration under Grant Agreement No. DTFH61-06-G-00004. Any opinions, findings, and conclusions or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the Federal Highway Administration.
Developed by:
The American Traffic Safety Services Association (ATSSA) 15 Riverside Parkway, Suite 100 Fredericksburg, VA 22406-1022 (800) 272-8772
Posted: 7/15/2010
Categories: Performance Measurement | Work Zone Safety | FHWA Work Zone Safety Grant Program
