Executive Summary Brief Description The Roadside Inspection and Traffic Enforcement programs are two of the Federal Motor Carrier Safety Administration’s (FMCSA) key safety programs. The Roadside Inspection program consists of roadside inspections performed by qualified safety inspectors following the guidelines of the North American Standard, which were developed by FMCSA and the Commercial Vehicle Safety Alliance. Most roadside inspections are conducted by the States under the Motor Carrier Safety Assistance Program (MCSAP). There are six levels of inspections that include a vehicle component, a driver component, or both. The Traffic Enforcement program is composed of two distinct activities: a traffic stop as a result of a moving violation and a roadside inspection. FMCSA, in cooperation with the Volpe National Transportation Systems Center, has developed an analytic model to measure the effectiveness of roadside inspections and traffic enforcements in terms of crashes avoided, injuries avoided, and lives saved. Traffic enforcements and roadside inspections are considered interventions and this analytic model is known as the Intervention Model. This model provides FMCSA management with information to address the requirements of the Government Performance and Results Act of 1993 (GPRA), which obligates Federal agencies to measure the effectiveness of their programs as part of the budget cycle process. It also provides FMCSA and State safety program managers with a quantitative basis for optimizing the allocation of safety resources in the field. The Intervention Model is based on the premise that interventions to correct vehicle and driver defects, defined as roadside inspections and traffic enforcements, directly and indirectly contribute to a reduction in crashes. The model includes two submodels that are used for measuring these different effects: • Direct effects are based on the assumption that vehicle and/or driver defects discovered and then corrected as the result of interventions reduce the probability that these vehicles/drivers will be involved in subsequent crashes. The model calculates direct-effect-prevented crashes according to the number and type of violations detected during an intervention. • Indirect effects are the by-products of the carriers’ increased awareness of FMCSA programs and the potential consequences that the programs could impose if steps are not taken to ensure and/or maintain higher levels of safety. In order to measure indirect effects, which are essentially changes in behavior involving driver preparation, practices and vehicle maintenance, the model calculates responses to exposure to the programs and the resulting reduction in potentially crash-causing violations. This model, which measures the effectiveness of the Roadside Inspection and Traffic Enforcement programs, when combined with the Compliance Review Effectiveness Model (http://ai.fmcsa.dot.gov/CarrierResearchResults/PDFs/ProgramEffectiveness/CREM_O6.pdf), forms a powerful performance measurement capability that plays a significant role in resource allocation decisions regarding FMCSA’s safety programs. Methodology This model is based on the premise that the two programs - Roadside Inspection and Traffic Enforcement - directly and indirectly contribute to the reduction of crashes. As a result, the model includes two submodels that are used for measuring these different effects. Direct effects are based on the assumption that vehicle and/or driver defects discovered and then corrected as the result of interventions (roadside inspections and traffic enforcements) reduce the probability that these vehicles/drivers will be involved in subsequent crashes. Indirect effects are considered to be the by-products of the carriers' increased awareness of FMCSA programs and the potential consequences that these programs impose if steps are not taken to ensure and/or maintain high levels of safety. Figure ES-1 provides an overview of the Intervention Model. Figure ES-1. Overview of the Intervention Model Direct Effects This section describes the methodology employed to estimate the number of direct-effect crashes avoided. Conceptually, the approach at the heart of the Direct Effects Submodel is straightforward. Since the occurrence of a single violation implies a certain degree of crash risk, each inspection that uncovers at least one violation can be interpreted as having reduced the risk associated with its noted violation(s). The model expresses this risk reduction in terms of the likelihood of a crash being avoided by each inspection violation that was noted and corrected. For an individual intervention, the avoided crash probability will be dependent upon the number and type of violations. Multiple violations will have a compounding effect, thereby increasing the likelihood of a prevented crash. By accounting separately for the two types of violations (roadside and traffic enforcement) and summing the portions of crashes avoided for all inspections within each group, it is possible to estimate direct-effect crashes that have been avoided due to the programs. The Direct Effects Submodel is composed of three major steps: input data selection, assignment of crash risk probabilities, and calculation of direct results. Input Data Selection One year of intervention data is extracted from the Motor Carrier Management Information System (MCMIS) database. This database contains roadside inspection and traffic enforcement information compiled from federal and state safety agencies. This data also includes the violations (if any) that were cited during the intervention. While interventions are not required to have violations associated with them, in practice about 75% of all interventions do have one or more violations. This violation data is the key component in the model as it represents the defects that were identified and subsequently corrected as a part of the program. This data is also used in the determination of which interventions were conducted under the Traffic Enforcement Program (i.e. traffic enforcements) and which were conducted under the Roadside Inspection Program. An inspection with a traffic enforcement driver violation is classified as traffic enforcement with a driver and/or vehicle roadside inspection component(s). All other inspections are classified as entirely driver and/or vehicle roadside inspections. Assignment of Crash Risk Probabilities In the model, the assumption is made that observed deficiencies (i.e. violations) discovered at the time of the intervention can be converted into crash risk probabilities. This assumption is based on the premise that detected defects represent varying degrees of mechanical or judgmental faults, and, further, that some are more likely than others to play a contributory role in motor carrier crashes. The assumption is that these deficiencies can be noted and ranked into discrete risk categories, each with a probability that quantifies the potential for a crash for all deficiencies in that category. The risk categories and their descriptions are as follows: • Risk Category 1 - The violation is the potential single, immediate factor leading to a crash. • Risk Category 2 - The violation is the potential single, eventual factor leading to a crash. • Risk Category 3 - The violation is a potential contributing factor leading to a crash. • Risk Category 4 - The violation is an unlikely potential contributing factor leading to a crash. • Risk Category 5 - The violation has little or no connection to crashes. The risk categories were designed such that each category represents a different order of magnitude of likelihood of contributing to a crash. Using this information and the latest available data, crash risk probabilities were developed for each risk category by out-of-service indicator and by violation type (driver or vehicle). Each probability is an estimate of the portion of a crash avoided when an inspection uncovers a particular violation or inversely the number of violations of that type that would need to be uncovered before one crash could be prevented. Calculation of Direct Results The likelihood of an avoided crash for each inspection is calculated by using the crash reduction probabilities of each of the violations cited during the inspection. An inspection with multiple violations will have a greater likelihood of an avoided crash than will an inspection with a single violation, assuming all the violations are in the same risk category. This result reflects the belief that multiple violations compound the safety hazard posed from driver deficiencies and/or vehicle defects. Once the number of crashes avoided for each inspection has been calculated, the next step in the calculation of the results is to compute the number of lives saved and injuries avoided as a result of those crashes avoided. This is done by computing national averages of fatalities per crash and injuries per crash using MCMIS data. These averages are then multiplied by the number of crashes avoided resulting in the number of lives saved and injuries avoided. Indirect Effects The fundamental premise of the indirect-effects approach is that once carriers have been exposed to interventions, they will change their behavior. This change in behavior will result in higher levels of compliance, fewer future violations, and, therefore, a reduction in the number of crashes. This section presents a summary of the methods used in the model to arrive at program indirect effects. The deterrent-effects part of the model - that is, the Indirect Effects Submodel - follows a similar process to that of the Direct Effects Submodel. Indirect effects, by their nature, defy measurement. However, changes in behavior represented by changes in the number of violations recorded for a carrier over time can be used to identify and evaluate the results of the indirect effects. In other words, if a carrier receives fewer and fewer violations as it is subjected to more inspections, it will be determined that compliance behavior has been affected and the resulting likelihood of crashes has been reduced. To measure these effects, multiple successive years of intervention data are required. The Indirect Effects Submodel compares carrier performance in a base year to the year after in order to measure the effects of the exposure to interventions in the base year on compliance. What is sought is an improvement, i.e., a reduction, in the likelihood of a crash resulting from increasingly fewer violations being recorded. The difference between the totals is calculated as the indirect-effect crashes-avoided. Depending upon the initiating intervention, it is tallied as indirect-effect crashes avoided for either the Roadside Inspection or Traffic Enforcement programs. Figure ES-2 illustrates the processes involved in assessing the indirect effects of the model. Figure ES-2. Indirect Effect Approach Input Data Selection Instead of one year of intervention data, like the Direct Effects Submodel, two years of intervention data are required. Again this includes the interventions as well as any associated violations. The first year of data selected is the base year. This is the year in which the effectiveness of the interventions will be estimated. The second year is the year after the base year and is used for comparison purposes in order to determine the change in carrier performance. Crash Risk Probability Assignment In this step, the two years of intervention data is analyzed and the violations are assigned to their appropriate risk category. Calculation of Results The crashes avoided are calculated for both years of data by carrier for each program using the same algorithm as the Direct Effects Model. This is where the two submodels diverge in their approach. A standard set of filtering criteria is used to eliminate carriers with insufficient data for a comparison. Once the filtering is complete, the difference between the crashes avoided estimated in the base year and the crashes avoided estimated in the subsequent year is computed for each carrier and program. These carrier-level results are then summed in order to arrive at program-level results for the difference in crashes from the base year to the subsequent year. This change in crashes is converted to a percentage difference then applied to the number of interventions conducted in the base year. The results of the computation are the estimated number of crashes avoided for each program. The determination of lives saved and injuries avoided is done in the exact same way as it was for the direct effects, that is national level fatalities and injuries per crash are used to estimate the lives saved and injuries avoided. The safety benefits estimated by this part of the model represent the indirect effect of the intervention program activities conducted in the base year, which is the activity year that was used in the direct effects calculation. The only drawback to this method of calculating the indirect effects is that it requires an additional year of data after the activity year. For example, in order to compute the indirect effects for the 2005 interventions, it would require 2006 intervention data as well. Instead of waiting until this data is available to release results, an average of the prior two years indirect effects benefits (as a percentage of the total benefits) are used to project the indirect effects. For example, to project the indirect benefits for the Roadside Inspection program for 2005 the percent of indirect benefits in the Roadside Inspection program for 2003 and 2004 are averaged. Once the additional year of activity data is available the indirect effect benefits are updated and used in the subsequent years calculations. 2005 Intervention Model Results The model was implemented to estimate the crashes avoided, lives saved, and injuries avoided as a result of activities performed during the 2005 calendar year. The direct effects were calculated exactly as described in the previous section. The indirect effects for each program were projected from the 2003 - 2004 indirect effects results. Over those two years the indirect effects on average accounted for 23% of the total Roadside Inspection program benefits and 14% of the total Traffic Enforcement program benefits. The direct and indirect results are combined and presented at two different levels, the national level and the state level. National Level Table ES-1 provides a comparison of the program activity level at the national level for the current analysis year (2005) as well as two historical years (2003 - 2004). In general, the activity levels of the two programs have remained relatively constant over the past few years; however, there is a noticeable shift from roadside inspections to traffic enforcements. Table ES-1. Program Exposure 2003 - 2005 2003 2004 2005 Roadside Inspections 2,215,762 2,211,875 2,194,567 Traffic Enforcements 791,157 803,032 827,719 Total Interventions 3,006,919 3,014,907 3,022,286 Table ES-2 presents the benefits of the two programs in the current analysis year (2005) as well as two years of historical results (2003 - 2004). There are a number of noteworthy observations that warrant some additional discussion, found in the Analysis section. Table ES-2. Program Effectiveness 2003 - 2005 2003 2004 2005 Crashes Avoided Roadside Inspection 12,667 9,606 9,256 Traffic Enforcement 4,484 9,067 9,215 Total 17,151 18,673 18,471 Injuries Avoided Roadside Inspection 9,647 7,004 6,418 Traffic Enforcement 3,415 6,611 6,390 Total 13,062 13,615 12,807 Lives Saved Roadside Inspection 534 371 344 Traffic Enforcement 188 351 343 Total 722 722 687 Figure ES-3 displays the trends in intervention benefits, crashes avoided and lives saved, from 2000 to 2005. Overall, the number of crashes avoided has shown an increasing trend, while the number of lives saved has decreased in recent years. Figure ES-3. Crash and Fatality Trends State Level The model's flexibility lends itself to finer divisions of examination, such as scrutiny by state, which then can be used to guide the allocation of MCSAP resources and the design of state safety programs. Because many states manage their intervention program differently, it is also important to analyze state level totals as well as the national totals. The national totals have the ability to obscure state level trends that may occur because of the differences in how the programs are administered. Table ES-10 through Table ES-12 at the end of the document provide detailed results for interventions conducted: • in all fifty states, • in the District of Columbia, American Samoa, the Northern Mariana Islands, Puerto Rico and • by federal staff (denoted by US). These tables provide intervention counts, total estimated benefits (crashes avoided, lives saved, injuries avoided), and normalized estimated benefits (benefits per thousand interventions. Analysis This section is devoted to the analysis of the model results. The current analysis year (2005) has shown a minor increase of 0.25% in the number of interventions compared with the previous year (2004), while the number of crashes avoided resulting from the interventions has decreased by 4.85%. The reason for the decreased marginal contribution per inspection is a combination of the shift in intervention type and the indirect contribution, which are further explored in the following sections. Program Activity The activity data reveals that there has been a slight shift in the program exposure from roadside inspections to traffic enforcements (Table ES-1), while the overall activity has remained fairly constant. Comparing 2005 to 2004 the number of roadside inspections has decreased by 17,308 and the number of traffic enforcements has increased by 24,687. The shift in program exposure has carried over to the program effectiveness; the crashes avoided due to roadside inspections has decreased by 350 relative to 2004 while the crashes avoided due to traffic enforcements has increased by 148 relative to 2004. The shift in program exposure has further been influenced by the indirect effects. Indirect Effect Trends The decrease in the number of crashes avoided is largely attributable to the decrease in the indirect contribution over the most recent years in the Roadside Inspection program. Table ES-3 provides a comparison of the direct and indirect program benefits for the Roadside Inspection program over 2002 - 2004. The percent of direct crashes avoided has shown a steady increase from about 75% in 2002 to about 78% in 2004. The indirect percentage has shown the opposite trend decreasing by approximately 3% from 2002 to 2004. Table ES-3. Roadside Inspection Program Benefits 2002-2004 2002 2003 2004 Crashes Avoided % of Total Crashes Avoided % of Total Crashes Avoided % of Total Direct 6,558 74.57% 6,840 76.69% 7,265 77.87% Indirect 2,236 25.43% 2,079 23.31% 2,065 22.13% Total 8,795 8,919 9,300 Table ES-4 displays a comparison of the direct and indirect crashes avoided for the Traffic Enforcement program. The Traffic Enforcement program has remained fairly constant in the percent allocation of direct and indirect benefits. Table ES-4. Traffic Enforcement Program Benefits 2002 - 2004 2002 2003 2004 Crashes Avoided % of Total Crashes Avoided % of Total Crashes Avoided % of Total Direct 7,298 85.80% 7,269 85.45% 7,764 86.21% Indirect 1,208 14.20% 1,238 14.55% 1,242 13.79% Total 8,505 8,506 9,005 The indirect effect percentages used in the model come from a two-year average of the years prior to the benefit year. For example, the estimated indirect benefits for 2005 use the two-year average of the 2003 and 2004 indirect benefits as a percentage of total crashes avoided. Table ES-5 displays these two-year averages for both programs. Table ES-5. Two Year Average of Indirect Benefits as a Percentage of Total Crashes 2001 - 2002 Average 2002 - 2003 Average 2003 - 2004 Average Roadside Inspections 25.72% 24.37% 22.72% Traffic Enforcements 14.66% 14.37% 14.17% Figure ES-4 clearly illustrates the decreasing trend in indirect effects for the Roadside Inspection program. The figure also depicts the large difference between the indirect contribution for each enforcement type. The indirect contribution of roadside inspections is much greater than the indirect contribution of the Traffic Enforcement program. Figure ES-4. Indirect Effects Trends The loss of 350 crashes avoided from the Roadside Inspection program from 2004 to 2005 stems from the shift in program exposure and the diminishing contribution from the roadside inspection indirect effects. The effectiveness of the Roadside Inspection program carries over to the total program effectiveness, which has shown a net loss of 202 crashes avoided from 2004 to 2005. Crash Severity Trends The program effectiveness in 2005 has shown a decrease relative to 2004 in terms of injuries avoided and lives saved (Table ES-2). In the previous three years the number of lives saved has not been proportional to the number of crashes avoided. The major reason for this behavior is the model relies on crash severity statistics from actual crashes reported during the current activity year and previous activity year as described in the methodology section of this document. Over the past few years the average number of fatal crashes and fatalities per crash have been decreasing according to MCMIS data. It is plausible that some of this decrease has resulted from increases in the safety of roads and vehicles. In the past few years, FMCSA has placed a greater emphasis on reporting injury and towaway crashes, which would also account for a decrease in the percentage of fatal crashes and subsequently the expected number of fatalities per crash. Table ES-6 displays the decreasing percentage of fatal and injury crashes from 2002 to 2005. The trend shows the percent of fatal and injury crashes has decreased from 2002 to 2005 while the percent of tow away crashes has shown an increase of almost 6% over these years. Table ES-6. Crash Severity Shares Year Fatal Crash Injury Crash Tow away Crash 2002 3.94% 47.76% 48.30% 2003 3.41% 47.18% 49.41% 2004 3.05% 44.98% 51.97% 2005 3.18% 42.78% 54.03% Table ES-7 shows the two year average of fatal, injury and tow away crash shares to smooth out any year to year fluctuations and display the percentages used in the model. Table ES-7. Two Year Average of Crash Severity Shares Time Range Fatal Crash Injury Crash Tow away Crash 2002 - 2003 3.68% 47.47% 48.86% 2003 - 2004 3.23% 46.08% 50.69% 2004 - 2005 3.12% 43.88% 53.00% Table ES-8 displays the average number of fatalities and injuries that are present in either fatal or injury crashes. In general, there has not been any substantial changes in the number of fatalities per crash and injuries per fatal or injury crash. Table ES-8. Average Number of Fatalities and Injuries by Year Year Fatalities/Fatal Crash Injuries/Fatal Crash Injuries/Injury Crash 2002 1.223 1.131 1.515 2003 1.178 1.031 1.521 2004 1.214 1.174 1.489 2005 1.169 0.968 1.519 Table ES-9 shows the two year average of number of fatalities and injuries in fatal or injury crashes to eliminate any yearly inconsistencies and display the numbers used in the model. Table ES-9. Two Year Average of Fatalities and Injuries Time Range Fatalities/Fatal Crash Injuries/Fatal Crash Injuries/Injury Crash 2002 - 2003 1.200 1.081 1.518 2003 - 2004 1.196 1.102 1.505 2004 - 2005 1.192 1.071 1.504 The above table ES-9 clearly shows that there is not much variation in the number of fatalities per fatal crash, around 1.2, injuries per fatal crash, around 1.1, or injuries per injury crash, around 1.5. However, there is a obvious decrease in the percent of fatal and injury crashes and rise in the percent of tow away crashes given by table ES-7. The drop in number of fatalities and injuries avoided due to interventions is attributable to the decreased share of fatal and injury crashes. Table ES-10. 2005 Roadside Inspection and Traffic Enforcement Program Benefits Table ES-11. 2005 Roadside Inspection Program Benefits Figure ES-5. 2005 Traffic Enforcement Benefits FMCSA Intervention Model Executive Summary Calendar Year 2005 Prepared for: Federal Motor Carrier Safety Administration Office of Information Management Analysis Division 400 Seventh Street, SW Washington, DC 20590 Prepared by: John A. Volpe National Transportation Systems Center Office of System and Economic Assessment Motor Carrier Safety Assessment Division, RTV-3E Kendall Square 55 Broadway Cambridge, MA 02142 Document Number: FMCSA-RRA-07-015