Data Collection & Statistical Analysis
A critical element of your CRR plan will be the collection of data and information subsequent to program implementation. If you have conducted a thorough risk assessment, you should be able to compare previous statistics with the results found following implementation of your CRR program.
Statistical analysis is used to make assumptions about a population or data source, and can illustrate significant differences in averages or changes over time. Statistical analysis is a higher level of evaluation, and is more meaningful from a scientific perspective when it is statistically significant. For example, if a community had two fire fatalities in the first year, and then one fire fatality in the second year, the result would be a 50% reduction. Although, technically this is accurate, this is not statistically valid, and misleading.
Statistical analysis does not necessarily provide proof that the program is effective, but instead provides evidence that it may be working. Be cautious about making claims that may not be valid or significant.
Trend Analysis
In statistics, a trend analysis—or trending—typically refers to techniques for extracting an underlying pattern of behavior in a time series. Trending illustrates the fluctuation and changes in outcomes and outputs over time. It is an important outcome measure, as over time, it can indicate changes not due to random chance or normal variances.
The preceding figure is an example of a chart illustrating a 10-year trend in kitchen fires. The chart shows that the frequency of kitchen fires started to decline in the years following the implementation of a CRR program targeted towards kitchen fires.
Although there may be specific changes seen in the impact evaluation, it does not necessarily mean there will be reductions in the outcome evaluation. A number of factors can affect outcomes, which is one reason to measure some outcomes on a per-capita basis.
The following illustration demonstrates the difference between changes and reductions. In 2014, the city had a population of 100,000 persons, with 10 fire fatalities. That represented a per capita rate of fire fatalities of 1/10,000 population. In 2015, the population had risen to 150,000 persons with 11 fire fatalities—or a per-capita rate of 1/13,326 persons.
Although the total number of fire fatalities had risen by 10%, the per capita rate of fire fatalities had actually decreased by nearly 27%.
The following illustration is another example of trending, but represents a comparison of local, state, and national averages (also represents and is linked to benchmarking; see “Benchmarking” below).
Benchmarking
Benchmarking is a process by which one organization (i.e., fire department) compares its results or performance against another organization. Usually, this focuses on a specific performance or evaluation metric, and compares one organization’s results to other’s best practices. The figure above is an example of benchmarking linked to trending.
Caution must be used when comparisons are made among agencies. You must ensure that the same parameters are be used between each organization. A good example is the measurement of cardiac arrest survival outcomes. Not all organizations use the same parameters when determining the percentage of patients that they successfully resuscitate.
The table on the following page demonstrates the difference in cardiac arrest survival between two communities. When viewing the survival rates between the two fire departments, it appears that Fire Department A has a significantly better rate of survival than Fire Department B.
| Fire Department A | Fire Department B |
|---|---|
|
Percent Survived to Discharge: 28% Measurement Components
|
Percent Survived to Discharge: 12% Measurement Components
|
However, this is not an accurate comparison, as Fire Department A only measures the percentage patients who survived that were found in ventricular fibrillation (VF); had a witnessed event; and in whom bystander CPR was performed. In contrast, Fire Department B measured its results on all cardiac arrest patients.
The following illustration is an example of benchmarking fire incident-rates per capita among various fictitious cities throughout the United States.
Performance Measures
Performance standards are utilized consistently throughout fire and emergency services. The National Fire Protection Association (NFPA), Commission on Fire Accreditation International (CFAI), Insurance Services Office (ISO), Commission on Accreditation of Ambulance Services (CAAS), and Governmental Accounting Standards Board (GASB) have each developed recommended performance standards related to fire protection, prevention, and other emergency services.
For example, NFPA Standard 1710 describes travel-time standards for the arrival of the first engine at a fire suppression incident.7 The CFAI defines recommendations for a community risk analysis, as well as for fire prevention and life safety programs. The ISO includes recommended distances of fire stations to commercial properties.
While most of these performance criteria represent consensus standards, they are usually not mandatory, but do provide guidance on potential goals that can be utilized by your department. However, your organization must determine reasonable performance goals based on the characteristics of your community.
The table below is an example of performance measures, which can overlap with evaluation types (process, impact, and outcome).
| Performance Measures & Outcomes | Current Goal | 2013 | 2014 | 2015 | Trend |
|---|---|---|---|---|---|
| Incident Response | |||||
| Total Calls for Service | Decrease FPY | 21,203 | 21,326 | 23,184 | Improving |
| Urban Response ≤ 5 minutes | 90% | 57% | 59% | 61% | |
| Suburban Response ≤ 6 minutes | 90% | 55% | 54% | 52% | |
| Rural Response ≤ 8 minutes | 90% | 71% | 69% | 73% | |
| False Alarms as percent of Total Calls | < 5% | 3.8% | 3.4% | 4.7% | |
| Emergency Medical Services | |||||
| Cardiac Arrest Survival (witnessed & VF) | Increase FPY | 13.6% | 26.5% | 57% | Improving |
| ALS: Percent calls to nursing & other facilities | Decrease FPY | 14.3% | 8.3% | 7.9% | |
| ALS Average On-Scene Time | Decrease FPY | 17:06 | 18:00 | 16:03 | Worse |
FPY: From previous year
Green: Goal met
Yellow: Missed goal, but close
Red: Missed goal totally, or bad trend
Process measures are related to outputs and inputs, but usually not outcomes. However, often there are times when there is overlap. The figure below illustrates an example of performance reporting and the types for each.
| Minimize the Effect of Fires | 2013 | 2014 | 2015 | Type |
|---|---|---|---|---|
| Percent of fires confined to room of origin | 49% | 50% | 40% | Outcome |
| Average dollar loss from structure fires | $15,523 | $12,585 | $41,462 | Outcome |
| Average total fire-response on-scene time | 44:53 min | 55:11 min | 47:11 min | Output |
| Fire Prevention | ||||
| Determine cause of fires | 78% | 90% | 89% | Outcome |
| Conduct safety presentations | 241 | 186 | 183 | Output |
| Number of effective fire inspections | 5,669 | 5,780 | 6,006 | Output |
| Average percent knowledge gain | 35% | 37% | 42% | Impact |
Finally, your evaluation does not require complex, sophisticated charts and graphs. Good data is imperative to monitoring your program accurately, and lends credibility to your organization. Depending on your capabilities and resources, it will be up to your organization to determine what measures to utilize. Consider asking for assistance from individuals outside your organization, who may have expertise (e.g., statisticians, industry consultants, etc.) that may help with your CRR program evaluation.