Monitoring the Plan

It will be critical that you collect sufficient data to determine whether your efforts are having the intended effect. If you have collected measurable information from your risk assessment, ongoing data collection should enable you to compare the results from before and after implementation of the CRR program.

The primary reasons for monitoring and evaluating your CRR plan include:

• It allows for timely adjustments through an ongoing review of the program progress.
• Assists in the decision-making process.
• It helps to assess if the program goals and objectives are being achieved.
• It may uncover unexpected benefits and problems.
• Provides data and/or other information to demonstrate value and success.
• Results can be shared with the community, within your organization, key stakeholders, and others.

Evaluation Stages

There are four stages of evaluation used in CRR programs. In simple terms, they can be defined as follows:

Formative Stages—Identification of problems, needs, or a risk assessment and/or research. Associated with the problem identification phase (i.e., risk assessment step).

Process Stages—Implementation, workload, efficiency, and program satisfaction.

Impact Stages— Measures knowledge gain, risk-reduction behaviors, and adoption of laws and policies. Short-term analysis used to measure results.

Outcome Stages—Changes or reductions in loss data for deaths, injuries, property loss, or responses. Long-term analysis used to measure results. May take years to see reductions.

Impact Evaluation Measures

Impact evaluation can be applied to the four primary disciplines within fire prevention: public education, code enforcement, plan review, and fire investigation. As mentioned, this stage measures short-term results. Methods for conducting an impact evaluation may include:

• Surveys
• Questionnaires
• Direct observation
• Group discussions
• Focus groups

Impact evaluation will tend to be different among each of the four disciplines. For example, a public education program may evaluate:

• Improvements in safety knowledge, attitudes, and beliefs of the participants.
• Observed and documented changes in behavior (hazard reduced or safety increased).
• Introduction and/or adoption of fire-safety legislation.

Examples of an impact evaluation in code enforcement may include:

• Number of code violations noted and abated.
• Percentage of fires in which pending, uncorrected violations were present at the time of the fire.
• Enforcement of fire safety legislation and regulation.

Outcome Evaluation Measures

Outcome evaluation is the final step in the process. This is where everything comes together. While an impact evaluation looks for changes, the outcome evaluation looks for reductions resulting from those changes. Examples of outcome measures among the four prevention disciplines might include:

Public Education

• Reduction in fire incidents per 1,000 residents in the target population.
• Reduction in fire deaths per 1,000 residents in the target population.
• Reduction in medical costs per 1,000 residents in the target population.

Code Enforcement

• Reduction in percent of total fire losses occurring in inspectable occupancies.
• Reduction in fire deaths per 1,000 residents of inspectable occupancies.
• Reduction in number of structural fires per 1,000 residents of inspectable occupancies.
• Reduction in inspectable property structure fires with at least $25,000 in loss.

Plan Review Program

• Reduction in fire incidents in reviewed occupancies.
• Reduction in property damage costs from fire in reviewed occupancies.

Fire Investigation Program

• Increase in percentage of fires where cause is determined.

Figure 8 illustrates how evaluation can be used throughout the life of a program. Often in the fire service, the formative stage is not considered, and the program begins with implementation. A strategic CRR programs looks at all four stages of evaluation.

As mentioned previously, outcomes take time to recognize. They may be seen as a reduction in losses through number of incidents, deaths, injuries or per capita losses. Alternatively, the result may not show a reduction, but instead demonstrate an increase—indicating the need for a modification of the CRR program.