Conducting an Evaluation

Evaluation is an ongoing assessment of a drinking water advisory protocol. It is the comparison of Standard Operating Procedure (SOP) criteria to performance. The collection and analysis of subjective experiences along with the analysis of objective forms of data or information provide the foundation for evaluation. Even the most basic evaluation provides insight and can improve future advisories and overall communication.

The scope of an advisory and a water system’s resources and capacities guide the evaluation process. The time, resources, and expertise needed for an evaluation vary with the scope. Evaluations can be conducted over time and in different stages. Evaluations draw on many sources of information, including, operational reports, debriefings, surveys, and public comment. See the Debriefing Evaluation Form. [DOCX - 2 pages]

Different types of data are described below:

• Quantitative data, such as water quality data, Web analytics, and epidemiology statistics.
• Qualitative data, such as customer comments, media reports, staff memos, etc.

See Appendix C: Online Resources, Data Management for links to information on qualitative data collection.

See the Advisory Feedback Guide [DOCX - 4 pages] and Call Center Data Collection Framework [DOCX - 2 pages] for collecting information from partners.

Surveys

Water systems should consider surveying their customers about the advisory. Surveys can be used to determine the effectiveness of the advisory and to determine whether the advisory influenced personal actions.

• Customer service surveys may include a section related to an advisory.
• Surveys can be designed to measure perceptions, actions, and communication effects and outcomes.
• The perceptions and recall of respondents will change with the amount of time that passes between an advisory and a survey.
• Surveys can be short and focused or longer to gather a full range of data.
• Online surveys offer a fast and efficient way to solicit and analyze feedback from a wide range of respondents.

Consult the Post-Advisory Community Survey [DOCX - 6 pages] or CASPER (Community Assessment for Public Health Emergency Response) for ideas about what to ask in a survey and for methods to use when conducting a survey.

Data Management

Manage data as it is collected. Data management activities may include:

• Storing data as hard and electronic copies in a central location for easy access.
• Requesting information from partners involved with the advisory.
• Asking partners about limitations or privacy policies that might limit the ability to report the data.
• Working with database experts to ensure that the database structure is optimal for doing the planned analysis.

Data sets are valuable when they are applied to improve future responses. The information collected should be analyzed, synthesized, and reported in a useful format. Use the data to develop questions for public outreach and for the basis of evaluation. Develop a plan for data analysis that will give the answers needed for improvement and decision making.

Appendix C: Online Resources, Data Management contains links to websites on data frameworks.

Table 2 presents examples of different types of resources and data that can be used to evaluate communication activities during a drinking water advisory.

Table 2: Examples of Types and Sources of Drinking Water Advisory Data