Executive Summary

This document is a “how to” guide for planning and implementing evaluation activities. The manual, based on CDC’s Framework for Program Evaluation in Public Health, is intended to assist managers and staff of public, private, and community public health programs to plan, design, implement and use comprehensive evaluations in a practical way. The strategy presented in this manual will help assure that evaluations meet the diverse needs of internal and external stakeholders. Such needs include assessing and documenting program implementation, outcomes, efficiency and cost-effectiveness of activities, and taking action based on evaluation results to increase the impact of programs.

Why Evaluate Public Health Programs?

Public health programs aim to prevent or control disease, injury, disability and death. Over time, as this task has become more complex, programs themselves have become more complex. Increasingly, public health programs address large problems, the solutions to which must engage large numbers of community members and organizations in a vast coalition. More often than not, public health problems—which in the last century might have been solved with a vaccine or change in sanitary systems—involve significant and difficult changes in attitudes and risk/protective behavior of consumers and/or providers.

In addition, the context in which public health programs operate has become more complex. Programs that work well in some settings fail dismally in others because of the fiscal, socioeconomic, demographic, interpersonal, and inter-organizational settings in which they are planted. At the same time that programs have become more complex, the demands for accountability from policymakers and other stakeholders have increased.

These changes in the environment in which public health programs operate mean that strong program evaluation is essential now more than ever. There is no one “right” evaluation. Rather, a host of evaluation questions may arise over the life of a program that might reasonably be asked at any point in time. Addressing these questions about program effectiveness means paying attention to documenting and measuring the implementation of the program and its success in achieving intended outcomes, and using such information to be accountable to key stakeholders.

Program Implementation

Evaluation encourages us to examine the operations of a program, including which activities take place, who conducts the activities, and who is reached as a result. In addition, evaluation will show how faithfully the program adheres to implementation protocols. Through program evaluation, we can determine whether activities are implemented as planned and identify program strengths, weaknesses, and areas for improvement.

For example, a treatment program may be very effective for those who complete it, but the number of participants may be low. Program evaluation may identify the location of the program or lack of transportation as a barrier to attendance. Armed with this information, program managers can move the class location or meeting times or provide free transportation, thus enhancing the chances the program will actually produce its intended outcomes.

Program Effectiveness

The CDC and the Federal government have identified goals that public health programs should work toward to prevent or reduce morbidity and mortality. Comprehensive public health programs use multiple strategies to address these goals. Typically, strategies are grouped into program components that might include such elements as community mobilization, policy and regulatory action, strategic use of media and health communication, and funding of frontline programs. Program evaluation documents progress on program goals and the effectiveness of various strategies in producing this progress.

Program Accountability

Program evaluation is a tool with which to demonstrate accountability to an array of stakeholders who may include funding sources, policymakers, state, and local agencies implementing the program, and community leaders. Depending on the needs of stakeholders, program evaluation findings may demonstrate that the program makes a contribution to reducing morbidity and mortality or relevant risk factors; or that money is being spent appropriately and effectively; or that further funding, increased support, and policy change might lead to even more improved health outcomes. By holding programs accountable in these ways, evaluation helps ensure that the most effective approaches are maintained and that limited resources are spent efficiently.

This manual integrates insights from previous Framework-based manuals developed by CDC’s Office on Smoking and Health, [1] and Division of Nutrition and Physical Activity [2] for their grantees and state and local partners, and by the Center for the Advancement of Community Based Public Health for community health programs [3] as well as additional insights from conducting trainings and facilitations based on the Framework. The document is organized around Framework’s six steps:

• Engage Stakeholders
• Describe The Program
• Focus The Evaluation
• Gather Credible Evidence
• Justify Conclusions
• Ensure Use of Evaluation Findings and Share Lessons Learned

Each chapter illustrates the main points using examples inspired by real programs at the Federal, state, and local levels.