Skip Navigation Links
Centers for Disease Control and Prevention
 CDC Home Search Health Topics A-Z

Preventing Chronic Disease: Public Health Research, Practice and Policy

View Current Issue
Issue Archive
Archivo de números en español








Emerging Infectious Diseases Journal
MMWR


 Home 

Volume 5: No. 2, April 2008

TOOLS AND TECHNIQUES
Adapting Logic Models Over Time: The Washington State Heart Disease and Stroke Prevention Program Experience


TABLE OF CONTENTS


Translation available Este resumen en español
  Ce résumé est en français
  本摘要繁體中文版
  本摘要简体中文版
Print this article Print this article
E-mail this article E-mail this article:



Send feedback to editors Send feedback to editors
Download this article as a PDF Download this article as a PDF (479K)

You will need Adobe Acrobat Reader to view PDF files.


Navigate This Article
Abstract
Introduction
Overview of the WaHDSPP
History of Logic Model Use by the WaHDSPP for Planning and Evaluation
Relationship Among Activities, Evaluation Plans, and the Logic Model
Lessons Learned and Future Directions
Author Information
References
Table


Marilyn Sitaker, MPH, Jan Jernigan, PhD, Susan Ladd, MS, Miriam Patanian, MPH

Suggested citation for this article: Sitaker M, Jernigan J, Ladd S, Patanian M. Adapting logic models over time: the Washington State Heart Disease and Stroke Prevention Program experience. Prev Chronic Dis 2008;5(2). http://www.cdc.gov/pcd/issues/2008/
apr/07_0249.htm
. Accessed [date].

Abstract

Logic models are graphic representations of the relationship between program activities and their intended effects and are used for both program planning and evaluation. Logic models can provide an important foundation for program evaluation by identifying evaluation questions that most appropriately assess program processes and outcomes and by guiding measurement decisions. We demonstrate how logic models can be used to plan program evaluation by describing the adoption of logic modeling by the Washington State Heart Disease and Stroke Prevention Program (WaHDSPP) and by specifying the changes in process and use of logic models since the program’s initial funding. Our paper describes how a logic model was used in generating the program evaluation plan for the WaHDSPP, including the identification of evaluation questions and development of indicators to track progress effectively. We describe the use of evaluation results, as well as steps state programs can take to use logic models in program evaluation.

Back to top

Introduction

The benefits of logic models for program planning and evaluation are evident. As a result, descriptions of logic models are becoming more commonplace in program and evaluation literature, and funding announcements now routinely encourage their development and use. For example, when the Centers for Disease Control and Prevention’s (CDC’s) Division for Heart Disease and Stroke Prevention (DHDSP) released its 5-year Funding Opportunity Announcement (FOA) for 2007 for state programs to address heart disease and stroke, applicants were required to develop logic models. However, although application requirements now more commonly require logic models, the subsequent use of these models once grants have been awarded may be inconsistent. We describe the Washington State Heart Disease and Stroke Prevention Program’s (WaHDSPP’s) experience with program logic models and their use in evaluation planning. We also describe how the development and use of program logic models evolved over time. By describing how the WaHDSPP developed its logic model and used it to construct an evaluation plan, we hope to demonstrate the link between development and use. Subsequently, we hope to demonstrate the utility of logic models in public health programs and stimulate the construction of better and more usable evaluation plans.

Back to top

Overview of the WaHDSPP

The goal of the WaHDSPP is to build statewide support for programs targeting people who have heart disease or history of stroke, as well as people who are at high risk for developing either condition (e.g., people with hypertension, people with high blood cholesterol, people with diabetes). The WaHDSPP first received funding from CDC’s DHDSP in 2003. From 2003 through 2006, the program was staffed by a full-time program manager and half-time epidemiologist. During this time, the program relied heavily on cross-program staff from the Washington State Department of Health, internal and external partners, and the WaHDSPP advisory council to implement its activities.

Back to top

History of Logic Model Use by the WaHDSPP for Planning and Evaluation

Logic models were not a requirement for funding by CDC, and during the first 2 years (2003–2004) of the program, no logic model existed to describe or assist in evaluating the WaHDSPP. Activities conducted by the WaHDSPP were chosen on the basis of their alignment to CDC DHDSP program priorities, the existing program capacity, relationships with internal and external partners, and previous experience. During this time, the WaHDSPP focused on capacity building and needs assessment activities, including developing The Burden of Heart Disease and Stroke in Washington State (1) (referred to hereafter as “the Burden document”), forming a statewide advisory council, and conducting an environmental scan of extant population resources for prevention and control of heart disease and stroke. Using the CDC DHDSP framework included in A Public Health Action Plan to Prevent Heart Disease and Stroke (2) (referred to hereafter as "the Action Plan") and findings from the environmental scan and the Burden document, the advisory council developed the Washington State Public Health Action Plan for Heart Disease and Stroke Prevention and Management (3) (referred to hereafter as “the State Plan”). Because projects had not been implemented and the logical link to expected outcomes of these capacity activities had not been identified through a logic model, the WaHDSPP did not engage in any program evaluation activities during 2003–2004.

Although not a requirement for continued funding, logic model use was encouraged by the DHDSP to assist in program planning and evaluation, and the DHDSP provided training and technical assistance to states for the development and use of logic models. Recognizing the benefits of logic models, the WaHDSPP developed a simple logic model that described its planned activities and expected outcomes as part of the 2005–2006 continuation application. Each activity in the 2005–2006 logic model (Figure 1) was linked to an essential public health service objective from the WaHDSPP State Plan. However, as the only staff members for the WaHDSPP, the program manager and epidemiologist produced the logic model in isolation, with no input from cross-program staff or partners.

Flow chart

[ View enlarged image and descriptive text ]

Figure 1. Washington State Specific Logic Model Framework for Heart Disease and Stroke Prevention Program. (HDSPP indicates Heart Disease and Stroke Prevention Program; CDC, Centers for Disease Control and Prevention; and HD&S, heart disease and stroke.)

To assist in evaluation planning, a logic evaluation plan (LEP) was developed in Excel (Microsoft Corporation, Redmond, Washington), linking evaluation measures to each activity and outcome in the logic model. The 2005–2006 LEP for one activity, the Washington State Collaborative, is shown in Figure 2. The evaluation included some process measures, as well as measures of progress toward short-, medium-, and long-term outcomes. The LEP not only provided simple measures for assessing program activities and outcomes, it also helped the WaHDSPP identify and address measurement gaps. For example, to measure the impact of the Washington State Collaborative on the medium-term outcome better management of chronic conditions, the WaHDSPP developed and implemented a statewide survey of health care providers to assess implementation of the planned care model by primary care physicians and to assess providers’ use of evidence-based guidelines.

Flow chart

[ View enlarged image and descriptive text ]

Figure 2. Washington State Heart Disease and Stroke Prevention Program (WaHDSPP) Logic Evaluation Plan, 2005–2006. (WSC indicates Washington State Collaborative; CVD, cardiovascular disease; FHCQ, Foundation for Healthcare Quality; HBP, high blood pressure; BRFSS, Behavioral Risk Factor Surveillance System; HD, heart disease; PA, physical activity; and CHARS, Comprehensive Hospital Abstract Reporting System.)

Planning for 2006–2007 improved previous years’ efforts for several reasons. First, the planning process was a collaborative effort. Second, evaluation and surveillance data, collected as part of the previous years’ evaluation efforts, were used in program planning efforts. Given the involvement of internal and external partners in the WaHDSPP, the program staff realized the potential benefits of the partners’ input in planning program activities and evaluation efforts. As a result, partners and key stakeholders were invited to participate in the planning retreat for the 2006–2007 continuation application. The involvement of partners and stakeholders responsible for implementing program activities added depth and richness to the discussion. This collaborative process allowed for an in-depth discussion of the barriers to and facilitators of implementing program activities, factors that affect outcome attainment, and future program directions. These discussions were immensely helpful in developing work plans that linked to successful outcomes and that were both feasible and appropriate, given WaHDSPP resources.

The WaHDSPP program manager scheduled time during the planning retreat to review surveillance, evaluation, and assessment results from the previous year, ensuring that these data were taken into account in the proposed program activities. Involving partners and stakeholders in this process provided an opportunity to discuss results with people who were the most knowledgeable about the activities, which led to a better understanding of program activities by the entire planning committee and more realistic approaches to program improvement. In addition, being part of the program planning effort increased buy-in and “ownership” of the WaHDSPP. Once work plans were developed, the logic model and the LEP were updated to reflect proposed activities and outcomes. Collaborating on the logic model revisions provided new clarity in the short-, intermediate-, and long-term objectives for both staff and partners, and this clarity facilitated the selection of indicators.

The planning retreat for 2007–2008 focused on responding to the FOA released by the DHDSP and included newly hired WaHDSPP staff. As during the previous year, key partners and stakeholders were involved in planning, and evaluation and surveillance data were used to guide planning discussions. One new feature in the planning process involved categorizing proposed activities to align with the spectrum of prevention outlined in the Action Plan. The Action Plan, developed collaboratively by the DHDSP and its national partners, describes a comprehensive approach to addressing heart disease and stroke, from preventing risk factors to preventing recurrent cardiovascular events. The FOA encouraged states to use the Action Plan to guide development of their applications. By comparing proposed activities with the Action Plan, the WaHDSPP was able to identify a gap in its interventions, leading to the development of a new objective for the assessment of rehabilitation capacity.

Having additional program staff available to develop work plans for the application enabled partners to collaborate more closely than in years past to revise the logic model. The updated logic model (Figure 3) demonstrated the WaHDSPP’s alignment with the Action Plan, categorizing activities and their outcomes as either primary prevention, acute event, or secondary prevention, and reflected more mature thinking on the part of the WaHDSPP about the specific changes expected as a result of program activities.

Flow chart

[ View enlarged image and descriptive text ]

Figure 3. Washington State Heart Disease and Stroke Prevention Program (WaHDSPP) Logic Evaluation Plan, 2007–2008. (HDSPP indicates Heart Disease and Stroke Prevention Program; CDC, Centers for Disease Control and Prevention; EMS TAC, Emergency Medical Service Technical Advisory Committee; NCQA, National Committee for Quality Assurance; EPHS, essential public health services; MIS, management information system; HD&S, Heart Disease and Stroke; MI, myocardial infarction.)

A second feature of the 2007–2008 planning process involved the use of the CDC DHDSP’s Developing an Evaluation Plan (4). This guide is one in a series developed by the DHDSP to assist states in their evaluation efforts. The WaHDSPP used the guide to systematically develop evaluation questions, leading to more well-rounded evaluation plans for individual activities and for the overall program (Table). The evaluation terms used by the program were revised to be consistent with those provided by CDC; the term measures was replaced with indicators and medium-term outcomes were referred to as intermediate outcomes. The evaluation plan replaced the LEP and provided more details on methods and data sources that would be used to answer key evaluation questions.

Back to top

Relationship Among Activities, Evaluation Plans, and the Logic Model

This section describes an outreach activity proposed for 2007–2008 and how it is being evaluated on the basis of the logic model. The African American Awareness and Screening Project takes place in barbershops and hair salons that have predominantly African American clients in two Washington counties with large African American populations. The project, which began in 2006, consists of training barbers and stylists to provide information about hypertension and the signs and symptoms of heart attack and stroke to their clients. Blood pressure readings are taken, and clients with high blood pressure are encouraged to see a health care provider.

Links to the program logic model

As shown in the 2007–2008 logic model, the African American Awareness and Screening Project directly contributes to the supporting (i.e., short-term) outcome of increased public awareness of heart disease and stroke signs and symptoms. This activity is also linked to the intermediate outcomes of increased recognition of symptoms and reduced time between recognition of symptoms and taking action to call 911 and to greater control of risk factors (e.g., high blood pressure, elevated cholesterol levels), on the basis of the theory that increased awareness of high blood pressure and high cholesterol levels will lead to improved control of these risk factors.

Evaluation approach

Evaluation planning focused on the stage of development of the activity. Because the African American Awareness and Screening Project was implemented in 2006, evaluation efforts targeted the implementation of the activity and assessment of the outcomes of increased public awareness of signs and symptoms of heart attack and stroke, as well as control of risk factors. Furthermore, the logic model assisted in developing the evaluation questions. By looking at the logic model to determine expected activities and outcomes, WaHDSPP staff were able to develop and prioritize the questions that the evaluation should answer, including the fidelity of the project, the impact of the project, and lessons learned and implemented.

The evaluation involves a mixed-methods approach, including both qualitative and quantitative data, and includes 1) the identification of facilitators of and barriers to the project through key stakeholder interviews, 2) a review of quarterly progress reports to evaluate fidelity to the original project plan, and 3) focus group sessions with participating barbers and stylists to evaluate trainings, project implementation, and perceptions about what worked and what did not work. Outcome evaluation will be conducted through review of screening results and follow-up to determine how many screened participants with high blood pressure were treated by a health care provider.

Back to top

Lessons Learned and Future Directions

Reflecting on the past 6 years of program activity, WaHDSPP staff identified several key lessons that have facilitated the use of programmatic logic models:

  1. Logic models have assisted the WaHDSPP in developing its theory of change. By stating the theory of change, the program can better identify intermediate steps — and related indicators — that precede long-term outcomes and identify the incremental steps that precede short-term outcomes, allowing the program to monitor progress in a more proximal manner.
  2. The WaHDSPP has used the annual occasion of preparing a continuing application as a time for collective reflection, not only on how to better refine existing activities or develop new ones, but also as a time to critically examine and improve the logic model and evaluation tools.
  3. The logic model now categorizes activities and outcomes according to their place in the spectrum of prevention, allowing WaHDSPP staff to visually track the expected impacts resulting from program activities over time and to think more clearly about how achievements in impacts at one stage of prevention may facilitate achievements at another stage by shrinking the size of the vulnerable population (5).
  4. Periodic reflection allows the program to incorporate new knowledge, such as information and resources from the CDC DHDSP, peers, and the literature to enhance evaluation planning.
  5. Logic modeling is integral to program planning, implementation, and evaluation. The model set forth in CDC’s six-step Framework for Program Evaluation in Public Health describes this collaborative approach (6).
  6. Evaluation and logic model development should be conducted in partnership with program stakeholders, with results feeding directly back into ongoing program planning and progress monitoring. Logic models and evaluation plans are dynamic tools to guide the program in carrying out activities but should always be developed and refined in partnership with activity work plans and the key staff and partners involved in the work.

These lessons have been incredibly valuable in informing and improving the direction of the WaHDSPP and demonstrate how logic models are useful in public health program planning and evaluation.

Back to top

Author Information

Corresponding Author: Susan Ladd, MS, Behavioral Scientist, Division for Heart Disease and Stroke Prevention, Centers for Disease Control and Prevention, 4770 Buford Highway NE, Mailstop K-47, Atlanta, Georgia 30341. Telephone: 770-488-5448. E-mail: sladd@cdc.gov.

Author Affiliations: Marilyn Sitaker, Miriam Patanian, Washington State Department of Health, Olympia, Washington; Jan Jernigan, Division for Heart Disease and Stroke Prevention, Centers for Disease Control and Prevention, Atlanta, Georgia.

Back to top

References

  1. Sitaker M, Kemple A, Hartwick N. The burden of heart disease and stroke in Washington State. Olympia (WA): Washington State Department of Health, Community and Family Health; 2004. http://www.doh.wa.gov/cfh/heart_stroke/publications/ hdsp_burden.pdf.* Accessed August 9, 2007.
  2. Centers for Disease Control and Prevention. A public health action plan to prevent heart disease and stroke. Atlanta (GA): U.S. Department of Health and Human Services, Centers for Disease Control and Prevention; 2003. http://www.cdc.gov/dhdsp/library/action_plan/index.htm.
  3. Washington State Department of Health. Washington state public health action plan for heart disease and stroke prevention and management. Olympia (WA): Washington State Department of Health, Community and Family Health. http://www.doh.wa.gov/cfh/heart_stroke/state_plan.htm.* Accessed August 9, 2007.
  4. Centers for Disease Control and Prevention. Developing an evaluation plan. Atlanta (GA): U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, Division for Heart Disease and Stroke Prevention; 2006. http://www.cdc.gov/DHDSP/state_program/evaluation_guides/ evaluation_plan.htm.
  5. Milstein B, Jones A, Homer JB, Murphy D, Essien J, Seville D. Charting plausible futures for diabetes prevalence in the United States: a role for system dynamics simulation modeling. Prev Chronic Dis 2007;4(3).  http://www.cdc.gov/pcd/issues/2007/jul/06_0070.htm.
  6. Centers for Disease Control and Prevention. Framework for program evaluation in public health. MMWR Recomm Rep 1999;48(RR-11):1-40.

Back to top

 


Table

Return to your place in the textTable. Evaluation Plana, Washington Heart Disease and Stroke Prevention Program (WaHDSPP)b, 2007–2008
Questions Indicators Data Sources Data Collected by Data Analysis Communicating Results
To Whom? How?
What key lessons were learned from evaluation of the previous barber/hair stylist outreach program conducted by WaHDSPP? Product = List of key lessons learned Written summary July 2007–August 2007 Document review CDC Steering Council

Community partners in King and Pierce counties

Progress reports to funding agency (CDC)

Calls with CDC Project Officer

Present results to Steering Council

Meeting with local partners

Was the hair stylist outreach program for community centers and faith-based organizations conducted as planned? Process measures; feedback from stylists  Quarterly progress reports and stylist focus group September 2007–February 2008 Analysis of focus group data
What was the impact of the barber/hair stylist outreach program? Of those whose screening results indicate high BP, the number who were treated by a health care provider Interviews with sample of clients with high BP results September 2007–February 2008 Analysis of survey responses

CDC indicates Centers for Disease Control and Prevention; BP, blood pressure.
a Based on the CDC Division for Heart Disease and Stroke Prevention Evaluation Guide. b Program promotes awareness of signs and symptoms of heart disease and stroke and targets African Americans (supporting outcomes: increased public awareness of risk factors, signs and symptoms of heart disease and stroke, and calling 911; intermediate outcomes: controlled risk factors, increased recognition of symptoms, and decreased time between recognizing symptoms and calling 911).

Back to top

*URLs for nonfederal organizations are provided solely as a service to our users. URLs do not constitute an endorsement of any organization by CDC or the federal government, and none should be inferred. CDC is not responsible for the content of Web pages found at these URLs.

 




 



The opinions expressed by authors contributing to this journal do not necessarily reflect the opinions of the U.S. Department of Health and Human Services, the Public Health Service, the Centers for Disease Control and Prevention, or the authors’ affiliated institutions. Use of trade names is for identification only and does not imply endorsement by any of the groups named above.


 Home 

Privacy Policy | Accessibility

CDC Home | Search | Health Topics A-Z

This page last reviewed March 30, 2012

Centers for Disease Control and Prevention
National Center for Chronic Disease Prevention and Health Promotion
 HHS logoUnited States Department of
Health and Human Services