Skip directly to search Skip directly to A to Z list Skip directly to page options Skip directly to site content

How Do You Know Which Health Care Effectiveness Research You Can Trust? A Guide to Study Design for the Perplexed

Figure 1. Healthy user bias, a type of selection bias, is demonstrated in a study of 3,415 patients with pneumonia (and at high risk for flu and its complications), where elderly flu vaccine recipients were already healthier than nonrecipients. Figure is based on data extracted from Eurich et al (13).

CharacteristicVaccinated, %Not Vaccinated, %
Physically independent9584
Received pneumococcal vaccine689
Former smoker4230
Statin user3525

 

Return to Article



 

The figure shows that the rate of flu-related death was 48% lower and the rate of hospitalizations for pneumonia or influenza was 27% lower among vaccinated elderly people than among unvaccinated elderly people.

Figure 2. A weak cohort study comparing the risk of death or hospitalization for pneumonia or flu among vaccinated versus unvaccinated elderly: example of failure to control for healthy users. Figure is based on data extracted from Nichol et al (15). 

 

Return to Article



 

Bar chart shows a reduced risk of death of 45% before flu season, of 39% during flu season, and of 26% after flu season.

Figure 3. Healthy user bias: a strong controlled study disproving the effects of the flu vaccine on all-cause mortality in the elderly during the flu “off season” (control period). The cohort study compared vaccinated elderly and unvaccinated elderly. Figure is based on data extracted from Campitelli et al (17).

 

Return to Article



 

Figure 4. Example of selection bias: underlying differences between groups of medical providers show how they are not comparable in studies designed to compare providers using EHRs with providers not using EHRs. Figure is based on data extracted from Simon et al (23) and Decker et al (24). Abbreviation: EHR, electronic health record.

CharacteristicPercentage Using Electronic Health Records
Size of practice
Large (≥7 physicians)52
Small (1–3 physicians)29
Type of hospital
Teaching hospital40
Nonteaching hospital14
Age of physician, y
≤4546
46–5537
>5526

 

Return to Article



 

Figure 5. Example of weak post-only cross-sectional study that did not control for selection bias: the study observed differences between practices with EHRs and practices with paper records after the introduction of EHRs but did not control for types of providers adopting EHRs. Note the unlikely outcome for nonsmoker. Figure is based on data extracted from Cebul et al (26). Abbreviations: BMI, body mass index; EHR, electronic health record.

Health OutcomePercentage of Patients Achieving Outcome
Electronic Health Record–Based PracticePaper-Based Practice
Blood pressure control (<140/80 mm Hg)5639
Weight control (body mass index <30)3334
Nonsmoker8252

 

Return to Article



 

Figure 6. Differences in patient characteristics between EHR-based practices and paper-based practices in a weak post-only cross-sectional study that did not control for selection bias. Abbreviation: EHR, electronic health record. Figure is based on data extracted from Cebul et al (26).

Patient CharacteristicPercentage of Patients Achieving Outcome
Electronic Health Record–Based PracticePaper-Based Practice
Medicaid (poor)723
Nonwhite4485
Medicare (elderly)3720
Commercial health insurance4810

 

Return to Article



 

A diagram shows how randomization often ensures a fair comparison when assessing the effects of an intervention. The intervention begins with a population (eg, patients, health centers). A picture of hand flipping a coin illustrates how randomization often eliminates selection bias. The flip of the coin decides who is randomized to the intervention and who is not. Each arm of the study (intervention and no intervention) produces results, and the results are compared.

Figure 7. Randomized controlled trial: the “gold standard” of research design.

 

Return to Article



 

Figure 8. A strong randomized controlled trial of the effect of health information technology on the prevention of drug-related injuries among nursing home residents. Intervention participants received computerized warnings about unsafe combinations of drugs. Figure is based on data extracted from Gurwitz et al (30).

Type of InjuryNo. of Injuries per 100 Residents per Month
InterventionNo Intervention (Control)
Nonpreventable10.810.4
Preventable4.03.9

 

Return to Article



 

Figure 9. Elderly people who begin benzodiazepine therapy (recipients) are already sicker and more prone to fractures than nonrecipients. Figure is based on data extracted from Luijendijk et al (35).

Patient CharacteristicPercentage Increase in Risk (Hazard Ratio) Benzodiazepine Recipients vs Nonrecipients
Female67
Depression53
Hypertension29
Pain-related joint complaints45
Health self-reported as worse than that of peers50
Current smoker36

 

Return to Article



 

Among previous users, the authors found 63 fractures and odds of a hip fracture of 0.9. Among current users, the authors found 73 fractures and odds of a hip fracture of 1.75.

Figure 10. Weak post-only epidemiological study suggesting that current users of benzodiazepines are more likely than previous users to have hip fractures. Figure is based on data extracted from Ray et al (32).

 

Return to Article



 

The figure presents 4 graphs. See text for complete description.

Figure 11. Several examples of effects that can be detected in interrupted time-series studies. The blue bar represents an intervention.

 

Return to Article



 

This figure consists of 2 graphs. The first shows a 60% decrease in benzodiazepine use in New York among female users after policy was implemented but no similar decrease in New Jersey. The second graph shows no change in the risk of hip fracture for either New York or New Jersey after policy was implemented.

Figure 12. Benzodiazepine (BZ) use and risk of hip fracture among women with Medicaid before and after regulatory surveillance restricting BZ use in New York State. A BZ user was defined as a person who had received at least 1 dispensed BZ in the year before the policy. From Annals of Internal Medicine, Wagner AK, Ross-Degnan D, Gurwitz JH, Zhang F, Gilden DB, Cosler L, et al. Effect of New York State regulatory action on benzodiazepine prescribing and hip fracture rates. 2007;146(2):96–103 (33). Reprinted with the permission of American College of Physicians, Inc.

 

Return to Article



 

Figure 13. Underreporting of calories and fat consumption due to social desirability among women and men. Figure is based on data extracted from Hebert et al (38). Fat intake was measured as the absolute percentage change for every 1% change in social desirability bias. The zero-line indicates no underreporting.

MeasureUnderreporting
WomenMen
Caloric intake, kcal−68.0−38.9
Fat intake, percentage−33−13

 

Return to Article



 

Figure 14. Study that contaminated intervention group by unwittingly tipping parents off to the socially desired outcome: fewer hours of television time per day for children. Figure is based on data extracted from Taveras et al (40).

TimingNo. of Self-Reported Hours of Television per Day
InterventionNo Intervention
Before intervention2.672.44
After intervention2.132.36

 

Return to Article



 

The number of hours per week in television and computer use declined in 15 months by 17.5 hours in the intervention group and 3.4 hours in the control group. This difference in decline between the 2 groups was significant.

Figure 15. Strong randomized controlled trial design using an electronic device that caused an involuntary reduction in television and computer use. The difference in decline in viewing between the intervention group and control group was significant. Figure is based on data extracted from Epstein et al (42).

 

Return to Article



 

From preintervention to postintervention, the percentage of patients who received beta blockers increased from 60% to 78%, the percentage who received thrombolytics increased from 67% to 79%, and the percentage who received lidocaine decreased from 25% to 12%.

Figure 16. Percentage of acute myocardial infarction patients who received essential life-saving drugs (β blockers and thrombolytics) and a drug linked with increased mortality (lidocaine) in control hospitals before and after an intervention. Figure is based on data extracted from Soumerai et al (43).

 

Return to Article



 

Figure 17. Example of a weak post-only study of a hospital safety program and mortality that did not control for history. Narrow bar shows start of quality of care program. There is no evidence that data are available for the years leading up to the program. The study did not define the intervention period other than to state that planning occurred in 2003. Figure is based on data extracted from Pryor et al (45). Abbreviation: FY, fiscal year.

Fiscal YearDeaths per 100 Discharges
1999Unknown
2000Unknown
2001Unknown
2002Unknown
2003Unknown
20042.2
20052.1
20062.0
20071.9
20081.9
20091.9
20101.8

 

Return to Article



 

Figure 18. Example of a strong time-series design that controlled for history bias in the Institute for Healthcare Improvement’s 100,000 Lives Campaign. Figure is based on data from the Agency for Healthcare Research and Quality (48).

YearDeaths per 100 Discharges
19932.72
19942.63
19952.58
19962.54
19972.46
19982.50
19992.46
20002.37
20012.32
20022.24
20032.22
20042.13
2005 (Quality of care program began in January 2005)2.09
20062.04
20071.94
20082.03
20091.92
20101.90
20111.91

 

Return to Article



 

The opinions expressed by authors contributing to this journal do not necessarily reflect the opinions of the U.S. Department of Health and Human Services, the Public Health Service, the Centers for Disease Control and Prevention, or the authors' affiliated institutions.
Top