Cognitive Interviewing

Cognitive interviewing is a popular method for evaluating survey questions in the Federal Government. It offers a detailed depiction of meanings and processes used by respondents to answer questions—which ultimately impacts the survey data. As such, the method offers insight that can enhance understandings of question validity and response error.

Characteristics of a Cognitive Interview study:

• Purposive sample selection: Because the goal is to identify the presence of problems, as opposed to making estimations or causal statements, a randomly drawn sample is not required.
• Qualitative methodologies are used to collect and analyze data
• Typically Small samples: 20-50 respondents

Follow-up questions or cognitive probes are used to examine thought processes.

• Probes can target specific processes.
  • Comprehension: What do you think 'abdomen' refers to?
  • Recall: How did you remember that?
  • Judgment: How sure are you about that?
  • Response: How did you pick an answer to that?
• Probes can also be more general and elicit narrative information about respondent's circumstances.
  • Why did you answer that way?
  • How did you come up with your answer?
  • Tell me more about that.

Findings of a Cognitive Interview study:

• Are based on narrative information about how respondents answered questions.
• Are grounded in the experiences of the respondents.
• Provide insight into question interpretation and patterns of calculation.

Analysis of Qualitative Interview Data

• Simultaneous pursuit of data collection & analysis
• Constructing analytic codes & themes from data (grounded theory)
• Make comparisons during each stage of analysis, for example…
  • Explore question response in 1 interview (comprehension; recall; judgment; response)
  • Compare that finding to the next interview (is it the same? If not, how is it different?)

Levels of Analysis in Cognitive Interviewing

Level 1 - Conducting Interviews

Level 2 - Summarizing Interview Notes

Level 3 - Comparing Across Respondents

Level 4 - Comparing Across Groups

Level 5 - Drawing Conclusions about Question Performance