Modernizing our Public Health Surveillance Systems

By: Paula Yoon, ScD, MPH, EIS ‘94
Director, Division of Health Informatics and Surveillance

Disease surveillance is the foundation of public health practice providing essential data to inform decision making and respond to health threats. In the past, surveillance systems were fairly simple and might require disease-specific data collection forms (paper or electronic) that were completed by epidemiologists and sent by fax or email, databases to store data that were often entered by hand, and tools to analyze and chart or map the data. Surveillance today is more complex. Data collection systems need to capture standardized data from electronic medical records or other sources, package that data into structured messages, transport the messages between systems, validate the messages for accuracy and completeness, unpack the messages and transform the data, provision the data into databases, and pull the data from the databases for analysis and visualization. And, it all needs to be done accurately and in near real-time, or as fast as possible. As an example, the figure below depicts the data flow for nationally notifiable diseases that are reported to CDC by state health agencies.

CDC’s surveillance systems serve critical public health functions but many of our systems use aging technologies that have been patched together over time and need to be rebuilt or replaced. We need systems that are less complex than the monolithic systems we use now, can easily be updated, and can be extended for multiple data collections purposes (see blog "Developing Forward Looking Software at CDC"). With this in mind, we are exploring technologies such as application programming interfaces (APIs) and microservices that are used successfully by many other organizations to deliver timely, high-quality data when and where it is needed.

The biggest challenge we face in modernizing our surveillance systems is not the technology; it is the changes that we need to make in our infrastructure and culture. We need to employ new methodologies such as agile development and DevOps for building, testing, and deploying our systems. We need to modernize our approach to IT governance to ensure systems and data are secure, and can handle automated iterative testing, discrete functionality monitoring, frequent deployment of software updates, and on-demand scaling. Equally as important, we need to enhance the IT workforce and change the way we procure and manage IT contracts. Current rules and funding cycles make it difficult to procure cutting-edge experience and expertise, coordinate across multiple IT projects, and transition systems from one contractor to another. Regardless of whether we build, borrow, or buy software, we need a workforce with knowledge of advanced industry standards and best practices to provide oversight of contract work and to make informed decisions about the technologies we need to build robust surveillance systems.

HHS Idea Lab programs such as the Entrepreneurs-in-Residence (EIRs) are helping us to address these challenges. Our two EIRs—a software architect and a data architect—are creating an R&D team to pilot and test new software, developing a metadata repository to coordinate and standardize the data we collect, drafting architectural blueprints to guide future software development, and establishing new processes like DevOps to improve collaboration across the agency. The innovation and disruption brought by the EIRs has been just the spark we needed.

The modernization of our surveillance systems won’t happen overnight but we are moving in the right direction with the ultimate goal of providing the right data to the right person at the right time for effective public health action.

If you have any comments or questions about this blog, we invite you to submit your feedback to surveillancepractice@cdc.gov with the subject line as the title of this blog.