Health Information Technology:

More Detailed Plans Needed for the Centers for Disease Control and Prevention's Redesigned BioSense Program

GAO-09-100, Nov 20, 2008

Additional Materials:

Contact:

Valerie C. Melvin
(202) 512-6304
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

In 2003, the Centers for Disease Control and Prevention (CDC), an agency within the Department of Health and Human Services (HHS), developed an electronic syndromic surveillance system called BioSense that uses health-related data to identify patterns of disease symptoms prior to specific diagnoses. In late 2007, CDC began to redesign the program to improve collaboration with stakeholders and address identified management weaknesses. Pursuant to House Report 110-231, GAO evaluated the BioSense program, focusing on the cost and timeline estimates and performance measures and benchmarks for implementing the program, among other objectives. To accomplish this, GAO analyzed relevant program documentation and interviewed CDC officials responsible for developing and implementing BioSense.

While CDC identified annual and long-term cost and timeline estimates and performance measures for the initial design of BioSense, these estimates and measures did not reflect the implementation of its redesigned program. CDC subsequently developed a draft plan for the redesigned program that described high-level cost and timeline estimates; however, the estimates are not reliable, and the plan did not include performance measures. According to best practices, cost estimates should be well-documented, comprehensive and accurate, and must be credible before they can be considered to be reliable. However, CDC's cost estimates for the redesigned program are not reliable because they are only partially documented, are not comprehensive and accurate, and therefore are not credible. Best practices for reliable timeline estimates include the identification of resources to complete each task, establishment of a critical path, and analysis of risks to the schedule. However, the agency has not implemented these practices, resulting in timelines for the redesigned program that are not reliable. The Office of Management and Budget directs agencies to define outcome-based performance measures to gauge program results early enough for stakeholder review, and industry experts describe the need for stakeholder input in developing performance measures in order to monitor performance. While CDC established performance measures and benchmarks for the initial implementation of the BioSense program, it has not yet developed outcome-based performance measures to monitor the progress of the redesigned program and does not intend to complete their development until the end of 2009. Until program officials develop reliable cost and timeline estimates and outcome-based performance measures for the redesigned BioSense program, they will lack key components needed to effectively manage the program, increasing the risk that the agency will perpetuate weaknesses identified in its initial implementation of the program and related system.

Status Legend:

More Info
  • Review Pending-GAO has not yet assessed implementation status.
  • Open-Actions to satisfy the intent of the recommendation have not been taken or are being planned, or actions that partially satisfy the intent of the recommendation have been taken.
  • Closed-implemented-Actions that satisfy the intent of the recommendation have been taken.
  • Closed-not implemented-While the intent of the recommendation has not been satisfied, time or circumstances have rendered the recommendation invalid.
    • Review Pending
    • Open
    • Closed - implemented
    • Closed - not implemented

    Recommendations for Executive Action

    Recommendation: To ensure that CDC defines reliable plans for effectively managing the development and implementation of the redesigned BioSense program and its related system, the Director of CDC should instruct the Director of the National Center for Public Health Informatics to develop reliable cost estimates for the program that are well-documented, comprehensive and accurate, and credible while in the planning phase of the redesigned BioSense program.

    Agency Affected: Department of Health and Human Services: Public Health Service: Centers for Disease Control

    Status: Open

    Comments: In September 2009, CDC provided us with documentation describing its efforts to develop reliable cost estimates for the BioSense program following practices outlined in GAO's Cost Estimating and Assessment Guide. However, the agency had not yet completed reliable cost estimates for the redesigned BioSense program. The cost estimates provided were high-level and detailed estimates had not been prepared. In July 2010, CDC officials confirmed that current efforts to "repurpose" the program will require the agency to revise its cost estimates for the program's new direction. CDC officials reported that they expected to provide detailed cost estimates to reflect the agency's revised plans for BioSense in November 2010. However, as of July 2011, CDC had not provided GAO with any cost estimates. Additionally, CDC officials stated that they are currently redesigning the BioSense program and system again, and redefining the overall BioSense governance structure to a community of practice model. They further stated that the cost estimates for the newly redesigned program have not been developed, and they did know when they would be available for GAO's review.

    Recommendation: To ensure that CDC defines reliable plans for effectively managing the development and implementation of the redesigned BioSense program and its related system, the Director of CDC should instruct the Director of the National Center for Public Health Informatics to develop reliable timeline estimates for implementing the program while in the planning phase of the redesigned BioSense program.

    Agency Affected: Department of Health and Human Services: Public Health Service: Centers for Disease Control

    Status: Open

    Comments: In September 2009, CDC provided us a work breakdown structure that identified specific timeline estimates for each of the major BioSense tasks and a critical path. BioSense program officials also provided assessments of risks to the completion of the tasks identified in the work breakdown structure. In March 2010, CDC officials provided GAO with an updated work breakdown structure for BioSense with milestones updated through 2010, along with identification of risks. However, none of the risk assessments were based on statistical analyses. Specifically, the risk assessments did not identify the probability of each risk event occurring, time variances to allow for contingencies, or the expected impact on the tasks that would be affected by the occurrence of each event. In June 2010, CDC entered into a contract for development of a detailed project plan, assumptions and a risk mitigation plan for the newly repurposed BioSense program based on stakeholder involvement. The agency intended to provide these plans to GAO in August 2010. However, in our continuing efforts to follow-up with them, BioSense program officials did not deliver these plans. Further, in July 2011, CDC officials stated that the BioSense program was being redesigned again and that new schedules had not yet been estimated. Until program officials define timelines and work breakdown structures for the newly redesigned program and thoroughly analyze risks to their schedules, they will not have completed activities that are essential to the development of reliable timeline estimates for implementing the redesigned program.

    Recommendation: To ensure that CDC defines reliable plans for effectively managing the development and implementation of the redesigned BioSense program and its related system, the Director of CDC should, with stakeholder input, instruct the Director of the National Center for Public Health Informatics to develop outcome-based performance measures that address all phases of the program and that focus the success of the program on CDC's federal, state, and local partners' use of the system for responding to public health events while in the planning phase of the redesigned BioSense program.

    Agency Affected: Department of Health and Human Services: Public Health Service: Centers for Disease Control

    Status: Open

    Comments: In July 2009, CDC provided us a strategic plan for the BioSense program that included information on each of the program's goals and strategies for fiscal years 2008-2012. The strategic plan included milestones that matched up with each strategy, and a grid that showed the years in which each milestone would be relevant. However, the plan did not contain outcome-based performance measures for achieving the program's goals. CDC noted that the intent of the BioSense strategic plan was to provide direction for a future detailed program plan that is intended to include such performance measures. In July 2010, CDC officials informed us that BioSense staff were working in concert with their contractor and external stakeholders to develop an updated strategic plan for BioSense that was to include outcome-based performance measures for each of the program's milestones. At that time, the agency intended to share this plan with GAO upon completion in December 2010. However, program officials did not provide the plan in response to our continuing follow-up efforts. In July 2011, CDC officials stated that the full strategic plan for the newly designed BioSense program, including performance measures, would not be available until autumn 2011.