FBI Following a Number of Key Acquisition Practices on New Case Management System but Improvements Still Needed
GAO-07-912, Jul 31, 2007
The Sentinel program is intended to replace and expand on the Federal Bureau of Investigation's (FBI) failed Virtual Case File (VCF) project and thereby meet the bureau's pressing need for a modern, automated capability to support its field agents and intelligence analysts' investigative case management and information sharing requirements. Because of the FBI's experience with VCF and the importance of Sentinel to the bureau's mission operations, GAO was asked to conduct a series of reviews on the FBI's management of Sentinel. This review focuses on the FBI's (1) use of effective practices for acquiring Sentinel and (2) basis for reliably estimating Sentinel's schedule and costs. To address its objectives, GAO researched relevant best practices, reviewed FBI policies and procedures, program plans and other program documents, and interviewed appropriate program officials.
The FBI is managing its Sentinel program according to a number of key systems acquisition best practices. For example, the FBI has followed best practices when soliciting offers from contractors to lead the development of Sentinel; it has also followed the practices in evaluating the offers and making a contract award decision. In addition, it has established and is following effective processes to proactively identify and mitigate program risks before they have a chance to become actual cost, schedule, or performance problems. Further, it has taken a range of steps to effectively define expectations for its prime contractor and to measure performance against these expectations and related incentives and hold the contractor accountable for results. However, the bureau has not done the same for one key aspect of tracking and overseeing its program management support contractors. In particular, it has not established performance and product quality standards for these support contractors. According to FBI officials, such standards are not necessary because they monitor their support contractors on a daily basis, including the review and approval of all work products. By not implementing this practice, GAO believes that the FBI's monitoring does not adequately ensure that Sentinel support contractors are performing important program management functions effectively and efficiently. The FBI's policies, procedures, and supporting tools that form the basis for Sentinel's schedule and cost estimates do not adequately reflect key best practices. While the FBI has issued an information technology (IT) program management handbook, related guidance, and tools that define how IT program schedules and costs are to be estimated, this handbook and related material do not, for example, address such key practices as having a historical database of program schedule and cost estimates to inform future estimates. As a result, the reliability of Sentinel's schedule and cost estimates is questionable. GAO's analyses of the Sentinel cost estimates and program officials' statements confirm this. For example, the analyses show that the estimates do not include all relevant costs, such as a technology refresh, and are not grounded in fully documented methodologies or a corporate history of experiences on other IT programs. FBI officials agreed that they need to update their IT program management handbook and related materials to incorporate schedule and cost estimating best practices and to establish a historical database of its estimating experiences on IT programs. Until FBI takes these steps, IT programs, such as Sentinel, are unlikely to have reliable schedule and cost estimates to support informed investment decision making, and their actual progress is unlikely to track closely to estimates.
- Closed - implemented
- Closed - not implemented
Recommendations for Executive Action
Recommendation: To strengthen the FBI's management of its Sentinel program, the FBI Director should instruct the bureau's CIO to work with Sentinel support contractors, where feasible, to establish and implement performance standards in statements of work relative to the quality and timeliness of products and the performance of services.
Agency Affected: Department of Justice: Federal Bureau of Investigation
Status: Closed - Not Implemented
Comments: In its comments on the draft of this report, FBI did not concur with this recommendation, stating that the Sentinel's program management office provides sufficient government oversight of its support contractors. Moreover, in August 2011, FBI officials identified changes to the program that, in their view, make this recommendation no longer relevant. They stated that the Sentinel program is now being developed largely by an in-house team under direct management of senior officials. This approach meant a shift away from the support contractors and contract vehicles in use at the time of our report and towards FBI staff. They also reported that FBI IT management is monitoring the quality and timeliness of Sentinel products and the performance of services for Sentinel's work products using a statement of objectives for the internal team and a Project Baseline Agreement. Nevertheless, FBI continues to use support contractors in its Sentinel program office and has not provided evidence that it has established performance standards for their statements of work. As a result, FBI has not demonstrated implementation of this recommendation.
Recommendation: To strengthen the FBI's management of its Sentinel program, the FBI Director should instruct the bureau's CIO to revise the IT handbook and related guidance to address schedule and cost estimating best practices that are identified in this report as not being addressed in FBI policies and procedures and ensure that these best practices are fully employed on all major IT programs, including Sentinel.
Agency Affected: Department of Justice: Federal Bureau of Investigation
Status: Closed - Implemented
Comments: In commenting on the draft of this report, FBI agreed with this recommendation, and has subsequently taken actions to implement it. For example, the bureau updated its IT program guidance (the Project Management Handbook) in September 2008 to require that IT programs follow leading cost and schedule estimating practices, including those described in GAO's Cost Assessment Guide. The handbook now describes the four characteristics of a high-quality and reliable cost estimate (comprehensive, well-documented, accurate, and credible). In addition, the handbook now describes leading practices associated with effective schedule estimating, including capturing key activities, assigning resources to the activities, establishing their critical path, and integrating the activities horizontally and vertically. Furthermore, the handbook states that the Office of IT Program Management has established an historical/lessons learned database with critical information from other projects and programs that should be used for comparison purposes when creating schedule and cost estimates. We could not verify whether these practices for cost and schedule estimation were fully employed on individual programs. Nevertheless, we consider this recommendation largely implemented based on the direction and guidance established in the handbook.