Many Analyses of Alternatives Have Not Provided a Robust Assessment of Weapon System Options
GAO-09-665: Published: Sep 24, 2009. Publicly Released: Oct 26, 2009.
Department of Defense (DOD) weapon programs often experience significant cost and schedule problems because they are allowed to start with too many technical unknowns and not enough knowledge about the development and production risks they entail. GAO was asked to review the department's Analysis of Alternatives (AOA) process--a key first step in the acquisition process intended to assess the operational effectiveness, costs, and risks of alternative weapon system solutions for addressing a validated warfighting need. This report (1) examines whether AOAs have been effective in identifying the most promising options and providing a sound rationale for weapon program initiation, (2) determines what factors have affected the scope and quality of AOAs, and (3) assesses whether recent DOD policy changes will enhance the effectiveness of AOAs. To meet these objectives, GAO efforts included collecting information on AOAs from 32 major defense acquisition programs, reviewing guidance and other documents, and interviewing subject matter experts.
Although an AOA is just one of several inputs required to initiate a weapon system program, a robust AOA can be a key element to ensure that new programs have a sound, executable business case. Many of the AOAs that GAO reviewed did not effectively consider a broad range of alternatives for addressing a warfighting need or assess technical and other risks associated with each alternative. For example, the AOA for the Future Combat System program, one of DOD's large and most complex development efforts, analyzed the operational performance and cost of its alternatives but failed to compare the technical feasibility and risks, assuming that the technologies would perform as forecasted. Without a sufficient comparison of alternatives and focus on technical and other risks, AOAs may identify solutions that are not feasible and decision makers may approve programs based on limited knowledge. While many factors can affect cost and schedule outcomes, we found that programs that had a limited assessment of alternatives tended to have poorer outcomes than those that had more robust AOAs. The narrow scope and limited risk analyses in AOAs can be attributed in part to program sponsors choosing a solution too early in the process, the compressed timeframes that AOAs are conducted under, and the lack of guidance for conducting AOAs. While AOAs are supposed to provide a reliable and objective assessment of viable weapon solutions, we found that service sponsors sometimes identify a preferred solution or a narrow range of solutions early on, before an AOA is conducted. The timing of AOAs has also been problematic. Some AOAs are conducted under compressed timeframes in order to meet a planned milestone or weapon system fielding date and are conducted concurrently with other key activities required to become a program of record. This can short-change a comprehensive assessment of risks and preclude effective cost, schedule, and performance trade offs from taking place prior to beginning development. Furthermore, while DOD has an opportunity to influence the scope and quality of AOAs, it has not always provided guidance for conducting individual AOAs. Recognizing the need for more discipline in weapon systems acquisition, DOD recently revised its overall acquisition and requirements policies. If implemented properly, the revised policies could provide a better foundation for planning and starting new programs with sound, knowledge-based business cases. Included in the revised acquisition policy are several mechanisms to improve the AOA process. For example, the policy revisions should help ensure that DOD direction is provided before AOAs are started and that they are conducted at an early point in the acquisition process where their results can inform decisions affecting program initiation. While these policy changes are promising, DOD must ensure that they are consistently implemented and reflected in decisions on individual programs. Furthermore, more specific criteria and guidance for how AOAs should be conducted may need to be developed to ensure they meet their intended objectives and provide an in-depth assessment of alternatives.
- Review Pending
- Closed - implemented
- Closed - not implemented
Recommendations for Executive Action
Recommendation: To further strengthen the effectiveness of AOAs in helping DOD establish sound business cases for major weapon programs, the Secretary of Defense should establish specific criteria and guidance for how AOAs should be conducted, including how technical and other programmatic risks should be assessed and compared.
Agency Affected: Department of Defense
Status: Closed - Implemented
Comments: In providing comments on this report, the agency concurred with this recommendation and indicated that the role of the analysis of alternatives (AOA) had been codified in revisions to the department's acquisition policy in December 2008. Subsequently, the Weapon Systems Acquisition Reform Act of 2009, specified that the Director of Cost Assessment and Program Evaluation (DCAPE) is responsible for leading the development of study guidance for AoAs. DOD clarified the functions and responsibilities of the DCAPE in a policy directive in 2012 (DODD 5105.84, May 11, 2012), including authority to guide and assess AoAs in support of military acquisition program decisions. Specifically, DCAPE is responsible for ensuring that AoAs consider trade-offs among effectiveness, suitability, and costs of alternatives, by (a) developing AoA study guidance for major defense acquisition programs, (b) approving AoA study plans for each program, and (c) evaluating the adequacy of each AoA conducted, to include a determination on whether the analysis is consistent with the AoA guidance. This broad policy guidance should contribute to improvements in the conduct of AOAs, however, GAO has not assessed the nature and extent of DCAPE's guidance for specific AOAs.
Recommendation: To further strengthen the effectiveness of AOAs in helping DOD establish sound business cases for major weapon programs, the Secretary of Defense should ensure that AOAs are completed and approved before program requirements--key performance parameters and attributes--are finalized and approved.
Agency Affected: Department of Defense
Status: Closed - Implemented
Comments: In providing comments on this report, the agency concurred with this recommendation. DOD revised its acquisition policy for weapon system programs in December 2008. One key element of the revised policy is that an AOA is required to be completed before the initiation of weapon system development. In addition, DOD is required by legislation to certify to Congress, among other things, that an AOA was completed before program initiation. GAO has tracked a number of future major defense acquisition programs as part of its annual assessment of selected weapon system programs (GAO-13-294SP) to gain insights into the implementation of recent acquisition reform initiatives. GAO has found that many of these future programs appear to have conducted AOAs prior to the start of system development and formal program initiation.