Weapon System Requirements:
Detailed Systems Engineering Prior to Product Development Positions Programs for Success
GAO-17-77: Published: Nov 17, 2016. Publicly Released: Nov 17, 2016.
The Department of Defense expects to spend more than $1.4 trillion to develop and procure its current portfolio of major weapon system programs, such as the F-35 Joint Strike Fighter and the KC-46A refueling tanker. DOD's major programs have historically come in over cost and behind schedule.
Our latest look at 9 programs found that delays and overruns might be averted through early, detailed systems engineering—a disciplined process for ensuring that program requirements can be met with available resources.
We recommended DOD submit systems engineering plans to Congress when requesting new program funding.
F-35 Fighter Jet
Picture of an F-35 fighter jet in flight.
What GAO Found
GAO's analysis of nine case studies identified four factors that frame the challenge posed by a given weapon system's requirements: acquisition approach, technology status, design maturity, and system interdependency. Systems engineering is the primary means for determining whether and how that challenge can be met. It is a disciplined learning process that translates requirements into specific design features and thus identifies key risks to be resolved. GAO's prior best practices work has found that if detailed systems engineering is done early, a program can resolve such risks through trade-offs and additional investments before a program starts. A key point in systems engineering where this match can be assessed is the preliminary design. As shown below, establishing a preliminary design through early detailed systems engineering portends better program outcomes than doing so after program start.
Timing of Systems Engineering for Problematic and Successful Programs
GAO's analysis of selected Department of Defense (DOD) programs illustrates the relationship among the four factors, systems engineering, and program outcomes. Programs with modest requirements and early detailed systems engineering had better outcomes. For example, the Small Diameter Bomb Increment I program, which delivered within cost and schedule estimates, had an incremental approach, mature technologies, a derivative design, and detailed systems engineering before development began. Programs that began with more challenging requirements and insufficient systems engineering reported worse outcomes. For example, the F-35 Lightning II, which has encountered significant cost and schedule problems, began development with a single-step approach, a highly complex design, immature technologies, and little systems engineering.
Understanding the dynamic between a program's requirements, risks, and the requisite systems engineering effort has important implications for oversight. A particular challenge is that Congress often must consider requests to authorize and fund a new program in advance of the start of product development, when the business case would be better established. DOD policy requires that DOD decision makers have information about a proposed program's risk factors and systems engineering status, in a systems engineering plan, at the start of a new program. However, it is not clear whether Congress also has this information at that time. A systems engineering plan could provide more robust information to Congress when considering a budget request to start a new program. In commenting on a draft of this report DOD disagreed.
Why GAO Did This Study
Cost and schedule growth in DOD major defense acquisition programs persist, and some acquisition reform proponents believe such growth is due to unplanned changes in program requirements (commonly referred to as "requirements creep"). GAO found in June 2015 that cost and schedule growth are often more directly related to a lack of systems engineering, which, if done, would reduce risk by introducing discipline and rigor into the process of defining and understanding a program's initial requirements.
House Armed Services Committee Report 114-102 contained a provision for GAO to review the DOD requirements process. This report (1) identifies a framework for assessing the challenge posed by weapon system requirements and the extent of systems engineering done before product development begins; (2) illustrates the relationship between systems engineering and program outcomes; and (3) assesses implications for program oversight. GAO analyzed a non-generalizable sample of nine case studies. GAO assessed the extent to which systems engineering was conducted before development by reviewing program requirements and analyzing cost and schedule documentation for each case study. GAO also reviewed prior GAO work and interviewed DOD officials.
What GAO Recommends
To support oversight and inform budget decisions, Congress should consider requiring DOD to report on systems engineering status of each major acquisition program in the department's annual budget request.
For more information, contact Michael J. Sullivan at (202) 512-4841 or firstname.lastname@example.org.
Matter for Congressional Consideration
Comments: Congress has not yet taken action on the matter for consideration. The Department of Defense (DOD) non-concurred and as of March 16, 2018, DOD officials reported they have closed this recommendation because they do not agree that the systems engineering plan is the most effective means to provide Congress insight into the program's risk. DOD officials stated that the timing of the systems engineering plan and any updates are not aligned to inform a budget decision that could occur as much as 18 months prior to program initiation; and existing statutory certifications and reports, such as 2366a and 2366b requirements, submitted to Congress contain adequate information regarding program risk and technical maturity. GAO will continue to monitor.
Matter: To enhance program oversight and provide more robust input to budget deliberations, Congress should consider requiring DOD to report on each major acquisition program's systems engineering status in the department's annual budget request, beginning with the budget requesting funds to start development. The information could be presented on a simple timeline--as done for the case studies in this report--and at a minimum should reflect the status of a program's functional and allocated baselines as contained in the most current version of the program's systems engineering plan.