This is the accessible text file for GAO report number GAO-12-339 
entitled 'Defense Management: Guidance and Progress Measures Are 
Needed to Realize Benefits from Changes in DOD's Joint Requirements 
Process' which was released on February 24, 2012. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as 
part of a longer term project to improve GAO products' accessibility. 
Every attempt has been made to maintain the structural and data 
integrity of the original printed product. Accessibility features, 
such as text descriptions of tables, consecutively numbered footnotes 
placed at the end of the file, and the text of agency comment letters, 
are provided but may not exactly duplicate the presentation or format 
of the printed version. The portable document format (PDF) file is an 
exact electronic replica of the printed version. We welcome your 
feedback. Please E-mail your comments regarding the contents or 
accessibility features of this document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

United States Government Accountability Office: 
GAO: 

Report to Congressional Committees: 

February 2012: 

Defense Management: 

Guidance and Progress Measures Are Needed to Realize Benefits from 
Changes in DOD's Joint Requirements Process: 

GAO-12-339: 

GAO Highlights: 

Highlights of GAO-12-339, a report to congressional committees. 

Why GAO Did This Study: 

The Department of Defense’s Joint Requirements Oversight Council 
(JROC) is charged with assisting in the prioritization of capability 
needs from a joint perspective and helping guide investments. The JROC 
is supported by the Joint Capabilities Integration and Development 
System (JCIDS) process. However, a congressional committee and GAO 
have expressed concerns about the extent to which JCIDS has been 
effective in prioritizing capability needs. The Ike Skelton National 
Defense Authorization Act for Fiscal Year 2011 required GAO to provide 
a report on the effectiveness of JCIDS in several areas. In addition 
to responding to this direction, GAO has more broadly evaluated the 
extent to which (1) the Joint Staff has developed and implemented an 
analytic approach to prioritize capability needs and (2) the JROC has 
considered aspects of the availability and operational support of 
weapon systems-—called sustainment—-when validating the requirements 
of proposed capability solutions. To do so, GAO analyzed capability 
documents, reviewed relevant guidance and law, and interviewed 
officials. 

What GAO Found: 

After studying the Joint Capabilities Integration and Development 
System (JCIDS) process since September 2010, the Joint Staff began 
initiating actions in October 2011 to better prioritize capability 
needs and align those needs with available budgetary resources. 
Specifically, according to Joint Staff officials, the Joint 
Requirements Oversight Council (JROC) has begun to consider the 
benefits and affordability of new capabilities within the context of 
joint capability areas and to evaluate possible duplication before 
validating new capability requirements. The Joint Staff has begun to 
implement a new approach to support JROC prioritization of capability 
needs, but the new approach is still evolving and has not been fully 
developed and clearly documented. New guidance does not clearly 
outline goals of the new approach, develop and communicate the 
analytic approach envisioned to support JROC decision making, or set 
out criteria and accompanying measures of progress. GAO previously 
reported that JCIDS’s ability to prioritize needs could be improved if 
it had an analytic approach to reviewing and validating proposals that 
would help ensure that the most important capability needs of the 
department are addressed. Until the Joint Staff takes steps to fully 
develop, document, and institutionalize the new analytic approach, it 
is not clear whether the current momentum for improving the JCIDS 
process will be sustained. 

JCIDS guidance in effect through December 2011 required that sponsors 
of potential major defense acquisition programs address sustainment 
information in capability development documents according to four 
metrics—materiel availability, operational availability, reliability, 
and ownership cost. Each of these metrics includes a set of 
potentially reportable criteria or data, which are listed as review 
criteria and are suggested, but not clearly required by the guidance, 
to be included in the metric. Based on GAO’s analysis of six 
capability development documents, GAO found that all of the documents 
provided information on the four required sustainment metrics, but the 
completeness of information for all of the metrics’ key criteria 
varied. Further, in some cases information that should have been 
included, according to Department of Defense officials, was not 
provided. The Joint Staff issued updated JCIDS guidance in January 
2012, but the guidance still does not clearly require program sponsors 
to report on the individual criteria for each of the four sustainment 
metrics. Without complete and detailed information on each of the 
individual criteria elements, the JROC may not have the information it 
needs to make the most informed decisions when validating the 
requirements of proposed solutions intended to mitigate capability 
gaps. 

What GAO Recommends: 

GAO recommends that the Vice Chairman of the Joint Chiefs of Staff (1) 
revise and implement JCIDS guidance to reflect recent changes to the 
process and establish criteria and measures for determining the 
relative importance of capability needs and (2) require program 
sponsors to address each criterion in JCIDS guidance related to 
sustainment in capability documents. DOD partially concurred with 
GAO’s recommendations. 

View GAO-12-339. For more information, contact John Pendleton at (202) 
512-3489 or pendletonj@gao.gov. 

[End of section] 

Contents: 

Letter: 

Background: 

The Joint Staff Has Initiated Efforts to Prioritize Capability Needs, 
but Implementation Processes Are Not Fully Developed and Clearly 
Documented: 

JROC Has Not Always Had Complete Information on Future Costs to 
Sustain Proposed Capabilities before Approving Documentation for 
Program Development: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Observations on Reporting Provisions in the National 
Defense Authorization Act for Fiscal Year 2011: 

Appendix II: Scope and Methodology: 

Appendix III: Comments from the Department of Defense: 

Appendix IV: GAO Contact and Staff Acknowledgments: 

Tables: 

Table 1: Elements of a Results-Oriented Management Framework: 

Table 2: Examples of Review Criteria for Each of the Four Sustainment 
Metrics: 

Table 3: Section 862(b)(1)(A)--Evaluate the Effectiveness of JCIDS in 
Achieving Timeliness in Delivering Capability to the Warfighter: 

Table 4: Section 862(b)(1)(B)--Evaluate the Effectiveness of JCIDS in 
Achieving Efficient Use of the Investment Resources of the Department 
of Defense: 

Table 5: Section 862(b)(1)(C)--Evaluate the Effectiveness of JCIDS in 
Achieving Control of Requirements Creep--and Section 862(b)(1)(D)--
Evaluate the Effectiveness of JCIDS in Achieving Responsiveness to 
Changes Occurring after the Approval of a Requirements Document 
(Including Changes to the Threat Environment, the Emergence of New 
Capabilities, or Changes in the Resources Estimated to Procure or 
Sustain a Capability): 

Table 6: Section 862(b)(1)(E)--Evaluate the Effectiveness of JCIDS in 
Achieving Development of the Personnel Skills, Capacity, and Training 
Needed for an Effective and Efficient Requirements Process: 

Table 7: Section 862(b)(2)(A)--the Time That Requirements Documents 
Take to Receive Approval through JCIDS: 

Table 8: Section 862(b)(2)(B)--the Quality of Cost Information 
Considered in JCIDS and the Extent of Its Consideration: 

Table 9: Section 862(b)(2)(C)--the Extent to Which JCIDS Establishes a 
Meaningful Level of Priority for Requirements: 

Table 10: Section 862(b)(2)(D)--the Extent to Which JCIDS Is 
Considering Trade-offs between Cost, Schedule, and Performance 
Objectives: 

Table 11: Section 862(b)(2)(E)--the Quality of Information on 
Sustainment Considered in JCIDS and the Extent to Which Sustainment 
Information Is Considered: 

Table 12: Section 862(b)(2)(F)--an Evaluation of the Advantages and 
Disadvantages of Designating a Commander of a Unified Combatant 
Command for Each JROC-Interest Requirements Document to Provide a 
Joint Evaluation Task Force for Certain Purposes: 

Figure: 

Figure 1: Interaction between DOD's Requirements Generation and 
Acquisition Processes: 

Abbreviations: 

CJCSI: Chairman of the Joint Chiefs of Staff Instruction: 

DOD: Department of Defense: 

JCIDS: Joint Capabilities Integration and Development System: 

JROC: Joint Requirements Oversight Council: 

[End of section] 

United States Government Accountability Office: 
Washington, DC 20548: 

February 24, 2012: 

Congressional Committees: 
Like the rest of the federal government, the Department of Defense 
(DOD) is operating in a constrained budget environment and facing 
difficult decisions about how to invest its resources to meet national 
security objectives. The Joint Requirements Oversight Council (JROC) 
directly and indirectly supports the Chairman of the Joint Chiefs of 
Staff’s role as the principal military advisor to the President and 
other senior national security officials. The JROC consists of the 
Vice Chairman of the Joint Chiefs of Staff and other senior military 
officials and, among other duties, is expected to assist in the 
prioritization of joint military requirements and ensure that resource 
levels needed to fulfill those requirements are consistent with the 
level of priority. These duties can help ensure that proposed weapon 
systems are prioritized to create an affordable portfolio. The JROC 
uses the Joint Capabilities Integration and Development System 
(JCIDS), a process created in 2003 to guide the development of 
capabilities from a joint perspective, to help it identify capability 
gaps and validate the requirements of proposed capability solutions to 
mitigate those gaps. JCIDS operates in conjunction with two other DOD 
processes—the acquisition process, which facilitates the development 
and acquisition of proposed capabilities, and the budgeting process, 
through which these capabilities are funded. The development and 
acquisition of weapon systems consume a substantial portion of DOD’s 
budget. From fiscal years 2011 through 2016, DOD plans to spend almost 
$390 billion on its current portfolio of major defense acquisition 
programs.[Footnote 1] Moreover, DOD spends billions of dollars each 
year to sustain these weapon systems. As we have reported, costs to 
operate and support these systems make up at least 70 percent of a 
system’s life cycle costs.[Footnote 2] The House Armed Services 
Committee and GAO have expressed concerns about the extent to which 
JCIDS has been effective in assigning priorities to capabilities 
proposed to meet the needs of the joint force. 

A provision in the Ike Skelton National Defense Authorization Act for 
Fiscal Year 2011 required GAO to provide a report on the effectiveness 
of JCIDS in several areas, such as prioritizing joint requirements and 
considering information on costs of sustaining future programs. 
[Footnote 3] In appendix I, we have reprinted the elements of the 
provision to be covered by the report and our response to each one. 
[Footnote 4] As part of our analysis, we also evaluated (1) the extent 
to which the Joint Staff has developed and implemented an analytic 
approach to prioritize capability needs and (2) the extent to which 
the JROC has considered aspects of the availability and operational 
support requirements of weapon systems—-called sustainment--when 
validating the requirements of proposed capability solutions. 

To assess the extent to which the Joint Staff has developed and 
implemented an analytic approach to prioritize capability needs, we 
reviewed relevant law and Joint Staff policy documents and related 
guidance on the roles and responsibilities of the JCIDS process and 
the JROC as it pertains to prioritization. We interviewed Joint Staff 
officials to discuss changes to its approach for managing requirements 
within capability portfolios and making recommendations for trade-offs 
among alternatives. We also reviewed prior GAO reports that discussed 
capability development and requirements prioritization and compared 
the current efforts to minimize risks and identify unnecessary overlap 
and duplication with what we reported in the past. To assess the 
extent to which the JROC has considered aspects of the availability 
and operational support requirements of weapon systems-—called 
sustainment-—when validating the requirements of proposed capability 
solutions, we reviewed DOD and Joint Staff policy documents and 
related guidance on the requirement to develop sustainment metrics for 
capability documents for certain programs processed through JCIDS. We 
interviewed DOD and Joint Staff officials to discuss the preparation, 
presentation, and consideration of sustainment data. We also selected a 
nonprobability sample of six requirements documents for programs that 
have been required to report sustainment metrics to determine whether 
the documents contained the required metrics and the supporting 
information included with the metrics. 

We conducted this performance audit from April 2011 through February 
2012 in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit 
to obtain sufficient, appropriate evidence to provide a reasonable 
basis for our findings and conclusions based on our audit objectives. 
We believe that the evidence obtained provides a reasonable basis for 
our findings and conclusions based on our audit objectives. Additional 
details on our scope and methodology are in appendix II. 

Background: 

JCIDS; the Planning, Programming, Budgeting and Execution process; and 
the Defense Acquisition System broadly make up DOD's overall defense 
acquisition management framework. JCIDS was implemented in 2003 to 
guide future defense programs from a joint capabilities perspective. 
JCIDS is one of the first steps in DOD's acquisition processes; JCIDS 
participants work to identify and determine whether to validate the 
need for capabilities proposed by the services, the defense agencies, 
and the combatant commands. Once a requirement is validated, the 
services rely on the DOD's Planning, Programming, Budgeting and 
Execution process, through which DOD allocates financial resources 
across the department--including the services--to identify funding for 
validated capability solutions. DOD then manages the development and 
procurement of proposed capabilities through the Defense Acquisition 
System. 

DOD implemented the JCIDS process in 2003 in an effort to assist the 
JROC by changing DOD's requirements validation process from a service-
specific perspective to a joint capabilities perspective. The JROC, 
which is chaired by the Vice Chairman of the Joint Chiefs of Staff, 
consists of a general or admiral from each of the military services 
and may include combatant commanders or deputy commanders when 
directed by the JROC Chairman.[Footnote 5] The JROC is charged with 
assisting the Chairman of the Joint Chiefs of Staff with a number of 
tasks, including (1) identifying, assessing, and approving joint 
military requirements to meet the national military strategy; (2) 
establishing and assigning priority levels for joint military 
requirements; and (3) reviewing the estimated level of resources 
required to fulfill each joint military requirement and ensuring that 
the resource level is consistent with the requirement's priority, 
among others. The JROC also assists acquisition officials in 
identifying alternatives to any acquisition programs that meet joint 
military requirements for the purposes of certain statutory 
provisions, addressing matters such as cost growth. In 2009, the 
Weapon Systems Acquisition Reform Act expanded the role of the JROC by 
directing it to assist the Chairman of the Joint Chiefs of Staff in 
ensuring that trade-offs among cost, schedule, and performance 
objectives are considered for joint military requirements and 
establishing an objective for the overall period of time within which 
an initial operational capability should be delivered to meet each 
joint military requirement.[Footnote 6] 

The JROC reviews requirements for programs designated as JROC-interest 
based on their expected cost and complexity[Footnote 7] and, under 
guidance in effect through December 2011, also reviewed programs at 
the request of certain senior DOD officials. Within JCIDS, the JROC is 
supported in its duty to review and validate joint capability needs by 
the Joint Capabilities Board and six Functional Capabilities 
Boards.[Footnote 8] The Joint Capabilities Board is chaired by the 
Director of the Joint Staff's Directorate for Force Structure, 
Resources, and Assessment, and each Functional Capabilities Board is 
chaired by a general/flag officer or civilian equivalent. The Joint 
Capabilities Board reviews capability documents before they are passed 
on to the JROC for its review and also serves as the validation 
authority for certain programs that do not reach JROC-interest 
thresholds,[Footnote 9] although the JROC may review any JCIDS 
document or other issues requiring joint resolution. Functional 
Capabilities Boards are responsible for reviewing proposed 
requirements specific to joint capability areas, such as protection, 
logistics, or battlespace awareness. 

In JCIDS, the JROC and its supporting organizations review 
requirements documents related to capability gaps and the major 
defense acquisition programs intended to fill those gaps prior to key 
acquisition milestones. These requirements documents--initial 
capabilities documents, capability development documents, and 
capability production documents for materiel solutions and change 
recommendations for nonmateriel solutions--are submitted into the 
JCIDS process by capability sponsors.[Footnote 10] The initial 
capabilities document identifies a specific capability gap, or set of 
gaps, and if a materiel solution is required, helps inform the initial 
stages of the acquisition process, which include an analysis of the 
alternative solutions to fulfilling the capability need and the 
selection of a preferred system concept. When the technology 
development phase of the acquisition process is complete, a program 
sponsor completes a capability development document that includes more 
detail on the desired capabilities of the proposed system and defines 
the system's key performance parameters or attributes against which 
the delivered increment of capability will be measured. Finally, the 
sponsor prepares a capability production document to describe the 
actual performance of the system that will deliver the required 
capability. Figure 1 depicts how JCIDS reviews align with the 
acquisition process. 

Figure 1: Interaction between DOD's Requirements Generation and 
Acquisition Processes: 

[Refer to PDF for image: illustration] 

Capabilities assessment: 
Initial capabilities document. 

Analysis of alternatives: 
Analysis of alternatives[A]. 

Technology development: 
Capability development document.  

Engineering and manufacturing development: 
Capability production document. 

Production: 

Source: GAO analysis of DOD policy.      

[A] The 2012 JCIDS guidance directs the Functional Capabilities Boards 
to review the analysis of alternatives and other analyses for certain 
capability requirements. See Chairman of the Joint Chiefs of Staff 
Instruction 3170.01H, Joint Capabilities Integration and Development 
System, enc. A, para. 2(c)(1)(b) (Jan. 10, 2012) (hereinafter cited as 
CJCSI 3170.01H (Jan. 10, 2012)). 

[End of figure] 

The House Armed Services Committee and a panel established by the 
committee have discussed long-standing challenges with the JCIDS 
process and the JROC's fulfillment of its statutory responsibilities. 
The House Armed Services Committee, in a report accompanying a bill 
for the National Defense Authorization Act for Fiscal Year 2008, 
described a legislative provision that would allow for joint decision 
making as opposed to service-centric budget considerations by 
incorporating clear priorities and budget guidance into the JROC 
process.[Footnote 11] In 2009, the House Armed Services Committee 
established a panel on defense acquisition reform because of a sense 
that the acquisition system was not responsive enough for today's 
needs, not rigorous enough in protecting taxpayers, and not 
disciplined enough in the acquisition of weapon systems for tomorrow's 
wars. The panel received testimony that the Joint Staff lacked some of 
the analytical expertise necessary to ensure that the JCIDS process 
rigorously vets proposed requirements. Additionally, since 2008 we 
have reported on these challenges. We reported in 2008 that the JCIDS 
process was not effective in prioritizing capability gaps, and we 
noted that capability needs continued to be proposed and defined by 
the services with little involvement from the joint community. We 
recommended that the Secretary of Defense direct the Chairman of the 
Joint Chiefs of Staff to develop an analytic approach within JCIDS to 
better prioritize and balance the capability needs of the military 
services, combatant commands, and other defense components.[Footnote 
12] DOD partially agreed with our recommendation but did not fully 
implement it, and prioritization remains service driven. More 
recently, in June 2011 we reported that the JROC did not always 
consider trade-offs among cost, schedule, and performance objectives; 
prioritize requirements; consider redundancies across proposed 
programs; or prioritize and analyze capability gaps in a consistent 
manner.[Footnote 13] We recommended that the JROC require higher-
quality resource estimates from requirements sponsors to ensure that 
best practices are being followed, provide a sound basis to ensure 
that trade-offs are considered, prioritize requirements across 
proposed programs, and address potential redundancies during 
requirements reviews, among other steps. DOD partially agreed with our 
recommendations and commented that improvements to the quality of 
resource estimates would be addressed in upcoming changes to the JCIDS 
process. 

In May 2011, we also reported that combatant command[Footnote 14] 
officials raised concerns that JCIDS focuses more on long-term service-
centric capability gaps than on combatant commands' more immediate and 
largely joint gaps.[Footnote 15] JCIDS was designed as a deliberate 
process to meet longer-term joint needs. To address urgent needs, DOD 
established a separate process--the joint urgent operational needs 
process in 2005. The joint urgent operational needs process was 
intended to respond to needs associated with combat operations in 
Afghanistan and Iraq and the War on Terror. The revised JCIDS guidance 
canceled separate guidance for joint urgent operational needs and 
incorporates and describes the joint urgent operational needs process. 
Urgent operational needs, as defined by the new JCIDS guidance, are 
capability requirements needed for ongoing or anticipated contingency 
operations that if left unfulfilled could potentially result in loss 
of life or critical mission failure.[Footnote 16] For this report, we 
focus on requirements that have not been identified as urgent and 
instead follow the deliberative JCIDS process. 

The Joint Staff Has Initiated Efforts to Prioritize Capability Needs, 
but Implementation Processes Are Not Fully Developed and Clearly 
Documented: 

The Joint Staff is undertaking efforts to improve the ability to 
prioritize capability needs from a joint perspective through JCIDS and 
to align those needs with available budget resources. However, 
implementation processes for JCIDS's new approach to managing 
requirements and considering affordability are still evolving and have 
not been fully developed and clearly documented. 

Changes in JCIDS Processes Are Intended to Improve Prioritization of 
Capability Needs and Guide Future Investments: 

Determining priorities among joint requirements has been a 
responsibility of the JROC since Congress amended section 181 of Title 
10 of the U.S. Code in 2008 to require the JROC to assist in 
establishing and assigning priority levels for joint military 
requirements and to help ensure that resource levels associated with 
those requirements are consistent with the level of priority.[Footnote 
17] DOD officials acknowledge that JCIDS has been ineffective in 
helping the JROC carry out these responsibilities. We have previously 
reported that JCIDS's ability to align resources to balance competing 
needs could be improved if it had an analytic approach that provided a 
means to review and validate proposals to ensure that the most 
important capability needs of the department are being addressed. We 
further said that such an approach should establish criteria and 
measures for identifying capability gaps and determining the relative 
importance of capability needs. Finally, the approach should result in 
measurable progress in allocating resources in order to eliminate 
redundancies, gain efficiencies, and achieve a balanced mix of 
executable programs.[Footnote 18] 

DOD officials told us that downward pressure on the defense budget has 
led the Joint Staff to change how the JCIDS process is used to 
strengthen its ability to support JROC members in making trade-off 
decisions among requirements and balancing risks across the force 
within expected resources. In fall 2011, according to officials, the 
incoming Vice Chairman of the Joint Chiefs of Staff, as the Chairman 
of the JROC, began to make changes in the JCIDS processes to focus on 
what capabilities currently exist and weigh the benefits of investing 
in new capabilities with their estimated costs early in the review 
process. The Joint Staff issued draft guidance in October 2011 and 
began implementation based on the draft guidance. The Joint Staff 
issued the final guidance in January 2012. Descriptions of these 
changes and their implementation follow: 

* New capabilities will be considered as part of a "capability 
portfolio approach." Under the portfolio approach, officials stated 
that JROC members are to ensure that proposed investments in 
capabilities address joint needs or they will not be validated to 
proceed to the acquisition process. In addition to validating 
capability proposals, according to Joint Staff officials, the JROC has 
begun examining how the services' existing programs can support joint 
operations to reduce duplication of capabilities. As of December 2011, 
according to Joint Staff officials, the last five closed meetings of 
the JROC began conversations about how to meet requirements by 
considering available capabilities and the costs and benefits of 
proposed programs. As a result, according to Joint Staff officials, at 
least five classified programs have been reviewed and altered by 
either comparing redundant capabilities, reducing capacity, adjusting 
delivery schedules, or directing follow-on analysis before moving 
programs forward. To support JROC decision making, officials reported 
that Functional Capabilities Boards will be tasked with examining 
requirements, their associated capability gaps, and proposed solutions 
within their capability portfolios, and independently assessing how a 
proposed capability fits into its corresponding joint capability area. 
The Functional Capabilities Boards previously had responsibility for 
identifying, assessing, and prioritizing (if required) joint 
capability needs proposals within assigned joint capability areas but, 
according to officials, have not always carried out these 
responsibilities. Previously, Functional Capabilities Boards acted 
primarily as technical reviewers of requirements documents, and 
program sponsors briefed the JROC on the attributes of the program. 
However, the new guidance does not specify how the independent 
assessment is to be conducted, and it is too soon to tell how the 
Functional Capabilities Boards will respond to the new requirement. 
Officials reported that Functional Capabilities Boards are expected to 
develop methodologies on a case-by-case basis. DOD officials said that 
the analytic approach and uses of analytic information will evolve 
over time. 

DOD has previously attempted to manage capabilities departmentwide 
through a portfolio approach, but has never fully implemented the 
approach. In 2006, DOD established an effort to manage resources 
across capability areas by establishing capability portfolio managers 
to enable DOD to develop and manage capabilities across the department 
rather than by military service or individual program, and by doing 
so, to improve the interoperability of future capabilities, minimize 
capability redundancies and gaps, and maximize capability 
effectiveness. However, as we reported in 2008, capability portfolio 
managers make recommendations on capability development issues within 
their portfolios but do not have independent decision-making 
authority. In 2011, Joint Staff officials told us that they were 
unaware of capability portfolio managers' active involvement in the 
JCIDS process. 

* Attendance at JROC meetings will be limited to key decision makers 
and stakeholders. Beginning in October 2011, the JROC Chairman began 
to limit attendance at JROC meetings to facilitate candid discussion 
among senior leaders about priorities for joint requirements and 
alternative solutions. Joint Staff officials told us that previously, 
meetings were open to a broad range of interested parties and service 
sponsors provided briefings on their proposals for new capabilities. 
Under the new approach being implemented, a representative of one of 
the Functional Capabilities Boards will provide a briefing on the 
proposal to the JROC. The representatives would then present the 
board's independent assessment of benefits, costs, and risk for the 
JROC to discuss and decide upon. According to JCIDS officials, JROC 
members are expected to make decisions from the perspective of the 
joint force and avoid taking a service-centric approach. 

* Affordability of proposals will be a primary factor in validation 
decisions. JROC members have always been expected to consider the 
resource implications of validation decisions, but officials stated 
that until recently, these considerations have not been a focus 
because capabilities were not competing with each other for funding. 
According to officials, the Functional Capabilities Boards have been 
directed to take similar steps to ensure that capability proposals not 
only meet technical requirements but also represent the most efficient 
alternative for providing a capability within the joint capability 
area without creating duplication or overlap. According to JCIDS 
officials, the JROC is also reconsidering previous validation 
decisions and asking for changes to proposals to minimize 
costs.[Footnote 19] We reviewed recent Functional Capabilities Board 
briefings to the JROC, which provided information on how needs might 
be met with current capabilities and alternatives that might meet 
needs while minimizing costs. However, it is too soon to assess how 
the JROC will consider affordability of programs when making 
validation decisions. 

New JCIDS Processes Are Not Fully Developed and Clearly Documented: 

The Vice Chairman of the Joint Chiefs of Staff has sought some policy 
changes and, according to officials, provided other direction to 
implement changes in the JCIDS process, but the new approach to 
managing requirements and considering affordability is still evolving 
and has not been fully developed and clearly documented. According to 
Joint Staff officials, the dynamic fiscal environment and the 
evolutionary method being used to develop the new approach and 
implementation processes make it important that decision makers 
maintain flexibility in decision making. We believe that the new 
approach has promise in positioning the JROC to more accurately 
identify capability gaps and trade-offs, but it has not been fully 
developed to include steps to ensure that the approach is fully 
implemented, that the intent is fully communicated to all stakeholders 
involved, and that the results of the new approach will be measurable. 

We have previously reported that key practices for results-oriented 
management involve leadership from top officials as well as the 
involvement of stakeholders at all levels throughout a period of 
transition.[Footnote 20] We have also reported that in order to 
demonstrate a successful results-oriented framework, officials must 
include clearly defined measures to assess intended outcomes.[Footnote 
21] As shown in table 1, key practices for supporting change should 
include, among other actions, obtaining and sustaining support from 
senior leadership to facilitate the transformation, establishing clear 
lines of communication between all affected parties, and demonstrating 
value and credibility of new processes through the use of metrics. 

Table 1: Elements of a Results-Oriented Management Framework: 

Principle: Ensure top leadership drives the transformation; 
Critical elements: 
* Define and articulate a succinct and compelling reason for change; 
* Balance continued delivery of services with transformation 
activities. 

Principle: Establish a communication strategy to create shared 
expectations and report related progress; 
Critical elements: 
* Communicate early and often to build trust; 
* Ensure consistency of message; 
* Encourage two-way communication; 
* Provide information to meet specific needs of employees. 

Principle: Align goals and measures with agency and departmentwide 
goals; 
Critical elements: 
* Each of the agency's strategic plan goals is supported by 
performance measures; 
* Measures align with departmentwide goals; 
* Goals and measures cascade from the corporate level to the lowest 
level of the agency. 

Principle: Demonstrate results; 
Critical elements: 
* Includes a combination of output-and outcome-oriented measures; 
* Measures are clearly defined; 
* Provides trend data to demonstrate progress against targeted 
performance. 

Source: GAO. 

[End of table] 

According to Joint Staff officials, the Chairman of the JROC has been 
a driving force in positioning the JROC to take on the responsibility 
of aligning needs and balancing risk and resources, which fulfills one 
of the key steps for results-oriented management, but the approach is 
still new and officials have not completed all of the steps that 
facilitate institutional acceptance and implementation of the new 
approach. The Joint Staff has begun the process of change by 
articulating a clear rationale for change--that the JROC can more 
effectively represent the warfighters' requirements and make strategic 
trade-off decisions as budgets stay flat or decrease by taking a more 
active role in shaping an affordable joint force. However, best 
practices for managing a results-oriented change state that goals and 
procedures should be communicated to stakeholders throughout the 
organization so that they understand how they should implement the new 
approach and how the organization will measure progress. The Joint 
Staff issued guidance that outlines new procedures intended to 
establish an approach for prioritizing capabilities from a joint 
perspective and to increase the timeliness of JCIDS reviews by 
categorizing proposals according to the level of urgency of the need 
and streamlining procedures for urgent needs. However, the guidance 
does not clearly outline criteria and measures for demonstrating 
progress toward meeting the goal of aligning needs with available 
resources or clearly communicate the goals and the analytic approach 
envisioned to support JROC decision making. Further, the guidance does 
not describe how the proposed change will affect the services, 
combatant commands, and other stakeholders. Finally, the JCIDS 
guidance does not establish criteria and measures for demonstrating 
progress toward the goal of creating a balanced portfolio of programs 
that takes into account needs, risks, and available resources, nor do 
other documents provided to us by DOD. Measures such as the proportion 
of requirements that address joint priority needs versus service-
specific needs, the savings obtained through the elimination of 
redundant capabilities, and the comparison of estimated costs of a 
proposed program with the actual costs of operating the program over 
its life cycle could be helpful in assessing whether the process is 
balancing requirements with available resources or whether further 
adjustments to the process are needed. 

Considering new capabilities across the department in the context of 
joint capability areas can help DOD begin to identify priorities for 
future investment. However, unless the Joint Staff takes steps to 
define and institutionalize the new approach by adhering to the key 
principles of results-oriented management, it is not clear whether the 
current momentum in implementing an analytic approach through JCIDS 
will be sustained. 

JROC Has Not Always Had Complete Information on Future Costs to 
Sustain Proposed Capabilities before Approving Documentation for 
Program Development: 

Even though sustainment costs make up a significant portion of the 
total ownership costs of a weapon system, the JROC has not always had 
complete information on such costs when validating documentation used 
in the decision to initiate program development. "Sustainment" as a 
category represents a range of activities intended to ensure the 
availability of weapon systems and to support their operations, 
including some logistics and personnel services. During the 
identification of capability gaps and consideration of selected 
alternatives, it is difficult for the sponsors to provide detailed 
information on program capabilities and cost estimates. As a major 
defense acquisition program moves toward the development stage, JCIDS 
requires that more complete and accurate sustainment information be 
presented in capability development documents. According to DOD 
officials, decision makers need more accurate cost information to 
assess whether the benefit of a proposed capability justifies the cost 
of sustaining the capability over its life.[Footnote 22] JCIDS 
guidance requires that sponsors of potential major defense acquisition 
programs[Footnote 23] include sustainment information in capability 
development documents,[Footnote 24] which detail proposed solutions to 
fulfill capability needs. 

The JROC has generally relied on sponsor-provided assessments of 
sustainment information in the capability development documents to 
make its validation decisions, but these documents have not always 
included all the information suggested in JCIDS guidance or sufficient 
detail to enable the JROC to assess the quality of the information. A 
DOD manual[Footnote 25] regarding the development of sustainment 
information suggests that when sustainment requirements and underlying 
assumptions are not clearly documented, subsequent decisions about the 
project may be based on incorrect assumptions. Prior GAO work[Footnote 
26] suggests that gaps in joint warfighting capabilities and proposals 
to fulfill the gaps should be clearly identified to decision-making 
bodies, such as the JROC, to inform deliberations. Further, 
information should be complete so those making the important decisions 
may do so as effectively as possible. 

JCIDS guidance requires sponsors of major defense acquisition programs 
to address sustainment based on four metrics--materiel availability, 
operational availability, reliability, and ownership cost (renamed 
operation and support cost in January 2012).[Footnote 27] The guidance 
includes a series of criteria, which are listed as review criteria in 
the guidance. The criteria provide additional information on each 
metric. For example, JCIDS guidance lists as review criteria for the 
materiel availability metric whether there is a clear definition and 
accounting for the intended service life of the program, an 
identification of planned downtime, and a comparison of downtime value 
with experiences of other analogous systems, among other criteria. 
Table 2 outlines examples of key review criteria within each of the 
four sustainment metrics under the guidance in effect through December 
2011.[Footnote 28] 

Table 2: Examples of Review Criteria for Each of the Four Sustainment 
Metrics: 

Sustainment metric: Materiel availability; 
Total number of review criteria: 8; 
Examples of review criteria: 
* Accounts for the total population of end items being acquired for 
operational use; 
* Identifies and includes planned time periods during which the weapon 
system is not operationally available to perform an assigned mission; 
* Identifies sources of data and models used to establish and track 
materiel availability. 

Sustainment metric: Operational availability; 
Total number of review criteria: 5; 
Examples of review criteria: 
* Provides specific definitions for what constitutes failure; 
* Addresses time periods during which the weapon system is 
nonfunctional as a result of failure; 
* Addresses specific issues associated with failure, including time 
needed to recover a weapon system after suffering a failure and time 
to perform diagnostics. 

Sustainment metric: Reliability; 
Total number of review criteria: 8; 
Examples of review criteria: 
* Defines how time intervals for assessing the weapon system will be 
measured; 
* Identifies sources for baseline reliability data and addresses 
whether the proposed reliability value is consistent with comparable 
systems; 
* Addresses whether the proposed reliability of the weapon system is 
consistent with intended operational use of system. 

Sustainment metric: Ownership cost; 
Total number of review criteria: 9; 
Examples of review criteria: 
* Defines through analysis which type of ownership cost structure was 
used for assessing program cost; 
* Includes all required costs for a program, regardless of funding 
source; 
* Includes an approach to monitor, collect, and validate operation and 
support cost data. 

Source: GAO analysis of DOD guidance. 

[End of table] 

Program sponsors provide initial information on the sustainment 
metrics for proposed capability solutions when they submit a 
capability development document, one of three capability documents the 
JROC considers in its review and potential validation of capability 
proposals. Officials from both the Joint Staff and the Office of the 
Under Secretary of Defense for Acquisition, Technology and Logistics 
review the capability development documents and, according to 
officials, verify that all required sustainment elements have been 
included before the documents are validated by the JROC. Officials 
also provide their independent assessments of the quality of the cost 
estimates to the JROC.[Footnote 29] Officials from the Office of the 
Under Secretary of Defense for Acquisition, Technology and Logistics 
said they work with program sponsors to ensure that life cycle 
sustainment planning and costs are as accurate as possible. Officials 
from the Office of the Under Secretary of Defense for Acquisition, 
Technology and Logistics and the Joint Staff told us that they 
consider reported sustainment information important to a program's 
development and review all reported information. 

The JCIDS Manual notes that listed criteria, information, and 
activities cannot necessarily be applied to all systems. Sponsors have 
a degree of latitude in determining which items are applicable for 
their specific concept, technology, system, or combination of these. 
For example, a program sponsor for a major defense acquisition program 
is required to report a measure for operational availability, but 
would not necessarily have to report on the respective criteria, such 
as addressing downtime associated with failure, including recovery 
time or movement of maintenance teams to the work site. Because the 
guidance does not specifically require program sponsors to report on 
the individual criteria, they generally include some, but not all, of 
the individual criteria. 

Our analysis of six capability development documents found that all of 
the documents provided information on all of the required sustainment 
metrics. However, we found that the completeness of information 
reported for all of the metrics' key criteria varied. Specifically, 
none of the documents included complete information for each of the 
four sustainment metrics' review criteria elements. In addition, each 
of the documents had some common omissions; for example, none of the 
six capability documents we reviewed included information on all nine 
of the ownership cost metric's criteria elements. Further, several of 
these documents only included information on one criteria element for 
a single metric, and none reported information on all of the elements 
for any of the metrics. Finally, when information on the metrics' key 
criteria was provided, the level of detail varied among the documents. 
For example, for some criteria, some documents provided a paragraph of 
supporting information and analysis whereas others provided single-
sentence responses that did not provide as substantial detail for the 
supporting criteria. According to officials, for the capability 
documents we reviewed, there were cases in which information on some 
of the suggested criteria should have been included but were not. 

Joint Staff officials and officials from the Office of the Under 
Secretary of Defense for Acquisition, Technology and Logistics noted 
that while criteria for each of the required sustainment metrics may 
not be applicable to every program, it would be beneficial to the JROC 
if the services reported on the criteria for each metric outlined in 
the guidance or indicated a reason why a specific criterion was not 
applicable. Officials from the Office of the Under Secretary of 
Defense for Acquisition, Technology and Logistics told us that when 
they conduct their reviews of sponsor-reported sustainment 
information, not all of the supporting documentation they need for a 
thorough independent assessment is available in the capability 
development document. These officials said that they can generally 
find detailed documentation on sustainment planning and costs from 
sources outside of the JCIDS process, but that the information is not 
always readily available within the JCIDS database. Ultimately, review 
efficiency could be improved if all the information were available in 
the JCIDS database. Updated JCIDS guidance issued in January 2012 
still does not clearly require program sponsors to report on the 
individual criteria for each of the sustainment metrics. The Joint 
Staff is developing a new reporting tool intended to provide a 
standard format for reporting sustainment information. The tool will 
require program sponsors to at least minimally address each of the 
four sustainment metrics in order to submit the capability development 
document for review through JCIDS, according to officials. However, 
officials stated that this tool will not require that sponsors address 
each of the individual criteria elements within the four sustainment 
metrics. 

Without complete and detailed information on each of the individual 
criteria elements, the JROC may not be in the best position to weigh 
the costs and benefits of a proposal within a capability portfolio. 
The quality and completeness of the data that sponsors provide through 
the capability development document in the JCIDS process will become 
more important as the JROC increases its examination of the benefits 
of programs balanced against their associated costs. As we have 
previously reported, incomplete and inaccurate sustainment information 
has been a long-standing problem for DOD.[Footnote 30] In November 
2009, a DOD team assessing weapon system acquisition reform reported 
that DOD lacked valid, measurable sustainment information to 
accurately assess how programmatic decisions affected life cycle costs 
and made recommendations to improve weapon system life cycle 
sustainment.[Footnote 31] Until the JROC requires program sponsors to 
report complete sustainment information, including both the overall 
metrics and the supporting criteria, the JROC may not always have the 
complete and detailed information it needs to make the most informed 
decisions. 

Conclusions: 

The prospect of declining budgets has amplified the need for DOD to 
prioritize among capability gaps and to use its resources to maximize 
the capabilities of the joint force. The Chairman of the JROC has 
begun to take steps to better balance risks across the joint force by 
examining proposals for new capabilities within the context of 
existing joint capability areas and to consider affordability, 
including sustainment costs, as a factor in validating requirements 
proposals. The revised approach is new and evolving, but in order for 
it to achieve the intended results of prioritizing capability needs 
and aligning those needs with available resources, the Joint Staff 
needs to take steps to fully develop the approach and document it more 
explicitly. Specifically, DOD does not yet have a documented 
implementation plan with measures of success that support change. In 
addition, having good sustainment information is a key element needed 
to improve JCIDS's success over the long term. Sustainment costs 
historically represent 70 percent of a system's life cycle costs, but 
DOD has been making decisions with incomplete information on 
sustainment and does not require that sponsors address all of the 
criteria outlined in JCIDS guidance. Until the JROC has developed and 
fully documented an approach for prioritizing capability needs and 
aligning these needs with available resources and has complete 
sustainment information associated with the operation of new 
capabilities, it will not be in the best position to align resources 
with priorities or balance costs with benefits in affordable 
investment plans. 

Recommendations for Executive Action: 

To help sustain momentum for efforts to bring a capability portfolio 
approach to the JCIDS process and to improve the quality of 
sustainment information reported in capability development documents, 
we recommend that the Vice Chairman of the Joint Chiefs of Staff, as 
the JROC Chairman, take the following two actions: 

* Revise and implement guidance to reflect changes to the JCIDS 
process as well as to establish criteria and measures for determining 
the relative importance of capability needs across capability areas 
and assessing progress. 

* Explicitly require that program sponsors address each of the 
criteria outlined for the individual sustainment metrics when 
submitting capability development documents. 

Agency Comments and Our Evaluation: 

The Joint Staff provided written comments on a draft of this report. 
In its comments, the Joint Staff said our report represented a fair 
and objective assessment of the JCIDS process. It partially agreed 
with both of our recommendations, citing ongoing and planned changes 
to the joint requirements development process. However, the comments 
did not detail any specific steps that DOD plans to take to address 
our recommendations. The comments are reprinted in their entirety in 
appendix III. The Joint Staff also provided technical comments, which 
we have incorporated into the report as appropriate. 

The Joint Staff partially concurred with our recommendation that the 
Vice Chairman of the Joint Chiefs of Staff revise and implement 
guidance to reflect changes to the JCIDS process as well as to 
establish criteria and measures for determining the relative 
importance of capability needs across capability areas and assessing 
progress. In its written response, the Joint Staff described recent 
initiatives to substantially change the joint requirements development 
process to require that capability requirements be evaluated within a 
capability portfolio by Functional Capability Boards, the Joint 
Capabilities Board, and the JROC. The Joint Staff also discussed its 
planned efforts to improve prioritization of capability needs and 
stated that JROC reviews will incorporate an evolving portfolio 
assessment tool. The Joint Staff expects that the departmentwide 
priorities outlined in DOD's strategic guidance as well as a revised 
process for assessing capability gaps and combatant command priorities 
will enable the JROC to make more informed decisions about priorities. 
While we agree that the Joint Staff has taken important steps to 
enable prioritization of capabilities, such as addressing 
prioritization in a new enclosure in its revised JCIDS Manual, the 
enclosure does not explicitly outline implementation processes. We 
continue to believe that clear guidance that establishes criteria for 
determining priority levels and measures for demonstrating progress 
will be essential in sustaining momentum toward the goal of creating a 
balanced portfolio of programs that takes into account needs, risks, 
and available resources. Moreover, providing guidance that fully 
documents the new procedures for assigning priority levels to 
capability gaps is an essential step toward clarifying how the 
procedures will be implemented. 

The Joint Staff also partially concurred with our recommendation that 
the Vice Chairman of the Joint Chiefs of Staff explicitly require that 
program sponsors address each of the criteria outlined for the 
individual sustainment metrics when submitting capability development 
documents. According to its written comments, the Joint Staff criteria 
for the sustainment metrics were designed to guide the development of 
requirements, but were not intended to be prescriptive because 
individual programs are unique and criteria applicable to one problem 
may not apply to another. We agree that each of the criteria may not 
be applicable to every program. However, if program sponsors addressed 
each criterion in some manner, including explaining that a criterion 
is not applicable to the program, the JROC would be assured that 
program sponsors considered all criteria when developing program 
proposals. Further, the Joint Staff commented that JCIDS reviews of 
capability development documents by Joint Staff and Office of the 
Secretary of Defense officials ensure that a document is thoroughly 
vetted for consideration by the JROC. It suggested that the inclusion 
of analyses and rationale for sustainment metrics development in 
capability development documents might be duplicative because this 
information is contained in acquisition documents that exist outside 
of JCIDS. However, as we noted in our report, the documents that 
contain the analysis and rationale for the required sustainment 
metrics are not necessarily reviewed by or available to the JROC 
members during their consideration of a capability development 
document. We continue to believe that the inclusion of a sponsor-
provided rationale for each metric criterion would enhance the 
thoroughness and efficiency of the JROC's review of sustainment 
information through JCIDS. 

We are sending copies of this report to interested congressional 
committees, the Secretary of Defense, the Vice Chairman of the Joint 
Chiefs of Staff, and other interested parties. In addition, the report 
is available at no charge on the GAO website at [hyperlink, 
http://www.gao.gov]. 

If you or your staff have any questions about this report, please 
contact me at (202) 512-3489 or pendletonj@gao.gov. Contact points for 
our Offices of Congressional Relations and Public Affairs may be found 
on the last page of this report. GAO staff who made key contributions 
to this report are listed in appendix IV. 

John Pendleton Director, Defense Capabilities and Management: 

List of Committees: 

The Honorable Carl Levin: 
Chairman: 
The Honorable John McCain: 
Ranking Member: 
Committee on Armed Services: 
United States Senate: 

The Honorable Daniel K. Inouye: 
Chairman: 
The Honorable Thad Cochran: 
Ranking Member: 
Subcommittee on Defense: 
Committee on Appropriations: 
United States Senate: 

The Honorable Howard P. "Buck" McKeon: 
Chairman: 
The Honorable Adam Smith: 
Ranking Member: 
Committee on Armed Services: 
House of Representatives: 

The Honorable C.W. "Bill" Young: 
Chairman: 
The Honorable Norman D. Dicks: 
Ranking Member: 
Subcommittee on Defense: 
Committee on Appropriations: 
House of Representatives: 

[End of section] 

Appendix I: Observations on Reporting Provisions in the National 
Defense Authorization Act for Fiscal Year 2011: 

Section 862 of the Ike Skelton National Defense Authorization Act for 
Fiscal Year 2011 requires the Comptroller General to carry out a 
comprehensive review of the Joint Capabilities Integration and 
Development System (JCIDS) and to submit to the congressional defense 
committees a report on the review.[Footnote 32] This appendix, in 
conjunction with the letter, addresses each of the reporting 
provisions as described in the act. Specifically, section 862 requires 
the following contents for the review: 

Purpose. The purpose of the review is to evaluate the effectiveness of 
JCIDS in achieving the following objectives: 

* Timeliness in delivering capability to the warfighter. 

* Efficient use of the investment resources of the Department of 
Defense (DOD). 

* Control of requirements creep.[Footnote 33] 

* Responsiveness to changes occurring after the approval of a 
requirements document[Footnote 34] (including changes to the threat 
environment, the emergence of new capabilities, or changes in the 
resources estimated to procure or sustain a capability). 

* Development of the personnel skills, capacity, and training needed 
for an effective and efficient requirements process. 

Matters considered. In performing the review, the Comptroller General 
is required to gather information on and consider the following 
matters: 

* The time that requirements documents take to receive approval 
through JCIDS. 

* The quality of cost information considered in JCIDS and the extent 
of its consideration. 

* The extent to which JCIDS establishes a meaningful level of priority 
for requirements. 

* The extent to which JCIDS is considering trade-offs between cost, 
schedule, and performance objectives. 

* The quality of information on sustainment considered in JCIDS and 
the extent to which sustainment information is considered. 

* An evaluation of the advantages and disadvantages of designating a 
commander of a unified combatant command for each requirements 
document for which the Joint Requirements Oversight Council (JROC) is 
the validation authority to provide a joint evaluation task force to 
participate in a materiel solution and to: 

- provide input to the analysis of alternatives; participate in 
testing (including limited user tests and prototype testing);provide 
input on a concept of operations and doctrine; provide end user 
feedback to the resource sponsor; and participate, through the 
combatant commander concerned, in any alteration of the requirement 
for such solution. 

Section 862 also provided definitions for JCIDS, requirements 
document, requirements creep, and materiel solution.[Footnote 35] 

Tables 3 through 12 contain our response to each of the requirements 
mandated by the Ike Skelton National Defense Authorization Act for 
Fiscal Year 2011.[Footnote 36] The provision mandating our report 
defined the JCIDS process by referring to the JCIDS guidance in effect 
from March 2009 through December 2011.[Footnote 37] Accordingly, our 
response to the mandated elements as presented in this appendix 
generally focuses on JCIDS under that guidance. However, we also 
provide some information relating to JCIDS as described in the revised 
guidance issued in January 2012. In addition, our assessments 
generally focused on those programs that were determined to be JROC-
interest, or those that were designated as major defense acquisition 
programs or major automated information systems and capabilities that 
have a potentially significant impact on interoperability in allied 
and coalition operations. Generally, these programs have greater costs 
or are more complex than smaller programs, and therefore provide an 
opportunity to assess the effectiveness of more aspects of the JCIDS 
process. 

Table 3: Section 862(b)(1)(A)--Evaluate the Effectiveness of JCIDS in 
Achieving Timeliness in Delivering Capability to the Warfighter: 

Summary assessment: 
The JCIDS guidance in effect through December 2011 did not directly 
facilitate the timely delivery of capabilities to the warfighter. 
Joint Staff officials are implementing new procedures to expedite 
reviews of urgent needs and establishing targets for anticipated 
delivery times for weapon systems to facilitate timely delivery of 
capabilities to meet needs. 

Key GAO observations: 
JROC review and validation of capability needs validated in 2010 and 
2011 ranged from 3 months to 17 months. Officials told us that the 
length of time it has taken for a capability need to be validated 
through JCIDS depended on several factors, including the quality and 
completeness of the original proposal and the maturity of the 
technology involved. According to officials, JCIDS reviews generally 
did not delay or expedite the delivery of a capability; instead, the 
development phases of the acquisition system took the most time 
between identifying a need and delivering a capability. JCIDS guidance 
in effect through December 2011 did not require the JROC to approve a 
weapon system's anticipated schedule when making decisions about 
validating the related requirements documents, and according to Joint 
Staff officials, the JROC typically has not tried to influence the 
delivery dates of capabilities. Recent changes to JCIDS processes 
require the JROC to set targets for the anticipated delivery schedule 
when validating a weapon system's capability development document and 
to revalidate the document if the delivery target is altered by more 
than 12 months. In addition, JCIDS now tailors requirements reviews 
according to the urgency of the need[A]. 

GAO scope and methodology: 
To evaluate the extent to which JCIDS had contributed to the timely 
delivery of capabilities to the warfighter, we reviewed policy 
documents about the joint requirements and acquisition processes. 
Using the Joint Staff's database of joint requirements documents 
reviewed through JCIDS, we selected a nonprobability sample of 12 
requirements documents for major weapon systems that the JROC reviewed 
from May 1, 2010, through May 31, 2011. We focused on major weapon 
systems because they were reviewed through each of the JCIDS phases, 
whereas other weapon systems are generally reviewed at the Joint 
Capabilities Board level but not by the JROC. Specifically, we 
analyzed the time that elapsed during the JCIDS review of four initial 
capability documents, five capability development documents, and three 
capability production documents. We interviewed DOD, Joint Staff, and 
service officials with detailed knowledge of the programs and the 
review process to confirm our findings. Finally, we reviewed JCIDS 
guidance issued in January 2012. 

Sources: DOD and GAO. 

[A] See CJCSI 3170.01H, enc. A, para. 1(d)(2) (Jan. 10, 2012), and 
JCIDS Manual, enc. E (Jan. 19, 2012). Specifically, the guidance 
establishes different document staffing and validation processes where 
a need is a joint urgent operational need or a joint emergent 
operational need. In 2005, DOD established an urgent-needs, expedited 
review process outside JCIDS because the JCIDS process was ineffective 
at addressing urgent wartime needs; the revised JCIDS instruction 
cancels the instruction driving that process. See CJCSI 3170.01H, 
para. 2(b) (Jan. 10, 2012) (canceling Chairman of the Joint Chiefs of 
Staff Instruction 3470.01, Rapid Validation and Resourcing of Joint 
Urgent Operational Needs (JUONs) in the Year of Execution (July 15, 
2005)). 

[End of table] 

Table 4: Section 862(b)(1)(B)--Evaluate the Effectiveness of JCIDS in 
Achieving Efficient Use of the Investment Resources of the Department 
of Defense: 

Summary assessment: 
The JROC has not been effective at prioritizing capability needs or 
guiding the efficient use of resources to meet joint capability needs 
and balance resources and risks. The Joint Staff is taking steps to 
improve prioritization by analyzing joint requirements in the context 
of alternative existing and proposed programs within capability areas. 

Key GAO Observations: 
As required by law and reflected in Joint Staff guidance, the JROC has 
the responsibility to assist the Chairman of the Joint Chiefs of Staff 
in developing and assigning priority levels for joint military 
requirements and to help ensure that resource levels associated with 
those requirements are consistent with the level of priority. However, 
the JROC has not explicitly considered requirements' affordability in 
the context of DOD's budget. As we have reported since 2008, the JROC 
has not prioritized requirements, considered redundancies across 
proposed programs, or prioritized and analyzed capability gaps 
consistently through JCIDS. A key reason it did not prioritize needs 
was that the JROC reviewed requirements separately and largely left 
prioritization and trade-off decisions to the military services.[A] 
Further, DOD's ability to use JCIDS to ensure the efficient use of 
resources within and among the services has been hampered by the 
JROC's inability to enforce joint priorities in service budget 
decisions. According to officials, the Joint Staff has recently 
initiated efforts to assess the importance of each capability proposal 
JCIDS reviews in relation to existing capabilities and other related 
capability needs through a "capability portfolio" approach. Criteria 
for validation of a proposal would include the uniqueness of the 
proposal and its costs and benefits. JCIDS's Functional Capabilities 
Boards are already providing this assessment to the JROC for its 
consideration in closed meetings. However, guidance on how the 
Functional Capabilities Boards are to assess priorities among 
capabilities is not specific as to how to identify joint priorities. 
See the section of our letter on the Joint Staff's efforts to 
prioritize capability needs for additional information. 

Summary assessment: 
To evaluate the effectiveness of the JCIDS process in efficiently 
investing DOD's resources, we reviewed relevant legislation and policy 
documents and guidance and discussed issues with senior DOD officials. 
We also reviewed previous GAO assessments of JCIDS and acquisition 
issues detailed in multiple reports, including GAO-11-502; Defense 
Acquisitions: DOD's Requirements Determination Process Has Not Been 
Effective in Prioritizing Joint Capabilities, GAO-08-1060 (Washington, 
D.C.: Sept. 25, 2008); and High-Risk Series: An Update, GAO-11-278 
(Washington, D.C.: February 2011). 

Sources: DOD and GAO. 

[A] GAO, DOD Weapon Systems: Missed Trade-off Opportunities During 
Requirements Reviews, GAO-11-502 (Washington, D.C.: June 16, 2011). 

[End of table] 

Table 5: Section 862(b)(1)(C)--Evaluate the Effectiveness of JCIDS in 
Achieving Control of Requirements Creep--and Section 862(b)(1)(D)--
Evaluate the Effectiveness of JCIDS in Achieving Responsiveness to 
Changes Occurring after the Approval of a Requirements Document 
(Including Changes to the Threat Environment, the Emergence of New 
Capabilities, or Changes in the Resources Estimated to Procure or 
Sustain a Capability): 

Summary assessment: 
The JCIDS process does not actively prevent or encourage requirements 
creep,[A] as these changes are driven by the program sponsor and may 
not appear during review of documents. 

Key GAO observations: 
We have previously reported that despite the importance of well-
developed and stable requirements for obtaining good weapon program 
outcomes, program requirements often change during development. For 
example, the threat the program originally addressed may change, or 
the user and acquisition communities may change opinions on what is 
needed from a program.[B] Programs that modified critical system 
characteristics after development start experienced higher levels of 
cost growth and longer delays in delivering capabilities. The JROC 
typically oversees changes only for those system characteristics 
considered critical to delivering a capability.c In March 2011, we 
reported that 21 of 39 major defense acquisition programs we assessed 
had at least one change to a critical characteristic, either 
increasing or decreasing a requirement.d Specifically, 10 of the 21 
programs reported adding or enhancing a critical characteristic; 3 
reported reducing, deferring, or deleting a critical characteristic; 
and 8 reported making both types of changes. 
The acquisition community has taken some steps to address requirements 
creep and changes made to a program's attributes by establishing 
configuration steering boards for major acquisition programs to review 
and limit requirements changes and avoid cost increases. However, as 
we reported in July 2011, individual programs varied in how they used 
these boards to control requirements and mitigate cost and schedule 
risks[E]. 

GAO scope and methodology: 
To evaluate the effectiveness of JCIDS in controlling requirements 
creep and responding to changes, we reviewed policy documents on the 
joint requirements process and the acquisition system and our prior 
reviews of JCIDS and acquisition issues, including GAO-11-233SP. 

Sources: DOD and GAO. 

[A] We asked officials about their perspectives on the responsiveness 
of JCIDS to changes occurring after the approval of a requirements 
document and they had no specific insights to offer. Other than the 
examples presented in this table, neither we nor officials were able 
to identify specific examples regarding the effectiveness of JCIDS in 
achieving responsiveness to changes. 

[B] GAO, Defense Acquisitions: Assessments of Selected Weapon 
Programs, GAO-08-467SP (Washington, D.C.: Mar. 31, 2008). 

[C] Specifically, changes may only appear before the JROC and the 
Joint Capabilities Board for validation if they affect the key 
performance parameters, key system attributes, or other performance 
attributes. Changes to characteristics of a lower priority are 
typically addressed by the military services. JROC-interest programs 
with approved capability development and production documents must 
return to the JROC if they experience certain cost growth. 

[D] GAO, Defense Acquisitions: Assessments of Selected Weapon 
Programs, GAO-11-233SP (Washington, D.C.: Mar. 29, 2011). 

[E] GAO, Defense Acquisitions: DOD Can Improve Its Management of 
Configuration Steering Boards, GAO-11-640 (Washington, D.C.: July 7, 
2011). 

[End of table] 

Table 6: Section 862(b)(1)(E)--Evaluate the Effectiveness of JCIDS in 
Achieving Development of the Personnel Skills, Capacity, and Training 
Needed for an Effective and Efficient Requirements Process: 

Summary assessment: 
In response to a requirement in the John Warner National Defense 
Authorization Act for Fiscal Year 2007,[A] DOD developed and 
implemented a training program for personnel who develop requirements 
for major defense acquisition programs. DOD is currently planning to 
expand the training program to personnel involved in developing joint 
requirements for programs that do not meet the threshold to be major 
defense acquisition programs. This program helps to develop the 
personnel skills, capacity, and training for the requirements 
process.[B] 

Key GAO observations: 
The Office of the Under Secretary of Defense for Acquisition, 
Technology and Logistics, working with the Defense Acquisition 
University, developed the requirements management certification 
training program to enhance the ability of individuals to write, 
review, and validate requirements. Defense Acquisition University 
officials reported that about 1,000 personnel with responsibility for 
generating requirements for major defense acquisition programs took 
the requirements development training in 2008. Since then, training 
has been available to individuals involved in smaller programs, but 
DOD officials believe that the acquisition community's ability to 
identify and document joint requirements would be improved if all 
staff completed the training and applied the concepts in developing 
proposals. 
The curriculum addresses multiple topics related to generating 
requirements, including the JCIDS process for reviewing and approving 
documents, the collaboration between JCIDS and the Defense Acquisition 
System, and DOD's resource allocation process. The program presents a 
series of courses that can be tailored to each individual's level of 
involvement in the requirements generation process. The Defense 
Acquisition University provides the training courses and tracks 
individual completion of each course, according to officials, but 
beyond the required completion of DOD training, each component 
determines the specific steps necessary to certify requirements 
managers, such as additional training and on-the-job experience. 

GAO scope and methodology: 
To evaluate the effectiveness of JCIDS in developing the personnel 
skills, capacity, and training needed for an effective and efficient 
requirements process, we reviewed relevant legislation and Joint Staff 
policy documents to identify the requirement to implement training for 
personnel involved with developing joint requirements. We interviewed 
officials from the Office of the Under Secretary of Defense for 
Acquisition, Technology and Logistics; the Joint Staff; and the 
Defense Acquisition University to discuss DOD's approach for 
implementing the training, and we interviewed service and combatant 
command officials to discuss the extent to which the required training 
had been implemented. 

Sources: DOD and GAO. 

[A] See Pub. L. No. 109-364, § 801 (2006). 

[B] Neither we nor officials identified any specific programs under 
JCIDS itself to develop the personnel skills, capacity, and training 
for the requirements process. Officials identified no specific DOD-
wide development programs other than the one described in the table. 

[End of table] 

Table 7: Section 862(b)(2)(A)--the Time That Requirements Documents 
Take to Receive Approval through JCIDS: 

Summary assessment: 
JCIDS review of documents validated from May 1, 2010 through May 31, 
2011 took an average of 9 months to receive approval. Joint Staff and 
service officials believed that the length of the review period was 
generally appropriate given the cost and complexity of the 
capabilities proposed. New procedures designed to improve 
responsiveness set goals for the duration of JCIDS reviews, and are 
based on the urgency of the need. 

Key GAO observations: 
The average of 9 months for review of documents within JCIDS is 
consistent with our previous findings. We reported in 2008 that it 
took 9.5 months on average for the JROC to validate capability 
proposals submitted from 2003 through 2005 and 8 months for 
submissions from 2006 through 2008.[A] 
JCIDS guidance in effect through December 2011 specified timeliness 
goals for the initial commenting and adjudication phases of the 
process, but not for the later review phases.[B] However, goals for 
the commenting and adjudication phases were not always met. For most 
programs we analyzed, reviews exceeded timeliness goals for the 
commenting or adjudication periods, or both, by about 25 percent or 
more. The duration of the later review phases varied and accounted for 
the most time between submission of a proposal and validation. 
According to Joint Staff and service officials, the reasons for 
extended review times included need for additional information from 
the documents' service sponsors and the lack of sponsor personnel 
available to address comments and approve changes. We previously 
reported that given the size, complexity, and level of funding 
associated with a major weapon system, the investment of time for a 
thorough, joint review may be warranted.[C] 

GAO scope and methodology: 
To evaluate the time it took to review and approve requirements 
documents through JCIDS, we reviewed policy documents on the JCIDS 
process. We selected a nonprobability sample of 12 requirements 
documents--4 initial capabilities documents, 5 capability development 
documents, and 3 capability production documents--for major weapon 
systems[D] validated from May 1, 2010, through May 31, 2011, to 
determine the time it took the JROC to approve them. We focused on 
major weapon systems because they were reviewed through each of the 
JCIDS phases. We interviewed Joint Staff and service officials to 
discuss the time frames, including factors that contributed to 
timelines. 

Sources: DOD and GAO. 

[A] GAO, Defense Acquisitions: DOD's Requirements Determination 
Process Has Not Been Effective in Prioritizing Joint Capabilities, GAO-
08-1060 (Washington, D.C.: Sept. 25, 2008). 

[B] See JCIDS Manual, enc. D (Jan. 31, 2011). These reviews are 
performed by the Functional and Joint Capabilities Boards and the JROC. 

[C] GAO-08-1060. 

[D] We selected requirements for major weapon systems because these 
requirements are validated by the JROC whereas requirements for small 
programs are generally delegated to lower validation authorities. 

[End of table] 

Table 8: Section 862(b)(2)(B)--the Quality of Cost Information 
Considered in JCIDS and the Extent of Its Consideration: 

Summary assessment: 
The cost information considered during JCIDS reviews has not always 
been reliable, and the extent to which JCIDS has considered cost 
information for proposed programs was limited to funding 
considerations. 

Key GAO observations: 
As we reported in June 2011, the services did not consistently provide 
high-quality resource estimates to the JROC to aid its validation 
decisions and cost estimates were often unreliable when assessed 
against best practices criteria.[A] In some cases, the quality of 
service estimates varied from those validated by the services' cost 
analysis agencies to those that represented less rigorous rough-orders-
of-magnitude.b In most cases, the services had not effectively 
conducted uncertainty and sensitivity analyses, which establish 
confidence levels for resource estimates, based on the knowledge 
available, or examined the effects of changing assumptions and ground 
rules, all of which could further the JROC's efforts to provide a 
sound basis for making cost, schedule, and performance trade-off 
decisions. Moreover, we reported that the JROC had not explicitly 
considered a requirement's affordability in a broader context during 
JCIDS reviews. 
Owing to the poor quality of the estimates of programs we reviewed, we 
recommended in the June 2011 report that the Vice Chairman of the 
Joint Chiefs of Staff require that capability sponsors present 
resource estimates that have been reviewed by a military service's 
cost analysis organization to ensure that best practices are being 
followed. The Joint Staff partially concurred with our recommendation, 
stating that program office cost estimates were compared to 
independent cost estimates during reviews of capability development 
documents. However, it is not clear whether the JCIDS guidance issued 
in January 2012 fully responds to our recommendation from the June 
2011 report. Officials told us that the guidance does not explicitly 
address the reliability of cost information, but according to Joint 
Staff officials, the JROC intends to rely more heavily on estimates 
developed by the Director of Cost Assessment and Program Evaluation to 
inform its deliberations. 

GAO scope and methodology: 
To evaluate the quality of cost information considered in JCIDS and 
the extent of its consideration, we relied on prior GAO work--GAO-11-
502--that addressed the extent to which the JROC considered cost 
information. We also interviewed Joint Staff and Office of the Under 
Secretary of Defense for Acquisition, Technology and Logistics 
officials, and reviewed JCIDS guidance to characterize the JROC's 
approach for addressing our prior recommendation. 

Sources: DOD and GAO. 

[A] GAO, DOD Weapon Systems: Missed Trade-off Opportunities During 
Requirements Reviews, GAO-11-502 (Washington, D.C.: June 16, 2011). 

[B] A rough-order-of-magnitude estimate is a quick, high-level 
estimate that generally involves less time and effort than a budget-
quality estimate. 

[End of table] 

Table 9: Section 862(b)(2)(C)--the Extent to Which JCIDS Establishes a 
Meaningful Level of Priority for Requirements: 

Summary assessment: 
JCIDS has not established a meaningful level of priority for joint 
requirements, but it is adopting a new approach to prioritize joint 
requirements within the context of current capabilities. 

Key GAO observations: 
In 2008, we reported that JCIDS was ineffective in setting priorities 
across the services and that virtually all capability proposals 
submitted to JCIDS had been approved or validated.[A] In June 2011, we 
reported that the JROC still did not prioritize requirements during 
the JCIDS process, consider redundancies across proposed programs, or 
prioritize and analyze capability gaps in a consistent manner b DOD 
officials told us that prioritization across programs occurred 
primarily through DOD's budgeting process, which is the responsibility 
of the military services and the Office of the Under Secretary of 
Defense (Comptroller). JCIDS guidance in effect through December 2011 
required that capability sponsors prioritize capability gaps and 
identify redundancies when developing initial capabilities documents, 
but it did not specify the criteria sponsors should use to prioritize 
needs. Our review found that sponsors used different methods, with 
some ranking priorities in numerical order and others assigning 
priority categories. Some did not report on redundancies and overlap. 
The JROC and Joint Staff have acknowledged the need to improve the 
JROC's ability to prioritize capabilities through the JCIDS process. 
Officials noted that the Joint Staff is now implementing a new 
approach to impose prioritization of joint needs within the context of 
existing capabilities in joint capability areas. However, the new 
approach has not been fully developed and clearly documented, and the 
extent to which current momentum toward assigning priorities will be 
sustained is uncertain. See the section of our letter on the Joint 
Staff's efforts to prioritize capability needs for more information. 

GAO scope and methodology: 
To evaluate the extent to which JCIDS establishes a meaningful level 
of priority for requirements, we reviewed law and Joint Staff policy 
documents to understand the roles and requirements of the JCIDS 
process and the JROC as it pertains to prioritization. We interviewed 
Joint Staff officials to discuss changes to the approach for 
prioritizing capability requirements. We also reviewed prior GAO 
reports, including GAO-11-502 and GAO-08-1060, that discussed 
prioritization and compared the current efforts to prioritize with 
what was reported in the past. 

Sources: DOD and GAO. 

[A] GAO, Defense Acquisitions: DOD's Requirements Determination 
Process Has Not Been Effective in Prioritizing Joint Capabilities, GAO-
08-1060 (Washington, D.C.: Sept. 25, 2008). GAO's 2008 report assessed 
JCIDS under guidance prior to CJCSI 3170.01G. 

[B] GAO, DOD Weapon Systems: Missed Trade-off Opportunities During 
Requirements Reviews, GAO-11-502 (Washington, D.C.: June 16, 2011). 

[End of table] 

Table 10: Section 862(b)(2)(D)--the Extent to Which JCIDS Is 
Considering Trade-offs between Cost, Schedule, and Performance 
Objectives: 

Summary assessment: 
The JROC has not always considered trade-offs among cost, schedule, 
and performance objectives during JCIDS. It is premature to assess 
current initiatives for considering trade-offs within JCIDS. 

Key GAO observations: 
The Weapon Systems Acquisition Reform Act of 2009 amended section 181 
to require the JROC to assist the Chairman of the Joint Chiefs of 
Staff in ensuring the consideration of trade-offs among cost, 
schedule, and performance objectives for joint military 
requirements.[A] However, as we reported in June 2011, most trade-offs 
are made by the military services outside the JCIDS process as they 
analyze alternative solutions for meeting capability needs. The JROC's 
reviews of requirements do not align with these activities, and as a 
result, the JROC does not have an opportunity to provide the services 
input on trade-offs or on the proposed solution to a capability gap 
before it is selected, and significant time and resources may be 
expended before the JROC gets to formally weigh in during the JCIDS 
process. However, we discussed several instances where the JROC held 
requirements firm during reviews of programs reporting substantial 
cost growth. According to officials, by accepting increased cost and 
schedule delays, the JROC essentially traded cost and possibly 
schedule for performance. We made recommendations to improve the 
JROC's ability to make trade-offs, including establishing a mechanism 
to review the results of the services' alternatives earlier in the 
process, among other actions.[B] 
The Joint Staff issued new JCIDS guidance intended to better position 
the JROC for considering cost, schedule, and performance trade-offs 
and provide improved accuracy early in the process. As explained in 
DOD documentation, changes include the following: 
* Conducting a review of the program sponsor's analysis of 
alternatives study--including study results, such as cost- , schedule-
, and performance-level recommendations--prior to an acquisition 
program proceeding to the next phase of development; 
* Considering cost, schedule, performance, and quantity targets in 
making validation decisions and annotating decisions in JROC 
memorandums, as appropriate; 
* Reviewing requirements that have unacceptably deviated from 
schedule, quantity, cost, and performance parameters. 

GAO scope and methodology: To evaluate the extent to which JCIDS is 
considering trade-offs between cost, schedule, and performance 
objectives, we relied primarily on GAO's June 2011 report--GAO-11-502-
-that discussed the extent to which the JROC has considered trade-offs 
within programs. We also reviewed Joint Staff briefings and guidance 
documents to identify changes in guidance with regard to cost, 
schedule, and performance trade-offs since June 2011. 

Sources: DOD and GAO. 

[A] See Pub. L. No. 111-23, § 201(b)(1)(C) (2009) (codified at 10 
U.S.C. § 181(b)(1)(C)). Because of this statutory responsibility, we 
focused on the JROC. 

[B] GAO, DOD Weapon Systems: Missed Trade-off Opportunities During 
Requirements Reviews, GAO-11-502 (Washington, D.C.: June 16, 2011). 

[End of table] 

Table 11: Section 862(b)(2)(E)--the Quality of Information on 
Sustainment Considered in JCIDS and the Extent to Which Sustainment 
Information Is Considered: 

Summary assessment: 
The quality of information on sustainment planning that program 
sponsors provided JCIDS varied and the JROC has not systematically 
considered sustainment before validating programs.[A] 

Key GAO observations: 
The JCIDS Manual[B] in effect through December 2011 required that 
program sponsors of major defense acquisition programs develop 
information on sustainment based on four factors--materiel 
availability, operational availability, reliability, and ownership 
cost[C]--and listed descriptive criteria, against which documents 
would be reviewed, for program sponsors to consider.d We have 
previously reported that the rigor of the analysis supporting 
sustainment planning and cost estimates varied. Our current review of 
six capability development documents showed that all the sponsors 
addressed the information required, but the level of detail varied. 
For example, the Navy provided an ownership cost estimate for its 
proposed Littoral Combat Ship capability but not information to 
support the estimate or a description of plans to monitor, collect, or 
validate the ownership cost data. Further, the level of detail 
provided for the supporting criteria varied, with some documents 
providing a paragraph of supporting information for some criteria and 
others providing single-sentence responses. Additionally, none of the 
documents we reviewed included a clear definition of the type of cost 
structure used to determine the ownership cost estimates for the 
proposed programs. 
JCIDS has not generally sought detailed information to present to the 
JROC for consideration before validating proposals. Although the 
review criteria associated with each of the four required sustainment 
metrics may not be applicable to every program, according to DOD 
officials the information could be useful for JROC validation 
discussions and decisions. Without complete information, the JROC may 
not always have the sustainment information it needs to effectively 
manage and validate capability documents. See the section of our 
letter on sustainment information available to the JROC for more 
information. 

GAO scope and methodology: 
To assess the quality of reported sustainment information and the 
extent to which it is being considered in JCIDS, we worked with DOD 
officials to identify six capability development documents that 
included sustainment information. We reviewed the documents to 
determine whether and the extent to which sustainment information was 
included and interviewed DOD officials to discuss reported sustainment 
data. We also reviewed relevant law and Joint Staff policy documents 
to understand reporting requirements. 

Sources: DOD and GAO. 

[A] Under a version of the JCIDS Manual in effect through December 
2011, the four sustainment factors were required to be developed for 
major defense acquisition programs. Applicability of some of the 
information for acquisition category II and lower programs was left to 
the determination of the sponsor. Under the current version of the 
manual, issued January 19, 2012, acquisition category II and lower 
programs, with materiel solutions, are to include either the 
sustainment metric or sponsor-defined sustainment metrics. As a 
consequence, under both versions, sustainment information may not 
necessarily be considered or have been considered outside the context 
of major defense acquisition programs. 

[B] The JCIDS Manual required the inclusion of sustainment information 
in capability development documents beginning in 2007. 

[C] The current version of the JCIDS Manual renamed the fourth factor 
as "operation and support cost." 

[D] See JCIDS Manual, enc. B (Jan. 31, 2011). 

[End of table] 

Table 12: Section 862(b)(2)(F)--an Evaluation of the Advantages and 
Disadvantages of Designating a Commander of a Unified Combatant 
Command for Each JROC-Interest Requirements Document to Provide a 
Joint Evaluation Task Force for Certain Purposes: 

Summary assessment: 
Developing a joint evaluation task force could have some advantages 
for combatant commands, such as advancing a dialogue among the 
commands, but the disadvantages of burdening commands with additional 
duties of participation may outweigh the benefits. 

Key GAO observations: 
Combatant command officials reported that participation in a task 
force might provide greater opportunities for the commands to provide 
input, but they noted that the commands already participate in 
identification and discussion of capability requirements through the 
current JCIDS process and other forums.[A] Specifically, commands can 
participate in: 
* experimentation and testing efforts and analysis of alternative 
teams; 
* discussions of capability proposals that deviate from cost, 
schedule, or quantity targets in existing forums; 
* advocating for desired material solutions as end users through 
integrated priority lists and service component collaboration; and; 
* commenting on concept of operations and doctrine through 
participation in regular Functional Capabilities Board meetings. 
DOD officials consistently cited the following three key disadvantages 
to formation of joint evaluation task forces: 
* Implementation of the task force would represent a shift in 
responsibilities from the military services--which have historically 
completed these duties--to the combatant commands; 
* Combatant commands have limited manpower and few acquisition-trained 
specialists to appropriately fill the role of combat developer; 
* It is unclear how the leadership responsibilities would be divided 
among the combatant commands, particularly where a solution is of 
interest to multiple commands. 
In response to congressional questions about establishing such a task 
force, the current Vice Chairman of the Joint Chiefs of Staff stated 
that such a task force could result in an additional layer of 
coordination and staffing between the existing requirements and 
acquisition processes, further delaying the identification of 
capability gaps and subsequent fielding of systems addressing those 
gaps. 

GAO scope and methodology: 
To evaluate the advantages and disadvantages of implementing a joint 
evaluation task force, we reviewed a March 2010 report prepared by the 
House Armed Services Committee Panel on Defense Acquisition Reform and 
interviewed Joint Staff and combatant command officials to discuss 
advantages and disadvantages of the concept. 

Sources: DOD and GAO. 

Note: See the introduction to this appendix for the full text of this 
requirement, including the specific purposes identified. 

[A] Among the findings of a May 2011 GAO report was that combatant 
commands noted the importance of participating in the development of 
joint requirements, but questioned the value of what they described as 
a resource-intensive and time-consuming process that is not always 
responsive to their more immediate capability needs. They also 
questioned the value of a process that results in decisions that while 
influential, are advisory to the acquisition and budget process driven 
by service investment priorities. See GAO, Defense Management: 
Perspectives on the Involvement of Combatant Commands in the 
Development of Joint Requirements, GAO-11-527R (Washington, D.C.: May 
20, 2011). Although the report did not address involvement in a joint 
evaluation task force, it provides combatant command perspectives on 
the overall JCIDS process, including views on greater combatant 
command involvement. 

[End of table] 

[End of section] 

Appendix II: Scope and Methodology: 

To assess the extent to which the Joint Staff has developed and 
implemented an analytic approach to prioritize capability needs, we 
reviewed relevant legislation and Joint Staff guidance on the roles 
and requirements of the JCIDS process and the JROC as it pertains to 
assigning levels of priority to capability proposals. Specifically, we 
reviewed section 181 of the U.S. Code, Title 10; Chairman of the Joint 
Chiefs of Staff Instruction 3170.01G; Chairman of the Joint Chiefs of 
Staff Instruction 3170.01H (which we reviewed in draft form); and the 
JCIDS Manual. We then compared the current and prior versions of the 
instruction and manual to identify changes in the guidance with 
respect to prioritization of capability proposals. We met with 
officials from the Joint Staff; Department of the Air Force; 
Department of the Army; Department of the Navy; and Office of the 
Under Secretary of Defense for Acquisition, Technology and Logistics 
to discuss their perspectives on the implementation of changes to 
JCIDS with respect to prioritizing capability requirements. In order 
to understand how the JROC is implementing its new approach for 
prioritizing capabilities, we reviewed briefing materials presented at 
a JROC forum in November 2011. To corroborate our understanding of the 
documents we reviewed, we conducted interviews with Joint Staff and 
Office of the Under Secretary of Defense for Acquisition, Technology 
and Logistics officials. To understand the Joint Staff's recent 
internal review of JCIDS, we reviewed the charter and recommendations, 
and met with Joint Staff officials to discuss how those 
recommendations from the review might affect the JROC's prioritization 
of capability proposals. We also reviewed prior reports by GAO, the 
House Armed Services Committee, and the Defense Business Board that 
discussed prioritization of capability proposals through JCIDS, and 
compared the JROC's current efforts to prioritize with what has been 
reported in the past. We assessed whether the guidance in the JCIDS 
Manual and JCIDS instruction (in draft form during our review and 
issued in January 2012) on prioritization meets the intent of 
recommendations contained in our prior reports. 

To assess the extent to which the JROC has considered aspects of the 
availability and operational support requirements of weapon systems--
called sustainment--when validating the requirements of proposed 
capability solutions, we reviewed relevant DOD and Joint Staff policy 
documents and related guidance outlining the requirement to develop 
and report sustainment metrics for capability documents. Specifically, 
we reviewed the reporting requirements for major defense acquisition 
programs processed through JCIDS.[Footnote 38] To determine JCIDS 
reporting requirements for capability development documents and to 
understand the JCIDS process, we reviewed the JCIDS Manual enclosure 
pertaining to sustainment and instructions from the Chairman of the 
Joint Chiefs of Staff. We also reviewed prior GAO work on related 
topics. Further, we interviewed DOD and Joint Staff officials to 
discuss preparation, presentation, and consideration of sustainment 
data. 

We also conducted a case study analysis of select capability 
development documents that included sustainment information. We sought 
a universe of all capability development documents subject to 
reporting the sustainment key performance parameter validated since 
2007, when JCIDS began requiring program sponsors to include this 
information. We initially obtained a universe of 22 JCIDS capability 
development documents from the Office of the Under Secretary of 
Defense for Acquisition, Technology and Logistics/Logistics and 
Materiel Readiness. We relied on this provided list of documents 
because the Joint Staff's Knowledge Management/Decision Support 
database did not produce reliable results of all requirements 
documents containing sustainment information. We narrowed the list by 
eliminating programs that have been downgraded, truncated, or 
canceled; programs in which the sustainment data were not at the key 
performance parameter level or were more narrowly defined by only one 
type of platform; programs whose information was contained in 
capability production documents rather than capability development 
documents; and programs whose capability development documents were 
entered in JCIDS before the full implementation of the sustainment key 
performance parameter requirement. Additionally, several programs did 
not have supporting documentation that would allow a review; Joint 
Staff officials we met with stated that these documents were not 
available either because of a misidentification of the type of 
capability document that was being reviewed (a capability production 
document as opposed to a capability development document), or because 
the document was not included in the Joint Staff's database for JROC 
review. These factors led to a refined universe of 12 requirements 
documents. We then randomly selected capability development documents 
for two programs per service--Army, Air Force, and Navy--resulting in 
a total of six programs to serve as case studies. Because this was a 
nonprobability sample of programs, the results are not generalizable 
to all programs; however, they are illustrative of the kinds of issues 
that are possible in such programs. In order to assess reported 
sustainment information in the six selected cases, we performed a 
content analysis of the documentation available for the six cases. Two 
GAO analysts independently reviewed each of the six capability 
development documents, assessing whether each of the individual 
elements of the JCIDS Manual sustainment metrics was included, coding 
the inclusion of each metric as "yes," "no," "partial," and "don't 
know." The two analysts then discussed and reconciled all initial 
disagreements regarding the assigned codes. We then discussed the 
results of this content analysis with officials from the Joint Staff 
and the Office of the Under Secretary of Defense for Acquisition, 
Technology and Logistics/Logistics and Materiel Readiness to verify 
that the results of our analysis were valid. 

We conducted this performance audit from April 2011 through February 
2012 in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit 
to obtain sufficient, appropriate evidence to provide a reasonable 
basis for our findings and conclusions based on our audit objectives. 
We believe that the evidence obtained provides a reasonable basis for 
our findings and conclusions based on our audit objectives. 

[End of section] 

Appendix III: Comments from the Department of Defense: 

The Joint Staff: 
Washington, DC: 

February 8, 2012: 

Reply ZIP Code: 20318-0300. 

Mr. John H. Pendleton: 
Director, Defense Capabilities and Management: 
U.S. Government Accountability Office: 
441 G Street, NW: 
Washington, D.C. 20548: 

Dear Mr. Pendleton: 

Thank you for the opportunity to review GAO Draft Report GAO-12-339,
"Defense Management: Guidance and Progress Measures are Needed to 
Realize Benefits From Changes in DOD's Joint Requirements Process," 
dated 6 January 2012 (GAO Code 351609). The DOD response to the 
report's recommendations is enclosed. 

I believe the report reflects a fair and objective assessment of the 
current Joint Staff's Joint Capabilities Integration and Development 
System (JCIDS) process during the GAO assessment period. It 
appropriately highlights the active efforts to improve the JCIDS/Joint 
Requirements Oversight Council process and procedures to include 
recently updated guidance in Chairman of the Joint Chiefs of Staff 
Instruction (CJCSI) 3170.0111, CJCSI 5123.01F, and the JCIDS Manual.
The Joint Staff point of contact is Mr. Randolph Wood; J-8/Requirements
Management Division; 703-614-9628; randolph.wood@js.pentagon.mil. 

Sincerely, 

Signed by: 

Craig A. Franklin: 
Major General, USAF: 
Vice Director, Joint Staff: 

Enclosure: 

[End of letter] 

Enclosure: 

GAO Draft Report Dated 6 January 2012: 
GAO-12-339 (GAO CODE 351609): 

"Defense Management: Guidance And Progress Measures Are Needed To 
Realize Benefits From Changes In DOD's Joint Requirements Process" 

Department of Defense Comments TO THE GAO Recommendations: 

Recommendation 1: The Government Accountability Office (GAO) 
recommends that the Vice Chairman of the Joint Chiefs of Staff, as the 
Joint Requirements Oversight Council (JROC) Chairman, revise and 
implement guidance to reflect changes to the Joint Capabilities 
Integration and
Development System (JCIDS) process as well as to establish criteria 
and measures for determining the relative importance of capability 
needs across capability areas and assessing progress. 

DOD Response: Partially concur. 

Substantial changes to the joint requirements development process have 
recently been made after more than a year of review, discussions, and
deliberations. With regard to prioritization of capability needs and 
assessing progress, the Director, Joint Staff recently approved and 
signed the JCIDS instruction and JROC Charter instruction (10 January 
2012). Changes outlined in these instructions and further detailed in 
the revised JCIDS Manual (19 January 2012) provide rust steps towards 
an improved prioritization methodology by the Functional Capability 
Boards, Joint Capabilities Board, and JROC to take into account 
capability requirements in an expanded portfolio view. Capability gap 
and solution discussions presented for the JROC will incorporate an 
evolving portfolio assessment tool that provides a view of the 
proposed gap/solution with regard to its strategic guidance importance 
along with other similar solutions already being developed or 
available to the joint force. 

In addition to the DOD priorities presented in the recently released 
Strategic Guidance for the Department (January 2012), the revised 
Capability Gap Assessment/Integrated Priority List (CGA/IPL) process 
will provide further direction for sponsors and other JCIDS 
implementing bodies to focus on the most important areas as capability 
gaps/solutions are proposed, reviewed, and validated, or not. During 
this current review of Combatant Command (CCMD) CGA/IPL inputs (for 
FYs 14-18), the process includes the incorporation of the Chairman's 
Risk Assessment (CRA). By associating the capability gaps identified 
in the CCMD IPLs with strategic and military risks from the CRA, the
JROC will be able to prioritize and make informed decisions as to 
which gaps are the most critical and which are not of sufficient 
priority to justify additional actions or resources. This evaluation 
and the decisions resulting from this revised CGA/IPL process should 
further inform and assist in future JCIDS prioritization methodology 
and results. 

Recommendation 2: The GAO recommends that the Vice Chairman of the
Joint Chiefs of Staff, as the JROC Chairman, explicitly require that 
program sponsors address each of the criteria outlined for the 
individual sustainment metrics when submitting capability development 
documents. 

DOD Response: Partially concur. 

The criteria outlined for the sustainment metric key performance 
parameter and key system attributes within JCIDS were designed to be a 
"shaping" mechanism for the requirements community. They were not 
designed to be prescriptive. Each program is unique, and criteria 
applicable to one program may not be applicable to another. The 
individual criteria are a guideline to ensure critical 
thinking/evaluation has occurred in development and review of a 
program's sustainment metric. Capability development documents (CDDs) 
are vetted through the Joint Staff and Office of the Secretary of 
Defense (OSD), providing opportunities to address omissions/ 
discrepancies. Reviews are held internal to OSD as well as with 
program sponsors. These reviews enable the Joint Staff to ensure a 
complete and thoroughly vetted document is presented to the JROC for 
requirements approval/validation of the proposed capability solution. 
Additionally, the analyses and rationale for metric development 
resides in documents other than the CDD. Documents such as the 
Reliability, Availability, Maintainability, and Cost report, Life 
Cycle Support Plan, Independent Cost Estimate, and Program Life Cycle 
Cost Estimate contain more detailed documentation supporting the 
development of the sustainment metric. DOD policy discourages 
replication of data and analysis across multiple documents. 

[End of section] 

Appendix IV: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

John Pendleton, (202) 512-3489 or pendletonj@gao.gov: 

Staff Acknowledgments: 

In addition to the contact named above, Margaret Morgan, Assistant 
Director; Melissa Blanco; Mae Jones; Kate Lenane; Jennifer Madison; 
Ron Schwenn; Michael Shaughnessy; Michael Silver; Jennifer Spence; 
Amie Steele; and Kristy Williams made key contributions to this report. 

[End of section] 

Footnotes: 

[1] Major defense acquisition programs are those programs identified 
by DOD that are estimated to require eventual total expenditure for 
research, development, test, and evaluation, including all planned 
increments, of more than $365 million or expenditure for procurement, 
including all planned increments, of more than $2.19 billion in fiscal 
year 2000 constant dollars. Programs may also be designated by DOD as 
major defense acquisition programs. 

[2] GAO, Defense Management: DOD Needs Better Information and Guidance 
to More Effectively Manage and Reduce Operating and Support Costs of 
Major Weapon Systems, [hyperlink, 
http://www.gao.gov/products/GAO-10-717] (Washington, D.C.: July 20, 
2010). 

[3] See Pub. L. No. 111-383, § 862 (2011). 

[4] We provided a draft of this report to the congressional defense 
committees on January 6, 2012, to satisfy this requirement. 

[5] The following officials serve as advisors to the JROC on matters 
within their authority and expertise: the Under Secretary of Defense 
for Acquisition, Technology and Logistics; the Under Secretary of 
Defense (Comptroller); the Under Secretary of Defense for Policy; the 
Director of Cost Assessment and Program Evaluation; the Director of 
Operational Test and Evaluation; and such other civilian officials of 
DOD as designated by the Secretary of Defense. See 10 U.S.C. § 
181(d)(1). The JROC must also seek and consider input from the 
commanders of the combatant commands in carrying out several of its 
missions. See § 181(d)(2). 

[6] See Pub. L. No. 111-23, §§ 201(b)(1)(C), 201(b)(4) (2009) 
(codified at 10 U.S.C. § 181). Both actions are to be taken in 
consultation with certain officials. See id. 

[7] The JROC-interest designation applies to all programs that are 
estimated to require eventual total expenditures for research, 
development, test, and evaluation, including all planned increments, 
of more than $365 million or expenditures for procurement, including 
all planned increments, of more than $2.19 billion in fiscal year 2000 
constant dollars, or that are designated as major defense acquisition 
programs or special interest. The JROC-interest designation also 
applies to major automated information systems programs, which are 
programs designated as such or estimated to exceed certain expenditure 
thresholds. Finally, the JROC-interest designation applies to all 
joint doctrine, organization, training, materiel, leadership and 
education, personnel, facilities, and policy change recommendations 
and programs and capabilities that have a potentially significant 
impact on interoperability in allied and coalition partner operations. 
For JROC-interest documents, recommendations on capability needs are 
passed on to the Joint Capabilities Board that reviews and, if 
appropriate, forwards them with recommendations to the JROC for 
validation and approval. 

[8] Functional Capabilities Boards assess needs and make 
recommendations about validating capability gaps for documents to be 
validated by the JROC or the Joint Capabilities Board. 

[9] Under the JCIDS guidance prior to January 10, 2012, the Joint 
Capabilities Board served as the validation authority for Joint 
Capabilities Board-interest programs, which were defined as all 
acquisition category II and below programs where the capabilities, 
systems, or both associated with the document would have affected the 
joint force and an expanded joint review was required. See Chairman of 
the Joint Chiefs of Staff Instruction 3170.01G at GL-6 (Mar. 1, 2009) 
(hereinafter cited as CJCSI 3170.01G (Mar. 1, 2009)). Under the 
current version of the guidance, Joint Capabilities Board-interest 
documents are those describing acquisition category II and below 
programs that have a potentially significant impact on 
interoperability (interagency, allied, partner nation, coalition, 
etc.). See Manual for the Operation of the Joint Capabilities 
Integration and Development System, enc. C, para. 3(e)(2) (Jan. 19, 
2012) (hereinafter cited as JCIDS Manual (Jan. 19, 2012)). Under both 
the previous and present versions of the guidance, programs that are 
not designated as JROC-or Joint Capabilities Board-interest are 
validated by the sponsor. See JCIDS Manual, enc. C, para. 3(e)(3)-(5) 
(Jan. 19, 2012). 

[10] Sponsors are generally the military services, but can also be 
other DOD agencies or combatant commands. 

[11] See H.R. Rep. No. 110-146, at 381 (2007) (accompanying H.R. 
1585). The provision, which revised parts of section 181 of Title 10, 
U.S. Code, was included as amended in the National Defense 
Authorization Act for Fiscal Year 2008. See Pub. L. No. 110-181, § 942 
(2008). 

[12] GAO, Defense Acquisitions: DOD's Requirements Determination 
Process Has Not Been Effective in Prioritizing Joint Capabilities, 
[hyperlink, http://www.gao.gov/products/GAO-08-1060] (Washington, 
D.C.: Sept. 25, 2008). 

[13] GAO, DOD Weapon Systems: Missed Trade-off Opportunities During 
Requirements Reviews, [hyperlink, 
http://www.gao.gov/products/GAO-11-502] (Washington, D.C.: June 16, 
2011). 

[14] Combatant commands are DOD's operational commands. Of the nine 
combatant commands, the following six have geographic 
responsibilities: U.S. Africa Command, U.S. Central Command, U.S. 
European Command, U.S. Northern Command, U.S. Pacific Command, and 
U.S. Southern Command. The following three have functional 
responsibilities: U.S. Special Operations Command, U.S. Strategic 
Command, and U.S. Transportation Command. 

[15] GAO, Defense Management: Perspectives on the Involvement of 
Combatant Commands in the Development of Joint Requirements, 
[hyperlink, http://www.gao.gov/products/GAO-11-527R] (Washington, 
D.C.: May 20, 2011). 

[16] See CJCSI 3170.01H, at GL-7 (Jan. 19, 2012). 

[17] See Pub. L. No. 110-181, § 942(a) (codified as amended at 10 
U.S.C. § 181(b)). 

[18] GAO, Best Practices: An Integrated Portfolio Management Approach 
to Weapon System Investments Could Improve DOD's Acquisition Outcomes, 
[hyperlink, http://www.gao.gov/products/GAO-07-388] (Washington, D.C.: 
Mar. 30, 2007). 

[19] We are not including specific examples of issues and programs 
under review because of the sensitivity of deliberations and security 
classification concerns. 

[20] GAO, Results-Oriented Cultures: Implementation Steps to Assist 
Mergers and Organizational Transformations, [hyperlink, 
http://www.gao.gov/products/GAO-03-669] (Washington, D.C.: July 2, 
2003). 

[21] GAO, Defense Management: Tools for Measuring and Managing Defense 
Agency Performance Could Be Strengthened, [hyperlink, 
http://www.gao.gov/products/GAO-04-919] (Washington, D.C.: Sept. 13, 
2004). 

[22] Our prior work has found that the operation and support costs of 
sustaining a fielded system account for about 70 percent of a system's 
life cycle costs. See [hyperlink, 
http://www.gao.gov/products/GAO-10-717]. 

[23] During our review, the Joint Staff revised guidance on 
sustainment reporting requirements. Under the guidance in effect 
through December 2011, which we used for our review, sustainment 
metrics were to be developed for all acquisition category I programs 
involving materiel solutions and acquisition category II and below 
programs as determined by the sponsor. See Manual for the Operation of 
the Joint Capabilities Integration and Development System, app. B to 
enc. B, para. 2 (February 2009, updated Jan. 31, 2011) (hereinafter 
cited as JCIDS Manual (Jan. 31, 2011)). According to the recently 
revised JCIDS Manual, sustainment metrics must be developed for all 
acquisition category I programs. The manual also states that 
acquisition category II and below programs, with material solutions, 
shall include the sustainment metrics or sponsor-defined sustainment 
metrics. See JCIDS Manual, app. E to enc. B, para. 2 (Jan. 19, 2012). 
Under both versions of the manual, these metrics are developed in the 
form of key performance parameters and key system attributes. 

[24] The JCIDS Manual required the inclusion of sustainment 
information in capability development documents beginning in 2007. 

[25] DOD, Reliability, Availability, Maintainability, and Cost 
Rationale Report Manual (Washington, D.C.: June 1, 2009). 

[26] GAO, Defense Acquisition: Employing Best Practices Can Shape 
Better Weapon System Decisions, [hyperlink, 
http://www.gao.gov/products/GAO/T-NSIAD-00-137] (Washington, D.C.: 
Apr. 26, 2000). 

[27] See JCIDS Manual, app. B to enc. B (Jan. 31, 2011). See also 
JCIDS Manual, app. E to enc. B (Jan. 19. 2012). The version of the 
JCIDS Manual in effect through December 2011 defined "materiel 
availability" as a measure of the percentage of the total inventory of 
a system that is operationally capable (ready for tasking) of 
performing an assigned mission at a given time, based on materiel 
condition; "operational availability" as a measure of the percentage 
of time that a system or group of systems within a unit are 
operationally capable of performing an assigned mission and can be 
expressed as (uptime/(uptime + downtime)); "reliability" as a measure 
of the probability that the system will perform without failure over a 
specific interval and must be sufficient to support the warfighting 
capability needed; and "ownership cost" as a measure that provides 
balance to the sustainment solution by ensuring that the operations 
and support costs associated with availability are considered in 
making decisions, and should cover the planned lifecycle timeframe, 
consistent with the timeframe used in the materiel availability 
metric. The revised guidance largely maintained these descriptions. 

[28] The revised guidance issued in January 2012 includes many of the 
same review criteria. See JCIDS Manual, app. E to enc. B (Jan. 19, 
2012). As previously noted, the revised guidance renamed the ownership 
cost metric as operation and support cost. 

[29] The Joint Staff Directorate for Logistics (J4), Maintenance 
Division, is the Joint Staff entity responsible for evaluating the 
sustainment key performance parameter. In this report, GAO refers to 
the sustainment key performance parameter as sustainment information. 

[30] [hyperlink, http://www.gao.gov/products/GAO-10-717]. 

[31] DOD, Weapon System Acquisition Reform Product Support Assessment 
(Washington, D.C.: Nov. 1, 2009). This report was produced by a DOD 
Product Support Assessment Team led by the Office of the Deputy Under 
Secretary of Defense for Logistics and Materiel Readiness. 

[32] See Pub. L. No. 111-383, § 862(a) (2011). The statute 
specifically defined JCIDS by reference to Chairman of the Joint 
Chiefs of Staff Instruction 3170.01G. See § 862(c)(1). 

[33] Requirements creep is defined in the statute as "the addition of 
new technical or operational specifications after a requirements 
document is approved." See § 862(c)(3). 

[34] The term requirements document is defined in the statute as "a 
document produced in JCIDS that is provided for an acquisition program 
to guide the subsequent development, production, and testing of the 
program and that--(A) justifies the need for a materiel approach, or 
an approach that is a combination of materiel and non-materiel, to 
satisfy one or more specific capability gaps; (B) details the 
information necessary to develop an increment of militarily useful, 
logistically supportable, and technically mature capability, including 
key performance parameters; or (C) identifies production attributes 
required for a single increment of a program." See § 862(c)(2). 

[35] See § 862(c). 

[36] See Pub. L. No. 111-383, § 862. 

[37] See § 862(c)(1) (defining JCIDS as "the system for the 
assessment, review, validation, and approval of joint warfighting 
requirements that is described in Chairman of the Joint Chiefs of 
Staff Instruction 3170.01G"). 

[38] Under the version of the JCIDS Manual in effect during the time 
of our review, sustainment metrics were to be developed for all 
acquisition category I programs involving materiel solutions and 
acquisition category II and below programs as determined by the 
sponsor. See JCIDS Manual, app. B to enc. B, para. 2 (Jan. 31, 2011). 
The current version of the manual states that acquisition category II 
and below programs, with materiel solutions, shall include the 
sustainment metrics or sponsor-defined sustainment metrics. See JCIDS 
Manual, app. E to enc. B, para. 2 (Jan. 19. 2012). 

[End of section] 

GAO’s Mission: 

The Government Accountability Office, the audit, evaluation, and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the 
performance and accountability of the federal government for the 
American people. GAO examines the use of public funds; evaluates 
federal programs and policies; and provides analyses, recommendations, 
and other assistance to help Congress make informed oversight, policy, 
and funding decisions. GAO’s commitment to good government is 
reflected in its core values of accountability, integrity, and 
reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO’s website [hyperlink, http://www.gao.gov]. Each 
weekday afternoon, GAO posts on its website newly released reports, 
testimony, and correspondence. To have GAO e-mail you a list of newly 
posted products, go to [hyperlink, http://www.gao.gov] and select “E-
mail Updates.” 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of 
production and distribution and depends on the number of pages in the 
publication and whether the publication is printed in color or black 
and white. Pricing and ordering information is posted on GAO’s 
website, [hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card, 
MasterCard, Visa, check, or money order. Call for additional 
information. 

Connect with GAO: 

Connect with GAO on facebook, flickr, twitter, and YouTube.
Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts.
Visit GAO on the web at [hyperlink, http://www.gao.gov]. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 
Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; 
E-mail: fraudnet@gao.gov; 
Automated answering system: (800) 424-5454 or (202) 512-7470. 

Congressional Relations: 

Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-4400
U.S. Government Accountability Office, 441 G Street NW, Room 7125
Washington, DC 20548. 

Public Affairs: 
Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149 
Washington, DC 20548.