This is the accessible text file for GAO report number GAO-09-518 
entitled 'Military Readiness: DOD Needs to Strengthen Management and 
Oversight of the Defense Readiness Reporting System' which was released 
on September 25, 2009. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to the Subcommittee on Readiness and Management Support, 
Committee on Armed Services, U.S. Senate: 

United States Government Accountability Office: 
GAO: 

September 2009: 

Military Readiness: 

DOD Needs to Strengthen Management and Oversight of the Defense 
Readiness Reporting System: 

Military Readiness: 

GAO-09-518: 

GAO Highlights: 

Highlights of GAO-09-518, a report to the Subcommittee on Readiness and 
Management Support, Committee on Armed Services, U.S. Senate. 

Why GAO Did This Study: 

The Department of Defense (DOD) reports data about the operational 
readiness of its forces. In 1999, Congress directed DOD to create a 
comprehensive readiness system with timely, objective, and accurate 
data. In response, DOD started to develop the Defense Readiness 
Reporting System (DRRS). After 7 years, DOD has incrementally fielded 
some capabilities, and, through fiscal year 2008, reported obligating 
about $96.5 million. GAO was asked to review the program including the 
extent that DOD has (1) effectively managed and overseen DRRS 
acquisition and deployment and (2) implemented features of DRRS 
consistent with legislative requirements and DOD guidance. GAO compared 
DRRS acquisition disciplines, such as requirements development, test 
management, and DRRS oversight activities, to DOD and related guidance, 
and reviewed the system’s current and intended capabilities relative to 
legislative requirements and DOD guidance. We did not evaluate DOD’s 
overall ability to assess force readiness or the extent that readiness 
data reflects capabilities, vulnerabilities, or performance issues. 

What GAO Found: 

DOD has not effectively managed and overseen the DRRS acquisition and 
deployment, in large part because of the absence of rigorous and 
disciplined acquisition management controls and an effective governance 
and accountability structure for the program. In particular, system 
requirements have not been effectively developed and managed. For 
example, user participation and input in the requirements development 
process was, until recently, limited, and requirements have been 
experiencing considerable change, are not yet stable, and have not been 
effectively controlled. In addition, system testing has not been 
adequately performed and managed. For example, test events for already 
acquired system increments, as well as currently deployed and operating 
increments, were not based on well-defined plans or structures, and 
test events have not been executed in accordance with plans or in a 
verifiable manner. Moreover, DRRS has not been guided by a reliable 
schedule of work to be performed and key activities to occur. These 
program management weaknesses can, in part, be attributed to long-
standing limitations in program office staffing and program oversight 
and accountability. Despite being a DOD-wide program, until April, 2009 
DRRS was not accountable to a DOD-wide oversight body, and it was not 
subject to DOD’s established mechanisms and processes for overseeing 
business systems. Collectively, these acquisition management weaknesses 
have contributed to a program that has fallen well short of 
expectations, and is unlikely to meet future expectations. 

DOD has implemented DRRS features that allow users to report certain 
mission capabilities that were not reported under the legacy system, 
but these features are not fully consistent with legislative 
requirements and DOD guidance; and DOD has not yet implemented other 
features. The geographic combatant commands are currently reporting 
their capabilities to execute most of their operations and major war 
plans in DRRS, and DOD is reporting this additional information to 
Congress. However, because DRRS does not yet fully interface with 
legacy systems to allow single reporting of readiness data, the 
military services have not consistently used DRRS’s enhanced capability 
reporting features. For example, as of May 2009, the Army and Navy had 
developed interfaces for reporting in DRRS, while the Marine Corps 
required units to only report in their legacy system. Recently, the 
Marine Corps also began developing an interface and has done limited 
reporting in DRRS. In addition, DRRS has not fully addressed the 
challenges with metrics that led Congress to require a new readiness 
reporting system. DRRS metrics are less objective and precise, and no 
more timely than the legacy system metrics. Users have also noted that 
DRRS lacks some of the current and historical data and connectivity 
with DOD’s planning systems necessary to manage and deploy forces. 
Until these limitations are fully addressed, DRRS will not have the 
full complement of features necessary to meet legislative and DOD 
requirements, and users will need to rely on legacy reporting systems 
to support mission-critical decisions. 

What GAO Recommends: 

GAO is making recommendations to address the risks facing DOD in 
acquiring and developing DRRS and increase the chance of success. DOD 
agreed or partially agreed with three of our eight recommendations, and 
disagreed with the remaining five because it stated that it was already 
taking actions in these areas. 

View [hyperlink, http://www.gao.gov/products/GAO-09-518] or key 
components. For more information, contact Sharon Pickup at (202) 512-
9619 or pickups@gao.gov, or Randolph C. Hite at (202) 512-3439 or 
hiter@gao.gov. 

[End of section] 

Contents: 

Letter: 

Background: 

DOD Disagreed with GAO's Prior Recommendation to Develop an 
Implementation Plan to Guide DRRS Development: 

DOD Has Not Effectively Managed and Overseen the Acquisition and 
Deployment of DRRS: 

Some DRRS Features Are Operational but Are Not Fully Consistent with 
Legislative Requirements and DOD Guidance: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Objectives, Scope, and Methodology: 

Appendix II: Detailed Analysis of DRRS Acquisition and Deployment 
Management and Oversight: 

Appendix III: Comments from the Department of Defense: 

Appendix IV: GAO Contacts and Staff Acknowledgments: 

Tables: 

Table 1: DRRS Capability Modules: 

Table 2: Organizations Interviewed during Our Review: 

Table 3: DRRS Satisfaction of Nine Schedule-Estimating Key Practices: 

Figures: 

Figure 1: Air Force and Marine Corps Dual Reporting Requirements to 
Meet Readiness Reporting Guidelines: 

Figure 2: Schedule for Developing and Implementing DRRS Capabilities: 

Figure 3: Changes in Estimated Dates for DRRS Capabilities: 

Abbreviations: 

CJCSI: Chairman of the Joint Chiefs of Staff Instruction: 

DIO: DRRS Implementation Office: 

DOD: Department of Defense: 

DRRS: Defense Readiness Reporting System: 

ESORTS: Enhanced Status of Resources and Training System: 

GSORTS: Global Status of Resources and Training System: 

JITC: Joint Interoperability Test Command: 

MAIS: Major Automated Information System: 

OSD: Office of the Secretary of Defense: 

SIPRNet: Secure Internet Protocol Router Network: 

TEMP: Test and Evaluation Master Plan: 

USD (P&R): Under Secretary of Defense for Personnel and Readiness: 

[End of section] 

United States Government Accountability Office:
Washington, DC 20548: 

September 25, 2009: 

The Honorable Evan Bayh: 
Chairman: 
The Honorable Richard Burr: 
Ranking Member: 
Subcommittee on Readiness and Management Support: 
Committee on Armed Services: 
United States Senate: 

To assess the ability of U.S. forces to execute the wartime missions 
for which they were designed, as well as other assigned missions, and 
to make decisions about deploying forces, the Department of Defense 
(DOD) relies heavily on readiness information derived from multiple 
information systems. Over the years, we and others have identified 
shortcomings in DOD's readiness assessment and reporting, such as 
limitations in the completeness and precision of readiness data and a 
tendency to focus on examining the status of personnel, equipment, and 
other resources rather than broader capabilities. Congress addressed 
DOD's readiness reporting in the Strom Thurmond National Defense 
Authorization Act for Fiscal Year 1999[Footnote 1] by adding section 
117 to Title 10 of the U.S. Code, directing the Secretary of Defense to 
establish a comprehensive readiness reporting system to measure, in an 
"objective, accurate, and timely manner," the capability of the armed 
forces to carry out the National Security Strategy prescribed by the 
President, the defense planning guidance provided by the Secretary of 
Defense, and the National Military Strategy prescribed by the Chairman 
of the Joint Chiefs of Staff. 

In June 2002, the Deputy Secretary of Defense directed[Footnote 2] the 
Under Secretary of Defense for Personnel and Readiness (USD P&R) to 
develop, field, maintain, and fund the Enhanced Status of Resources and 
Training System (ESORTS), which is the automated readiness reporting 
system within the Defense Readiness Reporting System (DRRS).[Footnote 
3] He also directed that DRRS build upon existing processes and 
readiness assessment tools to establish a capabilities-based, adaptive, 
near-real-time readiness reporting system. In addition, in June 2004, 
the Secretary of Defense directed USD (P&R) to develop DRRS in a manner 
that would support the data requirements of various users of readiness 
information, such as the Chairman of the Joint Chiefs of Staff, the 
combatant commanders, the Secretaries of the military departments, and 
the Chief of the National Guard Bureau, including their requirements 
for data on the availability, readiness, deployment, and redeployment 
of forces.[Footnote 4] 

USD (P&R) established a DRRS Implementation Office (DIO) to manage the 
system's acquisition, including managing system development and 
engaging the user community. The DIO has used support contractors to 
develop the system and reported obligating about $96.5 million for DRRS 
from fiscal year 2001 through fiscal year 2008. Some fielded system 
capabilities are currently being used to varying degrees by the user 
community. The DIO originally estimated that DRRS would achieve full 
operational capability in fiscal year 2007, but currently expects DRRS 
to reach full capability in 2014. In September 2008, the DIO projected 
that it would spend about $135 million through fiscal year 2014. 

Recognizing that DRRS was not yet fully deployed or operational and in 
light of our prior work on readiness-related issues, you asked us to 
review DOD's efforts to develop and implement DRRS, including the 
program's status, and the extent that DRRS addresses the challenges 
that led Congress to require a new system, such as the availability of 
information on capabilities, and the precision, timeliness, 
reliability, and objectivity of readiness metrics. In addressing these 
issues, we assessed the extent to which (1) DOD has effectively managed 
and overseen DRRS acquisition and deployment, and (2) features of DRRS 
have been implemented and are consistent with legislative requirements 
and DOD guidance. 

To determine the extent that DOD has effectively managed and overseen 
DRRS acquisition and deployment, we analyzed a range of program 
documentation and interviewed cognizant program and contractor 
officials relative to the following acquisition management disciplines: 
requirements development and management, test management, schedule 
reliability, and human capital. For each discipline, we compared key 
program documentation, such as a requirements management plan, test and 
evaluation master plans, and test reports to relevant DOD, federal, and 
related guidance, and we analyzed a statistical sample of individual 
requirements and test cases to determine consistency among them. In 
addition, we attended meetings of organizations established to monitor 
or govern DRRS development and reviewed information from meetings that 
we did not attend and interviewed officials associated with these 
meetings. 

To determine the extent to which the features of DRRS have been 
implemented and are consistent with legislative requirements and DOD 
guidance, we reviewed criteria such as the legislation that mandated a 
comprehensive DOD readiness reporting system, the DOD directive that 
established DRRS, program documentation and USD (P&R) guidance 
memorandums, DIO briefings to the readiness community, other 
departmental instructions, and directives and memorandums related to 
DRRS requirements and implementation. From these documents, we 
identified desired features of DRRS and compared them to documentary 
and testimonial evidence concerning system performance during meetings 
with a full range of officials responsible for developing and using the 
system. To obtain the developer's perspective, on numerous occasions 
throughout our review we met with officials from USD (P&R), the DIO, 
and the three current DRRS contractors. To obtain user perspectives, we 
met with and surveyed by questionnaire officials from the Joint Staff, 
the geographic and functional combatant commands, and the services, and 
also met with officials from USD (P&R). We also attended meetings of 
organizations established to monitor or govern DRRS development and 
analyzed information from meetings that we did not attend. We also 
directly observed the system's capabilities through our own limited use 
of the system and by observing others using the system. 

We did not evaluate the department's overall ability to assess the 
readiness of its forces or the extent that data contained in any of its 
readiness reporting systems, including DRRS and GSORTS, reflect 
capabilities, deficiencies, vulnerabilities, or performance issues. Our 
review focused on acquisition and program management issues, such as 
requirements management, schedule and human capital requirements, the 
current usage of DRRS, and the extent to which DRRS' features address 
legislative requirements and DOD guidance. 

We conducted this performance audit from April 2008 through August 2009 
in accordance with generally accepted government auditing standards. 
Those standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe that 
the evidence obtained provides a reasonable basis for our findings and 
conclusions based on our audit objectives. Additional details on our 
scope and methodology are in appendix I. 

Background: 

Historically, DOD has used its readiness assessment system to assess 
the ability of units and joint forces to fight and meet the demands of 
the national security strategy. DOD's readiness assessment and 
reporting system is designed to assess and report on military readiness 
at three levels--(1) the unit level; (2) the joint force level; and (3) 
the aggregate, or strategic, level. Using information from its 
readiness assessment system, DOD prepares and sends legislatively 
mandated Quarterly Readiness Reports to Congress. DRRS is DOD's new 
readiness reporting system that is intended to capture information from 
the previous system, as well as information about organizational 
capabilities to perform a wider variety of missions and mission 
essential tasks. DRRS is also intended to capture readiness information 
from defense agencies and installations, which were not required to 
report under the previous system. Some DRRS features are currently 
fielded and being used to varying degrees by the user community. 

DOD Collects a Wide Range of Readiness Information to Support Decision 
Makers Within and Outside DOD: 

Laws, directives, and guidance, including a DOD directive, Chairman of 
the Joint Chiefs of Staff Instruction (CJCSI), Secretary of Defense and 
USD (P&R) memorandums, and service regulations and messages, show that 
readiness information and data are needed to support a wide range of 
decision makers. These users of readiness data include Congress, the 
Secretary of Defense, the Chairman of the Joint Chiefs of Staff, the 
combatant commanders, the Secretaries of the military departments, and 
the Chief of the National Guard Bureau. 

The directives and guidance also list roles and responsibilities for 
collecting and reporting various types of readiness data. For example, 
CJCSI 3401.02A[Footnote 5] assigns the service chiefs responsibility 
for ensuring required global status of resources and training system 
(GSORTS) reports are submitted. GSORTS is DOD's legacy, resource-based 
readiness reporting system that provides a broad assessment of unit 
statuses based on units' abilities to execute the missions for which 
they were organized or designed as well as the current missions for 
which they may be employed. The information in the required GSORTS 
reports includes units' abilities to execute the missions for which 
they were organized or designed, as well as the status of their 
training, personnel, and equipment.[Footnote 6] In addition, DOD 
directive 7730.65, which established DRRS as DOD's new readiness 
reporting system, assigns the Secretaries of the military departments 
and the commanders of the combatant commands responsibilities for 
developing mission essential tasks for all of their assigned missions. 
[Footnote 7] 

Requirements and Intended Characteristics of DOD's New Readiness 
Reporting System: 

Prior to 1999, we identified challenges with DOD's existing readiness 
reporting system, GSORTS, and in 1999, Congress directed the Secretary 
of Defense to establish a comprehensive readiness reporting system. 
[Footnote 8] The legislation requires the system to measure in an 
objective, accurate, and timely manner the capability of the armed 
forces to carry out (1) the National Security Strategy prescribed by 
the President, (2) the defense planning guidance provided by the 
Secretary of Defense, and (3) the National Military Strategy prescribed 
by the Chairman of the Joint Chiefs of Staff. 

To address the requirements established by Congress, the Office of the 
Deputy Under Secretary of Defense (Readiness) began in 2001 to build 
consensus among DOD's senior readiness leaders for an improved 
readiness assessment system. For example, the Deputy's office 
distributed a list of key characteristics of the improved readiness 
assessment system to the leaders in advance of scheduled meetings. The 
system's key desired characteristics included allowing near-real-time 
access to readiness data and trends, enabling rapid, low-cost 
development using classified Internet technology, and reducing the 
reporting burdens on people. Since then various directives and 
memorandums have been issued regarding DRRS responsibilities, 
requirements, and related issues. For example: 

* On June 3, 2002, the Deputy Secretary of Defense established DOD's 
new readiness reporting system, as directed by Congress, by signing DOD 
Directive 7730.65. According to this directive, DRRS is intended to 
build upon DOD's existing processes and readiness assessment tools to 
establish a capabilities-based, near-real-time readiness reporting 
system. The DRRS directive assigned USD (P&R) responsibilities for 
developing, fielding, maintaining, and funding ESORTS (the tool to 
collect capability, resource, and training information) and overseeing 
DRRS to ensure accuracy, completeness, and timeliness of its 
information and data, its responsiveness, and its effective and 
efficient use of modern practices and technologies. In addition, the 
USD P&R is responsible for ensuring that ESORTS information, where 
appropriate, is integrated into DOD's planning systems and processes. 
The directive also states that until ESORTS becomes fully operational, 
the Chairman of the Joint Chiefs of Staff shall maintain the GSORTS 
database. 

* On June 25, 2004, the Secretary of Defense issued a memorandum, which 
directed USD (P&R) to develop DRRS to support data requirements 
identified by the Chairman of the Joint Chiefs of Staff, the combatant 
commanders, the Secretaries of the Military Departments, and the Chief, 
National Guard Bureau to include availability, readiness, deployment, 
and redeployment data.[Footnote 9] 

* On November 2, 2004, USD (P&R) issued a DRRS interim implementation 
guidance memorandum.[Footnote 10] In this memorandum, the 
undersecretary noted that he had established a DIO to provide reporting 
assistance for units. The memorandum also stated that combatant 
commanders would begin reporting readiness by mission essential tasks 
by November 30, 2004. The memorandum also directed the services to 
develop detailed implementing guidance for reporting and assessing 
mission essential task readiness in ESORTS within their respective 
services, and set a goal for the services to implement the mission 
essential task reporting process by September 30, 2005. To meet these 
mission essential task reporting requirements, USD (P&R) directed 
commanders to rate their organizational capabilities as (1) yes or "Y", 
(2) qualified yes or "Q", or (3) no or "N." A "Y" indicates that an 
organization can accomplish the rated tasks or missions to prescribed 
standards and conditions in a specified environment. It should reflect 
demonstrated performance in training or operations. A "Q" indicates 
that performance has not been demonstrated, and, although data may not 
readily support a "Y," the commander believes the organization can 
accomplish the rated task or mission to standard under most conditions. 
An "N" indicates that an organization cannot accomplish the rated task 
or mission to prescribed standards in the specified environment at the 
time of the assessment. 

* The November 2004 memorandum also stated that the expected transition 
from GSORTS to ESORTS was scheduled to begin in fiscal year 2005. 
According to the 2004 memorandum, the ESORTS module of DRRS would 
provide, among other things, visibility of the latest GSORTS 
information reported by units, and detailed resource information from 
authoritative data sources with the capability to aggregate or separate 
the data. This memorandum signaled a change in program direction. 
Although the 2002 DOD directive stated that DRRS is intended to build 
upon DOD's existing processes and readiness assessment tools, the 2004 
memorandum indicated that DRRS was to replace GSORTS, as the ESORTS 
module of DRRS captured both capabilities and resource data. 

Overview of DRRS Program Management and Oversight Structure: 

Since its establishment, the DIO has operated within the Office of USD 
(P&R) and has relied on multiple contractors. To provide governance of 
DRRS, and enhance communication between the development community, 
represented by the DIO and contractors, and the user community, which 
includes the Joint Staff, military services, and combatant commands, 
USD (P&R) established various bodies with representatives from the user 
community, including military services, combatant commands, and the 
defense agencies. Representatives from the Office of USD (P&R) and the 
Joint Staff currently serve as cochairs of the various bodies. DRRS 
Battle Staffs comprise colonels, Navy captains, and similar-graded 
civilians. They track DRRS development and identify issues with the 
system. At the one-star level, the DRRS General and Flag Officer 
Steering Committee discusses issues raised by the Battle Staff. In 
December 2007, USD (P&R) created a committee at the three-star level, 
referred to as the DRRS Executive Committee. Its charter, finalized 
about a year later in January 2009, calls for the committee to review 
and approve proposals and plans to establish policy, processes, and 
system requirements for DRRS, including approving software development 
milestones required to reach objectives. 

To ensure that leadership is provided for the direction, oversight, and 
execution of DOD's business transformation efforts, including business 
systems modernization efforts such as DRRS, DOD relies on several 
entities. These entities include the Defense Business Systems 
Management Committee, which is chaired by the Deputy Secretary of 
Defense and serves as the department's highest-ranking governance body 
and the approval authority for business systems modernization 
activities; the Investment Review Boards, which are chartered by the 
Principal Staff Assistants--senior leaders from various offices within 
DOD--and serve as the review and certification bodies for business 
system investments in their respective areas of 
responsibility;[Footnote 11] and the Business Transformation Agency, 
which is responsible for supporting the Investment Review Boards and 
for leading and coordinating business transformation efforts across the 
department. Among other things, the Business Transformation Agency 
supports the Office of the Under Secretary of Defense, Acquisition, 
Technology and Logistics in conducting system acquisition risk 
assessments.[Footnote 12] 

Disciplined System Acquisition and Oversight Are Keys to Program 
Success: 

Our research and evaluations of information technology programs, 
including business systems modernization efforts within DOD, have shown 
that delivering promised system capabilities and benefits on time and 
within budget largely depends on the extent to which key program 
management disciplines are employed by an adequately staffed program 
management office. Among other things, these disciplines include a 
number of practices associated with effectively developing and managing 
system requirements, adequately testing system capabilities, and 
reliably scheduling the work to be performed. They also include 
proactively managing the program office's human capital needs, and 
promoting program office accountability through executive-level program 
oversight. DOD acquisition policies and guidance, along with other 
relevant guidance, recognize the importance of these management and 
oversight disciplines.[Footnote 13] As we have previously reported, not 
employing these and other program management disciplines increases the 
risk that system acquisitions will not perform as intended and require 
expensive and time-consuming rework. 

DOD Disagreed with GAO's Prior Recommendation to Develop an 
Implementation Plan to Guide DRRS Development: 

In 2003, we reported that, according to USD (P&R) officials, DRRS was a 
large endeavor, and that development would be challenging and require 
buy-in from many users.[Footnote 14] We also reported that the program 
was only a concept without detailed plans to guide its development and 
implementation. Based on the status of the program at that time and 
DOD's past record on readiness reporting initiatives, we recommended 
that the Secretary of Defense direct the Office of USD (P&R) to develop 
an implementation plan that identified: 

* performance goals that are objective, quantifiable, and measurable; 

* performance indicators to measure outcomes; 

* an evaluation plan to compare program results with established goals; 
and: 

* milestones to guide DRRS development to the planned 2007 full 
capability date. 

DOD did not agree with our recommendation, stating that it had 
established milestones, cost estimates, functional responsibilities, 
expected outcomes, and detailed plans for specific information 
technology requirements and progress indicators. In evaluating the DOD 
comments, we noted that DOD had established only two milestones-- 
initial capability in 2004 and full capability in 2007--and did not 
have a road map explaining the steps needed to achieve full capability 
by 2007.[Footnote 15] 

DOD Has Not Effectively Managed and Overseen the Acquisition and 
Deployment of DRRS: 

DOD has not effectively managed and overseen the acquisition and 
deployment of DRRS in accordance with a number of key program 
management disciplines that are recognized in DOD acquisition policies 
and guidance, along with other relevant guidance, and are fundamental 
to delivering a system that performs as intended on time and within 
budget. In particular, DRRS requirements have not been effectively 
developed and managed, and DRRS testing has not been adequately 
performed and managed. Further, DRRS has not been guided by a reliable 
schedule of the work needed to be performed and the key activities and 
events that need to occur. These program management weaknesses can be 
attributed in part to long-standing limitations in program office 
staffing and oversight. As a result, the program has not lived up to 
the requirements set for it by Congress, and the department has not 
received value from the program that is commensurate with the time and 
money invested--about 7 years and $96.5 million. Each of these 
weaknesses are summarized below and discussed in detail in appendix II. 

DRRS Requirements Have Not Been Effectively Developed and Managed: 

According to DOD and other relevant guidance, effective requirements 
development and management includes, among other things, (1) 
effectively eliciting user needs early and continuously in the system 
life-cycle process, (2) establishing a stable baseline set of 
requirements and placing the baseline under configuration management, 
(3) ensuring that system requirements are traceable backward to higher 
level business or operational requirements (e.g., concept of 
operations) and forward to system design documents (e.g., software 
requirements specification) and test plans, and (4) controlling changes 
to baseline requirements. However, none of these conditions have been 
met on DRRS. Specifically, key users have only recently become fully 
engaged in developing requirements, and requirements have been 
experiencing considerable change and are not yet stable. Further, 
different levels of requirements and related test cases have not been 
aligned with one another, and changes to requirements have not been 
effectively controlled. As a result, efforts to develop and deliver 
initial DRRS capabilities have taken longer than envisioned and these 
capabilities have not lived up to user expectations. These failures 
increase the risk of future DRRS capabilities not meeting expectations 
and increase the likelihood that expensive and time-consuming system 
rework will be necessary. 

Recent Actions Have Been Taken to Address Limited User Involvement in 
Developing and Managing Requirements: 

Until recently, key users were not fully or effectively engaged in DRRS 
requirements development and management. One of the leading practices 
associated with effective requirements development is engaging system 
users early and continuously in the process of defining requirements. 
However, DIO officials and representatives from the military services 
and the Joint Staff agree that until recently, key users were not 
effectively engaged in DRRS requirements development and management, 
although they disagree at to why user involvement has suffered. 
Regardless, DRRS Executive Committee direction has improved the 
situation. Specifically, in January 2008, the committee directed the 
Joint Staff to conduct an analysis of DRRS capabilities, referred to as 
the "capabilities gap analysis," which involved the entire readiness 
community and resulted in 530 additional user requirements. In our 
view, this analysis is a positive step in addressing long-standing 
limited involvement by key DRRS users in defining requirements that has 
contributed to significant delays in the program, as discussed later in 
the report. 

DRRS Requirements Are Not Stable: 

As of April 2009, DRRS requirements continued to be in a state of flux. 
Establishing an authoritative set of baseline requirements prior to 
system design and development provides a stable basis for designing, 
developing, and delivering a system that meets its users' operational 
needs. However, the fact that these 530 user requirements have recently 
been identified means that the suite of requirements documentation 
associated with the system, such as the detailed system requirements, 
will need to change and thus is not stable. To illustrate, these 530 
requirements have not been fully evaluated by the DIO and the DRRS 
governance boards and according to program officials, have not yet been 
approved, and thus their impact on the program is not clear. 
Compounding this instability in the DRRS requirements is the fact that 
additional changes are envisioned. According to program officials, the 
changes resulting from the gap analysis and reflected in the latest 
version of the DRRS Concept of Operations, which was approved by the 
DRRS Executive Committee in January 2009, have yet to be reflected in 
other requirements documents, such as the detailed system requirements. 
Although defining and developing requirements is inherently an 
iterative process, having a baseline set of requirements that are 
stable is a prerequisite to effective and efficient system design and 
development. Without them, the DIO has not been able to deliver a 
system that meets user needs on time, and it is unlikely that future 
development and deployment efforts will produce better results. 

Alignment among Requirements and Related System Design and Testing 
Artifacts Has Not Been Demonstrated: 

During our review, DIO officials could not demonstrate that 
requirements and related system design and testing artifacts are 
properly aligned. One of the leading practices associated with 
developing and managing requirements is maintaining bidirectional 
traceability from high-level operational requirements through detailed 
lower-level requirements and design documents to test cases. We 
attempted on three separate occasions to verify the traceability of 
system requirements backwards to higher-level requirements and forward 
to lower-level software specifications and test cases, and each time we 
found that traceability did not exist. DIO and contractor officials 
attributed the absence of adequate requirements traceability to the 
ongoing instability in requirements and efforts to update program 
documentation. Without traceable requirements, the DIO does not have a 
sufficient basis for knowing that the scope of the design, development, 
and testing efforts will produce a system solution on time and on 
budget and that will meet users' operational needs and perform as 
intended. As a result, the risk is significant that expensive and time- 
consuming system rework will be required. 

Changes to Requirements Are Not Being Effectively Controlled: 

Since the inception of the program in 2002, DRRS has been developed and 
managed without a formally documented and approved process for managing 
changes to system requirements. Adopting a disciplined process for 
reviewing and accepting changes to an approved baseline set of 
requirements in light of the estimated costs, benefits, and risk of 
each proposed change is a recognized best practice. However, 
requirements management and change-control plans developed in 2006 by 
the DRRS software development contractor, according to DIO officials, 
were not adequate. To address this, the Joint Staff developed what it 
referred to as a conceptual requirements change-control process in 
February 2008, as a basis for the DIO to develop more detailed plans 
that could be implemented. In January 2009, the DIO drafted more 
detailed requirements management and configuration management plans, 
the latter of which the DIO updated in March 2009. However, the plans 
have yet to be approved and implemented. Until the DIO effectively 
controls requirements changes, it increases the risk of needed DRRS 
capabilities taking longer and costing more to deliver than necessary. 

DRRS Testing Has Not Been Adequately Performed and Key Test Management 
Structures and Controls Have Not Been Established: 

According to DOD and other relevant guidance, system testing should be 
progressive, meaning that it should consist of a series of test events 
that first focus on the performance of individual system components, 
then on the performance of integrated system components, followed by 
system-level tests that focus on whether the system (or major system 
increments) are acceptable, interoperable with related systems, and 
operationally suitable to users. For this series of related test events 
to be conducted effectively, each test event needs to be executed in 
accordance with well-defined test plans, the results of each test event 
need to be captured and used to ensure that problems discovered are 
disclosed and corrected, and all test events need to be governed by a 
well-defined test management structure. 

However, the DIO cannot demonstrate that it has adequately tested any 
of the DRRS increments, referred to as system releases and subreleases, 
even though it has already acquired and partially deployed a subset of 
these increments. Moreover, the DIO has yet to establish the test 
management structures and controls needed to effectively execute DRRS 
testing going forward. More specifically, the test events for already 
acquired, as well as currently deployed and operating, DRRS releases 
and subreleases were not based on well-defined plans. For example, the 
test plan did not include a schedule of activities to be performed or 
defined roles and responsibilities for performing them. Also, the test 
plan did not consistently include test entrance and exit criteria, a 
test defect management process, and metrics for measuring progress. 
Further, test events have not been fully executed in accordance with 
plans, or executed in a verifiable manner, or both. For example, 
although increments of DRRS functionality have been put into 
production, the DIO has not performed system integration testing, 
system acceptance testing, or operational testing on any DRRS release 
or subrelease. Moreover, the results of all executed test events have 
not been captured and used to ensure that problems discovered were 
disclosed to decision makers, and ultimately corrected. For example, 
the DIO has not captured the test results for at least 20 out of 63 
DRRS subreleases. Test results that were captured did not include key 
elements, such as entrance/exit criteria status and unresolved defects 
and applicable resolution plans. 

The DIO has also not established an effective test management structure 
to include, for example, a clear assignment of test management roles 
and responsibilities, or a reliable schedule of planned test events. 
Compounding this absence of test management structures and controls is 
the fact that the DIO has yet to define how the development and testing 
to date of a series of system increments (system releases and 
subreleases) relate to the planned development and testing of the 10 
system modules established in January 2009. (See table 1 for a list and 
description of these modules.) Collectively, this means that it is 
unlikely, that already developed and deployed DRRS increments can 
perform as intended and meet user operational needs. Equally doubtful 
are the chances that the DIO can adequately ensure that yet-to-be 
developed DRRS increments will meet expectations. 

Table 1: DRRS Capability Modules: 

Module: 1. Joint Mission Readiness; 
Definition: Enables selected users to see and assess the highest 
strategic-level "joint" readiness data. 

Module: 2. Unit Mission Readiness; 
Definition: Captures unit-level readiness data, as well as 
authoritative resources data (e.g., personnel, equipment) fed from 
external systems owned by military services. 

Module: 3. Asset Visibility; 
Definition: Allows authoritative resources data from military services 
to be provided in near-real-time, and certifies them. 

Module: 4. Business Intelligence; 
Definition: Provides the ability to query and analyze readiness and 
asset visibility data, based on the desires and needs of the user. 

Module: 5. Installation Readiness; 
Definition: Allows readiness reporting by installations, 
training/firing ranges and other infrastructure facilities. 

Module: 6. Readiness Reviews; 
Definition: Enables forcewide readiness reviews to be conducted, such 
as the Joint Combat Capability Assessment review process. 

Module: 7. Planning/Execution Support; 
Definition: Allows DRRS to interact with other planning systems and 
processes, such as the Global Force Management and Adaptive Planning 
and Execution communities. 

Module: 8. Historical Data; 
Definition: Focuses on the historical collection and presentation of 
information. Also includes the Continuity of Operations (COOP) 
capability. 

Module: 9. System Architecture; 
Definition: Meets the underlying information technology system 
specifications, such as standards for information assurance, 
interoperability, bandwidth, and other issues. 

Module: 10. End-User Usability; 
Definition: Fulfills end-user support of the DRRS application to 
include training, customer support, and documentation. 

Source: DOD. 

[End of table] 

DRRS Has Not Been Guided by a Reliable Schedule: 

The success of any program depends in part on having a reliable 
schedule that defines, among other things, when work activities will 
occur, how long they will take, and how they are related to one 
another. From its inception in 2002 until November 2008, the DIO did 
not have an integrated master schedule, and thus has long been allowed 
to proceed without a basis for executing the program and measuring its 
progress. In fact, the only milestone that we could identify for the 
program prior to November 2008 was the date that DRRS was to achieve 
full operational capability, which was originally estimated to occur in 
fiscal year 2007, but later slipped to fiscal year 2008 and then fiscal 
year 2011, and is now fiscal year 2014--a 7-year delay. 

Moreover, the DRRS integrated master schedule that was first developed 
in November 2008, and was updated in January 2009 and again in April 
2009 to address limitations that we identified and shared with the 
program office, is still not reliable. Specifically, our research has 
identified nine practices associated with developing and maintaining a 
reliable schedule.[Footnote 16] These practices are (1) capturing all 
key activities, (2) sequencing all key activities, (3) assigning 
resources to all key activities, (4) integrating all key activities 
horizontally and vertically, (5) establishing the duration of all key 
activities, (6) establishing the critical path for all key activities, 
(7) identifying float between key activities, (8) conducting a schedule 
risk analysis, and (9) updating the schedule using logic and durations 
to determine the dates for all key activities.[Footnote 17] The 
program's latest integrated master schedule does not address three of 
the practices and only partially addresses the remaining six. For 
example, the schedule does not establish a critical path for all key 
activities, nor does it include a schedule risk analysis, and it is not 
being updated using logic and durations to determine the dates for all 
key activities. In addition, the schedule introduces considerable 
concurrency across key activities and events for several modules, which 
introduces increased risk. These limitations in the program's latest 
integrated master schedule, coupled with the program's 7-year slippage 
to date and continued requirements instability, make it likely that 
DRRS will incur further delays. 

DIO Lacks Adequate Staffing and an Effective Approach to Meeting Its 
Human Capital Needs: 

The DIO does not currently have adequate staff to fulfill its system 
acquisition and deployment responsibilities, and it has not managed its 
staffing needs in an effective manner. Effective human capital 
management should include an assessment of the core competencies and 
essential knowledge, skills, and abilities needed to perform key 
program management functions, an inventory of the program's existing 
workforce capabilities, an analysis of the gap between the assessed 
needs and the existing capabilities, and plans for filling identified 
gaps. DIO performs a number of fundamental DRRS program management 
functions, such as acquisition planning, performance management, 
requirements development and management, test management, contractor 
tracking and oversight, quality management, and configuration 
management. To effectively perform such functions, program offices, 
such as the DIO, need to have not only well-defined policies and 
procedures and support tools for each of these functions, but also 
sufficient human capital to implement the processes and use the tools 
throughout the program's life cycle. However, the DIO is staffed with 
only a single full-time government employee--the DIO Director. All 
other key program office functions are staffed by either contractor 
staff or staff temporarily detailed, on an as-needed basis, from other 
DOD organizations. In addition, key positions, such as performance 
manager and test manager, have either not been established or are 
vacant. According to DIO and contractor officials, they recognize that 
additional program management staffing is needed but stated that 
requests for additional staff had not been approved by USD (P&R) due to 
competing demands for staffing. Further, they stated that the requests 
were not based on an assessment of the program's human capital needs 
and the gap between these needs and its onboard workforce capabilities. 
Until DIO adopts a strategic, proactive approach to managing its human 
capital needs, it is unlikely that it will have an adequate basis for 
obtaining the people it needs to effectively and efficiently manage 
DRRS. 

DOD Executive Oversight of DRRS Has Been Limited: 

A key principle for acquiring and deploying system investments is to 
establish a senior-level governance body to oversee the investment and 
hold program management accountable for meeting cost, schedule, and 
performance commitments. Moreover, for investments that are 
organization wide in scope and introduce new ways of doing business, 
like DRRS, the membership of this oversight body should represent all 
stakeholders and have sufficient organizational seniority to commit 
their respective organizations to any decisions reached. For 
significant system investments, the department's acquisition process 
provides for such executive governance bodies. For example, Major 
Automated Information Systems, which are investments over certain 
dollar thresholds or that are designated as special interest because 
of, among other things, their mission importance, are reviewed at major 
milestones by a designated milestone decision authority.[Footnote 18] 
These authorities are supported by a senior advisory group, known as 
the Information Technology Acquisition Board, which comprises senior 
officials from the Joint Staff, the military departments, and staff 
offices within the Office of the Secretary of Defense. In addition, all 
business system investments in DOD that involve more than $1 million in 
obligations are subject to review and approval by a hierarchy of DOD 
investment review boards that comprise senior DOD leaders, including 
the Defense Business Systems Management Committee, which is chaired by 
the Deputy Secretary of Defense. Through these executive oversight 
bodies and their associated processes, programs are to be, among other 
things, governed according to corporate needs and priorities, and 
program offices are to be held accountable for meeting cost, schedule, 
and performance expectations. 

Until April 2009, DRRS was not subject to any of DOD's established 
mechanisms and processes for overseeing information technology systems. 
As previously discussed, USD (P&R) established the DRRS Battle Staff, 
which is a group of midlevel military officers and civilians from DRRS 
stakeholder organizations, and it established a higher-ranked General 
and Flag Officer Steering Committee, consisting of stakeholder 
representatives. However, neither of these entities had specific 
oversight responsibilities or decision-making authority for DRRS. 
Moreover, neither was responsible for holding the program office 
accountable for results. According to meeting minutes and knowledgeable 
officials, these entities met on an irregular basis over the last 
several years, with as much as a 1-year gap in meeting time for one of 
them, to discuss DRRS status and related issues. 

In December 2007, USD (P&R) recognized the need for a more senior-level 
and formal governance body, and established the DRRS Executive 
Committee. Since January 2008, this committee, which consists of top- 
level representatives from stakeholder organizations, has met at least 
seven times. In January 2009, the DRRS Executive Committee's charter 
was approved by the Deputy Under Secretary of Defense (Readiness) and 
the three-star Director of the Joint Staff. According to the charter, 
the committee is to review and approve proposals and plans to establish 
policy, processes, and system requirements for DRRS, including 
approving software development milestones required to reach objectives. 
Consistent with its charter, the committee has thus far made various 
program-related decisions, including approving a DRRS concept of 
operations to better inform requirements development, and directing the 
Joint Staff to conduct an analysis to identify any gaps between DRRS 
requirements and user needs. However, the committee has not addressed 
the full range of acquisition management weaknesses previously 
discussed in this report, and it has not taken steps to ensure that the 
program office is accountable for well-defined program baseline 
requirements. 

More recently, the DOD Human Resources Management Investment Review 
Board and the Defense Business Systems Management Committee reviewed 
DRRS and certified and approved, respectively, the program to invest 
$24.625 million in fiscal years 2009 and 2010. These entities comprise 
senior leadership from across the department, including the Deputy 
Secretary of Defense as the Defense Business Systems Management 
Committee Chair, military service secretaries, the defense agency 
heads, principal staff assistants, and representatives from the Joint 
Staff and combatant commands. However, neither the Investment Review 
Board's certification nor the Defense Business Systems Management 
Committee's approval was based on complete and accurate information 
from USD (P&R). Specifically, the certification package submitted to 
both oversight bodies by the USD (P&R) precertification authority 
(Office of Readiness Programming and Assessment) stated that DRRS was 
on track for meeting its cost, schedule, and performance measures and 
highlighted no program risks despite the weaknesses discussed in this 
report. According to the chairwoman of the Investment Review Board, the 
board does not have a process or the resources to validate the 
information received from the programs that it reviews.[Footnote 19] 
Moreover, the chairwoman stated that program officials did not make the 
board aware of the results of our review that we shared with the DIO 
prior to either the Investment Review Board or Defense Business Systems 
Management Committee reviews. Since we briefed the chairwoman, the 
Investment Review Board has requested that the DIO provide it with 
additional information documenting DRRS compliance with applicable DOD 
regulations and statutes. 

According to USD (P&R) and DIO officials, DRRS was not subject to 
department executive-level oversight for almost 6 years because, among 
other things, they did not consider DRRS to be a more complex 
information technology system. Furthermore, because of the nature of 
the authority provided to the USD (P&R) in the DRRS charter, they did 
not believe it was necessary to apply the same type of oversight to 
DRRS as other information systems within DOD. This absence of effective 
oversight has contributed to a void in program accountability and 
limited prospects for program success. 

Some DRRS Features Are Operational but Are Not Fully Consistent with 
Legislative Requirements and DOD Guidance: 

DOD has implemented DRRS features that allow users to report certain 
mission capabilities that were not reported under the legacy system, 
but these features are not fully consistent with legislative 
requirements and DOD guidance; and DOD has not yet implemented other 
envisioned features of the system. While some users are consistently 
reporting enhanced capability information, reporting from other users 
has been inconsistent. In addition, DRRS has not fully addressed the 
challenges with metrics that were identified prior to 1999 when 
Congress required DOD to establish a new readiness reporting system. 
Users have also noted that DRRS lacks some of the current and 
historical data and connectivity with DOD's planning systems necessary 
to manage and deploy forces. 

DOD Is Providing Congress with DRRS Capability Data from the Combatant 
Commands, but Services Have Not Consistently Reported Capability Data: 

The geographic combatant commands are capturing enhanced capability 
data in DRRS, and DOD's quarterly readiness reports to Congress 
currently contain this information, as well as information that is 
drawn from DOD's legacy readiness reporting system, GSORTS. However, 
the military services have not consistently used the enhanced 
capability reporting features of DRRS. Because DRRS does not yet fully 
interface with legacy systems to allow single reporting of readiness 
data, the Army and Navy developed additional system interfaces and are 
reporting in DRRS. Until May 2009, the Marine Corps directed its units 
to report only in the legacy system to avoid the burden of dual 
reporting. The Air Force chose not to develop an interface and 
instructed its units to report in both DRRS and the legacy system. 

DRRS and GSORTS both contain capabilities information and resource 
(numbers of personnel, equipment availability, and equipment condition) 
and training data. However, DRRS currently provides more capabilities 
data than GSORTS. When Congress directed DOD to establish a new 
readiness reporting system, GSORTS was already providing capability 
information concerning unit capabilities to perform the missions for 
which they were organized or designed.[Footnote 20] More recently, some 
of the military services began reporting limited capability information 
on unit capabilities to perform missions other than those that they 
were organized or designed to perform into GSORTS.[Footnote 21] 
However, DRRS is designed to capture capabilities on a wider variety of 
missions and mission essential tasks. For example, organizations can 
report their capabilities to conduct missions associated with major war 
plans and operations such as Operation Iraqi Freedom into DRRS, as well 
as their capabilities to perform the missions for which they were 
organized or designed. DRRS also captures capability information from a 
wider range of organizations than GSORTS. Although the primary 
(monthly) focus is on operational units and commands, DRRS collects and 
displays readiness information from defense agencies and installations. 

Geographic combatant commands--such as U.S. Central Command, U.S. 
Pacific Command, and U.S. Northern Command--are currently reporting 
their commands' capabilities to execute most of their operations and 
major war plans in DRRS. DOD reports this enhanced capability 
information from the geographic combatant commands in its Quarterly 
Readiness Report to Congress. The geographic combatant commands are 
also using DRRS to report their capabilities to perform headquarters- 
level, joint mission essential tasks, and some of these commands 
utilize DRRS as their primary readiness reporting tool. For example, 
U.S. Northern Command uses DRRS to assess risk and analyze capability 
gaps, and U.S. Pacific Command identifies critical shortfalls by 
evaluating mission essential task assessments that are captured in 
DRRS. 

While DRRS currently has the necessary software to collect and display 
these enhanced capability data from organizations at all levels 
throughout DOD, a variety of technical and other factors have hindered 
service reporting of capability data.[Footnote 22] As a result, the 
services have either developed their own systems to report required 
readiness data or have delayed issuing implementing guidance that would 
require their units to report standardized mission essential task data 
in DRRS. By 2005, DRRS was able to collect and display mission 
essential task information from any organizations that had access to a 
Secure Internet Protocol Router Network (SIPRNet) workstation.[Footnote 
23] In August 2005, USD (P&R) issued a memorandum that directed the 
services to ensure that all of their GSORTS-reporting units were 
reporting mission essential task capabilities in DRRS by September 30, 
2005.[Footnote 24] The memorandum stated that, for tactical units, 
mission essential tasks were to be drawn from the Service Universal 
Task List and standardized across like-type entities, such as tank 
battalions, destroyers, or F-16 squadrons.[Footnote 25] However, two 
factors that have hindered compliance with the memorandum's direction 
to report mission essential task capabilities in DRRS are discussed 
below. 

Lack of Direct Access to DRRS: 

While DRRS has been able to collect and display mission essential task 
data since 2005, some Army and Navy users did not have the means to 
directly access DRRS and update mission essential task assessments. For 
example, some ships lacked hardware necessary to be able to transmit 
their mission essential task data directly into DRRS while at sea. In 
addition, many National Guard units lacked, and still lack, direct 
access to the SIPRNet workstations that are necessary to directly input 
mission essential task data directly into DRRS. However, the Army and 
the Navy have developed systems, respectively designated DRRS-A and 
DRRS-N that interface with DRRS and thus allow all of their units to 
report mission essential task data. After Army and Navy units report 
mission essential task data in their respective DRRS-A and DRRS-N 
service systems, the services transmit these data to DRRS. As a result, 
Army and Navy officials told us that they are currently complying with 
the requirement to ensure that all their GSORTS-reporting units report 
mission essential task data in DRRS. 

Delays in Developing Software Tools to Input Data: 

Unlike the Army and the Navy, the Marine Corps and the Air Force have 
not developed their own systems to allow their units to use a single 
tool to enter readiness data to meet Office of the Secretary of 
Defense, Chairman of the Joint Chiefs of Staff, and service readiness 
reporting requirements. While the DIO has developed the software for 
users to enter mission essential task data into DRRS, the DIO has been 
unsuccessful in attempts to develop a tool that would allow Air Force 
and Marine Corps users to enter readiness data to meet all of their 
readiness reporting requirements through DRRS. As a result, rather than 
reducing the burden on reporting units, DRRS has actually increased the 
burden on Air Force and Marine Corps units because they are now 
required to report readiness information in both DRRS and GSORTS. On 
September 29, 2005, USD (P&R) issued a memorandum stating that DRRS is 
the single readiness reporting system for the entire Department of 
Defense and that legacy systems, such as GSORTS and associated service 
readiness systems, should be phased out.[Footnote 26] Since that time, 
officials have discussed whether to phase out GSORTS and tentative 
dates for this action have slipped several times. 

In 2001, the Office of the Deputy Undersecretary of Defense (Readiness) 
listed reducing reporting burdens as a key characteristic of its 
envisioned improved readiness assessment system. In an effort to 
eliminate this burden of dual reporting, the DIO began to develop a 
"current unit status" tool as a means for users to manage unit-specific 
readiness data and submit required reports in support of all current 
reporting guidelines.[Footnote 27] The tool was to minimize the burden 
associated with dual reporting by collecting, displaying, and 
integrating resource data from service authoritative data sources with 
GSORTS and DRRS. However, in December 2007, the DIO reported that it 
was unable to deliver the intended functionality of the "current unit 
status" tool. Instead, the DIO decided to develop an interim reporting 
tool, known as the SORTSREP tool, which would not provide the type of 
new capabilities envisioned for the "current unit status" tool, but 
would simply replicate the functionality of the input tool that the Air 
Force and Marines already used to input data into GSORTS. After delays, 
and 10 months of effort, the DIO delivered the SORTSREP tool to the 
Marine Corps for review. Based on this review, in December, 2008, the 
Marine Corps provided the developers and the DIO with 163 pages of 
detailed descriptions and graphics to explain the SORTSREP tool's 
deficiencies. It then informed the DIO that it would no longer expend 
energy and resources to review future versions of the SORTSREP tool and 
would instead look at leveraging the Army's or Navy's DRRS-A or DRRS-N 
systems. The Air Force also informed the DIO that it was no longer 
interested in the SORSTSREP tool, and said efforts should be focused on 
the "current unit status" tool instead. As a result, the Air Force and 
Marine Corps are currently faced with dual reporting requirements, as 
illustrated in figure 1. 

Figure 1: Air Force and Marine Corps Dual Reporting Requirements to 
Meet Readiness Reporting Guidelines: 

[Refer to PDF for image: illustration] 

Air Force units: 
RAS-IT: 
*GSORTS (Designed mission capabilities, resources, and training); 
DRRS (Missions and mission essential tasks). 

Marine Corps units: 
RAS-IT: 
*GSORTS (Designed mission capabilities, resources, and training); 
DRRS (Missions and mission essential tasks). 

TRAS-IT: Readiness Assessment System Input Tool. 

Source: GAO based on Air Force and Marine Corps information. 

[End of figure] 

On March 3, 2009, the Air Force Deputy Chief of Staff (Operations, 
Plans and Requirements) issued a memorandum that updated the Air 
Force's previous implementing guidance and directed all GSORTS- 
reporting units to begin assessing readiness in DRRS based on 
standardized core task lists within 90 days.[Footnote 28] As a result, 
Air Force units will report readiness in both DRRS and GSORTS until the 
DIO is able to deliver the intended functionality of the "current unit 
status" tool. 

While some Marine Corps units are reporting their capabilities in DRRS, 
the Marine Corps had not yet directed its units to report in the system 
as of May 2009. The Commandant of the Marine Corps had stated that he 
supported the development and implementation of DRRS, but that he would 
not direct units to assess mission essential tasks in DRRS until the 
system met its stated requirements and was accepted as the single 
readiness reporting system of record. Marine Corps officials said that 
they did not want to place a burden on operational units, which were 
fighting or preparing to fight a war, by requiring that they report 
readiness in two different systems. After we completed our audit work, 
on May 12, 2009, the Marine Corps issued an administrative message that 
required that units assess their mission essential tasks and missions 
in DRRS. The message stated that doing so would improve familiarity 
with DRRS, which will lead to an easier transition when the Marine 
Corps fields DRRS-Marine Corps (DRRS-MC).[Footnote 29] 

Without a viable tool for inputting data, DRRS is not fully integrated 
with GSORTS or with the service readiness reporting systems and it is 
not capable of replacing those systems since it does not capture the 
required data that are contained in those systems. 

DRRS Enhanced Capability Data Are Not Likely to Fully Address the 
Challenges with Metrics That Existed Prior to the Establishment of the 
System: 

While DRRS is being used to provide Congress with enhanced capability 
information, the quality of DRRS metrics still faces the same 
challenges, including limitations in timeliness, precision, and 
objectivity that existed prior to 1999 when Congress directed DOD to 
establish a new readiness reporting system. Section 117 of Title 10 of 
the U.S. Code directed the Secretary of Defense to establish a 
comprehensive readiness reporting system to measure the capability of 
the armed forces in an "objective, accurate, and timely manner." 
However, the enhanced capability data that are captured in DRRS and 
reported to Congress are no more timely than the readiness data that 
were being provided to Congress in 1999 using GSORTS. Furthermore, the 
metrics that are being used to capture the enhanced capability 
information are less objective and precise than the metrics that were 
used to report readiness in 1999. 

Timeliness: 

The statute directing the development of a new readiness reporting 
system requires that the reporting system measure in a timely manner 
the capability of the armed forces to carry out the National Security 
Strategy, the Secretary of Defense's defense planning guidance, and the 
National Military Strategy. The legislation also lists a number of 
specific requirements related to frequency of measurements and updates. 
For example, the law requires that the capability of units to conduct 
their assigned wartime missions be measured monthly, and that units 
report any changes in their overall readiness status within 24 hours of 
an event that necessitated the change in readiness status.[Footnote 30] 
In its DRRS directive, DOD assigned USD (P&R) responsibility for 
ensuring the timeliness of DRRS information and data, and it specified 
that DRRS was to be a near-real-time readiness reporting system. 

While DOD is reporting readiness information to Congress on a quarterly 
basis as required, and units are measuring readiness on a monthly 
basis, DRRS is not a near-real-time reporting system. Specifically, in 
DRRS, as in GSORTS, operational commanders assess the readiness of 
their organizations on a monthly basis or when an event occurs that 
changes the units' overall reported readiness. Thus, DRRS has not 
improved the timeliness of the key readiness data that are reported to 
Congress. According to USD (P&R) officials, DRRS data will be more 
timely than GSORTS data because DRRS will update underlying data from 
authoritative data sources between the monthly updates.[Footnote 31] 
However, DRRS is not yet capturing all the data from the authoritative 
data sources, and according to service officials, the service systems 
that support GSORTS also draw information from their service 
authoritative data sources between the monthly updates. Furthermore, 
the source and currency of some of the authoritative data that are 
currently in DRRS are not clearly identified. As a result, some users 
told us that they are reluctant to use DRRS data to support their 
decisions. 

Precision and Objectivity: 

We previously reported that the readiness information that DOD provided 
to Congress lacked precision, noting that GSORTS readiness measures 
that differed by 10 percentage points or more could result in identical 
ratings, with DOD often not reporting the detailed information behind 
the ratings outside of the department.[Footnote 32] For example, units 
that were at 90 and 100 percent of their authorized personnel strengths 
both were reported as P-1 in DOD's reports to Congress. In 2003, USD 
(P&R) recognized the imprecision of the reported metrics from GSORTS 
and noted that its efforts to develop DRRS would allow enhancements to 
reported readiness data. 

As previously noted, the DRRS capability information that DOD is 
reporting to Congress covers a broader range of missions than the 
GSORTS information that was provided in the past. However, when 
comparing the DRRS and GSORTS assessments of units' capabilities to 
perform the missions for which the units were organized or designed, 
DRRS assessments are actually less precise than the GSORTS assessments. 
Specifically, within GSORTS, overall capability assessments are grouped 
into four categories based on four percentage ranges for the underlying 
data. For example, commanders compare on-hand and required levels of 
personnel and equipment. Within DRRS, mission essential task 
assessments are reported on a scale that includes only three ratings-- 
"yes", "no", and "qualified yes," which can include any assessments 
that fall between the two extremes. 

The law directing DOD to establish a new readiness reporting system 
also requires that the system measure readiness in an objective manner. 
GSORTS assessments of units' capabilities to execute the missions for 
which they were organized or designed are based on objective personnel 
and equipment data and training information that may include both 
objective and subjective measures. Furthermore, the overall capability 
assessment in GSORTS is based on an objective rule that calls for the 
overall assessment to be the same level as the lowest underlying 
resource or training data level. For example, if a unit reported the 
highest personnel level (P-1) and the lowest training level (T-4), the 
rules in the GSORTS guidance instruct the commander to rate the unit's 
overall capability at the C-4 level. Because GSORTS contains these 
objective measures and rules, it is easy to evaluate reported readiness 
to see if it aligns with established reporting criteria.[Footnote 33] 

Within DRRS, organizations rate their capabilities based on mission 
essential tasks. These mission essentials tasks have conditions and 
standards associated with them. The conditions specify the types of 
environments that units are likely to face as they execute the tasks, 
such as weather conditions and political or cultural factors. Standards 
describe what it means for the unit to successfully execute the task 
under specified conditions. For example, a unit may have to achieve a 
90 percent success rate for measures associated with the task being 
assessed. In spite of these conditions and standards, DRRS mission 
assessments are often subjective rather than objective. In DRRS program 
guidance, DOD has defined mission essential tasks as tasks that are 
approved by the commander and that, based on mission analysis, are 
"absolutely necessary, indispensable, or critical to mission success." 
In prior briefings and reports to Congress, we have noted examples that 
highlight the subjective nature of DRRS mission assessments. For 
example, we noted that one commander used his professional judgment to 
decide that his command was "qualified" to execute a mission even 
though the preponderance of the "indispensable" tasks that supported 
that mission were rated as "no." In contrast, other commanders used 
their professional judgments to rate missions as "qualified" based on 
one or more "qualified" tasks among many "yes" tasks. 

DRRS Lacks the Complete Resource and Training Data and System 
Connectivity Some Users Need to Manage and Deploy Forces: 

DRRS does not have all of the resource, training, readiness data, and 
connectivity with the department's operations planning and execution 
system that the services, Joint Staff, and certain combatant commands 
need to manage and deploy forces. As a result, DRRS is not yet able to 
operate as the department's single readiness reporting system, as 
intended. The Secretary of Defense's and the Under Secretary of 
Defense's guidance documents recognize that DRRS needs to support the 
data requirements of multiple users. For example, the Secretary of 
Defense's June 25, 2004, memorandum directed USD (P&R) to develop DRRS 
to support the data requirements identified by the Chairman of the 
Joint Chiefs of Staff, the combatant commanders, the Secretaries of the 
military departments, and the Chief of the National Guard Bureau. 
[Footnote 34] Furthermore, the 2002 DRRS directive noted that DRRS was 
to build upon DOD's existing processes and readiness assessment tools 
and that ESORTS information (capability, resource, and training), where 
appropriate, is integrated into DOD's planning systems and processes. 
It also directed the Chairman of the Joint Chiefs of Staff to maintain 
the GSORTS database until key capabilities of DRRS become fully 
operational. 

Complete and Accurate Current and Historical Data: 

Officials with U.S. Joint Forces Command and U.S. Special Operations 
Command reported that historical data are needed to manage forces and 
provide users the ability to analyze readiness trends.[Footnote 35] 
Similarly, service officials stated a need for historical data so they 
can manage their forces and take action to address readiness issues. In 
2005, USD (P&R) reported that unit resource data, including detailed 
inventory and authorization data on personnel, equipment, supply, and 
ordnance were available in DRRS. However, in response to a survey we 
conducted in December 2008, the services and certain combatant commands 
stated that necessary current and historical resource and training data 
were not available in DRRS. For example, officials from all four 
services responded that DRRS, at that time, contained less than half of 
their GSORTS resources and training data. In addition, officials from 
U.S. Joint Forces Command, U.S. Special Operations Command, the U.S. 
Strategic Command, and the U.S. Transportation Command all responded 
that historical resource data were not available in DRRS. We confirmed 
that this information was still not available when we concluded our 
review, and in April, 2009, the DIO said it was still working on this 
data availability issue. 

Furthermore, user organizations have reported inaccuracies in the data 
that are available in DRRS. Marine Corps and U.S. Special Operations 
Command officials stated that inconsistencies between DRRS data and the 
data in other readiness systems have caused them to adjudicate the 
inconsistencies by contacting their subordinate units directly. Army 
officials noted that searches of DRRS data can produce different 
results than searches in the Army's data systems. For example, they 
noted that a DRRS search for available personnel with a particular 
occupational specialty produced erroneously high results because DRRS 
did not employ the appropriate business rules when conducting the 
search. Specifically, DRRS did not apply a business rule to account for 
the fact that an individual person can possess multiple occupational 
specialty codes but can generally fill only one position at a time. DIO 
officials informed us that they intend to correct issues with the 
accuracy of data drawn from external databases. However, the current 
version of the DRRS Integrated Master Schedule indicates that the 
ability of DRRS to provide the capability to correctly search, 
manipulate, and display current and historical GSORTS and mission 
essential task data will not be complete until June 2010. As a result, 
the reliability of the DRRS data is likely to remain questionable and a 
number of DOD organizations will likely continue to rely on GSORTS and 
other sources of readiness data to support their decision making. 

Connections to Planning Systems: 

One important DRRS function is integration with DOD's planning systems. 
Specifically, the 2002 DRRS directive requires USD (P&R) to ensure 
that, where appropriate, ESORTS information (capability, resource, and 
training) is compatible and integrated into DOD's planning systems and 
processes. Global force management is one of the DOD planning processes 
that is to be integrated with DRRS. Global Force Management is a 
process to manage, assess, and display the worldwide disposition of 
U.S. forces, providing DOD with a global view of requirements and 
availability of forces to meet those requirements. The integration of 
DRRS with global force management planning processes is supposed to 
allow DOD to link force structure, resources, and capabilities data to 
support analyses, and thus help global force managers fill requests for 
forces or capabilities. 

Officials from the four organizations with primary responsibilities for 
providing forces (U.S. Joint Forces Command, U.S. Special Operations 
Command, U.S. Strategic Command, and U.S. Transportation Command) all 
stated that they are unable to effectively use DRRS to search for units 
that will meet requested capabilities. These commands also reported 
that DRRS does not currently contain the information and tools 
necessary to support global force management. For example, officials 
from U.S. Northern Command told us that when they used DRRS to search 
for available helicopters of a certain type, they found thousands, but 
when U.S. Joint Forces Command did the same DRRS search they found 
hundreds. The current version of the DRRS Integrated Master Schedule 
indicates that DRRS will not be able to fully support global force 
management until March 2011. As a result, these commands continue to 
rely on GSORTS rather than DRRS to support their planning and sourcing 
decisions. 

Conclusions: 

DRRS is not currently and consistently providing timely, objective, and 
accurate information, and it is not exactly clear where the department 
stands in its efforts to meet this expectation because system 
requirements remain in a state of flux, and the program office lacks 
disciplined program management and results information due to a long- 
standing lack of rigor in its approach to acquiring and deploying 
system capabilities. This situation can be attributed, in part, to long-
standing limitations in the program office's focus on acquiring human 
capital skills needed to manage such a complex initiative. It can also 
be linked to many of years of limited program office oversight and 
accountability. Although program oversight has recently increased, 
oversight bodies have not had sufficient visibility into the program's 
many management weaknesses. 

DRRS is providing Congress and readiness users with additional mission 
and mission essential task capability data that were not available in 
GSORTS. However, after investing about 7 years and about $96.5 million 
in developing and implementing DRRS, the system's schedule has been 
extended, requirements are not stable, and the system still does not 
meet congressional and DOD requirements for a comprehensive readiness 
reporting system to assess readiness and help decision makers manage 
forces needed to conduct combat and contingency operations around the 
world. Given DRRS performance and management weaknesses, it is critical 
that immediate action be taken to put the program on track and position 
it for success. Without this action, it is likely that DRRS will cost 
more to develop and deploy than necessary and that DOD will not have a 
comprehensive reporting system that meets the needs of all the decision 
makers who rely on accurate, timely, and complete readiness 
information. 

Recommendations for Executive Action: 

To address the risks facing DOD in its acquisition and deployment of 
DRRS, and to increase the chances of DRRS meeting the needs of the DOD 
readiness community and Congress, we recommend that the Secretary of 
Defense direct the Deputy Secretary of Defense, as the Chair of the 
Defense Business Systems Management Committee, to reconsider the 
committee's recent approval of DRRS planned investment for fiscal years 
2009 and 2010, and convene the Defense Business Systems Management 
Committee to review the program's past performance and the DIO's 
capability to manage and deliver DRRS going forward. 

To fully inform this Defense Business Systems Management Committee 
review, we also recommend that the Secretary direct the Deputy 
Secretary to have the Director of the Business Transformation Agency, 
using the appropriate team of functional and technical experts and the 
established risk assessment methodology, conduct a program risk 
assessment of DRRS, and to use the findings in our report and the risk 
assessment to decide how to redirect the program's structure, approach, 
funding, management, and oversight. In this regard, we recommend that 
the Secretary direct the Deputy Secretary to solicit the advice and 
recommendations of the DRRS Executive Committee. 

We also recommend that the Secretary, through the appropriate chain of 
command, take steps to ensure that the following occur: 

1. DRRS requirements are effectively developed and managed with 
appropriate input from the services, Joint Staff, and combatant 
commanders, including (1) establishing an authoritative set of baseline 
requirements prior to further system design and development; (2) 
ensuring that the different levels of requirements and their associated 
design specifications and test cases are aligned with one another; and 
(3) developing and instituting a disciplined process for reviewing and 
accepting changes to the baseline requirements in light of estimated 
costs, benefits, and risk. 

2. DRRS testing is effectively managed, including (1) developing test 
plans and procedures for each system increment test event that include 
a schedule of planned test activities, defined roles and 
responsibilities, test entrance and exit criteria, test defect 
management processes, and metrics for measuring test progress; (2) 
ensuring that all key test events are conducted on all DRRS increments; 
(3) capturing, analyzing, reporting, and resolving all test results and 
test defects of all developed and tested DRRS increments; and (4) 
establishing an effective test management structure that includes 
assigned test management roles and responsibilities, a designated test 
management lead and a supporting working group, and a reliable schedule 
of test events. 

3. DRRS integrated master schedule is reliable, including ensuring that 
the schedule (1) captures all activities from the work breakdown 
structure, including the work to be performed and the resources to be 
used; (2) identifies the logical sequencing of all activities, 
including defining predecessor and successor activities; (3) reflects 
whether all required resources will be available when needed and their 
cost; (4) ensures that all activities and their duration are not 
summarized at a level that could mask critical elements; (5) achieves 
horizontal integration in the schedule by ensuring that all external 
interfaces (hand-offs) are established and interdependencies among 
activities are defined; (6) identifies float between activities by 
ensuring that the linkages among all activities are defined; (7) 
defines a critical path that runs continuously to the program's finish 
date; (8) incorporates the results of a schedule risk analysis to 
determine the level of confidence in meeting the program's activities 
and completion date; and (9) includes the actual start and completion 
dates of work activities performed so that the impact of deviations on 
downstream work can be proactively addressed. 

4. The DRRS program office is staffed on the basis of a human capital 
strategy that is grounded in an assessment of the core competencies and 
essential knowledge, skills, and abilities needed to perform key DRRS 
program management functions, an inventory of the program office's 
existing workforce capabilities, and an analysis of the gap between the 
assessed needs and the existing capabilities. 

5. DRRS is developed and implemented in a manner that does not increase 
the reporting burden on units and addresses the timeliness, precision, 
and objectivity of metrics that are reported to Congress. 

To ensure that these and other DRRS program management improvements and 
activities are effectively implemented and that any additional funds 
for DRRS implementation are used effectively and efficiently, we 
further recommend that the Secretary direct the Deputy Secretary to 
ensure that both the Human Resources Management Investment Review Board 
and the DRRS Executive Committee conduct frequent oversight activities 
of the DRRS program, and report any significant issues to the Deputy 
Secretary. 

Agency Comments and Our Evaluation: 

In written comments on a draft of this report, signed by the Deputy 
Under Secretary of Defense (Military Personnel Policy) performing the 
duties of the Under Secretary of Defense (Personnel and Readiness), DOD 
stated that the report is flawed in its assessment of DRRS, noting that 
DRRS is a net-centric application that provides broad and detailed 
visibility on readiness issues, and that achieving data sharing across 
the DOD enterprise was groundbreaking work fraught with barriers and 
obstacles, many of which have now been overcome. In addition, DOD 
stated that it was disappointed that the report did not address 
cultural impediments that it considers to be the root cause of many of 
the issues cited in the report and of many previous congressional 
concerns on readiness reporting. DOD further stated that the report 
instead focuses on past acquisition process and software development 
problems that it believes have now been remedied According to the 
department, this focus, coupled with inaccurate and misleading factual 
information included in the report, led us to develop an incomplete 
picture of the program. Notwithstanding these comments, DOD agreed with 
two of our recommendations and partially agreed with a third. However, 
it disagreed with the remaining five recommendations, and provided 
comments relative to each recommendation. DOD's comments are reprinted 
in their entirety in appendix III. 

In summary, we do not agree with DOD's overall characterization of our 
report or the positions it has taken in disagreeing with five of our 
recommendations, finding them to be inconsistent with existing guidance 
and recognized best practices on system acquisition management, 
unsupported by verifiable evidence, and in conflict with the facts 
detailed in our report. Further, we recognize that developing DRRS is a 
significant and challenging undertaking that involves cultural 
impediments. As a result, our report explicitly focuses on the kind of 
program management rigor and disciplines needed to address such 
impediments and successfully acquire complex systems, including 
effective requirements development and management and executive 
oversight. We also disagree that our report focuses on past issues and 
problems. Rather, it provides evidence that demonstrates a long- 
standing and current pattern of system acquisition and program 
oversight weaknesses that existed when we concluded our audit work and 
that DOD has not provided any evidence to demonstrate has been 
corrected. 

In addition, we would emphasize that we defined our objectives, scope, 
and methodology, and executed our audit work in accordance with 
generally accepted government auditing standards, which require us to 
subject our approach as well as the results of our audit work to proven 
quality assurance checks and evidence standards that require us to seek 
documentation rather than relying solely on testimonial evidence. While 
we support any departmental efforts, whether completed or ongoing, that 
would address the significant problems cited in our report, we note 
that DOD, in its comments, did not specifically cite what these efforts 
are or provide documentation to support that they have either been 
completed or are ongoing. Therefore, we stand by our findings and 
recommendations. Moreover, we are concerned that in light of the 
program's significant and long-standing management weaknesses, the 
department's decision not to pursue recommendations aimed at corrective 
actions for five of our eight recommendations will further increase 
risk to achieving program success, and is not in the best interests of 
the military readiness community or the U.S. taxpayer. Accordingly, we 
encourage the department to reconsider its position when it submits its 
written statement of the actions taken on our recommendations to the 
Senate Committee on Homeland Security and Governmental Affairs and the 
House Committee on Oversight and Government Reform , as well has the 
House and Senate Committees on Appropriations, as required under 31 
U.S.C. 720. 

DOD's specific comments on each recommendation, along with our 
responses to its comments follow. 

* The department did not agree with our recommendation for the Deputy 
Secretary of Defense, as the Chair of the Defense Business Systems 
Management Committee, to reconsider the committee's recent approval of 
DRRS planned investment for fiscal years 2009 and 2010, and to convene 
the Defense Business Systems Management Committee to review the 
program's past performance and the DIO's capability to manage and 
deliver DRRS going forward in deciding how best to proceed. In this 
regard, DOD stated that the Investment Review Board certification and 
Defense Business Systems Management Committee approval were granted in 
compliance with the established processes. It also added that oversight 
of the specific issues identified in this report are the responsibility 
of the DRRS Executive Committee, which it stated has and will continue 
to provide appropriate governance for this effort. It also stated that 
USD (P&R) will ensure that the program is compliant with all 
acquisition requirements prior to submission for further 
certifications. 

We do not question whether the Investment Review Board certification 
and Defense Business Systems Management Committee approval were 
provided in accordance with established processes, as this is not 
relevant to our recommendation. Rather, our point is that the 
Investment Review Board and Defense Business Systems Management 
Committee were provided, and thus based their respective decisions, on 
erroneous and incomplete information about DRRS progress, management 
weaknesses, and risks. Moreover, neither the Investment Review Board 
nor the Defense Business Systems Management Committee were informed 
about the findings in our report, even though we shared each of them 
with the DRRS program director and other DIO officials prior to both 
the Investment Review Board and the Defense Business Systems Management 
Committee deliberations. Therefore, while the Investment Review Board 
certification and the Defense Business Systems Management Committee 
approval were granted in accordance with established processes, they 
were not based on a full disclosure of facts. Moreover, while we 
support DOD's comment that it will ensure that the program is in 
compliance with all acquisition requirements prior to further 
certifications, nothing precludes the board or the committee from 
reconsidering their respective decisions in light of our report. With 
respect to DOD's comment that the DRRS Executive Committee has and will 
continue to provide appropriate governance for this effort, we do not 
disagree that the DRRS Executive Committee has an oversight role. 
However, the DRRS Executive Committee should not be solely responsible 
for oversight of the specific issues in our report. Both the Investment 
Review Board and the Defense Business Systems Management Committee 
provide additional layers of oversight pursuant to law[Footnote 36] and 
DOD policy.[Footnote 37] Accordingly, we stand by our recommendation as 
it appropriately seeks to have the Investment Review Board and Defense 
Business Systems Management Committee, in collaboration with the DRRS 
Executive Committee, act in a manner that is consistent with their 
respective roles as defined in law. In doing so, our intent is to 
promote accountability for DRRS progress and performance, and prompt 
action to address the many risks facing the program. 

* The department agreed with our recommendation for the Deputy 
Secretary of Defense, as the chair of the Defense Business Systems 
Management Committee, to have the Business Transformation Agency 
conduct a risk assessment of DRRS, and with the advice and 
recommendation of the DRRS Executive Committee, to use the results of 
this assessment and the findings in our report to decide how to 
redirect the program. In this regard, the department stated that this 
assessment will be complete by the middle of fiscal year 2010. 

* The department did not agree with our recommendation for ensuring 
that DRRS requirements are effectively developed and managed. In this 
regard, it stated that the program has an authoritative set of baseline 
requirements established with an effective governance process for 
overseeing the requirements management process, to include biweekly 
reviews as part of the DRRS configuration control process. 

We do not agree. At the time we concluded our work, DRRS requirements 
were not stable, as evidenced by the fact that an additional 530 
requirements had been identified that the DIO was still in the process 
of reviewing and had yet to reach a position on their inclusion, or 
process them through the DRRS change control governance process. 
Moreover, when we concluded our work, this change control process had 
yet to be approved by the DRRS governance structure. As we state in our 
report, the introduction of such a large number of requirements 
provided a compelling basis for concluding that requirements had yet to 
progress to the point that they could be considered sufficiently 
complete and correct to provide a stable baseline. Our recommendation 
also noted that the Secretary should take steps to ensure that the 
different levels of requirements be aligned with one another. DOD's 
comments did not address this aspect of our recommendation. 

* The department did not agree with our recommendation for ensuring 
that DRRS testing is effectively managed. In this regard, it stated 
that DRRS testing is already in place and performing effectively, and 
stated, among other things, that (1) the DIO goes through a rigorous 
testing regimen that includes documenting test plans with user test 
cases for each incremental release to include utilizing system 
integration, acceptance, interoperability, and operational testing; (2) 
user test cases and functionality are validated by designated testers 
independent of the developers prior to a deployment; and (3) for 
interoperability testing the DIO has a designated test director and the 
Joint Interoperability Test Command (JITC) is the designated 
interoperability and operational test activity. 

We do not agree. As our report concludes, DRRS testing has not been 
effectively managed because it has not followed a rigorous testing 
regimen that includes documented test plans, cases, and procedures. To 
support this conclusion, our report cites numerous examples of test 
planning and execution weaknesses, as well as the DIO's repeated 
inability to demonstrate through requisite documentation that the 
testing performed on DRRS has been adequate. Our report shows that test 
events for already acquired, as well as currently deployed and 
operating, DRRS releases and subreleases were not based on well-defined 
plans and DOD had not filled its testing director vacancy. Further, our 
report shows that test events were not fully executed in accordance 
with plans that did exist, or executed in a verifiable manner, or both. 
For example, although increments of DRRS functionality had been put 
into production, the program had no documentation (e.g., test 
procedures, test cases, test results) to show that the program office 
had performed system integration testing, system acceptance testing, or 
operational testing on any DRRS release or subrelease, even though the 
DIO's test strategy stated that such tests were to be performed before 
system capabilities became operational. Moreover, evidence showed that 
the results of all executed test events had not been captured and used 
to ensure that problems discovered were disclosed to decision makers, 
and ultimately corrected. With respect to DOD's comments that JITC is 
the designated lead for interoperability and operational testing, our 
report recognizes that JITC is to conduct both interoperability and 
operational testing before the system is deployed and put into 
production (i.e., used operationally). However, during the course of 
our audit, the DIO could not produce any evidence to show that 
interoperability and operational testing of all operating system 
increments had been conducted. 

* The department did not agree with our recommendation for ensuring 
that the DRRS integrated master schedule is reliable. In this regard, 
it stated that a process is already in place to ensure that the 
schedule is current, reliable, and meets all the criteria outlined in 
the recommendation. 

We do not agree. As our report states, an integrated master schedule 
for DRRS did not exist until November 2008, which was 2 months after we 
first requested one. Moreover, following our feedback to the DIO on 
limitations in this initial version, a revised integrated master 
schedule was developed in January 2009, which was also not reliable. 
Subsequently, a revised integrated master schedule was developed in 
April 2009. However, as we detail in our report, that version still 
contained significant weaknesses. For example, it did not establish a 
critical path for all key activities or include a schedule risk 
analysis, and was not being updated using logic and durations to 
determine the dates for all key activities. These practices are 
fundamental to producing a sufficiently reliable schedule baseline that 
can be used to measure progress and forecast slippages. In addition, 
the schedule introduced considerable concurrency across key activities 
and events for several modules, which introduces increased risk. 
Therefore, we stand by our recommendation. 

* The department partially agreed with our recommendation for ensuring 
that it has an effective human capital strategy. In this regard, it 
stated that actions are underway to add more full-time civilian support 
to the DIO, and that plans exist to convert some contractor to civilian 
billets during the 2010/2011 time frame. 

We support the department's actions and plans described in its comments 
to address the DIO human capital management limitations discussed in 
our report, but would note that they do not go far enough to 
systematically ensure that the program has the right people with the 
right skills to manage the program in both the near term and the long 
term. To accomplish this, the department needs to adopt the kind of 
strategic and proactive approach to DRRS workforce management that our 
report describes and our recommendation embodies. As our evaluations 
and research show, failure to do so increases the risk that the program 
office will not have the people it needs to effectively and efficiently 
manage DRRS. Therefore, we believe that the department needs to fully 
implement our recommendation. 

* The department did not agree with our recommendation to take steps to 
ensure that DRRS is developed and implemented in a manner that does not 
increase the reporting burden on units and addresses the timeliness, 
precision, and objectivity of metrics that are reported to Congress. In 
this regard, it stated that one of the primary tenets of DRRS has been 
to reduce reporting requirements on the war fighter. It also stated 
that DRRS is already using state-of-the-art technology to ensure that 
near-real-time data are available for the war fighters. Finally it 
stated that the DRRS governance structure that is currently in place 
ensures that DRRS development does not deviate from these core 
principles. 

While we recognize that a goal of DRRS is to reduce a reporting burden 
on the war fighter, we disagree with the department's position because 
the system has not yet achieved this goal. As our report states, while 
the DIO has developed the software for users to enter mission essential 
task data into DRRS, the DIO has been unsuccessful in attempts to 
develop a tool that would allow Air Force and Marine Corps users to 
enter readiness data to meet all of their readiness reporting 
requirements through DRRS. As a result, rather than reducing the burden 
on reporting units, DRRS actually increased the burden on Air Force and 
Marine Corps units because they were required to report readiness 
information in both DRRS and GSORTS. Without a viable tool for 
inputting data, DRRS is not fully integrated with GSORTS or with the 
service readiness reporting systems and it is not capable of replacing 
those systems since it does not capture the required data that are 
contained in those systems. In addition, the DRRS readiness data that 
are currently reported to Congress are not near-real-time data. 
Specifically, the periodicity for DRRS capability assessments is the 
same as the legacy GSORTS system's readiness reports--monthly or when 
an event occurs that changes a unit's overall readiness. Furthermore, 
our report shows that DRRS mission assessments are often subjective and 
imprecise because they are reported on a scale that includes only three 
ratings--"yes," "no," and "qualified yes," which can include any 
assessments that fall between the two extremes. Therefore, because 
additional actions are still needed to reduce reporting burdens and 
improve the timeliness, precision, and objectivity of the DRRS data 
that are reported to Congress, we stand by our recommendation. 

* The department agreed with our recommendation for ensuring that both 
the Human Resources Management Investment Review Board and the DRRS 
Executive Committee conduct frequent oversight activities of the DRRS 
program and report any significant issues to the Deputy Secretary of 
Defense. In this regard, the department stated that the USD (P&R) 
component acquisition executive is working with the program to ensure 
that it becomes fully compliant with all acquisition requirements. In 
addition, it stated that the acquisition executive will certify to the 
Human Resources Investment Review Board and the Deputy Chief Management 
Officer of compliance prior to submission of future certification 
requests. Further, it stated that the current DRRS governance process 
will provide sustained functional oversight of the program and that 
issues that arise in any of these areas will be elevated for review, as 
appropriate. We believe these are positive steps. 

We are sending copies of this report to the appropriate congressional 
committees; the Secretary of Defense; and the Director, Office of 
Management and Budget. The report will also be available at no charge 
on the GAO Web site at [hyperlink, http://www.gao.gov]. 

If you or your staffs have questions about this report, please contact 
us at pickups@gao.gov or hiter@gao.gov or at our respective phone 
numbers, (202) 512-9619 and (202) 512-3439. Contact points for our 
Offices of Congressional Relations and Public Affairs may be found on 
the last page of this report. GAO staff who made key contributions to 
this report are listed in appendix IV. 

Signed by: 

Sharon L. Pickup: 
Director, Defense Capabilities and Management: 

Signed by: 

Randolph C. Hite: 
Director, Information Technology Architecture and Systems Issues: 

[End of section] 

Appendix I: Objectives, Scope, and Methodology: 

Our objectives were to assess the extent to which (1) the Department of 
Defense (DOD) has effectively managed and overseen DRRS acquisition and 
deployment, and (2) features of the Defense Readiness Reporting System 
(DRRS) have been implemented and are consistent with legislative 
requirements and DOD guidance. 

We did not evaluate the department's overall ability to assess the 
readiness of its forces or the extent that data contained in any of its 
readiness reporting systems, including DRRS and the Global Status of 
Resources and Training System (GSORTS), reflect capabilities, 
deficiencies, vulnerabilities, or performance issues. Our review 
focused on acquisition and program management issues, such as 
requirements management, schedule, and human capital requirements; the 
current usage of DRRS; and the extent to which DRRS' features address 
legislative requirements and DOD guidance. 

To determine the extent to which the DRRS acquisition and deployment 
has been effectively managed and overseen, we focused on the following 
acquisition management areas: (1) requirements development and 
management, (2) test planning and execution, (3) DRRS schedule 
reliability, and (4) human capital planning. In doing so, we analyzed a 
range of program documentation, such as high-level and detailed-level 
requirements documentation, test plans and reports, the current DRRS 
schedule, and program management documentation and interviewed 
cognizant program and contractor officials. 

* To determine the extent to which the program had effectively 
implemented requirements development and management, we reviewed 
relevant program documentation, such as the concept of operations 
document, capability requirements document, software requirements 
document, requirements traceability matrix, configuration management 
plan, and the program management plan, as well as minutes of change 
control board meetings, and evaluated them against relevant 
guidance.[Footnote 38] Moreover, we reviewed briefing slides from 
meetings of DRRS oversight bodies in order to identify concerns about 
DRRS expressed by representatives from the DRRS community of users, as 
well as the efforts by the Joint Staff (at the direction of DRRS 
Executive Committee) to identify and address any gaps identified by 
users in the development of DRRS requirements. To determine the extent 
to which the program has maintained traceability backward to high-level 
business operation requirements and system requirements, and forward to 
system design specifications and test plans, we randomly selected 60 
program requirements and traced them both backward and forward. This 
sample was designed with a 5 percent tolerable error rate at the 95 
percent level of confidence so that, if we found zero problems in our 
sample, we could conclude statistically that the error rate was less 
than 5 percent. In addition, we interviewed program and development 
contractor officials to discuss the requirements development and 
management process. 

* To determine if the DRRS Implementation Office (DIO) is effectively 
managing DRRS testing, we reviewed relevant documentation, such as the 
DRRS Test and Evaluation Master Plans and test reports and compared 
them to DOD and other relevant guidance.[Footnote 39] Further, we 
reviewed developmental test plans and procedures for each release/ 
subrelease that to date has either been developed or fielded and 
compared them with best practices to determine whether test activities 
had been adequately documented. We also examined test results and 
reports for the already acquired, as well as currently deployed and 
operating, DRRS releases and subreleases and compared them against 
plans to determine whether they had been executed in accordance with 
plans. Moreover, we reviewed key test documentation, such as the 
Software Version Descriptions, and compared them against relevant 
guidance to determine whether defect data were being captured, 
analyzed, prioritized, and reported. We also interviewed program and 
contractor officials to gain clarity beyond what was included in the 
program documentation, including the Defense Information Systems 
Agency's Joint Interoperability Test Center in order to determine the 
results of their efforts to independently test DRRS interoperability. 
In addition, to determine the extent to which the program had 
effectively tested its system requirements, we observed the DIO's 
efforts to demonstrate the traceability of 60 program requirements to 
test cases and results. This sample was designed with a 5 percent 
tolerable error rate at the 95 percent level of confidence so that, if 
we found zero problems in our sample, we could conclude statistically 
that the error rate was less than 5 percent. 

* To determine the extent to which the program's schedule reflects key 
estimating practices that are fundamental to having a reliable 
schedule, we reviewed the DRRS integrated master schedules and schedule 
estimates and compared them with relevant guidance.[Footnote 40] We 
also used schedule analysis software tools to determine whether the 
latest schedule included key information, such as the activities 
critical to on-time completion of DRRS, a logical sequence of 
activities, and evidence that the schedule was periodically updated. We 
also reviewed the schedule to determine the time frames for completing 
key program activities and to determine any changes to key milestones. 
In addition, we shared the results of our findings with program and 
contractor officials and asked for clarifications. We then reviewed the 
revised schedule, prepared in response to the weaknesses we found, and 
compared it with relevant guidance. 

* To evaluate whether DOD is adequately providing for the DRRS 
program's human capital needs, we compared the program's efforts 
against relevant criteria and guidance, including our own framework for 
strategic human capital management.[Footnote 41] In doing so, we 
reviewed key program documentation, such as the program management plan 
and the DIO organizational structure to determine whether it reflected 
key acquisition functions and identified whether these functions were 
being performed by government or contractor officials. We interviewed 
key officials to discuss workforce analysis and human capital planning 
efforts. 

* To determine the level of oversight and governance available to the 
DRRS community of users, we attended relevant meetings, met with 
officials responsible for program certification, and reviewed relevant 
guidance and program documentation. Specifically, we attended Battle 
Staff meetings and analyzed briefing slides and meeting minutes from 
the DRRS Executive Committee, General and Flag Officer's Steering 
Committee, and Battle Staff meetings--the main DRRS governance bodies. 
In addition, we reviewed key DRRS certification and approval 
documentation provided by the Human Resources Management Investment 
Review Board, such as economic viability analyses and the business 
system certification dashboard and met with Investment Review Board 
officials to determine the basis for certifying and approving DRRS. 

To determine the extent to which the features of DRRS have been 
implemented and are consistent with legislative requirements and DOD 
guidance, we first examined the language of Section 117 of Title 10 of 
the United States Code, which directs the Secretary of Defense to 
establish a comprehensive readiness reporting system. We identified 
criteria for this system in DOD's directive formally establishing the 
system.[Footnote 42] We evaluated the system by conducting interviews--
see table 2 below for a list of these organizations--and receiving 
system demonstrations from members of the readiness community to 
determine how they used DRRS and how their usage compared with the 
criteria established for the system. We also conducted content and data 
analysis of system documents and briefing packages provided by the DIO 
and Joint Staff. In order to capture the broadest amount of data about 
the system we conducted a survey of readiness offices at all of the 
service headquarters, combatant commands, and the National Guard Bureau 
regarding how DRRS was currently being used and the types and amount of 
data available in the system.[Footnote 43] In addition, to track the 
development of DRRS capabilities, we attended Battle Staff meetings and 
analyzed documentation from meetings of all the DRRS governance bodies. 
We also searched for and extracted information from DRRS in order to 
support other GAO ongoing readiness reviews. While our usage of the 
system was not intended as a formal test of the system, our general 
observations concerning system functionality and the range of available 
data were consistent with the observations of most other users, which 
were noted in our survey. 

Table 2: Organizations Interviewed during Our Review: 

Office of the Under Secretary of Defense (Personnel & Readiness), 
Arlington, Virginia: 

U.S Army: 
* Office of the Deputy Chief of Staff (Operations), Arlington, Virginia.
* U.S. Army Forces Command, Fort McPherson, Georgia.
* U.S. Army Pacific, Fort Shafter, Hawaii.
* Installation Management Command, Pacific Region Office, Fort Shafter, 
Hawaii. 

U.S. Navy: 
* Headquarters Navy, Integrated Fleet Readiness, Arlington, Virginia.
* Fleet Forces Command, Norfolk, Virginia.
* U.S. Pacific Fleet, Pearl Harbor, Hawaii. 

U.S. Air Force: 
* Headquarters, U.S. Air Force (Readiness), Arlington, Virginia.
* Air Combat Command, Langley Air Force Base, Virginia.
* U.S. Pacific Air Forces, Readiness Branch, Hickam Air Force Base, 
Hawaii.
* 13th Air Force, Hickam Air Force Base, Hawaii. 

U.S. Marine Corps: 
* Headquarters Marine Corps (Readiness), Arlington, Virginia.
* Marine Forces Command, Norfolk, Virginia.
* Marine Forces Pacific, Camp Smith, Hawaii.
* Combat Logistics Battalion 3, Kaneohe Bay, Hawaii. 

U.S. Central Command, MacDill Air Force Base, Florida. 

U.S. Joint Forces Command, Norfolk, Virginia. 

U.S. Northern Command, Colorado Springs, Colorado. 

U.S. Pacific Command, Camp Smith, Hawaii. 

U.S. Special Operations Command, MacDill Air Force Base, Florida. 

Source: GAO. 

[End of table] 

We conducted our work from April 2008 through August 2009 in accordance 
with generally accepted government auditing standards. Those standards 
require that we plan and perform the audit to obtain sufficient, 
appropriate evidence to provide a reasonable basis for our findings and 
conclusions based on our audit objectives. We believe that the evidence 
obtained provides a reasonable basis for our findings and conclusions 
based on our audit objectives. 

[End of section] 

Appendix II: Detailed Analysis of DRRS Acquisition and Deployment 
Management and Oversight: 

Our research and evaluations of information technology programs have 
shown that the ability to deliver promised system capabilities and 
benefits on time and within budget largely depends on the extent to 
which key program management disciplines are employed by an adequately 
staffed program management office. Among other things, these 
disciplines include a number of practices associated with effectively 
developing and managing system requirements, adequately testing system 
capabilities, and reliably scheduling the work to be performed. They 
also include proactively managing the program office's human capital 
needs, and promoting program office accountability through effective 
program oversight. Department of Defense (DOD) acquisition policies and 
guidance, along with other relevant guidance, recognize the importance 
of these management and oversight disciplines.[Footnote 44] As we have 
previously reported, not employing these and other program management 
disciplines increases the risk that system acquisitions will not 
perform as intended and require expensive and time-consuming rework. 
Defense Readiness Reporting System (DRRS) acquisition and deployment 
has for years not been effectively managed in accordance with these key 
program management disciplines that are recognized in DOD and other 
relevant guidance, and are fundamental to delivering a system that 
performs as intended on time and within budget. 

DRRS Requirements Have Not Been Effectively Developed and Managed: 

Well-defined and well-managed requirements are a cornerstone of 
effective system development and acquisition. According to recognized 
guidance, documenting and implementing a disciplined process for 
developing and managing requirements can help to reduce the risks of 
producing a system that is not adequately tested, does not meet user 
needs, and does not perform as intended.[Footnote 45] Effective 
requirements development and management includes, among other things, 
(1) effectively eliciting user needs early and continuously in the 
system life-cycle process, (2) establishing a stable baseline set of 
requirements and placing this baseline under configuration management, 
(3) ensuring that system requirements are traceable backward to higher 
level business or operational requirements (e.g., concept of 
operations) and forward to system design documents (e.g., software 
requirements specification) and test plans, and (4) controlling changes 
to baseline requirements. 

DRRS requirements have not been effectively developed and managed. 
Specifically, (1) key users have only recently become engaged in 
developing requirements, (2) requirements have been experiencing 
considerable change and are not yet stable, (3) different levels of 
requirements and related test cases have not been aligned with one 
another, and (4) changes to requirements have not been effectively 
controlled. As a result, efforts to develop and deliver initial DRRS 
capabilities have taken longer than envisioned and these capabilities 
have not lived up to the readiness communities' expectations. These 
failures increase the risk of future DRRS capabilities not meeting 
expectations and ensure that expensive and time-consuming system rework 
will be necessary. 

Recent Actions Have Been Taken to Address Limited User Involvement in 
Developing and Managing Requirements: 

One of the leading practices associated with effective requirements 
development is engaging system users early and continuously in the 
process of defining requirements. As we have previously reported, 
assessing user needs early in the process increases the probability of 
success in defining, designing, and delivering a system that meets user 
needs and performs as intended.[Footnote 46] 

To the DRRS Implementation Office's (DIO) credit, the October 2008 DRRS 
Risk Management Plan recognizes this by stating that the success of 
DRRS depends on participation and support from the broad readiness 
community, which includes combatant commands, Joint Staff, and the 
military services. However, until recently, key users were not 
effectively engaged in DRRS requirements development and management, 
although reasons vary why they have not. Specifically, DIO officials 
told us that beginning in 2002, they reached out to all user groups-- 
combatant commands, Joint Staff, and the military services--in defining 
requirements. For example, they cited a July 2002 memorandum issued by 
the Office of the Under Secretary of Defense for Personnel and 
Readiness (USD P&R) that encouraged the Director of the Joint Chiefs of 
Staff, Deputy Commanders of the Combat Commands, Service Operations 
Deputies, and Directors of Defense Agencies to actively support the 
DRRS effort by ensuring that their organizations are represented at 
Battle Staff meetings.[Footnote 47] However, these officials told us 
that the military services and Joint Staff chose not to participate. 

In contrast, officials from these user groups told us their involvement 
had been limited by what they characterized as difficulties in 
submitting requirements through the DRRS governance boards that were in 
place at that time. For example, an official from the Joint Forces 
Command said that the Forces Battle Staff governance board did not meet 
for about a year between 2005 and 2006. Further, the official said that 
the meetings that were held did not offer users the opportunity to 
discuss their concerns or influence the requirements process. 
Similarly, an official from the Marine Corps cited a lack of clear and 
transparent communication from the DIO as a significant impediment. 

Notwithstanding this lack of stakeholder involvement in setting 
requirements, the Office of USD (P&R) developed and issued a DRRS 
concept of operations in 2004, which DIO officials told us was based on 
input from the combatant commands, relevant departmental directives, 
[Footnote 48] and DRRS governance boards (e.g., Battle Staff). In our 
view, this document provided a high-level overview of proposed DRRS 
capabilities from which more detailed requirements could be derived. 
However, the concept of operations was not approved by all key players 
in the readiness community. Specifically, DIO officials stated that the 
document had not been approved by the military services and the Joint 
Staff. According to these officials, the reason for not seeking all 
stakeholders' approval, and the decision to begin developing more 
detailed requirements in the absence of an approved concept of 
operations, was that the 2002 DRRS DOD directive provided a sufficient 
basis to begin developing and deploying what they anticipated being the 
initial versions of DRRS.[Footnote 49] 

In 2008, after 6 years of effort to define DRRS requirements and 
develop and deploy system capabilities, the Joint Staff, at the 
direction of the DRRS Executive Committee, conducted an analysis of 
DRRS capabilities--referred to as the "capabilities gap analysis." To 
the Joint Staff's credit, this analysis has appropriately focused on 
soliciting comments from the entire readiness community and on 
identifying any gaps between the DRRS requirements and the needs of 
this community. As will be discussed in the next section, this analysis 
resulted in 530 additional user requirements. 

The extended period of limited involvement by key DRRS users in 
defining a concept of operations and related capabilities and 
requirements has impeded efforts to develop a clear understanding of 
DRRS expectations, constraints, and limitations, which, in turn, has 
contributed to significant delays in providing the readiness community 
with needed system support. While the recent Joint Staff action to 
engage the entire DRRS user community is a positive step towards 
overcoming this long-standing problem, it remains to be seen whether 
this engagement will produce agreement and commitment across the entire 
readiness user community around DRRS requirements. 

DRRS Requirements Are Not Stable: 

As previously noted, establishing an authoritative set of baseline 
requirements prior to system design and development is necessary to 
design, develop, and deliver a system that performs as intended and 
meets users' operational needs.[Footnote 50] In general, a baselined 
set of requirements are those that are defined to the point that 
extensive changes are not expected, placed under configuration 
management, and formally controlled.[Footnote 51] 

DRRS requirements are currently in a state of flux. Specifically, the 
fact that 530 new user requirements have recently been identified means 
that the suite of requirements documentation associated with the system 
will need to be changed and thus are not stable. To illustrate, program 
officials told us that, as of late February 2009, these 530 new 
requirements had not been fully evaluated by the DIO and DRRS 
governance boards and thus not yet approved. As a result, their impact 
on the program is not clear. 

Compounding this instability in the DRRS requirements is the fact that 
additional changes are envisioned. According to program officials, the 
changes resulting from the gap analysis and reflected in the latest 
version of the DRRS concept of operations, which was approved by the 
DRRS Executive Committee in January 2009, have yet to be reflected in 
other requirements documents, such as the detailed system requirements. 

Although defining and developing requirements is inherently an 
iterative process, having a baseline set of requirements that are 
stable is a prerequisite to effective and efficient development of an 
operationally capable and suitable system. Without them, the DIO will 
not be able to deliver a system that meets user needs on time, and it 
is unlikely that future development and deployment efforts will produce 
better results. 

Alignment among Requirements and Related System Design and Testing 
Artifacts Has Not Been Demonstrated: 

One of the leading practices associated with developing and managing 
requirements is maintaining bidirectional traceability from high-level 
operational requirements (e.g., concept of operations and functional 
requirements) through detailed lower-level requirements and design 
documents (e.g., software requirements specification) to test cases. 
Such traceability is often accomplished through the use of a 
requirements traceability matrix, which serves as a crosswalk between 
different levels of related requirements, design, and testing 
documentation. The DRRS program management plan recognizes the 
importance of traceability, stating that requirements are to be 
documented and linked to acceptance tests, scripts, and criteria. 

Despite the importance of traceability, DIO officials could not 
demonstrate that requirements and related system design and testing 
artifacts are properly aligned. Specifically, we attempted on three 
separate occasions to verify the traceability of system requirements 
backward to higher-level requirements and forward to lower-level 
software specifications and test cases, and each time we found that 
traceability did not exist. Each attempt is discussed here: 

* In November 2008, our analysis of the requirements traceability 
matrix and the software requirements specification showed significant 
inconsistencies. For example, the traceability matrix did not include 
29 requirements that were included in the software requirements 
specification. As a result, we did not have an authoritative set of 
requirements to use to generate a random sample of requirements to 
trace. Program officials attributed the inconsistencies to delays in 
updating all the documents to reflect the aforementioned capability gap 
analysis. They also stated that these documents would be updated by 
December 2008. 

* In December 2008, we used an updated requirements traceability matrix 
to generate a randomized sample of 60 software requirements 
specifications and observed a DIO demonstration of the traceability of 
this sample. However, DIO officials were unable to demonstrate for us 
that these specifications could be traced backward to higher-level 
requirements and forward to test cases. Specifically, attempts to trace 
the first 21 requirements forward to test cases failed, and DIO 
officials stated that they could not trace the 60 requirements backward 
because the associated requirements documents were still being updated. 
According to the officials, 11 of the 21 could not be traced forward 
because these were implemented prior to 2006 and the related test 
information was not maintained by the program office but rather was at 
the development contractor's site. They added that the remaining 10 
either lacked test case information or test results. 

* In February 2009, we used an updated DIO-provided requirements 
traceability matrix, a capabilities requirement document, and software 
requirements specification to generate another randomized sample of 60 
detailed specifications. We then observed the development contractor's 
demonstration of traceability using the contractor's requirements 
management tool. Because of time constraints, this demonstration 
focused on 46 of the 60 requirements, and it showed that adequate 
traceability still did not exist. Specifically, 38 of the 46 could not 
be traced backward to higher-level requirements or forward to test 
cases. This means that about 83 percent of the DRRS specifications (95 
percent degree of confidence of being between 72 and 91 percent) were 
not traceable. Of the 38, 14 did not trace because of incomplete 
traceability documentation; 5 due to inconsistent traceability 
documentation; 3 due to requirements not being resident in the tracking 
tool; and 16 due to no actual development work being started. 

In addition, none of the 46 requirements were traceable to the January 
2009 concept of operations. According to contractor officials, this is 
because the newly developed capability requirements document is 
considered to be a superset of the concept of operations, and thus 
traceability to this new document is their focus. However, they were 
unable to demonstrate traceability to the requirements in either the 
capability requirements document or the concept of operations. Further, 
we also found numerous inconsistencies among the capabilities 
requirements document, software requirements specification, and the 
requirements traceability matrix. For example, 15 capabilities 
requirements listed on the traceability matrix were not listed in the 
capabilities requirements document, but were listed in the updated 
software requirements specification, dated February 2009. Further, one 
requirement listed in the traceability matrix was not listed in either 
of these documents. One possible reason for these inconsistencies is 
that the traceability matrix was prepared manually, rather than being 
automatically generated from the tool, which would increase the 
probability of these and other discrepancies caused by human error. 
Another reason cited by program officials is that test results that 
occurred prior to October 2006 had yet to be fully recorded in the 
contractor's tracking tool. 

DIO and contractor officials attributed the absence of adequate 
requirements traceability to the ongoing instability in requirements 
and magnitude of the effort to update the chain of preexisting and new 
requirements documentation. They added that they expect traceability to 
improve as requirements become more stable and the documentation is 
updated. Regardless, the DIO has and continues to invest in the 
development of DRRS in the absence of requirements traceability. 
Without traceable requirements, the DIO does not have a sufficient 
basis for knowing that the scope of the design, development, and 
testing efforts will produce a system solution on time and on budget 
and that will meet users' operational needs and perform as intended. As 
a result, the risk is significant that expensive and time-consuming 
system rework will be required. 

Changes to Requirements Are Not Being Effectively Controlled: 

Adopting a disciplined process for reviewing and accepting changes to 
an approved and authoritative baseline set of requirements in light of 
the estimated costs, benefits, and risk of each proposed change is a 
recognized best practice. Elements of a disciplined process include: 
(1) formally documenting a requirements change process; (2) adopting 
objective criteria for considering proposed changes, such as estimated 
cost or schedule impact; and (3) rigorously adhering to the documented 
change control process. 

Since the inception of the program in 2002, DRRS has been developed and 
managed without a formally documented and approved process for managing 
changes to system requirements. Further, while requirements management 
and change control plans were developed in 2006 by the DRRS software 
development contractor, according to DIO officials, the plans were not 
adequate. For example, the plans did not detail how DRRS user 
requirements were collected or how objective factors, such as cost, 
impacted development decisions. 

To address these problems, the Joint Staff developed what it referred 
to as a conceptual requirements change control process in February 
2008, which was to serve as a basis for the DIO to develop more 
detailed plans that could be implemented. Eleven months later, in 
January 2009, the DIO drafted more detailed plans--a DRRS requirements 
management plan and a DRRS requirements configuration management plan, 
the latter of which the DIO updated in March 2009. Specifically, the 
draft plans call for new DRRS requirements to be collected using an 
online tool and reviewed by the DIO to determine whether the 
requirement constitutes a major change to DRRS. Once approved, the DIO 
and the contractor are to provide the Battle Staff with a formatted 
report specifying the anticipated benefit of each new requirement and 
an initial analysis of the cost and performance impact. The Battle 
Staff then is to prioritize the requirement based on the DIO's impact 
analysis. If the issue cannot be resolved by the Battle Staff, it is to 
be elevated to the senior oversight bodies (i.e., the General Officer's 
Steering Committee and the DRRS Executive Committee). After a 
requirement has been approved, the software developer may prepare a 
more detailed "customer acceptance document" that analyzes the 
potential cost, schedule, and quality impact to DRRS objectives, which 
is then to be reviewed by the DIO at subsequent Change Control Board 
meetings. 

However, according to the user community and the DIO Director, the 
revised plans have not been submitted for review and approval to the 
DRRS community. Specifically, they stated that only a proposed process 
flow diagram was briefed at the Battle Staff, and according to them, 
the change control process was still being evaluated. Moreover, the DIO 
has yet to implement key aspects of its draft plans. For example, the 
DRRS Chief Engineer stated that until recently, the DIO had continued 
to accept changes to DRRS requirements that were submitted outside of 
the designated online tool. In addition, the reports that the Battle 
Staff are to use in making their requirement change determination do 
not include the anticipated benefit and estimated cost or schedule 
impact of new requirements. Rather, these reports only include the 
estimated number of hours necessary to complete work on a proposed 
requirement. Moreover, contractor officials and users from the Special 
Operations Command told us that cost or schedule impacts have rarely 
been discussed at the Battle Staff or Change Control Board meetings. 
Our analysis of minutes from change control meetings confirmed this. 
Furthermore, the DRRS Chief Engineer stated that the customer 
acceptance documents have only recently been used. 

Until the DIO effectively controls requirements changes, it increases 
the risk of needed DRRS capabilities taking longer and costing more to 
deliver than necessary. 

DRRS Testing Has Not Been Adequately Performed and Key Test Management 
Structures and Controls Have Not Been Established: 

Effective system testing is essential to successfully developing and 
deploying systems like DRRS. According to DOD and other relevant 
guidance, system testing should be progressive, meaning that it should 
consist of a series of test events that first focus on the performance 
of individual system components, then on the performance of integrated 
system components, followed by system-level tests that focus on whether 
the system (or major system increments) are acceptable, interoperable 
with related systems, and operationally suitable to users. [Footnote 
52] For this series of related test events to be conducted effectively, 
(1) each test event needs to be executed in accordance with well- 
defined test plans, (2) the results of each test event need to be 
captured and used to ensure that problems discovered are disclosed and 
corrected, and (3) all test events need to be governed by a well- 
defined test management structure. 

Despite acquiring and partially deploying a subset of DRRS increments, 
the DIO cannot demonstrate that it has adequately tested any of these 
system increments, referred to as system releases and subreleases. 
Specifically, (1) the test events for already acquired, as well as 
currently deployed and operating, DRRS releases and subreleases were 
not based on well-defined plans, and test events have not been fully 
executed in accordance with plans or executed in a verifiable manner, 
or both; (2) the results of executed test events have not been captured 
and used to ensure that problems discovered were disclosed to decision 
makers and ultimately corrected; and (3) the DIO has not established an 
effective test management structure to include, for example, a clear 
assignment of test management roles and responsibilities, or a reliable 
schedule of planned test events. Compounding this absence of test 
management structures and controls is the fact that the DIO has yet to 
define how the series of system releases and subreleases relate to its 
recent restructuring of DRRS increments into a series of 10 modules. 
Collectively, this means that it is unlikely that already developed and 
deployed DRRS capabilities can perform as intended and meet user 
operational needs. Equally doubtful are the chances that the DIO can 
adequately ensure that yet-to-be developed DRRS capabilities will meet 
expectations. 

DIO Has Not Adequately Planned and Executed Key Test Events: 

Key tests required for already developed and partially fielded DRRS 
increments either did not have well-defined test plans, or these tests 
have yet to be conducted. According to program documentation, system 
releases and subreleases have been subjected to what are described as 
30-day test cycles, during which: (1) a Software Test Plan is updated 
if applicable, (2) test procedures are developed and incorporated in 
the Software Test Description, (3) a series of developmental tests on 
each release/subrelease is performed, (4) weekly meetings are held to 
review software defects identified during testing, (5) final test 
results are summarized within the Software Test Report and Software 
Version Description, and (6) the release/subrelease is made available 
to users. 

However, the program office has yet to provide us with the 
developmental test plans and procedures for each release/subrelease 
that to-date has either been developed or fielded. Instead, it provided 
us with a Software Test Plan and two Software Test Descriptions that it 
said applied to two subreleases within release 4.0. However, available 
information indicates that DRRS subreleases total at least 63, which 
means that we have yet to receive the test plans and procedures for 61. 
Further, the test plan that we were provided is generic in nature, 
meaning that it was not customized to apply specifically to the two 
subreleases within Release 4.0. Moreover, the plan and procedures lack 
important elements specified in industry guidance.[Footnote 53] For 
example, the test plan does not include a schedule of activities to be 
performed or defined roles and responsibilities for performing them. 
Also, the test plan does not consistently include test entrance and 
exit criteria, a test defect management process, and metrics for 
measuring progress. 

Moreover, the DIO has yet to demonstrate that it has performed other 
key developmental and operational test events that are required before 
the software is fielded for operational use. According to DIO 
officials, developmental testing concludes only after system 
integration testing and system acceptance testing, respectively, are 
performed. Further, following developmental testing, the Joint 
Interoperability Test Command (JITC), which is a DOD independent test 
organization, is to conduct both interoperability and operational 
testing before the system is deployed and put into production (i.e., 
used operationally). Although increments of DRRS functionality have 
been put into production, the DIO has not performed system integration 
testing, system acceptance testing, or operational testing on any DRRS 
release or subrelease. Further, JITC documentation shows that while an 
interoperability test of an increment of DRRS functionality known as 
ESORTS was conducted, this test did not result in an interoperability 
certification.[Footnote 54] According to JITC and Joint Staff 
officials, this was because the DIO did not address JITC's identified 
limitations to the program's Information Support Plan, which identifies 
essential information-exchange sharing strategies between 
interdependent systems that are needed for interoperability 
certification. Without interoperability certification, the ability of 
the DRRS to exchange accurate and timely readiness data with other 
critical systems, such as the Joint Operation Planning and Execution 
System, cannot be ensured. 

Similarly, while DIO officials stated that acceptance testing has 
occurred for one increment of DRRS functionality known as SORTSREP, the 
DIO does not have either a finalized acceptance test plan or documented 
test results. [Footnote 55] Furthermore, the integrated master schedule 
(last updated in April 2009) shows that acceptance testing is not to 
occur until the July/August 2009 time frame, which is about 15 months 
later than originally envisioned. Moreover, this delay in acceptance 
testing has in turn delayed interoperability and operational testing by 
16 months (May/June 2008 to September/November 2009), according to the 
latest schedule. Program officials attributed the delays to Marine 
Corps and Air Force concerns about the quality of SORTSREP. 

Until the DIO has effectively planned and executed the series of tests 
needed to demonstrate the readiness of DRRS increments to operate in a 
production environment, the risk of fielded system increments not 
performing as intended and requiring expensive rework to correct will 
be increased, and DOD will continue to experience delays in delivering 
mission-critical system capabilities to its readiness community. 

DIO Has Not Adequately Documented and Reported Test Results: 

Available results of tests performed on already developed and at least 
partially deployed DRRS releases/subreleases show that the test results 
have not been effectively captured and analyzed, and have not been 
fully reported. Moreover, test results for other releases/subreleases 
do not exist, thus minimizing the value of any testing that has been 
performed. According to relevant guidance, effective system testing 
includes recording the results of executing each test procedure and 
test case as well as capturing, analyzing, correcting, and disclosing 
to decision makers problems found during testing (test defects). 
[Footnote 56] It also includes ensuring that test entry and exit 
criteria are met before beginning and ending, respectively, a given 
test event. 

The DIO does not have test results of all developed and tested DRRS 
releases and subreleases. Specifically, program officials provided us 
with the Software Test Reports and Software Version Descriptions that, 
based on program documentation, represent the full set of test results 
for three subreleases and a partial set of test results for 40 
subreleases within releases 1.0, 3.0, and 4.0. However, as noted 
earlier, DRRS subreleases total at least 63, which means that test 
reports and results for at least 20 subreleases do not exist. Moreover, 
the test reports and version descriptions that we received do not 
consistently include key elements provided for in industry guidance, 
such as a documented assessment of system capabilities and limitations, 
entrance/exit criteria status, an assessment as to whether the 
applicable requirements/thresholds were met, and unresolved defects and 
applicable resolution plans. [Footnote 57] This information is 
important because it assists in determining and disclosing to decision 
makers current system performance and efforts needed to resolve known 
problems, and provides program officials with a needed basis for 
ensuring that a system increment is ready to move forward and be used. 
Without this information, the quality and readiness of a system is not 
clear. 

Furthermore, the DIO does not have detailed test defect documentation 
associated with all executed DRRS test events. According to relevant 
guidance, defect documentation should, among other things, identify 
each issue discovered, assign each a priority/criticality level, and 
provide for each a strategy for resolution or mitigation. [Footnote 58] 
In lieu of detailed test defect documentation, program officials 
referred us to the above-mentioned Software Version Descriptions, and 
stated that additional information is available in an automated tool, 
known as the ISI BugTracker, that it uses to capture, among other 
things, defect data. However, these documents do not include the above- 
cited defect information, and defect data for each test event do not 
exist in the ISI BugTracker. 

Compounding the absence and limitations of test results are weaknesses 
in the program office's process for collecting such results during test 
execution. According to relevant guidance, test results are to be 
collected and stored according to defined procedures and placed under 
appropriate levels of control. [Footnote 59] Furthermore, these test 
results are to be reviewed against the source data to ensure that they 
are complete, accurate, and current. For DRRS, the program office is 
following a partially undocumented, manual process for collecting and 
storing test results and defects that involves a database and layers of 
documentation. As explained by program officials and documentation, the 
DIO initially documents defects and completed test case results 
manually on paper forms, and once the defect is approved by the test 
lead, it is input into a database. However, it does not have written 
procedures governing the entire process, and thus key controls, such as 
assigned levels of authority for database read/write access, are not 
clearly defined. Moreover, once the test results and defects are input 
into the database, traceability back to the original test data for data 
integrity checks cannot be established because the program office does 
not retain these original data sets. Program officials acknowledged 
these internal control weaknesses and stated that they intend to adopt 
a new test management tool that will allow them to capture in a single 
database test cases, test results, and test defects. 

Furthermore, the DIO's process for analyzing and resolving test defects 
has limitations. According to relevant guidance and the draft SORTSREP 
Test and Evaluation Master Plan (TEMP), defects should be analyzed and 
prioritized.[Footnote 60] However, the program office has not 
established a standard definition for defect priority levels identified 
during testing. For example, the various release/subrelease test 
reports (dated through January 2009) prioritize defects on a scale of 1-
3, where a priority 2 means critical but with a viable workaround. In 
contrast, the SORTSREP TEMP (dated January 2009) prioritizes defects on 
a scale of 1-5, where a priority 2 means an error that adversely 
affects the accomplishment of an operational or mission essential 
function in accordance with official requirements so as to degrade 
performance and for which no alternative work around solution exists. 
By not using standard priority definitions for categorizing defects, 
the program office cannot ensure that it has an accurate and useful 
understanding of the scope and magnitude of the problems it is facing 
at any given time, and it will not know if it is addressing the highest 
priority issues first. 

In addition, the DIO has not ensured that critical defects are 
corrected prior to concluding a given test event. According to relevant 
guidance and the draft SORTSREP TEMP, all critical and high defects 
should be resolved prior to the conclusion of a test event, and all 
test results should be reviewed for validity and completeness.[Footnote 
61] However, the DRRS release/subrelease test reports show that the DIO 
concluded five test events even though each had at least 11 open 
critical defects (priority 1 defects with no workaround). Moreover, 
these numbers of open critical defects are potentially higher because 
they do not include defects for which a solution was identified but the 
solution failed during regression testing and do not include defects 
that were dismissed because the program official was unable to recreate 
it. 

Until the DIO adequately documents and reports the test results, and 
ensures that severe problems discovered are corrected prior to 
concluding a given test event, the probability of incomplete test 
coverage, and insufficient and invalid test results, is increased, thus 
unnecessarily increasing the risk of DRRS not meeting mission needs or 
otherwise not performing as intended. 

DIO Has Not Established an Effective Test Management Structure: 

The DIO does not have an effective test management structure, to 
include a well-defined overall test management plan that clearly 
assigns test management roles and responsibilities, a designated test 
management lead and a supporting working group, and a reliable schedule 
of planned test events. According to relevant guidance, these aspects 
of test management are essential to adequately planning, executing, and 
reporting a program's series of test events. [Footnote 62] 

Although the program has been underway for 8 years, it did not have an 
overarching DRRS TEMP until very recently (February 2009), and this 
plan is still in draft and has yet to be approved. Further, this draft 
TEMP does not clearly define DRRS test management roles and 
responsibilities, such as those of the test manager, and it does not 
include a reliable schedule of test events that reflect the program's 
recent restructuring of its software releases/subreleases into 10 
modules. According to DIO officials, they recently decided not to 
approve this overarching TEMP. Instead, they said that they now intend 
to have individual TEMPs for each of the recently defined 10 modules, 
and to have supporting test plans for each module's respective 
developmental and operational test events. According to program 
documentation, three individual TEMPs are under development (i.e., 
SORTSREP tool and the Mission Readiness and Readiness Review modules). 
However, drafts of these TEMPs also do not clearly define test entrance 
and exit criteria, test funding requirements, an integrated test 
program schedule, and the respective test management roles and 
responsibilities. For example, while the draft SORTSREP TEMP identifies 
the roles and responsibilities of some players, such as the test 
manager, the personnel or organization that is to be responsible is not 
always identified. In addition, while the various players in the user 
community are identified (i.e., military services, combatant commands), 
their associated roles or responsibilities are not. 

Furthermore, the DIO has yet to designate a test management lead and 
establish an effective test management working group. According to 
relevant guidance, test management responsibility and authority should 
be assigned to an individual, and this individual should be supported 
by a working integrated product team that includes program office and 
operational testing representatives.[Footnote 63] Among other things, 
the working integrated product team is to develop an overall system 
test strategy. However, DIO officials told us that the test manager 
position has been vacant, and this position is now being temporarily 
filled by the program's chief engineer, who is a contractor. 
Furthermore, although DRRS system development began prior to 2004, a 
charter for a test and evaluation working integrated product team was 
not issued until February 2009. According to DIO officials, the delay 
in establishing the team has not had any impact because of 
corresponding delays in finalizing the program's overall test strategy. 
However, this statement is not consistent with the Defense Acquisition 
Guidebook, which states that two of the key products of the working 
integrated product team are the program's test strategy and TEMP. 
Further, JITC officials stated that the lack of a test manager and an 
active test and evaluation working integrated product team have reduced 
the effectiveness of DRRS testing activities. As a result, they stated 
that they have had to compensate by conducting individual meetings with 
the user community to discuss and receive documentation to support 
their operational and interoperability test planning efforts. 

Moreover, the DIO has yet to establish a reliable schedule of planned 
test events. For example, the schedule in the TEMPs is not consistent 
with either the integrated master schedule or the developmental test 
plans. Specifically, the draft SORTSREP TEMP (last updated in January 
2009) identifies SORTSREP developmental testing occurring through 
January 2009 and ending in early February 2009, while the integrated 
master schedule (last updated in April 2009) shows SORTSREP development 
testing as occurring in the July/August 2009 time frame. In addition, 
while program officials said that development testing for SORTSREP has 
occurred, the associated development test plans (e.g., system 
integration and system acceptance test plans) had no established dates 
for test execution, and are still in draft. As another example, a 
module referred to as "Mission Readiness" had no established dates for 
test execution in its TEMP, and while program documentation indicates 
that this module completed development testing in December 2008, the 
associated development test plans (e.g., system integration and system 
acceptance test plans) do not exist[Footnote 64].: 

In addition, the DIO has yet to define in its draft TEMPs how the 
development and testing to date of at least 63 subreleases relate to 
the development and testing of the recently established 10 system 
modules. According to Joint Staff and JITC officials, they do not know 
how the releases/subreleases relate to the modules, and attributed this 
to a lack of an approved description for each module that includes what 
functionality each is intended to provide. Furthermore, the high-level 
schedule in the TEMP does not describe what test events for the DRRS 
releases/subreleases that have already been developed and deployed 
relate to the development test efforts planned for the respective 
modules. These problems in linking release/subrelease test events to 
module test events limit the DIO and JITC in leveraging the testing 
already completed, which in turn will impact the program's ability to 
meet cost, schedule, and performance expectations. 

Collectively, the weaknesses in this program's test management 
structure increase the chances that the deployed system will not meet 
certification and operational requirements, and will not perform as 
intended. 

DRRS Has Not Been Guided by a Reliable Schedule: 

The success of any program depends in part on having a reliable 
schedule that defines, among other things, when work activities will 
occur, how long they will take, and how they are related to one 
another. As such, the schedule not only provides a road map for the 
systematic execution of a program, but also provides the means by which 
to gauge progress, identify and address potential problems, and promote 
accountability. From its inception in 2002 until November 2008, the DIO 
did not have an integrated master schedule. Moreover, the only 
milestone that we could identify for the program prior to November 2008 
was the date that DRRS was to achieve full operational capability, 
which was originally estimated to occur in fiscal year 2007, but later 
slipped to fiscal year 2008 and then fiscal year 2011, and is now 
fiscal year 2014--a 7-year delay. 

In addition, the DRRS integrated master schedule that was developed in 
November 2008, and was updated in January 2009 and again in April 2009 
to address limitations that we identified and shared with the program 
office, is still not reliable. Specifically, our research has 
identified nine practices associated with developing and maintaining a 
reliable schedule.[Footnote 65] These practices are (1) capturing all 
key activities, (2) sequencing all key activities, (3) assigning 
resources to all key activities, (4) integrating all key activities 
horizontally and vertically, (5) establishing the duration of all key 
activities, (6) establishing the critical path for all key activities, 
(7) identifying float between key activities, (8) conducting a schedule 
risk analysis, and (9) updating the schedule using logic and durations 
to determine the dates for all key activities.[Footnote 66] However, 
the program's latest integrated master schedule does not address three 
of the practices and only partially addresses the remaining six. For 
example, the schedule does not establish a critical path for all key 
activities, include a schedule risk analysis, and it is not being 
updated using logic and durations to determine the dates for all key 
activities. Further, it does not fully capture, sequence, and establish 
the duration of all key work activities; fully assign resources to all 
key work activities; fully integrate all of these activities 
horizontally and vertically; and fully identify the amount of float-- 
the time that a predecessor activity can slip before the delay affects 
successor activities--between these activities. These practices are 
fundamental to producing a sufficiently reliable schedule baseline that 
can be used to measure progress and forecast slippages. (See table 3 
for the results of our analyses relative to each of the nine 
practices.) 

Table 3: DRRS Satisfaction of Nine Schedule-Estimating Key Practices: 

Practice: Capturing all activities; 
Explanation: The schedule should reflect all key activities (e.g., 
steps, events, outcomes, etc.) as defined in the program's work 
breakdown structure, to include activities to be performed by both the 
government and its contractors; 
Practice met? Partially; 
GAO analysis: The schedule reflects many key activities as defined in 
the program's work breakdown structure. However, key activities are 
identified only as milestones rather than in terms of work to be 
performed and accomplished to achieve the milestones. Thus, the 
schedule does not account for all work to be performed. As a result, 
the reliability of the milestones is questionable. 

Practice: Sequencing all activities; 
Explanation: The schedule's activities need to be logically sequenced 
in the order that they are to be carried out. In particular, activities 
that must finish prior to the start of other activities (i.e., 
predecessor activities) as well as activities that cannot begin until 
other activities are completed (i.e., successor activities) should be 
identified. By doing so, interdependencies among activities that 
collectively lead to the accomplishment of key events or milestones can 
be established and used as a basis for guiding work and measuring 
progress; 
Practice met? Partially; 
GAO analysis: The schedule identifies some predecessor and successor 
activities, but not all. Specifically, out of 290 key activities, 139 
do not identify predecessor activities and 121 do not identify 
successor activities. DIO officials stated that this is because 
interdependencies are discussed and addressed at various meetings, and 
thus do not need to be in the schedule. However, recognition of such 
interdependencies in the schedule is necessary to logically link tasks 
and events and thereby calculate dates and predict changes in the 
future. Without proper linkages, tasks that slip early in the schedule 
do not transmit delays to tasks that should depend on them. By not 
logically linking all key activities, the schedule does not provide a 
sufficient basis for understanding the program as a whole, and 
confidence that the right dates or critical paths are represented is 
low. This means that the DIO cannot use its schedule to identify 
disconnects as well as hidden opportunities, and cannot otherwise 
promote efficiency and accuracy, and control the program by comparing 
actual to planned progress. 

Practice: Assigning resources to all activities; 
Explanation: The schedule should realistically reflect what resources 
(i.e., labor, material, and overhead) are needed to do the work, 
whether all required resources will be available when they are needed, 
and whether any funding or time constraints exist; 
Practice met?: Partially; 
GAO analysis: The schedule identifies 15 resources that are assigned to 
various activities. However, important information is not included, 
which hampers its usefulness. For example, one benefit of loading 
resource information is to ensure that resources in short supply will 
not be overscheduled in any time period. However, the schedule does not 
define the amount of each resource that is available, but instead 
assumes only one unit of each resource is available. As a result, 10 of 
the 15 types of resources (e.g., information assurance and test and 
evaluation) in the schedule are overscheduled and thus estimated 
completion dates are not realistic. Further, the schedule does not 
identify the costs of the resources. Knowing the cost of resources is 
important, because some resources, such as labor, can cost more if the 
program takes longer. 

Practice: Establishing the duration of all activities; Explanation: The 
schedule should realistically reflect how long each activity will take 
to execute. In determining the duration of each activity, the same 
rationale, historical data, and assumptions used for cost estimating 
should be used for schedule estimating. Further, these durations should 
be as short as possible, and they should have specific start and end 
dates. Excessively long periods needed to execute an activity (i.e., 
greater than 2-3 months) should prompt further decomposition of the 
activity so that shorter execution durations will result; 
Practice met? Partially; 
GAO analysis: The schedule establishes the duration of key activities 
and includes specific start and end dates. However, the activities are 
not always of short duration. For example, 23 activities have remaining 
durations that exceed 100 days (108 to 551 days). By having a schedule 
summarized at too high a level, the program runs the risk of masking 
critical work elements within the key activities associated with 
executing the program and fails to show the risk management approaches 
being used. Further, it risks masking the schedule's true critical 
path. 

Practice: Integrating schedule activities horizontally and vertically; 
Explanation: The schedule should be horizontally integrated, meaning 
that it should link the products and outcomes associated with already 
sequenced activities. These links are commonly referred to as "hand 
offs" and serve to verify that activities are arranged in the right 
order to achieve aggregated products or outcomes. The schedule should 
also be vertically integrated, meaning that traceability exists among 
varying levels of activities and supporting tasks and subtasks. Such 
mapping or alignment among levels enables different groups to work to 
the same master schedule; 
Practice met? Partially; 
GAO analysis: The schedule is not horizontally integrated, meaning that 
the activities are not arranged in the right order to achieve 
aggregated products or outcomes. This is because, as previously noted, 
many activities do not identify interdependencies (predecessor and 
successor activities). As a result, management lacks information on how 
a slippage in completing one activity will affect others. In contrast, 
the schedule is vertically integrated, meaning that for those key 
activities that are identified, traceability exists among varying 
levels of activities and that lower-level activities are within the 
constraints of higher-level activities. 

Practice: Establishing the critical path for all activities; 
Explanation: The critical path--the longest duration path through the 
sequenced list of key activities--should be identified using scheduling 
software. The establishment of a program's critical path is necessary 
for examining the effects of any activity slipping along this path. 
High-risk activities along or near the critical path should also be 
identified and associated time impacts of these risks should be 
reflected in the schedule; 
Practice met?: No; 
GAO analysis: The schedule does not identify a valid critical path 
because there is no logical link between the program's key activities. 
Instead, the activities are presented as being independent from one 
another. Without a critical path, management cannot know the activities 
critical to the on-time completion of DRRS, or the impact of any 
changes to activities on the path. 

Practice: Identifying float between activities; 
Explanation: The schedule should identify float time so that schedule 
flexibility can be determined. As a general rule, activities along the 
critical path typically have the least amount of float time; 
Practice met? Partially; 
GAO analysis: The schedule identifies the amount of float allocated to 
key activities. However, the amount of float associated with 56 of 
these activities is unusually large (100 - 461 days). Such large 
amounts of float time can be attributed to the fact that many of the 
activities do not identify linkages to other activities (predecessor or 
successor activities). In addition, the amount of float between 
activities is of questionable accuracy because activity dependencies 
(predecessor or successor activities) are often not identified. 
Instead, the start and completion dates for many activities are 
imposed, rather than based on logic, such as an activity not starting 
until its predecessor is completed. Further, it is unclear whether 
activities along the critical path have the least amount of float 
because, as previously noted, the schedule does not have a valid 
critical path. 

Practice: Conducting a schedule risk analysis; 
Explanation: A schedule risk analysis uses a good critical path method 
schedule and data about project schedule risks as well as Monte Carlo 
simulation[A] techniques to predict the level of confidence in meeting 
a program's completion date, the amount of time contingency needed for 
a level of confidence, and the identification of high-priority risks. 
This analysis focuses not only on critical path activities but also on 
other schedule paths that may become critical. Schedule reserve for 
contingencies should be calculated by performing a schedule risk 
analysis. As a general rule, the reserve should be held by the project 
or program manager and applied as needed to those activities that take 
longer than scheduled because of the identified risks. Reserves of time 
should not be apportioned in advance to any specific activity since the 
risks that will actually occur and the magnitude of their impact is not 
known in advance; 
Practice met? No; 
GAO analysis: The program office did not perform a schedule risk 
analysis. Without this analysis, the office cannot sufficiently 
understand the level of confidence in meeting the program's completion 
date and identifying reserves for contingencies. 

Practice: Updating the schedule using logic and durations to determine 
the dates; 
Explanation: The schedule should use logic and durations in order to 
reflect realistic start and completion dates for program activities. 
The schedule should be continually monitored to determine when 
forecasted completion dates differ from the planned dates, which can be 
used to determine whether schedule variances will affect downstream 
work. Maintaining the integrity of the schedule logic is not only 
necessary to reflect true status, but is also required before 
conducting a schedule risk analysis. The schedule should avoid logic 
overrides and artificial constraint dates that are chosen to create a 
certain result on paper. To ensure that the schedule is properly 
updated, individuals trained in critical path method scheduling should 
be responsible for updating the schedule's status; 
Practice met? No; 
GAO analysis: The realism of start and completion dates for some 
activities is questionable because, as previously noted, some 
activities do not identify logical relationships (i.e., 
interdependencies with other activities) or are of unusually lengthy 
duration. In addition, there is no "status date" information in the 
schedule (i.e., evidence the overall schedule is updated on a regular 
basis to capture actual start and completion dates). Without this 
information, the impact of deviations on downstream work cannot be 
fully understood and proactively addressed. In addition, some dates in 
the schedule were updated erroneously. For example, the schedule shows 
25 activities as having actual start dates in the future, including 6 
starting in 2010, even though the updated schedule was developed in 
April 2009. Likewise, there are 12 activities with actual 100 percent 
complete finish dates that range from May 2009 to July 2010. 

Source: GAO analysis of DOD data. 

[A] A Monte Carlo simulation allows the model's parameters to vary 
simultaneously according to their associated probability distribution. 
The result is a set of estimated probabilities of achieving alternative 
outcomes, given the uncertainty in the underlying parameters. 

[End of table] 

The limitations in the program's latest integrated master schedule, 
coupled with the program's 7-year slippage to date, make it likely that 
DRRS will incur further delays. Compounding these limitations is the 
considerable concurrency in the key activities and events in the 
schedule associated with the 10 recently identified system modules (see 
figure 2). For example, in 2010 alone, the program office plans to 
complete development testing on 2 modules and operational testing on 3 
modules, while also reaching initial operational capability on 3 
modules and full operational capability on 2 modules. By way of 
comparison, the program office had almost no concurrency across a 
considerably more modest set of activities and events over the last 5 
years, but nevertheless has fallen 7 years behind schedule. As 
previously reported, such significant overlap and concurrency among 
major program activities can create contention for limited resources 
and thus introduce considerable cost, schedule, and performance risks. 
[Footnote 67] 

Figure 2: Schedule for Developing and Implementing DRRS Capabilities: 

[Refer to PDF for image: illustrated table] 

Modules: Joint Mission Readiness: 
Begin planning, design, and development: Mid-2004; 
Complete developmental testing and evaluation: Late 2008; 
Complete operational testing and evaluation: Late 2009; 
Complete initial operational capability: Late 2004; 
Complete final operational capability: Early 2011. 

Modules: Unit Mission Readiness: 
Begin planning, design, and development: Mid-2004; 
Complete developmental testing and evaluation: Mid-2009; 
Complete operational testing and evaluation: Late 2009; 
Complete initial operational capability: Late 2009; 
Complete final operational capability: Mid-2010. 

Modules: Asset Visibility: 
Begin planning, design, and development: Early 2007; 
Complete developmental testing and evaluation: Mid-2010; 
Complete operational testing and evaluation: Late 2010; 
Complete initial operational capability: Late 2010; 
Complete final operational capability: Mid-2011. 

Modules: Business Intelligence: 
Begin planning, design, and development: Late 2008; 
Complete developmental testing and evaluation: Early 2011; 
Complete operational testing and evaluation: Mid-2011; 
Complete initial operational capability: Mid-2010; 
Complete final operational capability: Mid-2011. 

Modules: Installation Readiness: 
Begin planning, design, and development: Early 2009; 
Complete developmental testing and evaluation: Mid-2010; 
Complete operational testing and evaluation: Mid-2011; 
Complete initial operational capability: Mid-2009; 
Complete final operational capability: Mid-2011. 

Modules: Readiness Reviews: 
Begin planning, design, and development: Mid-2006; 
Complete developmental testing and evaluation: Late 2009; 
Complete operational testing and evaluation: Mid-2010; 
Complete initial operational capability: Mid-2007; 
Complete final operational capability: Mid-2010. 

Modules: Planning/Execution Support: 
Begin planning, design, and development: Early 2006; 
Complete developmental testing and evaluation: Late 2012; 
Complete operational testing and evaluation: Mid-2013; 
Complete initial operational capability: Mid-2013; 
Complete final operational capability: Late 2013. 

Modules: Historical Data: 
Begin planning, design, and development: Late 2008; 
Complete developmental testing and evaluation: Mid-2011; 
Complete operational testing and evaluation: Early 2012; 
Complete initial operational capability: Early 2012; 
Complete final operational capability: Late 2012. 

Modules: System Architecture: 
Begin planning, design, and development: Mid-2004; 
Complete final operational capability: Early 2012. 

Modules: End-User Usability: 
Begin planning, design, and development: Late 2004; 
Complete final operational capability: Mid-2009. 

Source: GAO analysis based on DOD data. 

[End of figure] 

In addition, the schedule remains unstable as evidenced by the degree 
of change it has experienced in just the past few months. For example, 
the January 2009 schedule had a full operational capability milestone 
of October 2011. By contrast, the April 2009 schedule has a December 
2013 milestone (see figure 3 below). Moreover, some milestones are now 
to occur much earlier than they were a few months ago. For example, the 
January 2009 schedule shows initial operational capability for 
"readiness reviews" to be June 2010. However, the April 2009 schedule 
shows that this milestone was attained in August 2007. Overall, 
multiple milestones for four modules were extended by at least 1 year, 
including two milestones that were extended by more than 2 years. Such 
change in the schedule in but a few months suggests a large degree of 
uncertainty, and illustrates the importance of ensuring that the 
schedule is developed in accordance with best practices. 

Figure 3: Changes in Estimated Dates for DRRS Capabilities: 

[Refer to PDF for image: illustration] 

Modules: Joint Mission Readiness; 
Complete final operational capability: Late 2009 to Early 2001. 

Modules: Unit Mission Readiness; 
Complete operational testing and evaluation: Early 2009 to Mid-2009. 

Modules: Asset Visibility; 
Complete developmental testing and evaluation: Late 2009 to Mid-2010; 
Complete operational testing and evaluation: Early 2010 to Late 2010; 
Complete initial operational capability: Early 2010 to Late 2010; 
Complete final operational capability: Late 2011 to Mid-2011. 

Modules: Business Intelligence; 
Complete developmental testing and evaluation: Mid-2010 to Early 2011; 
Complete operational testing and evaluation: Mid-2010 to Mid-2011; 
Complete initial operational capability: Mid-2010; 
Complete final operational capability: Mid-2011. 

Modules: Installation Readiness; 
Complete developmental testing and evaluation: Mid-2010; 
Complete operational testing and evaluation: Mid-201; 
Complete initial operational capability: Mid-2011 to Mid-2009; 
Complete final operational capability: Mid-2011 to Late 2011. 

Modules: Readiness Reviews; 
Complete operational testing and evaluation: Late 2009; 
Complete initial operational capability: Mid-2010 to Mid-2007; 
Complete final operational capability: Mid-2010. 

Modules: Planning/Execution Support; 
Complete developmental testing and evaluation: Early 2011 to Late 2012; 
Complete operational testing and evaluation: Mid-2011 to Mid-2013; 
Complete initial operational capability: Mid-2011 to Mid-2013; 
Complete final operational capability: Mid-2011 to Late 2013. 

Modules: Historical Data; 
Begin planning, design, and development: Mid-2009 to Early 2009; 
Complete developmental testing and evaluation: Late 2009 to Mid-2011; 
Complete operational testing and evaluation: Mid-2010 to Early 2012; 
Complete initial operational capability: Mid-2010 to Early 2012; 
Complete final operational capability: Mid-2010 to Late 2012. 

Modules: System Architecture; 
Complete final operational capability: Mid-2011 to Early 2012. 

Modules: End-User Usability; 
Complete final operational capability: Late 2008 to Mid-2009. 

Source: GAO analysis based on DOD data. 

[End of figure] 

DIO Lacks Adequate Staffing and an Effective Approach to Meeting its 
Human Capital Needs: 

As we have previously reported, effective human capital management is 
an essential ingredient to achieving successful program outcomes. 
[Footnote 68] Among other things, effective human capital management 
involves a number of actions to proactively understand and address any 
shortfalls in meeting a program's current and future workforce needs. 
These include an assessment of the core competencies and essential 
knowledge, skills, and abilities needed to perform key program 
management functions, an inventory of the program's existing workforce 
capabilities, and an analysis of the gap between the assessed needs and 
the existing capabilities. Moreover, they include explicitly defined 
strategies and actions for filling identified gaps, such as strategies 
for hiring new staff, training existing staff, and contracting for 
support services. 

The DIO is responsible for performing a number of fundamental DRRS 
program management functions. For example, it is responsible for 
acquisition planning, performance management, requirements development 
and management, test management, contractor tracking and oversight, 
quality management, and configuration management. To effectively 
perform such functions, program offices, such as the DIO, need to have 
not only well-defined policies and procedures and support tools for 
each of these functions, but also sufficient human capital to implement 
the processes and use the tools throughout the program's life cycle. 
Without sufficient human capital, it is unlikely that a program office 
can effectively perform its basic program management functions, which 
in turn increases the risk that the program will not deliver promised 
system capabilities and benefits on time and on budget. 

The DIO does not currently have adequate staff to fulfill its system 
acquisition and deployment responsibilities. In particular, the DIO is 
staffed with a single full-time government employee--the DIO Director. 
All other key program office functions are staffed by either contractor 
staff or staff temporarily detailed, on an as-needed basis, from other 
DOD organizations (referred to as "matrixed" staff). As a result, 
program management positions that the DIO itself has identified as 
critical to the program's success, such as configuration manager and 
security manager, are being staffed by contractors. Moreover, these 
contractor staff report to program management positions also staffed by 
contractors. Other key positions, such as those for performing 
acquisition management, requirements development and management, and 
performance management, have not even been established within the DIO. 
Furthermore, key positions, such as test manager, are vacant. These 
human capital limitations were acknowledged by the DRRS Executive 
Committee in November 2008. 

According to DIO and contractor officials, they recognize that 
additional program management staffing is needed. They also stated that 
while DRRS has been endorsed by USD (P&R) leadership and received 
funding support, past requests for additional staff have not been 
approved by USD (P&R) due to other competing demands for staffing. 
Further, DIO officials stated that the requests for staff were not 
based on a strategic gap analysis of its workforce needs and existing 
capabilities. Specifically, the program has not assessed its human 
capital needs and the gap between these needs and its onboard workforce 
capabilities. Until the program office adopts a strategic, proactive 
approach to managing its human capital needs, it is unlikely that it 
will have an adequate basis for obtaining the people it needs to 
effectively and efficiently manage DRRS. 

[End of section] 

Appendix III: Comments from the Department of Defense: 

Office Of The Under Secretary Of Defense: 
Personnel And Readiness: 
4000 Defense Pentagon: 
Washington, DC 20301-4000: 

September 3, 2009: 

Ms. Sharon Pickup: 
Director Defense Capabilities and Management: 
U.S. Government Accountability Office: 
Washington, D.C. 22548: 

Dear Ms. Pickup: 

This is the Department of Defense (DoD) response to the GAO draft 
report, GAO-09-518, "Military Readiness: DoD Needs to Strengthen 
Management and Oversight of the Defense Readiness Reporting System," 
dated July 1, 2009 (GAO Code 351217). 

Thank you for the opportunity to review this draft report. However, in 
our view, the report is flawed in its assessment of the Defense 
Readiness Reporting System (DRRS). DRRS is a true net-centric 
application that provides broad and detailed visibility on readiness 
issues. Achieving this data sharing across the DoD enterprise was 
groundbreaking work, fraught with barriers and obstacles - many of 
which have now been overcome. 

We were disappointed to learn that the report did not address the 
cultural impediments that are the root cause of many of the issues it 
cites, and the cause of so many previous congressional concerns on 
readiness reporting. Instead, the report focused on past issues and 
problems in the software development and acquisition processes that 
have now been remedied. We believe that this focus, coupled with 
inaccurate and misleading factual information included in the report, 
has lead the GAO to develop an incomplete picture of the program. 

Attached, please find the Department's detailed comments in response to 
the GAO's specific recommendation. 

Sincerely, 

Signed by: 

William J. Carr: 
Deputy Under Secretary of Defense (Military Personnel Policy): 
Performing the Duties of the Under Secretary of Defense (Personnel and 
Readiness): 

Attachment: As stated: 

[End of letter] 

GAO Draft Report - Dated July 1, 2009: 
GAO Code 351217/GAO-09-518: 

"Military Readiness: DoD Needs to Strengthen Management and Oversight 
of the Defense Readiness Reporting System" 

Department Of Defense Comments To The Recommendations: 

Recommendation 1: The GAO recommends that the Secretary of Defense 
direct the Deputy Secretary of Defense, as the Chair of the Defense 
Business Systems Management Committee, to reconsider the committee's 
recent approval of the Defense Readiness Reporting System (DRRS) 
planned investment for fiscal years 2009 and 2010, and convene the 
Defense Business Systems Management Committee to review the program's 
past performance and the DRRS Implementation Office's capability to 
manage and deliver DRRS going forward. 

DOD Response: Non-Concur. The IRB and DBSMC certifications were granted 
in compliance with the established certification process, which is 
designed to not be overly redundant with the acquisition process. 
Consistent with the Department's concept of tiered accountability, 
oversight of the specific issues identified by the GAO in this report 
are the responsibility of component responsible for the system. In this 
case, USD (P&R) chartered a DRRS Executive Committee (DEXCOM) to 
provide for governance of the effort in the readiness community, The 
DEXCOM has and will continue to provide appropriate governance for this 
effort. Additionally, the USD (P&R) will ensure that the program is 
compliant with all acquisition requirements prior to submission for 
further certifications. 

Recommendation 2: The GAO recommends that the Secretary of Defense 
direct the Deputy Secretary of Defense to have the Director of the 
Business Transformation Agency, using the appropriate team of 
functional and technical experts and the established risk assessment 
methodology, conduct a program risk assessment of DRRS, and to use the 
findings in GAO's report and the risk assessment to decide how to 
redirect the program's structure, approach, funding, management, and 
oversight. In this regard, GAO recommends that the Secretary of Defense 
direct the Deputy Secretary of Defense to solicit the advice and 
recommendations of the DRRS Executive Committee. 

DOD Response: Concur. The Department accepts the GAO's recommendation 
that a program risk assessment would be constructive. The Business 
Transformation Agency (BTA) will work with the DRRS Executive Committee 
(DEXCOM) to determine the scope of the assessment and will subsequently 
work with the DEXCOM to analyze its results. This assessment will be 
complete by the middle of Fiscal Year 2010. 

Recommendation 3: The GAO recommends that the Secretary of Defense 
through the appropriate chain of command, take steps to ensure that 
DRRS requirements are effectively developed and managed with 
appropriate input from the Services, Joint Staff, and Combatant 
Commanders, including: 
(a) establishing an authoritative set of baseline requirements prior to 
further system design and development; 

(b) ensuring that the different levels of requirements and their 
associated design specifications and test cases, are aligned with one 
another, and; 
(c) developing and instituting a disciplined process for reviewing and 
accepting changes to the baseline requirements in light of estimated 
costs, benefits, and risk. 

DOD Response: Non-Concur. The DRRS program has an authoritative set of 
baseline requirements already established. The current DRRS governance 
process, which includes membership from all CoComs, Services, Combat 
Support Agencies (CSAs), Joint Staff, and OSD oversees the DRRS 
requirements management process. These requirements are reviewed bi-
weekly as part of the DRRS configuration control process. 

Recommendation 4: The GAO recommends that the Secretary of Defense 
through the appropriate chain of command, take steps to ensure that 
DRRS testing is effectively managed, including: 

(a) developing lest plans and procedures for each system increment test 
event that include a schedule of planned test activities, defined roles 
and responsibilities, test entrance and exit criteria, test defect 
management processes, and metrics for' measuring test progress, 

(b) ensuring that all key lest events are conducted on all DRRS 
increments, 

(c) capturing, analyzing, reporting, and resolving all test results and 
test defects of all developed and tested DRRS increments; and, 

(d) establishing an effective test management structure that includes 
assigned test management roles and responsibilities, a designated test 
management lead and a supporting working group, and a reliable schedule 
of test events. 

DoD Response: Non-Concur. DRRS testing is already in place and 
performing effectively. The DIO goes through a rigorous testing regimen 
that includes documenting test plans with user test cases for each 
incremental release. The DRRS program utilizes System Integration 
Testing (SIT), System Acceptance Testing (SAT), Interoperability 
Testing (IOT) and Operational Testing (OT) with System Integration Test 
Plans and System Acceptance Test Plans. There are designated testers 
independent of the developers that validate the user test cases and 
functionality prior to a deployment decision. Finally, the D10 conducts 
a release review with the developers and the testers to determine if 
the increment is ready for release and meets all the exit criteria. For 
interoperability testing with systems external to DRRS, the DIO has a 
designated test director and the Joint Interoperability Test Command 
(JITC) is the designated Interoperability Test Activity and the 
Operational Test Activity. JITC is responsible for developing the 
Interoperability Test Plan and the Operational Test Plan. 

Recommendation 5: The GAO recommends t oat the Secretary of Defense 
through the appropriate chain of command, take steps to ensure that the 
DRRS integrated master schedule is reliable, including ensuring that 
the schedule: 

(a) captures all activities from the work breakdown structure, 
including the work to be performed and the resources to he used; 

(b) identifies the logical sequencing of all activities, including 
defining predecessor and successor activities; 

(c) reflects whether all required resources will he available when 
needed and their cost; 

(d) ensures that all activities and their duration are not summarized 
at a level that could mask critical elements; 

(e) achieves horizontal integration in the schedule by ensuring that 
all external interfaces (hand-offs) are established and 
interdependencies among activities are defined; 

(f) identifies float between activities by ensuing that the linkages 
among all activities are defined; 

(g) defines a critical path that runs continuously to the program's 
finish date; 

(h) incorporates the results of a schedule risk analysis to determine 
the level of confidence in meeting the program's activities and 
completion date, and; 

(i) includes the actual start and completion dotes of work activities 
per formed so that the impact of deviations on downstream work can he 
proactively addressed. 

DOD Response: Non Concur. A process is already in place to ensure the 
DRRS integrated master schedule is current, reliable, and meets all of 
the criteria outlined in the recommendation, With the approval of the 
DEXCOM charter and the renewed/refined Battle Staff and General Officer 
Steering Committee (GOSC), many of the program issues regarding Plan of 
Action and Milestones (POA&M) software development, integration, and 
testing in the GAO report are being address or have been corrected. 

Recommendation 6: The GAO recommends t nit the Secretary of Defense 
through the appropriate chain of command, take steps to ensure that the 
DRRS program office is staffed on the basis of a human capital strategy 
that is grounded in an assessment of the core competencies and 
essential knowledge, skills, and abilities needed to perform key DRRS 
program management functions, an inventory of the program office's 
existing workforce capabilities, and an analysis of the gap between the 
assessed needs and the existing capabilities. 

DOD Response: Partially concur. The DIO was intentionally kept small in 
order to maximize flexibility and responsiveness to customer 
requirements as well as to implement commercial best practices for 
agile software development. However, due to increase demands on DRRS 
through additional requirements and capabilities requested by users, 
additional billets, both military and civilian are needed to expedite 
the implementation of DRRS. To that end, the USD (P& R) is planning on 
adding full time civilian acquisition support for the DIO and the 
conversion of contractor to civilian billets during 2010/2011 timeframe 
as part of the Department's in-sourcing initiative. 

Recommendation 7: The GAO recommends brat the Secretary of Defense 
through the appropriate chain of command, take steps to ensure that 
DRRS is developed and implemented in a manner that does not increase 
the reporting burden on units and addresses the timeliness. precision, 
and objectivity of metrics that are reported to Congress. 

DOD Response: Non-Concur. One of the primary tenets for DRRS 
development has always been to reduce the reporting requirements on the 
war fighter. DRRS is already using state of the art technology in order 
to ensure that data which already exists is available within the DRRS 
enterprise in near-real time for the war fighters. The MRS governance 
structure that is currently in place ensures that DRRS development does 
not deviate from these core principles. 

Recommendation 8: The GAO recommends that the Secretary of Defense 
direct the Deputy Secretary of Defense to ensure that both. he Human 
Resources Management Investment Review Board and the DRRS Executive 
Committee conduct frequent oversight activities of the DRRS program. 
and report any significant issues to the Deputy Secretary of Defense. 

DOD Response: Concur. The Department is committed to ensuring that the 
proper level of oversight of the DRRS program is adhered to. As an 
Acquisition Category Ill level program, DRRS will be managed from an 
acquisition perspective by the Component Acquisition Executive (CAE) 
within USD(P&R). The USD (P&R) CAE is already working with the program 
to ensure that they become fully compliant with all acquisition 
requirements. This is consistent with tiered accountability. The CAE 
will certify to the IIRM IRB and the DCMO of compliance prior to 
submission of future certification requests. Finally, the current 
DEXCOM governance process will continue to provide sustained functional 
oversight of the DRRS program. Issues that arise in any of these areas 
will be elevated for review as appropriate. 

[End of section] 

Appendix IV: GAO Contacts and Staff Acknowledgments: 

GAO Contacts: 

Sharon L. Pickup, (202) 512-9619 or pickups@gao.gov Randolph C. Hite, 
(202) 512-6256 or hiter@gao.gov: 

Staff Acknowledgments: 

In addition to the contacts named above, key contributors to this 
report were Michael Ferren (Assistant Director), Neelaxi Lakhmani 
(Assistant Director), April Baugh, Mathew Butler, Richard J. Hagerman, 
Nicole Harms, James Houtz, John Lee, Stephen Pruitt, Terry Richardson, 
Karen Richey, Karl Seifert, and Kristy Williams. 

[End of section] 

Footnotes: 

[1] Pub. L. No. 105-261, §373 (1998). Codified at 10 U.S.C. §117. 

[2] Department of Defense Directive 7730.65, Department of Defense 
Readiness Reporting System (DRRS) (June 3, 2002). 

[3] ESORTS is an automated readiness reporting tool designed to collect 
capability and resource data, while DRRS is the broader system that, 
according to the 2002 DRRS Directive, measures and reports on the 
readiness of military forces and the supporting infrastructure to meet 
missions and goals assigned by the Secretary of Defense. However, 
ESORTS is now viewed as a tool within DRRS. 

[4] Secretary of Defense Memorandum, Policy Implementation to Establish 
Commander, USJFCOM (CDRUSJFCOM), as the Primary Joint Force Provider 
(JFP) (June 25, 2004). The U.S. military organizes its global presence 
into a series of geographic and functional combatant commands. The 
geographic combatant commands--U.S. Central Command, U.S. European 
Command, U.S. Northern Command, U.S. Pacific Command, U.S. Southern 
Command, and U.S. Africa Command --have authority over all U.S. 
military forces operating within a specified area of operation and are 
directly responsible for the performance of missions assigned to the 
command. The functional combatant commands--U.S. Joint Forces Command, 
U.S. Special Operations Command, U.S. Strategic Command, and U.S. 
Transportation Command--possess worldwide functional responsibilities, 
such as joint training, force provision, and global command and 
control. 

[5] Chairman of the Joint Chiefs of Staff Instruction 3401.02A, Global 
Status of Resources and Training System (GSORTS) (Feb. 27, 2004). 

[6] All combat, combat support, and combat service support units that 
have the potential to support an: operations, contingency, or single 
integrated operational plan; or a contingency operation are required to 
report. 

[7] A mission is a task or a series of tasks with a purpose. Mission 
essential tasks are those tasks that are absolutely necessary, 
indispensable, or critical to mission success. DRRS' capability data 
are based on commanders' assessments of their organizations' 
capabilities to carry out their missions and mission essential tasks. 

[8] The Strom Thurmond National Defense Authorization Act for Fiscal 
Year 1999, Pub. L. No. 105-261 §373 (1998), codified at 10 U.S.C. §117. 
Section 373 initially directed the Secretary to establish and implement 
the readiness reporting system no later than January 15, 2000. An 
amendment in the National Defense Authorization Act for Fiscal Year 
2000, Pub. L. No. 106-65, §361 changed the date from January 15, 2000, 
to April 1, 2000. 

[9] Secretary of Defense Memorandum, Policy Implementation to Establish 
Commander, USJFCOM (CDRUSJFCOM), as the Primary Joint Force Provider 
(JFP), (June 25, 2004). The memorandum's primary purpose was to direct 
the Commander of the U.S. Joint Forces Command to assume the duties of 
DOD's primary joint force provider. 

[10] Under Secretary of Defense for Personnel and Readiness Memorandum, 
Department of Defense Readiness Reporting System (DRRS) Interim 
Implementation Guidance (Nov. 2, 2004). USD (P&R) issued three 
subsequent memorandums between August 10, 2005, and August 23, 2006, 
which expanded and clarified reporting requirements. 

[11] The four Investment Review Boards are for (1) Financial 
Management, established by the Deputy Under Secretary of Defense for 
Financial Management; (2) Weapon Systems Lifecycle Management and 
Materiel Supply and Services Management; (3) Real Property and 
Installations Lifecycle Management, both established by the Under 
Secretary of Defense for Acquisition, Technology and Logistics; and (4) 
Human Resources Management, established by USD (P&R). 

[12] The purpose of a risk assessment is to reduce systemic risk and 
support informed decision making by focusing on delivering business 
capabilities rapidly and at a reduced cost, and identifying program 
vulnerabilities and assisting in developing mitigation solutions. The 
assessment is generally performed prior to major acquisition decisions, 
although assessments may be requested for other reasons. 

[13] See for example, Department of Defense Instruction 5000.02, 
Operation of the Defense Acquisition System (Dec. 8, 2008); Defense 
Acquisition University, Test and Evaluation Management Guide, 5th ed. 
(Fort Belvoir, Va.: January 2005); Institute of Electrical and 
Electronics Engineers, Standard 1012-2004 for Software Verification and 
Validation (New York, N.Y.: June 8, 2005); GAO, GAO Cost Estimating and 
Assessment Guide: Best Practices for Developing and Managing Capital 
Program Costs, [hyperlink, http://www.gao.gov/products/GAO-09-3SP] 
(Washington, D.C.: March 2009); and GAO, A Model of Strategic Human 
Capital Management, GAO-02-373SP (Washington, D.C.: Mar. 15, 2002). 

[14] The users of readiness data include the Secretary of Defense, the 
military services, the Chairman of the Joint Chiefs of Staff, the 
combatant commands, defense agencies, and field activities. 

[15] GAO, Military Readiness: New Reporting System Is Intended to 
Address Long-Standing Problems, but Better Planning Is Needed, 
[hyperlink, http://www.gao.gov/products/GAO-03-456] (Washington, D.C.: 
Mar. 28, 2003). 

[16] GAO, GAO Cost Estimating and Assessment Guide: Best Practices for 
Developing and Managing Capital Program Costs, [hyperlink, 
http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: March 2009). 

[17] Float is the amount of time an activity can slip before affecting 
the critical path. 

[18] A Major Automated Information System is a program or initiative 
that is so designated by the Assistant Secretary of Defense (Networks 
and Information Integration)/Chief Information Officer or that is 
estimated to require program costs in any single year in excess of $32 
million, total program costs in excess of $126 million, or total life- 
cycle costs in excess of $378 million in fiscal year 2000 constant 
dollars. 

[19] As previously noted, the Investment Review Board is responsible 
for all business system investments in DOD that involve more than $1 
million in obligations. 

[20] The primary GSORTS metric--the "C" rating--is a mission-focused 
metric that is based not only on a unit's resources but also on the 
unit's capabilities to execute training to standards, in a specified 
environment. 

[21] The GSORTS metric that captures information concerning a unit's 
capability to execute these other assigned or directed missions is the 
percent effective metric. This percent effective rating is a subjective 
assessment that is based on a commander's professional military 
judgment. It is based on a number of considerations but not strictly 
tied to resources or training performance. 

[22] The software to collect and display this enhanced capability data 
has been in place for several years and the DIO has stated that DRRS 
reached its initial operating capability when the system was able to 
collect and display this information. 

[23] The SIPRNet is operated by the Defense Information Systems Agency 
and is DOD's largest interoperable command and control data network. In 
addition to supporting the input of data to DRRS, the SIPRNet supports 
the Global Command and Control System, the Defense Message System, 
collaborative planning, and numerous other classified warfighter 
applications. 

[24] The memorandum--Under Secretary of Defense for Personnel and 
Readiness Memorandum, Department of Defense Readiness Reporting System 
(DRRS) Interim Implementation Guidance, (Aug. 10, 2005)--directed units 
report mission essential task capabilities in DRRS through the 
automated ESORTS reporting tool. 

[25] When missions or mission essential tasks are not standardized, 
capability searches may not yield desired results because similar units 
are measuring their capabilities against different task conditions and 
standards. 

[26] Under Secretary of Defense for Personnel and Readiness Memorandum, 
Status of Defense Readiness Reporting System (DRRS) Implementation 
(Sept. 29, 2005). 

[27] Current reporting guidelines require the services to report both 
in the Chairman of the Joint Chiefs of Staff's GSORTS and in OSD's 
DRRS. 

[28] Department of the Air Force Memorandum, Defense Readiness 
Reporting System (DRRS) Core Unit Mission Essential Task List (METL) 
Implementation Guidance (Mar. 3, 2009). 

[29] Marine Corps Administrative Message 0307//09, Update to Interim 
Defense Readiness Reporting System (DRRS) Policy and Procedures for 
Marine Corps Units and Installations (May 12, 2009). 

[30] The law also lists annual requirements for training 
establishments, defense installations, and facilities to report their 
capabilities, and a number of other monthly, quarterly, and annual 
requirements. 

[31] While certain data, such as personnel data, are captured in 
multiple data systems, the services or OSD designate a single data 
system as the authoritative database for each type of information. 

[32] [hyperlink, http://www.gao.gov/products/GAO-03-456]. 

[33] Chairman of the Joint Chiefs of Staff Instruction 3401.02A, Global 
Status of Resources and Training System (GSORTS) (Feb. 27, 2004), 
permits commanders to subjectively upgrade or downgrade their units' 
overall assessment, although not their individual resource levels. 
However, these changes are easy to identify and commanders are required 
to provide justifications for any changes. A recent GAO review of Army 
data found that commanders' subjective changes constituted less than 10 
percent of the total assessments. 

[34] The memorandum's primary purpose was to provide policy 
implementation establishing the Commander of the U.S. Joint Forces 
Command to assume the duties as DOD's primary joint force provider. 

[35] These historical data would include training data, personnel data, 
and equipment supply and condition data as well as data concerning unit 
capabilities to perform the missions they were organized and designed 
to perform. 

[36] Ronald W. Reagan National Defense Authorization Act for Fiscal 
Year 2005, Pub. L. No. 108-375, § 332 (2004) (codified at 10 U.S.C. §§ 
186 and 2222). 

[37] Department of Defense Instruction Number 5000.02, Operation of the 
Defense Acquisition System, Enclosure 11 (December 8, 2008). 

[38] The Capability Maturity Model® Integration for Development, 
developed by the Software Engineering Institute of Carnegie Mellon 
University, defines key practices that are recognized hallmarks for 
successful organizations that, if effectively implemented, can greatly 
increase the chances of successfully developing and acquiring software 
and systems. Carnegie Mellon Software Engineering Institute, Capability 
Maturity Model® Integration for Development, version 1.2 (Pittsburgh, 
Pa.: August 2006). 

[39] See for example, Department of Defense Instruction 5000.02, 
Operation of the Defense Acquisition System (Dec. 8, 2008); Defense 
Acquisition University, Test and Evaluation Management Guide, 5th ed. 
(Fort Belvoir, Va.: January 2005); Institute of Electrical and 
Electronic Engineers, Standard 1012-2004 for Software Verification and 
Validation (New York, N.Y.: June 8, 2005); and Software Engineering 
Institute, Capability Maturity Model Integration for Acquisition 
version 1.2 (Pittsburgh, Pa.: November 2007). 

[40] GAO, GAO Cost Estimating and Assessment Guide: Best Practices for 
Developing and Managing Capital Program Costs, [hyperlink, 
http://www.gao.gov/products/GAO-09-3SP] (Washington D.C.: March 2009). 

[41] GAO, A Model of Strategic Human Capital Management, [hyperlink, 
http://www.gao.gov/products/GAO-02-373SP] (Washington, D.C.: Mar. 15, 
2002). 

[42] Department of Defense Directive 7730.65, Department of Defense 
Readiness Reporting System (DRRS) (June 3, 2002). 

[43] The reporting officer for U.S. Pacific Command indicated that he 
felt that the GAO's survey did not "accurately and fully assess the 
usefulness and functionality of DRRS" and provided additional written 
information to support his survey answers. After consultation with 
GAO's office of Applied Research and Methods we determined the survey 
did assess DRRS functionality. In order to capture U.S. Pacific 
Command's concerns, we incorporated the information it provided into 
the report sections describing current DRRS usage. 

[44] See for example, Department of Defense Instruction 5000.02, 
Operation of the Defense Acquisition System (Dec. 8, 2008); Defense 
Acquisition University, Test and Evaluation Management Guide, 5th ed. 
(Fort Belvoir, Va.: January 2005), Institute of Electrical and 
Electronic Engineers, Standard 1012-2004 for Software Verification and 
Validation (New York, N.Y.: June 8, 2005); GAO, GAO Cost Estimating and 
Assessment Guide: Best Practices for Developing and Managing Capital 
Program Costs, [hyperlink, http://www.gao.gov/products/GAO-09-3SP] 
(Washington D.C.: March 2009); and GAO, A Model of Strategic Human 
Capital Management, [hyperlink, 
http://www.gao.gov/products/GAO-02-373SP] (Washington, D.C.: Mar. 15, 
2002). 

[45] The Capability Maturity Model® Integration for Development, 
developed by the Software Engineering Institute of Carnegie Mellon 
University, defines key practices that are recognized hallmarks for 
successful organizations that, if effectively implemented, can greatly 
increase the chances of successfully developing and acquiring software 
and systems. Carnegie Mellon Software Engineering Institute, Capability 
Maturity Model® Integration for Development, version 1.2 (Pittsburgh, 
Pa.: August 2006). 

[46] GAO, Secure Border Initiative: DHS Needs to Address Significant 
Risks in Delivering Key Technology Investment, [hyperlink, 
http://www.gao.gov/products/GAO-08-1086] (Washington, D.C.: Sept. 22, 
2008). 

[47] Office of the Under Secretary of Defense for Personnel and 
Readiness Memorandum, Implementing the Defense Readiness Reporting 
System (July 1, 2002). 

[48] Department of Defense Directive 7730.65, Department of Defense 
Readiness Reporting System (DRRS) (June 3, 2002). Additional 
requirements for DRRS have been established in DOD directives and 
instructions that govern information systems including information 
assurance, net-centric architecture, net-centric data strategy, and 
information system interoperability and supportability. 

[49] Department of Defense Directive 7730.65, Department of Defense 
Readiness Reporting System (DRRS) (June 3, 2002). 

[50] [hyperlink, http://www.gao.gov/products/GAO-08-1086]. 

[51] The Capability Maturity Model® Integration for Development, 
developed by the Software Engineering Institute of Carnegie Mellon 
University, defines key practices that are recognized hallmarks for 
successful organizations that, if effectively implemented, can greatly 
increase the chances of successfully developing and acquiring software 
and systems. Carnegie Mellon Software Engineering Institute, Capability 
Maturity Model® Integration for Development, Version 1.2 (Pittsburgh, 
Pa.: August 2006). 

[52] See for example, Department of Defense Instruction 5000.02, 
Operation of the Defense Acquisition System; Defense Acquisition 
University, Test and Evaluation Management Guide; Institute of 
Electrical and Electronics Engineers, Standard 1012-2004 for Software 
Verification and Validation; and Software Engineering Institute, 
Capability Maturity Model Integration for Acquisition version 1.2 
(Pittsburgh, Pa.: November 2007). 

[53] See for example, Institute of Electrical and Electronics 
Engineers, Standard 829-2008 Standard for Software and System Test 
Documentation (New York, N.Y.: July 18, 2008) and Software Engineering 
Institute, Capability Maturity Model Integration for Acquisition, 
version 1.2. 

[54] According to the Concept of Operations, ESORTS is to provide 
current readiness status for operational forces and defense support 
organizations in terms of their ability to perform their mission 
essential tasks. 

[55] According to the draft SORTSREP Test and Evaluation Master Plan, 
SORTSREP is a tool to capture and input military departments' readiness 
data requirements into a readiness reporting database. 

[56] See for example, Defense Acquisition University, Defense 
Acquisition Guidebook (Arlington, Va.: November 2006); Institute of 
Electrical and Electronics Engineers, Standard 1012-2004 for Software 
Verification and Validation; Software Engineering Institute, Capability 
Maturity Model Integration for Acquisition, version 1.2. 

[57] See for example, Institute of Electrical and Electronics 
Engineers, Standard 829-2008 Standard for Software and System Test 
Documentation and Software Engineering Institute, Capability Maturity 
Model Integration for Acquisition version 1.2. 

[58] See for example, Institute for Electrical and Electronics 
Engineers, Standard 829-2008 Standard for Software and System 
Documentation. 

[59] See for example, Software Engineering Institute, Capability 
Maturity Model Integration for Acquisition, version 1.2. 

[60] See for example, GAO, Office of Personnel Management: Improvements 
Needed to Ensure Successful Retirement Systems Modernization, 
[hyperlink, http://www.gao.gov/products/GAO-08-345] (Washington, D.C.: 
January 2008); Institute of Electrical and Electronics Engineers, 
Standard 1012-2004 for Software Verification and Validation; and 
Institute of Electrical and Electronics Engineers, Standard 1012-1986 
for Software Verification and Validation Plans (New York, N.Y.: Nov. 
14, 1986). 

[61] See for example, [hyperlink, 
http://www.gao.gov/products/GAO-08-345]; Institute of Electrical and 
Electronics Engineers, Standard 1012-2004 for Software Verification and 
Validation; and Institute of Electrical and Electronics Engineers, 
Standard 1012-1986 for Software Verification and Validation Plans. 

[62] See for example, Department of Defense Instruction 5000.02, 
Operation of the Defense Acquisition System; Defense Acquisition 
University, Defense Acquisition Guidebook; Software Engineering 
Institute, Capability Maturity Model Integration for Acquisition, 
version 1.2; Institute for Electrical and Electronics Engineers, 
Standard 829-2008 Software and System Test Documentation (New York, 
N.Y.: July 18, 2008). 

[63] See for example, Defense Acquisition University, Defense 
Acquisition Guidebook; Software Engineering Institute, Capability 
Maturity Model Integration for Acquisition, version 1.2; Institute for 
Electrical and Electronics Engineers, Standard 829-2008 Software and 
System Test Documentation. 

[64] According to the Mission Readiness TEMP, Mission Readiness is to 
create a common standard metric for assessing readiness by capability 
that can be uniformly applied throughout the department. 

[65] GAO, GAO Cost Estimating and Assessment Guide: Best Practices for 
Developing and Managing Capital Program Costs, [hyperlink, 
http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: March 2009). 

[66] Float is the amount of time an activity can slip before affecting 
the critical path. 

[67] GAO, Information Technology: Improvements for Acquisition of 
Customs Trade Processing System Continue, but Further Efforts Needed to 
Avoid More Cost and Schedule Shortfalls, GAO-08-46 (Washington, D.C.: 
Oct. 25, 2007) and Secure Border Initiative: SBInet Planning and 
Management Improvements Needed to Control Risks, [hyperlink, 
http://www.gao.gov/products/GAO-07-504T] (Washington, D.C.: Feb. 27, 
2007). 

[68] GAO, A Model of Strategic Human Capital Management, [hyperlink, 
http://www.gao.gov/products/GAO-02-373SP] (Washington, D.C.: Mar. 15, 
2002). 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "E-mail Updates." 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO’s Web site, 
[hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional 
information. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: fraudnet@gao.gov: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Ralph Dawn, Managing Director, dawnr@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: