This is the accessible text file for GAO report number GAO-08-896 
entitled 'DOD Business Systems Modernization: Important Management 
Controls Being Implemented on Major Navy Program, but Improvements 
Needed in Key Areas' which was released on September 8, 2008.

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to the Subcommittee on Readiness and Management Support, 
Committee on Armed Services, U.S. Senate: 

United States Government Accountability Office: 
GAO: 

September 2008: 

DOD Business Systems Modernization: 

Important Management Controls Being Implemented on Major Navy Program, 
but Improvements Needed in Key Areas: 

GAO-08-896: 

GAO Highlights: 

Highlights of GAO-08-896, a report to the Subcommittee on Readiness and 
Management Support, Committee on Armed Services, U.S. Senate. 

Why GAO Did This Study: 

The Department of Defense (DOD) has long been challenged in 
implementing key information technology (IT) management controls on its 
thousands of business system investments. For this and other reasons, 
GAO has designated DOD’s business systems modernization efforts as high-
risk. One of the larger business system investments is the Department 
of the Navy’s Enterprise Resource Planning (Navy ERP) program. 
Initiated in 2003, the program is to standardize the Navy’s business 
processes, such as acquisition and financial management. It is being 
delivered in increments, the first of which is to cost about $2.4 
billion over its useful life and be fully deployed in fiscal year 2013. 
GAO was asked to determine whether key IT management controls are being 
implemented on the program. To do this, GAO analyzed, for example, 
requirements management, economic justification, earned value 
management, and risk management. 

What GAO Found: 

DOD has implemented key IT management controls on its Navy ERP program 
to varying degrees of effectiveness. To its credit, the control 
associated with managing system requirements is being effectively 
implemented. In addition, important aspects of other controls have at 
least been partially implemented, including those associated with 
economically justifying investment in the program and proactively 
managing program risks. Nevertheless, other aspects of these controls, 
as well as the bulk of what is needed to effectively implement earned 
value management, which is a recognized means for measuring program 
progress, have not been effectively implemented. Among other things, 
these control weaknesses have contributed to the more than 2-year 
schedule delay and the almost $600 million cost overruns already 
experienced on the program since it began, and they will likely 
contribute to future delays and overruns if they are not corrected. 
Examples of the weaknesses are provided below. 

* Investment in the program has been economically justified on the 
basis of expected benefits that far exceed estimated costs ($8.6 
billion versus $2.4 billion over a 20-year life cycle). However, 
important estimating practices, such as using historical cost data from 
comparable programs and basing the cost estimate on a reliable schedule 
baseline were not employed. While these weaknesses are important 
because they limit the reliability of the estimates, GAO’s analysis 
shows that they would not have altered the estimates to the point of 
not producing a positive return on investment. 

* Earned value management has not been effectively implemented. To its 
credit, the program office has elected to implement program-level 
earned value management. In doing so, however, basic prerequisites for 
effectively managing earned value have not been executed. In 
particular, the integrated master schedule was not derived in 
accordance with key estimating practices, and an integrated baseline 
review has not been performed on any of the first increment’s releases. 

* A defined process for proactively avoiding problems, referred to as 
risk management, has been established, but risk mitigation strategies 
have not been effectively implemented for all significant risks, such 
as those associated with data conversion and organizational change 
management, as well the risks associated with the above-cited 
weaknesses. 

The reasons that program management and oversight officials cited for 
these practices not being executed range from the complexity and 
challenges of managing and implementing a program of this size to 
limitations in the program office’s scope and authority. 
Notwithstanding the effectiveness with which important aspects of 
several controls have been implemented, the above-cited weaknesses put 
DOD at risk of investing in a system solution that does not optimally 
support corporate mission needs and mission performance, and meet 
schedule and cost commitments. 

What GAO Recommends: 

GAO is making recommendations to the Secretary of Defense aimed at 
improving cost and schedule estimating, earned value management, and 
risk management weaknesses. DOD largely agreed with GAO’s 
recommendations and described actions planned or under way to address 
them. 

To view the full product, including the scope and methodology, click on 
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-896]. For more 
information, contact Randolph C. Hite at (202) 512-3439 or 
hiter@gao.gov. 

[End of section] 

Contents: 

Letter: 

Results in Brief: 

Background: 

Implementation of Key DOD and Related IT Acquisition Management 
Controls on Navy ERP Varies: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Objective, Scope, And Methodology: 

Appendix II: Comments From The Department Of Defense: 

Appendix III: GAO Contact And Staff Acknowledgments: 

Tables: 

Table 1: Description and Status of the Navy ERP Pilots: 

Table 2: Navy ERP Template 1 Releases: 

Table 3: Organizations Responsible for Navy ERP Oversight and 
Management: 

Table 4: Description of Business System IT Acquisition Management 
Controls: 

Table 5: Summary of Cost Estimating Characteristics That the Cost 
Estimate Satisfies: 

Table 6: Satisfaction of Schedule Estimating Key Practices: 

Figures: 

Figure 1: Navy ERP Time Line: 

Figure 2: Navy ERP Milestones for Beginning Deployment: 

Figure 3: Navy ERP Life Cycle Cost Estimates in Fiscal Years 2003, 
2004, and 2007: 

Figure 4: Cumulative Cost Variance for Navy ERP over a 17-Month Period: 

Figure 5: Cumulative Schedule Variance of the Navy ERP Program over a 
17-Month Period: 

Abbreviations: 

BEA: business enterprise architecture: 

BTA: Business Transformation Agency: 

CIO: Chief Information Officer: 

DBSMC: Defense Business Systems Management Committee: 

DOD: Department of Defense: 

DON: Department of the Navy: 

EVM: earned value management: 

FOC: full operational capability: 

GCSS-MC: Global Combat Support System-Marine Corps: 

IOC: initial operational capability: 

IRB: investment review board: 

IT: information technology: 

MDA: milestone decision authority: 

NAVAIR: Naval Air Systems Command: 

NAVSEA: Naval Sea Systems Command: 

NAVSUP: Naval Supply Systems Command: 

Navy ERP: Navy Enterprise Resource Planning: 

NTCSS: Naval Tactical Command Support System: 

OMB: Office of Management and Budget: 

PMB: performance measurement baseline: 

SAP: Systems Applications and Products: 

SPAWAR: Space and Naval Warfare Systems Command: 

TC-AIMS II: Transportation Coordinators' Automated Information for 
Movements System: 

[End of section] 

United States Government Accountability Office:
Washington, DC 20548: 

September 8, 2008: 

The Honorable Daniel K. Akaka: 
Chairman: 
The Honorable John Thune: 
Ranking Member: 
Subcommittee on Readiness and Management Support: 
Committee on Armed Services: 
United States Senate: 

The Honorable John Ensign: 
United States Senate: 

For decades, the Department of Defense (DOD) has been challenged in 
modernizing its timeworn business systems.[Footnote 1] In 1995, we 
designated DOD's business systems modernization program as high risk, 
and continue to do so today.[Footnote 2] Our reasons include the 
modernization's large size, complexity, and its critical role in 
addressing other long-standing transformation and financial management 
challenges. Other reasons are that DOD has yet to institutionalize key 
system modernization management controls, and it has not demonstrated 
the ability to consistently deliver promised system capabilities and 
benefits on time and within budget. 

Nevertheless, DOD continues to invest billions of dollars in thousands 
of business systems, including about a hundred that the department has 
labeled as business transformational programs, 12 of which account for 
about 50 percent of these programs' costs. The Navy Enterprise Resource 
Planning (Navy ERP) program is one such program. Initiated in 2003, 
Navy ERP is to standardize the Navy's acquisition, financial, program 
management, maintenance, plant and wholesale supply, and workforce 
management business processes across its dispersed organizational 
environment. As envisioned, the program consists of a series of major 
increments, the first of which includes three releases and is expected 
to cost approximately $2.4 billion over its 20-year life cycle and to 
be fully operational in fiscal year 2013. 

As agreed, our objective was to determine whether the Department of the 
Navy (DON)[Footnote 3] is effectively implementing information 
technology (IT) management controls on Navy ERP. To accomplish this, we 
focused on the program's first increment by analyzing a range of 
program documentation and interviewing cognizant officials relative to 
the following management areas: architectural alignment, economic 
justification, earned value management (EVM), requirements management, 
and risk management. We conducted this performance audit from June 2007 
to September 2008, in accordance with generally accepted government 
auditing standards. Those standards require that we plan and perform 
the audit to obtain sufficient, appropriate evidence to provide a 
reasonable basis for our findings and conclusions based on our audit 
objective. We believe that the evidence obtained provides a reasonable 
basis for our findings and conclusions based on our audit objective. 
Additional details on our objective, scope, and methodology are in 
appendix I. 

Results In Brief: 

DOD has implemented key IT management controls on its Navy ERP program 
to varying degrees of effectiveness. To its credit, one of the controls 
has been fully implemented; important aspects of other controls have 
not. Collectively, these management controls are to ensure that a given 
system investment represents the right solution to filling a mission 
need, meaning that the system is defined to (1) minimize overlap and 
duplication and maximize interoperability with related systems and (2) 
produce mission benefits commensurate with costs over its useful life. 
The controls are also to ensure that the system is acquired and 
deployed the right way, meaning that it is done in a way to maximize 
the chances of delivering defined system capabilities and benefits on 
time and within budget. Given that deployment of Navy ERP is more than 
2 years behind schedule and is to cost about $570 million[Footnote 4] 
more than was originally envisioned, these goals have already not been 
fully met, in part because DOD program management and oversight 
entities have not fully implemented several key IT management controls. 
As a result, the department has yet to adequately demonstrate that the 
program's first increment, as it has been defined, is the right 
solution, and it is likely that the department will continue to add to 
the program's cost overruns and schedule delays that the program has 
already experienced to date. The strengths and weaknesses associated 
with each of the IT management controls that we evaluated are described 
here: 

* Navy ERP compliance with DOD's federated business enterprise 
architecture (BEA) has not been sufficiently demonstrated. To its 
credit, the program office followed DOD's architecture compliance 
assessment guidance and used the related assessment tool. However, this 
guidance and tool do not adequately provide for addressing all relevant 
aspects of architectural compliance. Specifically, the program's 
compliance assessment (1) did not include all relevant architecture 
products, such as those that describe technical and system elements; 
(2) was not used to identify potential areas of duplication across 
programs; and (3) did not address compliance with the DON enterprise 
architecture. These important steps were not performed because of 
policy, guidance, and tool limitations, and because aspects of the 
corporate BEA and the DON enterprise architecture, which are both major 
components of DOD's federated BEA, have yet to be sufficiently defined 
to permit thorough compliance determinations in these areas. In 
addition, program oversight and approval authorities did not validate 
the program office's compliance assessments. As a result, the 
department does not have a sufficient basis for knowing if Navy ERP has 
been defined to minimize overlap with, and duplication of other 
programs' functions, and is being defined and designed to maximize 
interoperability among related programs. 

* Investment in Navy ERP has been economically justified on the basis 
of expected life cycle benefits that far exceed estimated life cycle 
costs ($8.6 billion versus $2.4 billion over a 20-year life cycle). 
However, these benefit estimates have not been subject to analysis of 
how uncertainty in assumptions and data could impact them, as 
prescribed in relevant guidance. According to program officials, such 
uncertainty analysis is not warranted because they have taken and 
continue to take steps to validate the assumptions and the data, such 
as using the latest budget data associated with retiring legacy 
systems, and monitoring changes to the systems' retirement dates. While 
these steps are positive, they do not eliminate the need for 
uncertainty analysis. Accordingly, we assessed key uncertainty 
variables and found that while the inherent uncertainty in the 
estimates would reduce expected savings, the reduction would be small 
relative to a total benefit estimate of $8.6 billion. 

With respect to the cost estimate, our analysis showed that it was not 
derived using all key estimating practices contained in relevant 
guidance. For example, the estimate was not grounded in a historical 
record of comparable data from similar programs and was not based on a 
reliable schedule baseline, which are both necessary to having a cost 
estimate that can be considered credible and accurate. These practices 
were not employed for various reasons, including DOD's lack of 
historical data from similar programs and the lack of an integrated 
master schedule for the program's first increment that includes all 
three releases. Notwithstanding the fact that these limitations could 
materially increase the $2.4 billion cost estimate, it is nevertheless 
unlikely that they would increase the cost estimate to a level close to 
the uncertainty-adjusted benefit expectations. Therefore, we have no 
reason to believe that Navy ERP will not produce a positive return on 
investment. 

* EVM, which is a means for determining and disclosing actual program 
performance in comparison with estimates, is not being effectively 
implemented in Navy ERP. To its credit, the program office has elected 
to implement program-level EVM.[Footnote 5] In doing so, however, basic 
EVM activities have not been executed, which has produced, and will 
likely continue to produce, actual program costs and schedules that do 
not track close to estimates. For example, an integrated baseline 
review, which is to verify that the program's costs and schedule are 
reasonable given the program's scope of work and associated risks, has 
not been performed on any of the first increment's releases. According 
to program officials, this is because of the time it took to establish 
program-level EVM capabilities and because of their focus on deploying 
and stabilizing the first release. However, they recently stated that 
one has been tentatively scheduled for August 2008. By not having an 
integrated master schedule that has been subject to a baseline review, 
as well as not employing other industry standards as discussed in this 
report, Navy ERP will be challenged in implementing EVM effectively, 
and cost overruns and lengthy schedule delays beyond those already 
experienced by the program will likely occur. Our analysis of the 
latest estimate for completing just the budgeted development work for 
all three releases, which is about $844 million, shows that this 
estimate will most likely have an overrun of about $152 million. 

* An important requirements management activity has been effectively 
implemented in Navy ERP. Specifically, the program office has ensured 
that system requirements for the first release are traceable, as 
evidenced by our analysis of 60 randomly selected system-level 
requirements in which we confirmed that 58 are traceable backward to 
operational requirements and forward to design specifications and test 
results. Such traceability is important because it increases the 
chances of delivering a system that performs as intended. Our analysis 
of requirements in this sample also confirmed that system requirements 
that had been reallocated from the first release to the other releases 
were traceable, thus demonstrating traceability among product releases. 

* The program office has defined a process for proactively managing 
risks that reflects key practices governing how this IT management 
control should be performed. However, it has not effectively 
implemented this process for all identified risks. In particular, steps 
taken to mitigate the risks associated with converting data from 
existing systems to the new system and positioning user organizations 
for the operational changes associated with the new system were not 
effective. According to program officials, these mitigation strategies 
could not be effectively implemented because the program office does 
not have the authority to compel the user organizations to execute 
their part of the mitigation strategy. As a result, the first user 
organization to receive the Navy ERP experienced significant problems 
during recent operational testing, and these problems will take 
additional time and resources to correct. In addition, not all known 
risks have been captured in the risk inventory. For example, the risks 
associated with the two above discussed control weaknesses (not having 
adequately demonstrated the program's architectural alignment and not 
having implemented program-level EVM according to industry practices) 
are not included in the risk inventory and are thus not being disclosed 
and addressed. This is important because not having effectively 
addressed such risks has contributed to schedule delays and will likely 
contribute to more cost and schedule shortfalls. 

Notwithstanding the effectiveness with which several program management 
controls have been implemented on Navy ERP, the above-cited weaknesses 
in implementing other management controls put DOD at risk of investing 
in a system solution that does not optimally support corporate mission 
needs and mission performance and continuing to fall short of program 
schedule and cost commitments. Accordingly, we are making 
recommendations to the Secretary of Defense aimed at addressing the 
cost and schedule estimating, EVM, and risk management weaknesses, 
thereby better ensuring that the program is managed to deliver the 
right solution, the right way. We are not making recommendations in 
this report for addressing the architecture compliance weaknesses 
because we recently completed other work that is more broadly focused 
on this management control across multiple programs. 

In written comments on a draft of this report, signed by the Deputy 
Under Secretary of Defense (Business Transformation) and reprinted in 
appendix II, the department stated that it concurred with two, and 
partially concurred with two, of our four recommendations. Further, it 
stated that it has taken steps to address some of our recommendations, 
adding that it is committed to implementing recommendations that 
contribute to the program's success. 

In partially concurring with two of the recommendations, DOD concurred 
with most aspects of both. Nevertheless, for our recommendation aimed 
at improving the program's cost estimates, it stated that it did not 
concur with one aspect--ensuring that future cost estimates reflect the 
risk of not having cost data for comparable programs. In doing so, DOD 
stated that while the program had limited comparable data on which to 
base the program's cost estimate, it had accounted for this limitation 
in the cost estimate's uncertainty analysis. We do not agree that this 
risk was reflected in the uncertainty analysis. We examined this 
analysis as part of our review and found that it did not recognize this 
risk. Moreover, DOD's comments did not include any evidence to the 
contrary. 

In addition, for our recommendation aimed at improving the program's 
integrated master schedule, the department partially concurred with one 
of the five components of the recommendation--defining a critical path 
that incorporates all three releases of the system. In doing so, DOD 
stated what our report already recognized, namely that the schedule 
reflects a separate critical path for each release and that this is due 
to the size and complexity of the releases. However, DOD offers no new 
information in its comments. Further, our report also recognizes that 
to be successful, large and complex programs that involve thousands of 
activities need to ensure that their schedules integrate these 
activities. In this regard, we support the department's commitment to 
explore the feasibility of implementing this part of our 
recommendation. 

While concurring with all components of our other two recommendations, 
the department nevertheless provided comments relative to the program's 
use of EVM to explain why Release 1.0 of the system was not subject to 
an integrated baseline review. For several reasons discussed in the 
agency comments section of this report, we do not agree with these 
additional comments. In addition, the department's comments described 
actions planned or under way to address our recommendations. If fully 
and properly implemented, DOD's described actions should go a long way 
in addressing the weaknesses that we identify in the report. 

Background: 

DON's primary mission is to organize, train, maintain, and equip combat-
ready naval forces capable of winning the global war on terrorism and 
any other armed conflict, deterring aggression by would-be foes, 
preserving freedom of the seas, and promoting peace and security. To 
support this mission, DON performs a variety of interrelated and 
interdependent business functions (e.g., acquisition and financial 
management), relying heavily on IT systems. In fiscal year 2008, DON's 
budget for business systems and associated infrastructure was about 
$2.7 billion, of which $2.2 billion was allocated to operations and 
maintenance of existing systems and the remaining $500 million to 
systems in development and modernization. Of the approximately 3,000 
business systems that DOD reports in its current inventory, DON 
accounts for 904, or about 30 percent, of the total. Navy ERP is one 
such system investment. 

Navy ERP: A Brief Description: 

In July 2003, the Assistant Secretary of the Navy for Research, 
Development, and Acquisition established Navy ERP to "converge" four 
separate pilot programs that were under way at four separate Navy 
commands.[Footnote 6] This program is to leverage a commercial, off- 
the-shelf software known as an enterprise resource planning product. 
Such products consist of multiple, integrated functional modules that 
perform a variety of business-related tasks, such as acquisition and 
financial management. Table 1 provides a brief description and status 
of each of the pilots. 

Table 1: Description and Status of the Navy ERP Pilots: 

Pilot: SIGMA; 
Description and status: Deployed at Naval Air Systems Command 
(NAVAIR)[A] to support and link such business functions as program 
management, contracting, financial, and human resources management. It 
was retired in the first quarter of fiscal year 2008. 

Pilot: CABRILLO; 
Description and status: Deployed at Space and Naval Warfare Systems 
Command (SPAWAR)[B] to support Navy Working Capital Fund financial 
management. It is to be retired in fiscal year 2010. 

Pilot: NEMAIS; 
Description and status: Deployed at Naval Sea Systems Command 
(NAVSEA)[C] to support regional maintenance, including intermediate-
level maintenance[D] management and human resources. It is to be 
retired in fiscal year 2011. 

Pilot: SMART; 
Description and status: Deployed at Naval Supply Systems Command 
(NAVSUP)[E] and NAVAIR to support national and local supply management, 
intermediate-level maintenance management and to interface with 
aviation depots. It was retired in fiscal year 2005. 

Source: GAO analysis of DON data. 

[A] NAVAIR is responsible for developing, delivering, and supporting 
aircraft and weapons used by sailors and marines. 

[B] SPAWAR is responsible for developing, delivering, and supporting 
specialized command and control technologies, business information 
technology, and space capabilities. 

[C] NAVSEA is responsible for acquiring and maintaining the Navy's 
ships and submarines. 

[D] Intermediate-level maintenance is for repair or maintenance of 
items that do not have to go to the depot level for major work but 
cannot be maintained or repaired at the organizational level. 

[E] NAVSUP is responsible for supply, fuel, and transportation, as well 
as other logistics programs. 

[End of table] 

According to DOD, Navy ERP is to address the Navy's long-standing 
problems related to financial transparency and asset visibility. 
Specifically, the program is intended to standardize the Navy's 
acquisition, financial, program management, maintenance, plant and 
wholesale supply, and workforce management business processes across 
its dispersed organizational components. When the program is fully 
implemented, it is to support over 86,000 users. 

Navy ERP is being developed in a series of increments using the Systems 
Applications and Products (SAP) commercial software package, augmented 
as needed by customized software. SAP consists of multiple, integrated 
functional modules that perform a variety of business related tasks, 
such as finance and acquisition. The first increment, called Template 
1, is currently the only funded portion of the program and consists of 
three releases: 1.0 Financial and Acquisition, 1.1 Wholesale and Retail 
Supply, and 1.2 Intermediate-Level Maintenance. Release 1.0 is the 
largest of the three releases in terms of the functional requirements 
being addressed. Specifically, it is to provide about 56 percent of 
Template 1 requirements. See table 2 for a description of these 
releases. 

Table 2: Navy ERP Template 1 Releases: 

Release: 1.0 Financial and Acquisition; 
Functionality: General Fund and Navy Working Capital Fund finance 
applications, such as billing, budgeting, and cost planning; 
Acquisition applications, such as activity based costing, contract 
awards, and budget exhibits; Workforce management applications, such as 
personnel administration, and training and events management. 

Release: 1.1 Wholesale and Retail Supply; 
Functionality: Wholesale applications, such as supply and demand 
planning, order fulfillment, and supply forecasting; Retail supply 
applications, such as inventory management, supply and demand 
processing, and warehouse management. 

Release: 1.2 Intermediate-Level Maintenance; 
Functionality: Maintenance applications, such as maintenance 
management, quality management, and calibration management. 

Source: GAO analysis of DON data. 

[End of table] 

DON estimates the life cycle cost for the program's first increment to 
be about $2.4 billion, including about $1 billion for acquisition, and 
$1.4 billion for operations and maintenance. The life cycle cost of the 
entire program has not yet been determined because future increments 
have not been defined. The program office reported that approximately 
$400 million was spent from fiscal year 2004 through fiscal year 2007 
on the first increment. For fiscal year 2008, about $200 million is 
planned to be spent. 

Program Oversight and Management Roles and Responsibilities: 

To manage the acquisition and deployment of Navy ERP, DON established a 
program management office within the Program Executive Office for 
Executive Information Systems. The program office manages the program's 
scope and funding and is responsible for ensuring that the program 
meets its objectives. To accomplish this, the program office is 
responsible for key program management areas, such as architectural 
alignment, economic justification, earned value management, 
requirements management, and risk management. In addition, various DOD 
and DON organizations share program oversight and review activities. A 
listing of key entities and their roles and responsibilities is in 
table 3. 

Table 3: Organizations Responsible for Navy ERP Oversight and 
Management: 

Entity: Under Secretary of Defense for Acquisition, Technology, and 
Logistics; 
Roles and responsibilities: Serves as the milestone decision authority 
(MDA), which according to DOD, has overall responsibility for the 
program, to include approving the program to proceed through its 
acquisition cycle on the basis of, for example, the acquisition plan, 
an independently evaluated economic analysis, and the Acquisition 
Program Baseline. 

Entity: Assistant Secretary of the Navy, Research, Development, and 
Acquisition; 
Roles and responsibilities: Serves as DON's oversight organization for 
the program, to include enforcement of Under Secretary of Defense for 
Acquisition, Technology, and Logistics policies and procedures. 

Entity: Department of the Navy, Program Executive Office for Executive 
Information Systems; 
Roles and responsibilities: Oversees a portfolio of large-scale 
projects and programs designed to enable common business processes and 
provide standard capabilities, to include reviewing the acquisition 
plan, economic analysis, and the Acquisition Program Baseline prior to 
approval by the MDA. 

Entity: Department of the Navy Chief Information Officer (CIO); 
Roles and responsibilities: Supports the department's planning, 
programming, budgeting, and execution processes by ensuring that the 
program has achievable and executable goals and conforms to financial 
management regulations, and DON, DOD, and federal IT policies in 
several areas (e.g., security, architecture, and investment 
management); works closely with the program office during milestone 
review assessments. 

Entity: Office of the Secretary of Defense, Office of the Director for 
Program Analysis and Evaluation; 
Roles and responsibilities: Verifies and validates the reliability of 
cost and benefit estimates found in economic analyses and provides its 
results to the MDA. 

Entity: Naval Center for Cost Analyses; 
Roles and responsibilities: Performs independent costs estimates. 

Entity: Defense Business Systems Management Committee (DBSMC); 
Roles and responsibilities: Serves as the highest ranking governance 
body for business systems modernization activities and approves 
investments costing more than $1 million, as, for example, being 
compliant with the BEA. 

Entity: Investment Review Board (IRB); 
Roles and responsibilities: Reviews business system investments and has 
responsibility for recommending certification for all business system 
investments costing more than $1 million that are asserted as compliant 
with the BEA. 

Entity: Business Transformation Agency (BTA); 
Roles and responsibilities: Coordinates business transformation efforts 
across DOD and supports the IRBs and DBSMC. 

Entity: Navy ERP Program Management Office; 
Roles and responsibilities: Performs day-to-day program management and, 
as such, is the single point of accountability for managing the 
program's objectives through development, deployment, and sustainment. 

Source: DOD. 

[End of table] 

Overview of Navy ERP's Status: 

The first increment of Navy ERP is currently in the production and 
deployment phase of the defense acquisition system.[Footnote 7] The 
system consists of five key program life cycle phases and three related 
milestone decision points. These five phases and related milestones, 
along with a summary of key program activities completed during or 
planned for each phase, are as follows: 

1. Concept Refinement: The purpose of this phase is to refine the 
initial system solution (concept) and create a strategy for acquiring 
the solution. This phase began in July 2003, at which time DON began to 
converge the four pilot programs into Navy ERP and developed its first 
cost estimate in September 2003. This phase of the program was combined 
with the next phase, thus creating a combined Milestone A/B decision 
point. 

2. Technology Development: The purpose of this phase is to determine 
the appropriate set of technologies to be integrated into the 
investment solution by iteratively assessing the viability of the 
various technologies while simultaneously refining user requirements. 
During the combined Concept Refinement and Technology Development 
phase, the program office prepared a concept of operations and 
operational requirements document; performed an analysis of 
alternatives, business case analysis, and economic analysis; and 
established its first Acquisition Program Baseline. It also selected 
SAP as the commercial off-the-shelf ERP software. The combined phase 
was completed in August 2004, when the MDA approved Milestone A/B to 
allow the program to move to the next phase. 

3. System Development and Demonstration: The purpose of this phase is 
to develop a system and demonstrate through developer testing that the 
system can function in its target environment. This phase was completed 
in September 2007, when Release 1.0 passed development testing and its 
deployment to NAVAIR began. This was 17 months later than the program's 
original schedule set in August 2004 but on time according to the 
revised schedule set in December 2006. 

In September 2004, the program office awarded a $176 million system 
integration contract to BearingPoint for full system design, 
development, and delivery using SAP's off-the-shelf product and related 
customized software. In January 2006, the program office (1) reduced 
the contractor's scope of work from development and integration of the 
first increment to only development of the first release and (2) 
assumed responsibility and accountability for overall system 
integration. According to the program office, reasons for this change 
included the need to change the development plan to reflect 
improvements in the latest SAP product released and the lack of 
authority by the contractor to adjudicate and reconcile differences 
among the various Navy user organizations (i.e., Navy commands). 

In December 2006, the program office revised its Acquisition Program 
Baseline to reflect an increase of about $461 million in the life cycle 
cost estimate due, in part, to restructuring the program (e.g., 
changing the order of the releases, changing the role of system 
integrator from contractor to the program office) and resolving 
problems related to, among other things, converting data from legacy 
systems to run on Navy ERP and establishing interfaces between legacy 
systems and Navy ERP. In addition, the program office awarded a $151 
million contract for Release 1.1 and 1.2 configuration and development 
to IBM in June 2007. In September 2007, prior to entering the next 
phase, the program revised its Acquisition Program Baseline again to 
reflect a $9 million decrease in the life cycle cost estimate and a 5- 
month increase in its program schedule. Soon after, the MDA approved 
Milestone C to move to the next phase. 

4. Production and Deployment: The purpose of this phase is to achieve 
an operational capability that satisfies the mission needs, as verified 
through independent operational test and evaluation, and to implement 
the system at all applicable locations. This phase began in September 
2007, focusing first on achieving initial operational capability (IOC) 
of Release 1.0 at NAVAIR by May 2008. This date is 22 months later than 
the baseline established for Milestone A/B in August 2004, and 4 months 
later than the new baseline established in September 2007. According to 
program documentation, these delays were due, in part, to challenges 
experienced at NAVAIR in converting data from legacy systems to run on 
the new system and implementing new business procedures associated with 
the system. 

In light of the delays at NAVAIR in achieving IOC, the deployment 
schedules for the other commands were also revised. Specifically, 
Release 1.0 is still to be deployed at NAVSUP on October 2008, but 
Release 1.0 deployment at SPAWAR is now scheduled 18 months later than 
planned (October 2009), and deployment at NAVSEA general fund and Navy 
Working Capital Fund is now scheduled to be 12 months later than 
planned (October 2010 and 2011, respectively). Because of the Release 
1.0 delays, Release 1.1 is now planned for deployment at NAVSUP 7 
months later than planned (February 2010). Release 1.2 is still 
scheduled to be released at Regional Maintenance Centers in October 
2010. 

The program office is currently in the process of again re-baselining 
the program, and DON plans to address any cost overruns through 
reprogramming of fiscal year 2008 DON funds.[Footnote 8] It estimates 
that this phase will be completed with full operational capability 
(FOC)[Footnote 9] by August 2013 (26 months later than the baseline 
established in 2004, and 5 months later than the re-baseline 
established in September 2007). 

5. Operations and Support: The purpose of this phase is to 
operationally sustain the system in the most cost-effective manner over 
its life cycle. In this phase, the program plans to provide centralized 
support to its users across all system commands. Each deployment site 
is expected to perform complementary support functions, such as data 
maintenance. 

Overall, Increment 1 was originally planned to reach FOC in fiscal year 
2011, and its estimated life cycle cost was about $1.87 billion. 
[Footnote 10] The estimate was later baselined[Footnote 11] in August 
2004 at about $2.0 billion.[Footnote 12] In December 2006 and again in 
September 2007, the program was re-baselined. FOC is now planned for 
fiscal year 2013, and the estimated life cycle cost is about $2.4 
billion (31 percent increase over the original estimate).[Footnote 13] 
Key activities for each phase are depicted in figure 1, changes in the 
deployment schedule are depicted in figure 2, and cost estimates are 
depicted in figure 3. 

Figure 1: Navy ERP Time Line: 

[See PDF for image] 

This figure is an illustration of the Navy ERP time line, as follows: 

Phase: Concept refinement and technology development; 
Program established: Prior to the 2004 plan. 

Phase: System development and demonstration; 
2004 plan: mid-2004 through mid-2006; 
* Template 1 contract rescoped to Release 1.0 in early 2006. 
2007 plan: mid-2004 through end 2007; 
* Template 1 contract awarded late 2004; 
* Rebaselined early 2007; 
* Releases 1.1 and 1,2 contract awarded, mid-2007; 
* Rebaselined, end 2007. 

Phase: production and deployment; 
2004 plan: mid-2006 through mid-2011; 
* IOC, late 2006; 
* FOC, late 2011; 
2007 plan: late 2007 through late 2013; 
* IOC, early 2008; 
* FOC, late 2013. 

Source: GAO analysis of DON data. 

[End of figure] 

Figure 2: Navy ERP Milestones for Beginning Deployment: 

[See PDF for image] 

This figure is an illustration of the Navy ERP milestones for beginning 
deployment, as follows: 

Release: 1.0, Financial and Acquisition: 
Milestone: NAVAIR: late 2007 (2007 plan); 
Milestone: NAVSUP: late 2008 (2007 plan and 2008 plan); 
Milestone: SPAWAR: mid-2008 (2007 plan); early 2010 (2008 plan); 
Milestone: NAVSEA dor General Fund: early 2010 (2007 plan); early 2011 
(2008 plan); 
Milestone: NAVSEA for Working Capital Fund: early 2011 (2007 plan); 
early 2012 (2008 plan). 

Release: 1.1, Wholesale and Retail Supply; 
Milestone: NAVSUP: late 2009 (2007 plan); mid-2010 (2008 plan). 

Release 1.2, Intermediate-level Maintenance; 
Milestone: Regional Maintenance Centers: early 2011 (2007 plan and 2008 
plan). 

Source: GAO analysis of DON data. 

[End of figure] 

Figure 3: Navy ERP Life Cycle Cost Estimates in Fiscal Years 2003, 
2004, and 2007: 

[See PDF for image] 

This figure is a vertical bar graph depicting the following data: 

Navy ERP Life Cycle Cost Estimates in Fiscal Years 2003, 2004, and 
2007: 
Fiscal year: 2003: $1.87 billion; 
Fiscal year: 2004: $1.99 billion; 
Fiscal year: 2007: $2.44 billion. 

Source: GAO analysis of DON data. 

[End of figure] 

Use of IT Acquisition Management Controls Maximizes Chances for 
Success: 

IT acquisition management controls are tried and proven methods, 
processes, techniques, and activities that organizations define and use 
to minimize program risks and maximize the chances of a program's 
success. Using these controls can result in better outcomes, including 
cost savings, improved service and product quality, and a better return 
on investment. For example, two software engineering analyses of nearly 
200 systems acquisition projects indicate that teams using systems 
acquisition controls that reflected best practices produced cost 
savings of at least 11 percent over similar projects conducted by teams 
that did not employ the kind of rigor and discipline embedded in these 
practices.[Footnote 14] In addition, our research shows that these 
controls are a significant factor in successful acquisition outcomes, 
including increasing the likelihood that programs and projects will be 
executed within cost and schedule estimates.[Footnote 15] 

We and others have identified and promoted the use of a number of IT 
acquisition management controls associated with acquiring IT 
systems.[Footnote 16] See table 4 for a description of several of these 
activities. 

Table 4: Description of Business System IT Acquisition Management 
Controls: 

Business system acquisition practice: Architectural alignment; To 
ensure that the acquisition is consistent with the organization's 
enterprise architecture; 
Description: Architectural alignment is the process for analyzing and 
verifying that the proposed architecture of the system being acquired 
is consistent with the enterprise architecture for the organization 
acquiring the system. Such alignment is needed to ensure that acquired 
systems can interoperate and are not unnecessarily duplicative of one 
another. 

Business system acquisition practice: Economic justification; To ensure 
that system investments are economically justified; 
Description: Economic justification is the process for ensuring that 
acquisition decisions are based on reliable analyses of the proposed 
investment's likely costs versus benefits over its useful life, as well 
as an analysis of the risks associated with actually realizing the 
acquisition's forecasted benefits for its estimated costs. Economic 
justification is not a one-time event but rather is performed 
throughout an acquisition's life cycle in order to permit informed 
investment decision making. 

Business system acquisition practice: Earned value management; To 
ensure that actual progress against cost and schedule expectations is 
being measured; 
Description: EVM is a tool that integrates the technical, cost, and 
schedule parameters of a contract and measures progress against them. 
During the planning phase, an integrated program baseline is developed 
by time phasing budget resources for defined work. As work is performed 
and measured against the baseline, the corresponding budget value is 
"earned." Using this earned value metric, cost and schedule variances, 
as well as cost and time to complete estimates, can be determined and 
analyzed. 

Business system acquisition practice: Requirements management; To 
ensure that requirements are traceable, verifiable, and controlled; 
Description: Requirements management is the process for ensuring that 
the requirements are traceable, verifiable, and controlled. 
Traceability refers to the ability to follow a requirement from origin 
to implementation and is critical to understanding the interconnections 
and dependencies among the individual requirements and the impact when 
a requirement is changed. Requirements management begins when the 
solicitation's requirements are documented and ends when system 
responsibility is transferred to the support organization. 

Business system acquisition practice: Risk management; To ensure that 
risks are identified and systematically mitigated; 
Description: Risk management is the process for identifying potential 
acquisition problems and taking appropriate steps to avoid their 
becoming actual problems. Risk management occurs early and continuously 
in the acquisition life cycle. 

Source: GAO. 

[End of table] 

Prior GAO Reviews Have Identified IT Acquisition Management Control 
Weaknesses on DOD Business System Investments: 

We have previously reported[Footnote 17] that DOD has not effectively 
managed a number of business system investments. Among other things, 
our reviews of individual system investments have identified weaknesses 
in such things as architectural alignment and informed investment 
decision making, which are also the focus areas of the Fiscal Year 2005 
Defense Authorization Act business system provisions.[Footnote 18] Our 
reviews have also identified weaknesses in other system acquisition and 
investment management areas--such as EVM, economic justification, 
requirements management, and risk management. 

In July 2007, we reported that the Army's approach for investing about 
$5 billion over the next several years in its General Fund Enterprise 
Business System, Global Combat Support System-Army Field/Tactical, 
[Footnote 19] and Logistics Modernization Program did not include 
alignment with the Army enterprise architecture or use of a portfolio-
based business system investment review process.[Footnote 20] Moreover, 
we reported that the Army did not have reliable analyses, such as 
economic analyses, to support its management of these programs. We 
concluded that, until the Army adopts a business system investment 
management approach that provides for reviewing groups of systems and 
making enterprise decisions on how these groups will collectively 
interoperate to provide a desired capability, it runs the risk of 
investing significant resources in business systems that do not provide 
the desired functionality and efficiency. Accordingly, we made 
recommendations aimed at improving the department's efforts to achieve 
total asset visibility and enhancing its efforts to improve its control 
and accountability over business system investments. The department 
agreed with our recommendations. 

We also reported that DON had not, among other things, economically 
justified its ongoing and planned investment in the Naval Tactical 
Command Support System (NTCSS)[Footnote 21] and had not invested in 
NTCSS within the context of a well-defined DOD or DON enterprise 
architecture. In addition, we reported that DON had not effectively 
performed key measurement, reporting, budgeting, and oversight 
activities and had not adequately conducted requirements management and 
testing activities. We concluded that, without this information, DON 
could not determine whether NTCSS, as defined, and as being developed, 
is the right solution to meet its strategic business and technological 
needs. Accordingly, we recommended that the department develop the 
analytical basis to determine if continued investment in NTCSS 
represents prudent use of limited resources and to strengthen 
management of the program, conditional upon a decision to proceed with 
further investment in the program. The department largely agreed with 
our recommendations. 

In addition, we reported that the Army had not defined and developed 
its Transportation Coordinators' Automated Information for Movements 
System II (TC-AIMS II)--a joint services system with the goal of 
helping to manage the movement of forces and equipment within the 
United States and abroad--in the context of a DOD enterprise 
architecture.[Footnote 22] In addition, we reported that the Army had 
not economically justified the program on the basis of reliable 
estimates of life cycle costs and benefits and had not effectively 
implemented risk management. As a result, we concluded that the Army 
did not know if its investment in TC-AIMS II, as planned, was warranted 
or represented a prudent use of limited DOD resources. Accordingly, we 
recommended that the department, among other things, develop the 
analytical basis needed to determine if continued investment in TC-AIMS 
II represents prudent use of limited defense resources. In response, 
the department agreed with our recommendations and has since reduced 
the program's scope by canceling future investments. 

Furthermore, in 2005, we reported that DON had invested approximately 
$1 billion in the four previously cited ERP pilots without marked 
improvement in its day-to-day operations.[Footnote 23] More 
specifically, we reported that the program office had not implemented 
an EVM system. We also identified significant challenges and risks as 
the project moved forward, such as developing and implementing system 
interfaces, converting data from legacy systems into the ERP system, 
meeting its estimated completion date of 2011 at an estimated cost of 
$800 million, and achieving alignment with DOD's BEA. To address these 
areas, we made recommendations that DOD improve oversight of Navy ERP, 
including developing quantitative metrics to evaluate the program. DOD 
generally agreed with our recommendations. 

Implementation Of Key DOD And Related IT Acquisition Management 
Controls On Navy ERP Varies: 

DOD IT-related acquisition policies and guidance, along with other 
relevant guidance, provide an acquisition management control framework 
within which to manage business system programs like Navy ERP. 
Effective implementation of this framework can minimize program risks 
and better ensure that system investments are defined in a way to 
optimally support mission operations and performance, as well as 
deliver promised system capabilities and benefits on time and within 
budget. To varying degrees of effectiveness, Navy ERP has been managed 
in accordance with aspects of this framework. However, implementation 
of key management controls has not been effective. Specifically, 

* compliance with DOD's federated BEA has not been sufficiently 
demonstrated; 

* investment in the program has been economically justified on the 
basis of expected life cycle benefits that will likely exceed estimated 
life cycle costs, although some estimating limitations nevertheless 
exist; 

* earned value management has not been effectively implemented; 

* an important requirements management activity has been effectively 
implemented; and: 

* a risk management process has been defined, but not effectively 
implemented for all risks. 

The reasons that program management and oversight officials cited for 
why these key practices have not been sufficiently executed range from 
limitations in the applicable DOD guidance and tools to the complexity 
and challenges of managing and implementing a program of this size. 
Each of these reasons is described in the applicable sections of this 
report. By not effectively implementing all the above key IT 
acquisition management functions, the program is at increased risk of 
(1) not being defined in a way that best meets corporate mission needs 
and enhances performance and (2) adding to the more than 2 years in 
program schedule delays and about $570 million in program cost 
increases experienced to date. 

Navy ERP Compliance with DOD's Federated BEA Has Not Been Sufficiently 
Demonstrated: 

DOD and other guidance,[Footnote 24] recognize the importance of 
investing in business systems within the context of an enterprise 
architecture.[Footnote 25] Moreover, the Fiscal Year 2005 Defense 
Authorization Act requires that defense business systems be compliant 
with DOD's federated BEA.[Footnote 26] Our research and experience in 
reviewing federal agencies show that not making investments within the 
context of a well-defined enterprise architecture often results in 
systems that are duplicative, are not well integrated, are 
unnecessarily costly to interface and maintain, and do not optimally 
support mission outcomes.[Footnote 27] 

To its credit, the program office has followed DOD's BEA compliance 
guidance.[Footnote 28] However, this guidance does not adequately 
provide for addressing all relevant aspects of BEA compliance. 
Moreover, DON's enterprise architecture, which is a major component of 
DOD's federated BEA, as well as key aspects of DOD's corporate BEA, 
have yet to be sufficiently defined to permit thorough compliance 
determinations. In addition, current policies and guidance do not 
require DON investments to comply with its enterprise architecture. 
This means that the department does not have a sufficient basis for 
knowing if Navy ERP has been defined to minimize overlap with and 
duplication of other programs' functionality and maximize 
interoperability among related programs. Each of these architecture 
alignment limitations is discussed here: 

* The program's compliance assessments did not include all relevant 
architecture products. In particular, the program did not assess 
compliance with the BEA's technical standards profile, which outlines, 
for example, the standards governing how systems physically communicate 
with other systems and how they secure data from unauthorized access. 
This is particularly important because systems like Navy ERP need to 
share information with other systems and, for these systems to 
accomplish this effectively and efficiently, they need to employ common 
standards. A case in point is the relationship between Navy ERP and the 
Global Combat Support System--Marine Corps (GCSS-MC) program.[Footnote 
29] Specifically, Navy ERP has identified 25 technical standards that 
are not in the BEA technical standards profile, and GCSS-MC has 
identified 13 technical standards that are not in the profile. Among 
these non-BEA standards are program-unique information sharing 
protocols, which could limit information sharing between Navy ERP and 
GCSS-MC, and with other systems. 

In addition, the program office did not assess compliance with the BEA 
products that describe system-level characteristics. This is important 
because doing so would create a body of information about programs that 
could be used to identify common system components and services that 
could potentially be shared by the programs, thus avoiding wasteful 
duplication. For example, our analysis of Navy ERP program 
documentation shows that it contains system functions related to 
receiving goods, taking physical inventories, and returning goods, 
which are also system functions cited by the GCSS-MC program. However, 
because compliance with the BEA system products was not assessed, the 
extent to which these functions are potentially duplicative was not 
considered. 

Furthermore, the program office did not assess compliance with BEA 
system products that describe data exchanges among systems. As we 
previously reported, establishing and using standard system interfaces 
is a critical enabler to sharing data.[Footnote 30] For example, Navy 
ERP program documentation indicates that it is to exchange inventory 
order and status data with other systems. System interfaces are 
important for understanding how information is to be exchanged between 
systems. However, since the program was not assessed for compliance 
with these products, it does not have the basis for understanding how 
its approach to exchanging information differs from that of other 
systems that it is to interface with. Compliance against each of these 
BEA products was not assessed because DOD's compliance guidance does 
not provide for doing so and, according to BTA officials, because some 
BEA system products are not sufficiently defined. According to these 
officials, BTA plans to continue to define these products as the BEA 
evolves. 

* The compliance assessment was not used to identify potential areas of 
duplication across programs, which DOD has stated is an explicit goal 
of its federated BEA and associated investment review and decision- 
making processes. More specifically, even though the compliance 
guidance provides for assessing programs' compliance with the BEA 
product that defines DOD operational activities, and Navy ERP was 
assessed for compliance with this product, the results were not used to 
identify programs that support the same operational activities and 
related business processes. Given that the federated BEA is intended to 
identify and avoid not only duplications within DOD components, but 
also between components, it is important that such commonality be 
addressed. For example, BEA compliance assessments for Navy ERP and 
GCSS-MC, as well as two Air Force programs (Defense Enterprise 
Accounting and Management System--Air Force and the Air Force 
Expeditionary Combat Support System) show that each program supports at 
least six of the same BEA operational activities (e.g., conducting 
physical inventory, delivering property and services) and three of 
these four programs support at least 18 additional operational 
activities (e.g., performing budgeting, managing receipt and 
acceptance). However, since the potential overlap among these and other 
programs was not assessed, these programs may be investing in 
duplicative functionality. Reasons for this were that the compliance 
guidance does not provide for such analyses to be conducted and 
programs have not been granted access rights to use this functionality 
in the compliance tool. 

* The program's compliance assessment did not address compliance 
against DON's enterprise architecture, which is one of the biggest 
members of the federated BEA. This is particularly important given that 
DOD's approach to fully satisfying the architecture requirements of the 
Fiscal Year 2005 Defense Authorization Act is to develop and use a 
federated architecture in which component architectures are to provide 
the additional details needed to supplement the thin layer of corporate 
policies, rules, and standards included in the corporate BEA.[Footnote 
31] As we recently reported,[Footnote 32] DON's enterprise architecture 
is not mature because, among other things, it is missing a sufficient 
description of its current and future environments in terms of business 
and information/data. However, certain aspects of an architecture 
nevertheless exist and, according to DON CIO officials, these aspects 
will be leveraged in its efforts to develop a complete enterprise 
architecture. For example, the FORCEnet architecture is intended to 
document Navy's technical infrastructure. Therefore, opportunities 
exist for DON to assess its programs in relation to these architecture 
products, and to understand where its programs are exposed to risks 
because products do not exist, are not mature, or at odds with other 
Navy programs. According to DOD officials, compliance with the DON 
architecture was not assessed because DOD compliance policy is limited 
to compliance with the corporate BEA, and a number of aspects of the 
DON enterprise architecture have yet to be sufficiently developed. 

* The program's compliance assessment was not validated by DON or DOD 
investment oversight and decision-making authorities. More 
specifically, neither the DOD IRBs nor the DBSMC, nor the BTA in 
supporting both of these investment oversight and decision-making 
authorities, reviewed the program's assessments. According to BTA 
officials, under DOD's tiered approach to investment accountability, 
these entities are not responsible for validating programs' compliance 
assessments. Rather, this is a component responsibility, and thus they 
rely on the military departments and defense agencies to validate the 
assessments. 

However, DON Office of the CIO, which is responsible for precertifying 
investments as compliant before they are reviewed by the IRB, did not 
validate any of the program's compliance assessments. According to 
Office of the CIO officials, they rely on Functional Area Managers to 
validate a program's compliance assessments. However, no DON policy or 
guidance exists that describes how the Functional Area Managers should 
conduct such validations. CIO officials stated that this is because 
these authorities do not have the resources that they need to validate 
the assessments, and because a number of aspects of the DON 
architecture are not yet sufficiently developed. 

Validation of program assessments is further complicated by the absence 
of information captured in the assessment tool about what program 
documentation or other source materials were used by the program office 
in making its compliance determinations. Specifically, the tool is only 
configured, and thus was only used, to capture the results of a 
program's comparison of program architecture products to BEA products. 
Thus, it was not used to capture the system products used in making 
these determinations. 

The limitations in existing BEA compliance-related policy and guidance, 
the supporting compliance assessment tool, and the federated BEA, put 
programs like Navy ERP at increased risk of being defined and 
implemented in a way that does not sufficiently ensure interoperability 
and avoid duplication and overlap. We recently completed a review 
examining multiple programs' compliance with the federated BEA, 
including Navy ERP, for the Senate Armed Services Committee, 
Subcommittee on Readiness and Management Support. We addressed the 
architectural compliance guidance, tool, and validation limitations as 
part of this review.[Footnote 33] 

Investment in Navy ERP Has Been Economically Justified, but Important 
Estimating Practices Were Not Implemented: 

The investment in Navy ERP has been economically justified on the basis 
of expected life cycle benefits that far exceed estimated life cycle 
costs. According to the program's benefit/cost analysis, Navy ERP will 
produce about $8.6 billion in estimated benefits for an estimated cost 
of about $2.4 billion over its 20-year life cycle. While these benefit 
estimates were not subject to any analysis of how uncertainty in 
assumptions and data could impact the estimates, as called for by 
relevant guidance, our examination of key uncertainty variables, such 
as the timing of legacy systems' retirement, showed that the savings 
impact would be relatively minor. However, the reliability of the cost 
estimate is limited because it was derived using several, but not all, 
key estimating practices. For example, the estimate was not grounded in 
a historical record of comparable data from similar programs and was 
not based on a reliable schedule baseline, which are both necessary to 
having a cost estimate that can be considered credible and accurate. 
These practices were not employed for various reasons, including DOD's 
lack of historical data from similar programs and the lack of an 
integrated master schedule for the program that includes all releases. 

Notwithstanding the fact that these limitations could materially 
increase the $2.4 billion cost estimate, it is nevertheless unlikely 
that these factors would increase the estimate to a level approaching 
the program's benefit expectations. Therefore, we have no reason to 
believe that Navy ERP will not produce a positive return on investment. 

Benefit Estimates Are Sufficiently Reliable Despite Absence of 
Uncertainty Analysis: 

Forecasting expected benefits over the life of a program is a key 
aspect of economically justifying an investment. The Office of 
Management and Budget (OMB) guidance[Footnote 34] advocates 
economically justifying investments on the basis of expected benefits, 
costs, and risks. Since estimates of benefits can be uncertain because 
of the imprecision in both the underlying data and modeling assumptions 
used, the guidance also provides for analyzing and reporting the 
effects of this uncertainty. By doing this, informed investment 
decision making can occur through the life of the program, and a 
baseline can be established against which to compare the accrual of 
actual benefits from deployed system capabilities. 

The most recent economic analysis, dated August 2007, includes 
monetized benefit estimates for fiscal years 2004-2023, in three key 
areas--about $2.7 billion in legacy system cost savings, $3.3 billion 
in cost savings from inventory reductions, and $2.7 billion in cost 
savings from labor productivity improvements. Collectively, these 
benefits total about $8.6 billion.[Footnote 35] 

The program office calculated expected benefits in terms of cost 
savings, which is consistent with established practices and guidance. 
For example, the program is to result in the retirement of 138 legacy 
systems (including the 4 pilot systems) between fiscal years 2005 and 
2015, and the yearly maintenance costs for a single system are expected 
to be as high as about $39 million.[Footnote 36] According to relevant 
guidance, cost saving estimates should also be analyzed in terms of how 
uncertainty in assumptions and data could impact them. However, the 
program office did not perform such uncertainty analysis. According to 
program officials, uncertainty analysis is not warranted because they 
have taken and continue to take steps to validate the assumptions and 
the data, such as using the latest budget data associated with the 
legacy systems, and monitoring changes to the systems' retirement 
dates. While these steps are positive, they do not eliminate the need 
for uncertainty analysis. Accordingly, we assessed key uncertainty 
variables, such as the timing of the legacy systems' retirement, and 
found that the retirement dates of some of these systems have changed 
since the estimate was prepared, due to, among other things, schedule 
delays in the program. While the inherent uncertainty in these dates 
would reduce expected savings (e.g., only $11 million based on the 134 
legacy systems that we examined),[Footnote 37] the reduction would be 
small relative to a total benefit estimate of $8.6 billion. 

The Cost Estimate Was Not Reliably Derived: 

A reliable cost estimate is a key variable in calculating return on 
investment, and it provides the basis for informed investment decision 
making, realistic budget formulation and program resourcing, meaningful 
progress measurement, proactive course correction, and accountability 
for results. According to OMB,[Footnote 38] programs must maintain 
current and well-documented cost estimates, and these estimates must 
encompass the full life cycle of the program. OMB states that 
generating reliable cost estimates is a critical function necessary to 
support OMB's capital programming process. Without reliable estimates, 
programs are at increased risk of experiencing cost overruns, missed 
deadlines, and performance shortfalls. 

Our research has identified a number of practices that are the basis of 
effective program cost estimating. We have issued guidance that 
associates these practices with four characteristics of a reliable cost 
estimate.[Footnote 39] Specifically, these four characteristics are as 
follows: 

* Comprehensive: The cost estimates should include both government and 
contractor costs over the program's full life cycle, from the inception 
of the program through design, development, deployment, and operation 
and maintenance to retirement. They should also provide an appropriate 
level of detail to ensure that cost elements are neither omitted nor 
double counted and include documentation of all cost-influencing ground 
rules and assumptions. 

* Well-documented: The cost estimates should have clearly defined 
purposes and be supported by documented descriptions of key program or 
system characteristics (e.g., relationships with other systems, 
performance parameters). Additionally, they should capture in writing 
such things as the source data used and their significance, the 
calculations performed and their results, and the rationale for 
choosing a particular estimating method or reference. Moreover, this 
information should be captured in such a way that the data used to 
derive the estimate can be traced back to, and verified against, their 
sources. The final cost estimate should be reviewed and accepted by 
management on the basis of confidence in the estimating process and the 
estimate produced by the process. 

* Accurate: The cost estimates should provide for results that are 
unbiased and should not be overly conservative or optimistic (i.e., 
they should represent most likely costs). In addition, the estimates 
should be updated regularly to reflect material changes in the program, 
and steps should be taken to minimize mathematical mistakes and their 
significance. Among other things, the estimate should be grounded in a 
historical record of cost estimating and actual experiences on 
comparable programs. 

* Credible: The cost estimates should discuss any limitations in the 
analysis performed due to uncertainty or biases surrounding data or 
assumptions. Further, the estimates' derivation should provide for 
varying any major assumptions and recalculating outcomes based on 
sensitivity analyses, and their associated risks and inherent 
uncertainty should be disclosed. Also, the estimates should be verified 
based on cross-checks using other estimating methods and by comparing 
the results with independent cost estimates. 

The $2.4 billion life cycle cost estimate for Navy ERP reflects many of 
the practices associated with a reliable cost estimate, including all 
practices associated with being comprehensive and well-documented, and 
several related to being accurate and credible (see table 5). However, 
several important practices related to accuracy and credibility were 
not performed. To be reliable, a cost estimate should be comprehensive, 
well-documented, accurate, and credible. 

Table 5: Summary of Cost Estimating Characteristics That the Cost 
Estimate Satisfies: 

Characteristic of reliable estimates: Comprehensive; 
Satisfied?[A]: Yes. 

Characteristic of reliable estimates: Well-documented; 
Satisfied?[A]: Yes. 

Characteristic of reliable estimates: Accurate; 
Satisfied?[A]: Partially. 

Characteristic of reliable estimates: Credible; 
Satisfied?[A]: Partially. 

Source: GAO analysis of DON data. 

[A] "Yes" means that the program office provided documentation 
demonstrating satisfaction of the criterion. "Partially" means that the 
program office provided documentation demonstrating satisfaction of 
part of the criterion. "No" means that the program office has yet to 
provide documentation demonstrating satisfaction of the criterion. 

[End of table] 

The cost estimate is comprehensive because it includes both the 
government and contractor costs specific to development, acquisition 
(nondevelopment), implementation, and operations and support over the 
program's 20-year life cycle. Moreover, the estimate clearly describes 
how the various subelements are aggregated to produce amounts for each 
cost category, thereby ensuring that all pertinent costs are included, 
and no costs are double counted. Finally, cost-influencing ground rules 
and assumptions, such as the program's schedule, labor rates, and 
inflation rates are documented. 

The cost estimate is also well-documented in that the purpose of the 
cost estimate is clearly defined, and the technical baseline includes, 
among other things, the hardware components and planned performance 
parameters. Furthermore, the calculations and results used to derive 
the estimate are documented, including descriptions of the 
methodologies used and evidence of traceability back to source data 
(e.g., vendor quotes, salary tables). Also, the cost estimate was 
reviewed by the Naval Center for Cost Analysis and the Office of the 
Secretary of Defense, Director for Program Analysis and Evaluation, 
which adds a level of confidence in the estimating process and the 
estimate produced. 

However, the estimate lacks accuracy because not all important 
practices related to this characteristic were performed. Specifically, 
while the estimate is grounded in documented assumptions (e.g., 
hardware refreshment every 5 years) and periodically updated to reflect 
changes to the program, it is not adequately grounded in historical 
experience with comparable programs. While the program office did 
leverage historical cost data from the Navy ERP pilot programs, program 
officials told us that the level of cost accounting on these programs 
did not provide sufficient data. As stated in our guide, estimates 
should be based on historical records of cost and schedule estimates 
from comparable programs, and such historical data should be maintained 
and used for evaluation purposes and future estimates on comparable 
programs. The importance of doing so is evident by the fact that Navy 
ERP's cost estimate has increased by about $570 million since fiscal 
year 2003, which program officials attributed to, among other things, 
site implementation costs (e.g., training and converting legacy system 
data) not included in the original cost estimate, schedule delays, and 
the lack of historical data from similar ERP programs. 

This lack of cost data for large-scale ERP programs is, in part, due to 
DOD not having a standardized cost element structure for these programs 
that can be used for capturing actual cost data, which is a 
prerequisite to capturing and maintaining the kind of historical data 
that can inform cost estimates on similar programs. This means that 
programs like Navy ERP will not be able to ground their cost estimates 
in actual costs from comparable programs. According to officials with 
the Defense Cost and Resource Center,[Footnote 40] such cost element 
structures are needed, along with a requirement for programs to report 
on their costs, but approval and resources have yet to be gained for 
either these structures or the reporting of their costs. We recently 
completed work that addressed standardization of DOD's ERP cost element 
structure and maintenance of a database for historical ERP cost data 
for use on ERP programs.[Footnote 41] 

Compounding the estimate's limited accuracy are limitations in its 
credibility. Specifically, while the estimate satisfies some of the key 
practices for a credible cost estimate (e.g., confirming key cost 
drivers, performing sensitivity analyses,[Footnote 42] and having an 
independent cost estimate prepared by the Naval Center for Cost 
Analysis that was within 11 percent of the program's estimate), the 
program lacks a reliable schedule baseline, which is a key component of 
a reliable cost estimate because it serves as the basis for future work 
to be performed. Other factors that limit confidence in the cost 
estimate's accuracy are (1) past increases in the program's cost 
estimate (as discussed earlier) and (2) trends in EVM data (as 
discussed later). Taken together, the program's cost estimate is not 
sufficiently credible and accurate and thus not reliable. 

While important cost estimating practices were not implemented, it is 
nevertheless unlikely that these limitations would materially increase 
the $2.4 billion cost estimate to a level approaching the program's 
$8.6 billion benefit expectations. 

Earned Value Management Has Not Been Effectively Implemented: 

Measuring and reporting progress against cost and schedule commitments 
(i.e., baselines) is a vital element of effective program management. 
EVM provides a proven means for measuring such progress and thereby 
identifying potential cost overruns and schedule delays early, when 
their impact can be minimized. 

To its credit, the program has elected to implement program-level EVM, 
which is a best practice that has rarely been implemented in the 
federal government. In doing so, however, basic EVM activities have not 
been executed. In particular, an integrated baseline review, which is 
to verify that the program's cost and schedule are reasonable given the 
program's scope of work and associated risks, has not been performed. 
Moreover, other accepted industry standards have not been sufficiently 
implemented, and surveillance of EVM implementation by an entity 
independent of the program office has not occurred. Not performing 
these important practices has contributed to the cost overruns and 
lengthy schedule delays already experienced on Release 1.0, and they 
will likely result in more. In fact, our analysis of the latest 
estimate to complete just the budgeted development work for all three 
releases, which is about $844 million, shows that this estimate will 
most likely be exceeded by about $152 million. 

The Program Has Elected to Implement Program-Level EVM: 

As we previously reported,[Footnote 43] EVM offers many benefits when 
done properly. In particular, it allows performance to be measured, and 
it serves as an early warning system for deviations from plans. It, 
therefore, enables a program office to mitigate the risks of cost and 
schedule overruns. OMB policy recognizes the use of EVM as an important 
part of program management and decision making.[Footnote 44] 

Implementing EVM at the program level rather than just the contract 
level is considered a best practice, and OMB recently began requiring 
it to measure how well a program's approved cost, schedule, and 
performance goals are being met. According to OMB, integrating 
government and contractor cost, schedule, and performance status should 
result in better program execution through more effective management. 
In addition, integrated EVM data can be used to better justify budget 
requests. 

To minimize the risk associated with its decision to transition 
responsibility for Navy ERP system integration from the contractor to 
the government and to improve cost and schedule performance, the 
program office elected in October 2006 to perform EVM at the program 
level. We support the use of program-level EVM. However, if not 
implemented effectively, this program-level approach will be of little 
value. 

An Integrated Baseline Review Has Not Been Performed: 

A fundamental aspect of effective EVM is the development of a 
performance measurement baseline (PMB), which represents the cumulative 
value of planned work and serves as the baseline against which 
variances are calculated. According to relevant best practice guidance, 
a PMB consists of: 

* a complete work breakdown structure; 

* a complete integrated master schedule, and: 

* accurate budgets for all planned work.[Footnote 45] 

To validate the PMB, an integrated baseline review is performed to 
obtain stakeholder agreement on the baseline. According to DOD guidance 
and best practices, such a review should be held within 6 months of a 
contract award and conducted on an as needed basis throughout the life 
of a program to ensure that the baseline reflects (1) all tasks in the 
statement of work, (2) adequate resources (staff and materials) to 
complete the tasks, and (3) integration of the tasks into a well-
defined schedule. Further, the contract performance reports that are to 
be used to monitor performance against the PMB should be validated 
during the integrated baseline review.[Footnote 46] 

The program office has satisfied some of the prerequisites for having a 
reliable PMB, such as developing a work breakdown structure and 
specifying the contract performance reports that are to be used to 
monitor performance. However, it has not conducted an integrated 
baseline review. Specifically, a review was not conducted for Release 
1.0, even though the contract was finalized about 30 months ago 
(January 2006). Also, while the review for Release 1.1 was recently 
scheduled for August 2008, this is 8 months later than when such a 
review should be held, according to DOD guidance and best practices. 
This means that the reasonableness of the program's scope and schedule 
relative to the program risks has not been assured, and has likely 
been, and will likely continue to be a primary contributor to future 
cost increases and schedule delays. 

According to program officials, a review was not performed on the first 
release because development of this release was largely complete by the 
time the program office established the underlying capabilities needed 
to perform program-level EVM. In addition, program officials stated 
that an integrated baseline review has yet to be performed on the other 
two releases because their priority has been on deploying and 
stabilizing the first release. In our view, not assuring the validity 
of the PMB precludes effective implementation of EVM. Until a review is 
conducted, DOD will not have reasonable assurance that the program's 
scope and schedule are achievable, and thus, additional cost and 
schedule overruns are likely. 

Industry EVM Standards Have Not Been Fully Implemented And 
Independently Surveilled: 

In 1996, DOD adopted industry EVM guidance[Footnote 47] that identifies 
32 essential practices organized into five categories: (1) 
organization; (2) planning, scheduling and budgeting; (3) accounting; 
(4) analysis and management reports; and (5) revisions and data 
maintenance. DOD requires that all programs' implementation of EVM 
undergo a compliance audit against the 32 industry practices. In 
addition, DOD policy and guidance[Footnote 48] state that independent 
surveillance of EVM should occur over the life of the program to 
guarantee the validity of the performance data and ensure that EVM is 
being used effectively to manage cost, schedule, and technical 
performance. 

On Navy ERP, compliance with the 32 accepted industry practices has not 
been verified, and surveillance of EVM by an independent entity has not 
occurred. Therefore, the program does not have the required basis for 
ensuring that EVM is being effectively implemented on Navy ERP. 
According to program officials, surveillance was performed by NAVAIR 
for Release 1.0. However, NAVAIR officials said that they did not 
perform such surveillance because they did not receive the Release 1.0 
cost performance data needed to do so. Program officials also stated 
that DON's Center for Earned Value Management[Footnote 49] has 
conducted an initial assessment of their EVM management system, and 
that they intend to have the Center perform surveillance. However, they 
did not have a plan for accomplishing this. Until compliance with the 
standards is verified and continuous surveillance occurs, and 
deviations are addressed, the program will likely continue to 
experience cost overruns and schedule delays. 

Estimated Schedule Baseline Was Not Derived In Accordance With All Key 
Schedule Estimating Practices: 

The success of any program depends in part on having a reliable 
schedule of when the program's work activities will occur, how long 
they will take, and how they are related to one another. As such, the 
schedule not only provides a road map for the systematic execution of a 
program but also provides the means by which to estimate costs, gauge 
progress, identify and address potential problems, and promote 
accountability. Our research has identified nine practices associated 
with effective schedule estimating.[Footnote 50] These practices are 
(1) capturing key activities, (2) sequencing key activities, (3) 
establishing the duration of key activities, (4) assigning resources to 
key activities, (5) integrating key activities horizontally and 
vertically, (6) establishing the critical path for key activities, (7) 
identifying "float time"[Footnote 51] between key activities, (8) 
distributing reserves to high-risk activities, and (9) performing a 
schedule risk analysis. 

The program's estimated schedule was developed using some of these 
practices, but several key practices were not fully employed that are 
fundamental to having a schedule that provides a sufficiently reliable 
basis for estimating costs, measuring progress and forecasting 
slippages. On the positive side, the schedule for the first two 
releases captures key activities and their durations and is integrated 
horizontally and vertically, meaning that multiple teams executing 
different aspects of the program can effectively work to the same 
master schedule. Moreover, for these two releases, the program has 
established float time between key activities and distributed schedule 
reserve to high-risk activities. However, the program has not 
adequately sequenced and assigned resources to key program activities. 
Moreover, the estimated schedule for the first increment is not 
grounded in an integrated master schedule of all the releases, and thus 
the schedule for this increment does not reflect the program's critical 
path of work that must be performed to achieve the target completion 
date. Also, it does not reflect the results of a schedule-risk analysis 
across all three releases with schedule reserve allocated to high-risk 
activities because such risks were not examined. See table 6 for the 
results of our analyses relative to each of the nine practices. 

Table 6: Satisfaction of Schedule Estimating Key Practices: 

Practice: Capturing key activities; 
Explanation: The schedule should reflect all key activities (e.g., 
steps, events, outcomes) as defined in the program's work breakdown 
structure, to include activities to be performed by both the government 
and its contractors; 
Satisfied?[A]: Yes; 
GAO analysis: The program's estimated schedules for the first two 
releases reflect both government and contractor activities, such as 
development and testing of the software components, as well as key 
milestones for measuring progress. 

Practice: Sequencing key activities; 
Explanation: The schedule should be planned so that it can meet 
critical program dates. To meet this objective, key activities need to 
be logically sequenced in the order that they are to be carried out. In 
particular, activities that must be finished prior to the start of 
other activities (i.e., predecessor activities), as well as activities 
that cannot begin until other activities are completed (i.e., successor 
activities) should be identified. By doing so, interdependencies among 
activities that collectively lead to the accomplishment of events or 
milestones can be established and used as a basis for guiding work and 
measuring progress; 
Satisfied?[A]: Partially; 
GAO analysis: The schedules for the first two releases include the 
logical sequencing of most, but not all, activities. For example, 234 
of 2,445 activities in the Release 1.1 schedule were not sequenced 
properly. By not identifying the correct interdependencies and properly 
sequencing key activities, the schedule may not facilitate the 
meaningful tracking of progress. 

Practice: Establishing the duration of key activities; 
Explanation: The schedule should realistically reflect how long each 
activity will take to execute. In determining the duration of each 
activity, the same rationale, historical data, and assumptions used for 
cost estimating should be used for schedule estimating. Further, these 
durations should be as short as possible, and they should have specific 
start and end dates. Excessively long periods needed to execute an 
activity should prompt further decomposition of the activity so that 
shorter execution durations will result; 
Satisfied?[A]: Yes; 
GAO analysis: The schedules for the first two releases establish 
realistic durations of key activities. For example, the schedule for 
Release 1.1 is based on historical data from Release 1.0, which 
provides a level of confidence that the durations are reasonable. 

Practice: Assigning resources to key activities; 
Explanation: The schedule should reflect what resources (i.e., labor, 
material, and overhead) are needed to do the work, whether all required 
resources will be available when they are needed, and whether any 
funding or time constraints exist; 
Satisfied?[A]: Partially; 
GAO analysis: The schedules for the first two releases include the 
allocation of resources, such as personnel, to activities, but it does 
not reflect whether all resources will be available when they are 
needed because the identified resources are shared across the three 
releases. Restated, personnel are assigned to activities across 
multiple releases, each of which is managed according to a separate 
schedule. Therefore, if one of the schedules were to be delayed, the 
other schedules that required the same resources would likely also be 
delayed. 

Practice: Integrating key activities horizontally and vertically; 
Explanation: The schedule should be horizontally integrated, meaning 
that it should link the products and outcomes associated with already 
sequenced activities. These links are commonly referred to as 
"handoffs" and serve to verify that activities are arranged in the 
right order to achieve aggregated products or outcomes. The schedule 
should also be vertically integrated, meaning that traceability exists 
among varying levels of activities and supporting tasks and subtasks. 
Such mapping or alignment among levels enables different groups to work 
to the same master schedule; 
Satisfied?[A]: Yes; 
GAO analysis: The schedules for the first two releases are both 
horizontally and vertically integrated, meaning that the activities 
across the multiple teams are arranged in the right order to achieve 
aggregated products or outcomes. In addition, traceability exists among 
varying levels of activities, which allows multiple teams (i.e., 
development, testing) to work to the same master schedule. 

Practice: Establishing the critical path for key activities; 
Explanation: Using scheduling software, the critical path--the longest 
duration path through the sequenced list of key activities--should be 
identified. The establishment of a program's critical path is necessary 
for examining the effects of any activity slipping along this path. 
Potential problems that might occur along or near the critical path 
should also be identified and reflected in the scheduling of the time 
for high-risk activities (see next practice); 
Satisfied?[A]: Partially; 
GAO analysis: While the program has established the critical path for 
the first two releases separately, it has not done so for the entire 
first increment. This is because the program maintains separate 
schedules for each release, and in doing so, cannot identify the 
longest duration of tasks through the entire first increment. Without 
doing so, the effects of any slippage along the critical path on future 
releases cannot be determined. 

Practice: Identifying "float time" between key activities; 
Explanation: The schedule should identify float time--the time that a 
predecessor activity can slip before the delay affects successor 
activities--so that schedule flexibility can be determined. As a 
general rule, activities along the critical path typically have the 
least amount of float time; 
Satisfied?[A]: Yes; 
GAO analysis: The schedules for the first two releases identify float 
time between key activities. 

Practice: Distributing reserves to high-risk activities; 
Explanation: The baseline schedule should include a buffer or a reserve 
of extra time. Schedule reserve for contingencies should be calculated 
by performing a schedule risk analysis (see next practice). As a 
general rule, the reserve should be applied to high-risk activities, 
which are typically found along the critical path; 
Satisfied?[A]: Partially; 
GAO analysis: The schedule allocates reserve for high-risk activities 
on the critical path for Release 1.0. However, because the program has 
not established a critical path for the first increment, it cannot 
allocate reserve for the high-risk activities on the entire program's 
critical path. 

Practice: Schedule risk analysis should be performed; 
Explanation: A schedule risk analysis should be performed using 
statistical techniques to predict the level of confidence in meeting a 
program's completion date. This analysis focuses not only on critical 
path activities but also on activities near the critical path, since 
they can potentially affect program status; 
Satisfied?[A]: Partially; 
GAO analysis: A schedule risk analysis on the entire program was not 
performed. A schedule risk analysis has been done on Release 1.0, and 
the program office plans to do one for Release 1.1. However, without 
analyzing the risks associated with the program's entire schedule, the 
program cannot determine the level of confidence in meeting the 
program's completion date. 

Source: GAO analysis of DON data. 

[A] "Yes" means that the program provided documentation demonstrating 
satisfaction of the practice. "Partially" means that the program 
provided documentation demonstrating satisfaction of part of the 
practice. "No" means that the program has yet to provide documentation 
demonstrating satisfaction of the practice. 

[End of table] 

According to program documentation, they have plans to address the 
logical sequencing of activities (to ensure that it reflects how work 
is to be performed), but program officials stated that they do not plan 
to combine all three releases into a single integrated master schedule 
for the entire first increment of the program because doing so would 
produce an overly complex and nonexecutable schedule involving as many 
as 15,000 activities. However, our research of and experience in 
evaluating major programs' use of EVM and integrated master schedules 
show that while large, complex programs necessitate schedules involving 
thousands of activities, successful programs ensure that their 
schedules integrate these activities. In our view, not adequately 
performing these practices does not allow the program to effectively 
assign resources, identify the critical path, and perform a schedule 
risk analysis that would allow it to understand, disclose, and 
compensate for its schedule risks. This means that the program is not 
well-positioned to understand progress and forecast its impact. To 
illustrate, the program recently experienced delays in deploying its 
first release at NAVAIR, which according to a recent operational test 
and evaluation report[Footnote 52] has significantly affected the 
schedule's critical path. These schedule impacts are because resources 
supporting the deployment at NAVAIR began to shift to the next 
scheduled deployment site and thus are no longer available to resolve 
critical issues at NAVAIR. Since the schedule baseline is not 
integrated across all releases, the impact of this delay on other 
releases, and thus the program as a whole, cannot be readily 
determined. 

Trends in EVM Data Show Pattern of Cost Overruns and Schedule 
Slippages: 

Program data show a pattern of actual cost overruns and schedule delays 
between January 2007 and May 2008. Moreover, our analysis of the data 
supports a most likely program cost growth of about $152 million to 
complete all three releases. 

Differences from the PMB are measured in both cost and schedule 
variances.[Footnote 53] Positive variances indicate that activities are 
costing less or are completed ahead of schedule. Negative variances 
indicate that activities are costing more or are falling behind 
schedule. These cost and schedule variances can then be used in 
forecasting the cost and time needed to complete the program. 

Based on program-provided data for the first increment over a 17-month 
period ending May 2008, the program has experienced negative cost 
variances. Specifically, while these cost variances have fluctuated 
during this period, they have consistently been negative. (See fig. 4.) 
Moreover, our analysis of the cost to complete just the budgeted 
development work (also known as the PMB) for all three releases, which 
is about $844 million,[Footnote 54] will be exceeded by between about 
$102 million and $316 million, with a most likely overrun of about $152 
million.[Footnote 55] In contrast, the program office reports that the 
overrun at completion will be $55 million but has yet to provide us 
with documentation supporting this calculation. Moreover, our 
calculation does not reflect the recent problems discovered during the 
operational test and evaluation at NAVAIR and thus the overrun is 
likely to be higher. 

Figure 4: Cumulative Cost Variance for Navy ERP over a 17-Month Period: 

[See PDF for image] 

This figure is a line graph depicting the following data: 

Date: January, 2007: 
Cost variance: $2.567 million. 

Date: February, 2007: 
Cost variance: $0.387 million. 

Date: March, 2007: 
Cost variance: -$1.628 million. 

Date: April, 2007: 
Cost variance: $0.905 million. 

Date: May, 2007: 
Cost variance: -$4.047 million. 

Date: June, 2007: 
Cost variance: -$15.995 million. 

Date: July, 2007: 
Cost variance: -$16.36 million. 

Date: August, 2007: 
Cost variance: -$15.2 million. 

Date: September, 2007: 
Cost variance: -$10.544 million. 

Date: October, 2007: 
Cost variance: -$12.828 million. 

Date: November, 2007: 
Cost variance: -$24.932 million. 

Date: December, 2007: 
Cost variance: -$24.747 million. 

Date: January, 2008: 
Cost variance: -$19.044 million. 

Date: February, 2008: 
Cost variance: -$19.073 million. 

Date: March, 2008: 
Cost variance: -$15.685 million. 

Date: April, 2008: 
Cost variance: -$15.171 million. 

Date: May, 2008: 
Cost variance: -$17.289 million. 

Source: GAO analysis based on Navy ERP data. 

[End of figure] 

During this same 17-month period, the program has experienced negative 
schedule variances and, since January 2008, they have almost doubled 
each month. Further, as of May 2008, the program had not completed 
about $24 million in scheduled work. (See fig. 5.) An inability to meet 
schedule performance is a frequent indication of future cost increases, 
as more spending is often necessary to resolve schedule delays. 

Figure 5: Cumulative Schedule Variance of the Navy ERP Program over a 
17-Month Period: 

[See PDF for image] 

This figure is a line graph depicting the following data: 

Date: January, 2007: 
Cost variance: -$1.258 million. 

Date: February, 2007: 
Cost variance: -$1.655 million. 

Date: March, 2007: 
Cost variance: -$1.885 million. 

Date: April, 2007: 
Cost variance: -$1.84 million. 

Date: May, 2007: 
Cost variance: -$2.384 million. 

Date: June, 2007: 
Cost variance: -$4.98 million. 

Date: July, 2007: 
Cost variance: -$6.938 million. 

Date: August, 2007: 
Cost variance: -$7.646 million. 

Date: September, 2007: 
Cost variance: -$4.542 million. 

Date: October, 2007: 
Cost variance: -$4.715 million. 

Date: November, 2007: 
Cost variance: -$1.863 million. 

Date: December, 2007: 
Cost variance: -$2.548 million. 

Date: January, 2008: 
Cost variance: -$4.504 million. 

Date: February, 2008: 
Cost variance: -$8.069 million. 

Date: March, 2008: 
Cost variance: -$10.93 million. 

Date: April, 2008: 
Cost variance: -$17.703 million. 

Date: May, 2008: 
Cost variance: -$25.62 million. 

Source: GAO analysis based on Navy ERP data. 

[End of figure] 

Because the program office has not performed important reliability 
checks, such as EVM validation and integrated baseline reviews, as 
discussed above, we cannot be certain that the PMB is reliable (i.e., 
reflects all the work to be done and has identified all the risks). As 
a result, the overrun that we are forecasting could be higher. 

[See PDF for image] 

[End of figure] 

By not executing basic EVM practices, the program has and will likely 
continue to experience cost and schedule shortfalls. Until the program 
office implements these important EVM practices, it will likely not be 
able to track actual program costs and schedules close to estimates. 

Important Requirements Management Activity Has Been Effectively 
Implemented: 

Well-defined and managed requirements are recognized by DOD guidance as 
essential, and they can be viewed as a cornerstone of effective system 
acquisition. One aspect of effective requirements management is 
requirements traceability.[Footnote 56] By tracing requirements both 
backward from system requirements to higher level business or 
operational requirements and forward to system design specifications 
and test plans, the chances of the deployed product satisfying 
requirements is increased, and the ability to understand the impact of 
any requirement changes and thus make informed decisions about such 
changes, is enhanced. 

The program office is effectively implementing requirements 
traceability for its 1,733 Release 1.0 system requirements. To verify 
this traceability, we randomly selected and analyzed 60 of the 1,733 
system requirements and confirmed that 58 of the 60 were traceable both 
backward to higher level requirements and forward to design 
specifications and test results. The remaining 2 had been allocated to 
the other releases, and thus we also confirmed the program's ability to 
maintain traceability between product releases. In doing so, the 
program utilized a tool called DOORS, which if implemented properly, 
allows each requirement to be linked from its most conceptual 
definition to its most detailed definition, as well as to design 
specifications and test cases. In effect, the tool maintains the 
linkages among requirement documents, design documents, and test cases 
even if requirements change. 

If DON continues to effectively implement requirements traceability, it 
will increase the chances that system requirements will be met by the 
deployed system. 

A Risk Management Process Has Been Defined, but All Risks Have Not Been 
Effectively Mitigated: 

Proactively managing program risks is a key acquisition management 
control and, if defined and implemented properly, it can increase the 
chances of programs delivering promised capabilities and benefits on 
time and within budget. To the program office's credit, it has defined 
a risk management process that meets relevant guidance. However, it has 
not effectively implemented the process for all identified risks. As a 
result, these risks have not been proactively mitigated and either have 
contributed to cost and schedule shortfalls, or could potentially 
contribute to such shortfalls. 

DOD acquisition management guidance, as well as other relevant guidance 
advocates identifying facts and circumstances that can increase the 
probability of an acquisition's failing to meet cost, schedule, and 
performance commitments and then taking steps to reduce the probability 
of their occurrence and impact.[Footnote 57] In brief, effective risk 
management consists of: (1) establishing a written plan for managing 
risks; (2) designating responsibility for risk management activities; 
(3) encouraging program-wide participation in the identification and 
mitigation of risks; (4) defining and implementing a process that 
provides for the identification, analysis, and mitigation of risks; and 
(5) examining the status of identified risks in program milestone 
reviews. 

The program office has developed a written plan for managing risks and 
established a process that together provide for the above-cited risk 
management practices. Moreover, it has largely followed its plan and 
process as per the following examples: 

* The program manager has been assigned overall responsibility for 
managing risks and serves as the chair of the risk management board. 
[Footnote 58] Also, a functional team lead (i.e., subject matter 
expert) is assigned responsibility for analyzing and mitigating each 
identified risk. 

* Program-wide participation in the identification, analysis, and 
mitigation of risks is encouraged. Specifically, a manager for each 
release is responsible for providing risk management guidance to the 
staff, which includes staff identification and analysis of risks. Also, 
according to the program office's risk management plan, all program 
personnel can submit a risk for approval. In addition, stakeholders 
participate in risk management activities during acquisition milestone 
reviews. 

* The program office has identified and categorized individual risks. 
As of June 2008, the risk database contained 15 active risks--3 high, 8 
medium, and 4 low.[Footnote 59] 

* Program risks are considered during program milestone reviews. For 
example, during the program's critical design review, which is a key 
event of the system development and demonstration phase, key risks 
regarding implementing new business processes and legacy system changes 
were discussed. Furthermore, the program manager receives a monthly 
risk report that describes the status of program risks. 

However, the program office has not consistently followed other aspects 
of its process. In particular, it has not effectively implemented steps 
for mitigating the risks associated with (1) converting data from 
NAVAIR's legacy systems to run on Navy ERP and (2) positioning NAVAIR 
for adopting the new business processes embedded in Navy ERP. As we 
have previously reported, it is important for organizations that are to 
operate and use commercial off-the-shelf software products, such as 
Navy ERP, to proactively manage and position themselves for the 
organizational impact of introducing functionality embedded in the 
commercial products. If they do not, the organization's performance 
will suffer.[Footnote 60] 

To the program office's credit, it identified numerous risks associated 
with data conversion and organizational change management and developed 
and implemented strategies that were intended to mitigate these risks. 
However, it closed these risks even though they were never effectively 
mitigated, as evidenced by the results of recently completed DON 
operational test and evaluation. According to the June 2008 operational 
test and evaluation report for NAVAIR, significant problems relating to 
both legacy system data conversion and adoption of new business 
processes were experienced. The report states that these problems have 
contributed to increases in the costs to operate the system, including 
unexpected manual effort. It further states that these problems have 
rendered the deployed version not operationally effective and that 
deployment of the system to other sites should not occur until the 
change management process has been analyzed and improved. It also 
attributed the realization of the problems to the program office and 
NAVAIR not having adequately engaged and communicated early with each 
other to coordinate and resolve differences in organizational 
perspectives and priorities and provide intensive pre-deployment 
preparation and training. Program officials acknowledged these 
shortcomings and attributed them to their limited authority over the 
commands. In this regard, they have previously surfaced these risks 
with department oversight and approval authorities, but actions were 
not taken by these authorities that ensured that the risks were being 
effectively mitigated. 

Beyond not effectively mitigating these risks, the program office has 
not ensured that all risks are captured in the risk inventory. For 
example, the inventory does not include the risks described in this 
report that are associated with not having adequately demonstrated the 
program's alignment to the federated BEA and not having implemented 
program-level EVM in a manner that reflects industry practices. This 
means that these risks are not being disclosed or mitigated. 

By not effectively addressing all risks associated with the program, 
these risks can and have become problems that contribute to cost and 
schedule shortfalls. Until all significant risks are proactively 
addressed, to include ensuring that all associated mitigation steps are 
implemented and that they accomplished their intended purpose, the 
program will likely experience further problems at subsequent 
deployment sites. 

Conclusions: 

DOD's success in delivering large-scale business systems, such as Navy 
ERP, is in large part determined by the extent to which it employs the 
kind of rigorous and disciplined IT management controls that are 
reflected in department policies and related guidance. While 
implementing these controls does not guarantee a successful program, it 
does minimize a program's exposure to risk and thus the likelihood that 
it will fall short of expectations. In the case of Navy ERP, living up 
to expectations is important because the program is large, complex, and 
critical to addressing the department's long-standing problems related 
to financial transparency and asset visibility. 

The effectiveness to which key IT management controls have been 
implemented in Navy ERP varies, with one control and several aspects of 
others being effectively implemented, and others less so. Moreover, 
those controls that have not been effectively implemented have, in 
part, contributed to the sizable cost and schedule shortfalls 
experienced to date on the program. Unless this changes, more 
shortfalls can be expected. 

While the program office is primarily responsible for ensuring that 
effective IT management controls are implemented, other oversight and 
stakeholder organizations share responsibility. For example, even 
though the program has not demonstrated its alignment with the 
federated BEA, it nevertheless followed established DOD architecture 
compliance guidance and used the related compliance assessment tool in 
assessing and asserting its compliance. The root cause for not 
demonstrating compliance thus is not traceable to the program office 
but rather is due to, among other things, the limitations of the 
compliance guidance and tool, and the program's oversight entities not 
validating the compliance assessment and assertion. Also, the reason 
that the program's cost estimate was not informed by the cost 
experiences of other programs of the same size and scope is because DOD 
does not have a standard ERP cost element structure and has not 
maintained a historical database of costs for like programs to use. In 
contrast, effective implementation of other management controls, such 
as implementing EVM, requirements traceability, and risk management is 
the responsibility of the program office. 

All told, addressing the management control weaknesses requires the 
combined efforts of the various organizations that share responsibility 
for managing and overseeing the program. By doing so, the department 
can better assure itself that Navy ERP will optimally support its 
performance goals and will deliver promised capabilities and benefits 
on time and within budget. 

Because we recently completed work that more broadly addresses the 
above cited architectural alignment and comparable program cost data 
limitations, we are not making recommendations in this report for 
addressing them. 

Recommendations for Executive Action: 

To strengthen Navy ERP management control and better provide for the 
program's success, we are making the following recommendations: 

* To improve the reliability of Navy ERP benefit estimates and cost 
estimates, we recommend that the Secretary of Defense direct the 
Secretary of the Navy, through the appropriate chain of command, to 
ensure that future Navy ERP estimates include uncertainty analyses of 
estimated benefits, reflect the risks associated with not having cost 
data for comparable ERP programs, and are otherwise derived in full 
accordance with the other key estimating practices, and economic 
analysis practices discussed in this report. 

* To enhance Navy ERP's use of EVM, we recommend that the Secretary of 
Defense direct the Secretary of the Navy, through the appropriate chain 
of command, to ensure that (1) an integrated baseline review on the 
last two releases of the first increment is conducted, (2) compliance 
against the 32 accepted industry EVM practices is verified, and (3) a 
plan to have an independent organization perform surveillance of the 
program's EVM system is developed and implemented. 

* To increase the quality of the program's integrated master schedule, 
we recommend that the Secretary of Defense direct the Secretary of the 
Navy, through the appropriate chain of command, to ensure that the 
schedule (1) includes the logical sequencing of all activities, (2) 
reflects whether all required resources will be available when needed, 
(3) defines a critical path that integrates all three releases, (4) 
allocates reserve for the high-risk activities on the entire program's 
critical path, and (5) incorporates the results of a schedule risk 
analysis for all three releases and recalculates program cost and 
schedule variances to more accurately determine a most likely cost and 
schedule overrun. 

* To improve Navy ERP's management of program risks, we recommend that 
the Secretary of Defense direct the Secretary of the Navy, through the 
appropriate chain of command, to ensure that (1) the plans for 
mitigating the risks associated with converting data from legacy 
systems to Navy ERP and positioning the commands for adopting the new 
business processes embedded in the Navy ERP are re-evaluated in light 
of the recent experience with NAVAIR and adjusted accordingly, (2) the 
status and results of these and other mitigation plans' implementation 
are periodically reported to program oversight and approval 
authorities, (3) these authorities ensure that those entities 
responsible for implementing these strategies are held accountable for 
doing so, and (4) each of the risks discussed in this report are 
included in the program's inventory of active risks and managed 
accordingly. 

Agency Comments and Our Evaluation: 

In written comments on a draft of this report, signed by the Deputy 
Under Secretary of Defense (Business Transformation) and reprinted in 
appendix II, DOD stated that it concurred with two of our four 
recommendations and partially concurred with the remaining two. 
Further, it stated that it has taken steps to address some of our 
recommendations, adding that it is committed to implementing 
recommendations that contribute to the program's success. The 
department's comments relative to both of the recommendations that it 
partially concurred with, as well as additional comments, are discussed 
below. 

For our recommendation associated with improving the program's benefit 
and cost estimates, DOD concurred with two of the recommendation's 
three parts, but it did not concur with one part--ensuring that future 
cost estimates reflect the risk of not having cost data for comparable 
programs. While acknowledging that the program had limited cost data 
from comparable programs on which to base its cost estimate, DOD stated 
that an uncertainty analysis had been applied to the estimate to 
account for the risk associated with not having such data. The 
department further stated that actual experience on the program will 
continue to be used to refine the program's cost estimating 
methodology. While we support DOD's stated commitment to using actual 
program cost experience in deriving future estimates, we do not agree 
that the latest estimate accounted for the risk of not having cost data 
from comparable programs. We examined the uncertainty analysis as part 
of our review, and found that it did not recognize this risk. Moreover, 
DOD's comments offered no new evidence to the contrary. 

For our recommendation associated with improving the program's schedule 
estimating, DOD concurred with four of the recommendation's five parts, 
and partially concurred with one part--ensuring that the schedule 
defines a critical path that integrates all releases. In taking this 
position, the department stated that a critical path has been 
established for each release rather than across all three releases, and 
it attributes this to the size and complexity of the program. We do not 
take issue with either of these statements, as they are already 
recognized in our report. However, DOD offers no new information in its 
comments. Further, our report also recognizes that to be successful, 
large and complex programs that involve thousands of activities need to 
ensure that their schedules integrate these activities. In this regard, 
we support the department's commitment to explore the feasibility of 
implementing this part of our recommendation. 

In addition, while stating that it concurred with all parts of our 
recommendation associated with improving the program's use of EVM, DOD 
nevertheless provided additional comments as justification for having 
not conducted an integrated baseline review on Release 1.0. 
Specifically, it stated that when it rebaselined this release in 
December 2006, the release's development activities were essentially 
complete and the release was in the latter stages of testing. Further, 
it stated that the risks associated with the Release 1.0 schedule were 
assessed 3 months after this rebaselining, and these risks were 
successfully mitigated. To support this statement, it said that Release 
1.0 achieved its "Go-Live" as scheduled at NAVAIR. We do not agree with 
these comments for several reasons. First, at the time of the 
rebaselining, about 9 months of scheduled Release 1.0 development 
remained, and thus the release was far from complete. Moreover, the 
significance of the amount of work that remained, and still remains 
today on Release 1.0 is acknowledged in DOD's own comment that the 
scheduled integrated baseline review for Release 1.1 will also include 
remaining Release 1.0 work. Second, the Release 1.0 contract was 
awarded in January 2006, and DOD's own guidance requires that an 
integrated baseline review be conducted within 6 months of a contract's 
award. Third, although DOD states that the program achieved "Go-Live" 
as scheduled on October 1, 2007, the program achieved initial 
operational capability 7 months later than established in the December 
2006 baseline. 

In addition to these comments, the department also described actions 
under way or planned to address our recommendations. We support the 
actions described, as they are consistent with the intent of our 
recommendations. If fully and properly implemented, these actions will 
go a long way in addressing the management control weaknesses that our 
recommendations are aimed at correcting. 

We are sending copies of this report to interested congressional 
committees; the Director, Office of Management and Budget; the 
Congressional Budget Office; the Secretary of Defense; and the 
Department of Defense Office of the Inspector General. We also will 
make copies available to others upon request. In addition, the report 
will be available at no charge on the GAO Web site at [hyperlink, 
http://www.gao.gov]. 

If you or your staff have any questions about this report, please 
contact me at (202) 512-3439 or hiter@gao.gov. Contact points for our 
Offices of Congressional Relations and Public Affairs may be found on 
the last page of this report. GAO staff who made major contributions to 
this report are listed in appendix III. 

Signed by: 

Randolph C. Hite: 
Director, Information Technology Architecture and Systems Issues: 

[End of section] 

Appendix I: Objective, Scope, and Methodology: 

[End of section] 

Our objective was to determine whether the Department of the Navy is 
effectively implementing information technology management controls on 
the Navy Enterprise Resource Planning (Navy ERP) program. To accomplish 
this, we focused on the first increment of Navy ERP and the following 
management areas (1) architectural alignment, (2) economic 
justification, (3) earned value management (EVM), (4) requirements 
management, and (5) risk management. 

* To determine whether Navy ERP was aligned with the Department of 
Defense's (DOD) federated business enterprise architecture (BEA), we 
reviewed the program's BEA compliance assessments and system 
architecture products, as well as Versions 4.0, 4.1, and 5.0 of the 
BEA, and compared them with the BEA compliance requirements described 
in the Fiscal Year 2005 Defense Authorization Act[Footnote 61] and 
DOD's BEA compliance guidance, and we evaluated the extent to which the 
compliance assessments addressed all relevant BEA products. We also 
determined the extent to which the program-level architecture 
documentation supported the BEA compliance assessments. We obtained 
documentation, such as the BEA compliance assessments from the Navy ERP 
and Global Combat Support System--Marine Corps programs, as well as the 
Air Force's Defense Enterprise Accounting and Management System and Air 
Force Expeditionary Combat Support System programs. We then compared 
these assessments to identify potential redundancies or opportunities 
for reuse and determined if the compliance assessments examined 
duplication across programs, and if the tool that supports these 
assessments is being used to identify such duplication. In doing so, we 
interviewed program officials and officials from the Department of the 
Navy, Office of the Chief Information Officer and reviewed recent GAO 
reports to determine the extent to which the programs were assessed for 
compliance against the Department of the Navy enterprise architecture. 
We also interviewed program officials and officials from the Business 
Transformation Agency and the Department of the Navy, including the 
logistics Functional Area Manager, and obtained guidance documentation 
from these officials to determine the extent to which the compliance 
assessments were subject to oversight or validation. 

* To determine whether the program had economically justified its 
investment in Navy ERP, we reviewed the latest economic analysis to 
determine the basis for the cost and benefit estimates. This included 
evaluating the analysis against Office of Management and Budget 
guidance and GAO's Cost Assessment Guide.[Footnote 62] In doing so, we 
interviewed cognizant program officials, including the program manager 
and cost analysis team, regarding their respective roles, 
responsibilities, and actual efforts in developing and/or reviewing the 
economic analysis. We also interviewed officials at the Office of 
Program Analysis and Evaluation and Naval Center for Cost Analysis as 
to their respective roles, responsibilities, and actual efforts in 
developing and/or reviewing the economic analysis. We did not verify 
the validity of the source data used to calculate estimated benefits, 
such as those data used to determine the yearly costs associated with 
legacy systems planned for retirement. 

* To determine the extent to which the program had effectively 
implemented EVM, we reviewed relevant documentation, such as contract 
performance reports, acquisition program baselines, performance 
measurement baseline, and schedule estimates and compared them with DOD 
policies and guidance.[Footnote 63] To identify trends that could 
affect the program baseline in the future, we assessed cost and 
schedule performance and, in doing so, we applied earned value analysis 
techniques[Footnote 64] to data from contract performance reports. We 
compared the cost of work completed with the budgeted costs for 
scheduled work over a 17-month period, from January 2007 to May 2008, 
to show trends in cost and schedule performance. We also used data from 
the reports to estimate the likely costs at completion of the program 
through established earned value formulas. This resulted in three 
different values, with the middle value being the most likely. We 
checked EVM data to see if there were any mathematical errors or 
inconsistencies that would lead to the data being unreliable. We 
interviewed cognizant officials from the Naval Air Systems Command and 
program officials to determine whether the program had conducted an 
integrated baseline review, whether the EVM system had been validated 
against industry guidelines,[Footnote 65] and to better understand the 
anomalies in the EVM data and determine what outside surveillance was 
being done to ensure that the industry standards are being met. We also 
reviewed the program's schedule estimates and compared them with 
relevant best practices[Footnote 66] to determine the extent to which 
they reflect key estimating practices that are fundamental to having a 
reliable schedule. In doing so, we interviewed cognizant program 
officials to discuss their use of best practices in creating the 
program's current schedule. 

* To determine the extent to which the program has effectively 
implemented requirements management, we reviewed relevant program 
documentation, such as the program management plan and baseline list of 
requirements. To determine the extent to which the program has 
maintained traceability backward to high-level business operation 
requirements and system requirements, and forward to system design 
specifications, and test plans, we randomly selected 60 program 
requirements and traced them both backward and forward. This sample was 
designed with a 5 percent tolerable error rate at the 95 percent level 
of confidence so that, if we found 0 problems in our sample, we could 
conclude statistically that the error rate was less than 5 percent. In 
addition, we interviewed program officials involved in the requirements 
management process to discuss their roles and responsibilities for 
managing requirements. 

* To determine the extent to which the program implemented risk 
management, we reviewed relevant risk management documentation, such as 
the program's risk management plan and risk database reports 
demonstrating the status of the program's major risks and compared the 
program office's activities with DOD acquisition management guidance 
and related industry practices.[Footnote 67] We also reviewed the 
program's mitigation process with respect to key risks to determine the 
extent to which these risks were effectively managed. In doing so, we 
interviewed cognizant program officials, such as the program manager 
and risk manager, to discuss their roles and responsibilities and 
obtain clarification on the program's approach to managing risks 
associated with acquiring and implementing Navy ERP. 

We conducted this performance audit at DOD offices in the Washington, 
D.C., metropolitan area and Annapolis, Md., from June 2007 to September 
2008, in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit 
to obtain sufficient, appropriate evidence to provide a reasonable 
basis for our findings and conclusions based on our audit objective. We 
believe that the evidence obtained provides a reasonable basis for our 
findings and conclusions based on our audit objective. 

[End of section] 

Appendix II: Comments from the Department of Defense: 

Office Of The Under Secretary Of Defense: 
Acquisition Technology And Logistics: 
3000 Defense Pentagon: 
Washington, DC 20301-3000: 

August 19, 2008: 

Mr. Randolph C. Hite: 
Director, Information Technology Architecture and Systems Issues: 
U.S. Government Accountability Office: 
441 G Street, N.W. 
Washington, DC 20548: 

Dear Mr. Hite: 

This is the Department of Defense (Dot)) response to the GAO draft 
report GAO-08-896, "Defense Business Systems Modernization: Important 
Management Controls Being Implemented on Major Navy Program, But 
Improvements Needed in Key Areas," dated July 18, 2008 (GAO Code 
310659). Detailed comments on the report recommendations are enclosed. 

The Department concurred with two recommendations and partially 
concurred with the remaining two. The Department has already taken 
steps to address some of GAO's recommendations and the Navy Enterprise 
Resource Planning (ERP) program office is committed to implementing 
recommendations that will contribute to the program's success. 

We appreciate the GAO's input on the Department's progress with its 
business transformation efforts and continue to value our partnership. 

Sincerely, 

Signed by: 

Paul A. Brinkley: 
Deputy Under Secretary of Defense (Business Transformation): 

Enclosure: As stated: 

GAO Draft Report Dated July 18, 2008: 
GAO-08-896 (GAO Code 310659): 

"DOD Business Systems Modernization: Important Management Controls 
Being Implemented On Major Navy Program, But Improvements Needed In Key 
Areas" 

Department Of Defense Comments To The GAO Recommendations: 

Recommendation 1: The GAO recommends that the Secretary of Defense 
direct the Secretary of the Navy, through the appropriate chain of 
command, to ensure that future Navy Enterprise Resource Planning (ERP) 
estimates include uncertainty analyses of estimated benefits, reflect 
the risk associated with not having cost data for comparable ERP 
programs, and are otherwise derived in full accordance with the other 
key estimating practices, and economic analysis practices discussed in 
this report. (Page 52/GAO Draft Report) 

DOD Response: Partially Concur. Future Navy ERP estimates will include 
uncertainty analyses of estimated benefits and be derived in full 
accordance with key estimating and economic analysis practices. In 
future estimates, uncertainty surrounding the assumptions driving the 
benefit estimate will have probability distributions applied. The point 
estimate reported will be generated using the expected values of the 
probability distributions. 

The Department does not concur with the second element of 
Recommendation 1, that future Navy ERP estimates reflect the risk 
associated with not having cost data for comparable ERP programs. At 
Milestone .A/B, the program estimate was based on commercial best 
practices and standards used to implement System Application Program 
(SAP) functionality as well as engineering inputs and expertise 
developed during the Navy's pilot ERP programs. This approach was 
considered the best approach given the limited comparable enterprise 
program system implementation data currently available in DoD. 
Furthermore, the industry standards and the functional scope definition 
had uncertainty analysis applied to account for the risk associated 
with not having comparable DoD programmatic data. As the program began 
execution, the methodology was and continues to be refined as actual 
experience is gained by the Navy ERP program. 

Recommendation 2: The GAO recommends that the Secretary of Defense 
direct the Secretary of the Navy, through the appropriate chain of 
command, to ensure that: (1) an integrated baseline review on the last 
two releases of the first increment is conducted: (2) compliance 
against the 32 accepted industry earned value management (EVM) 
practices is verified: and (3) a plan to have an independent 
organization perform surveillance of the program's EVM system is 
developed and implemented. (Page 52/GAO Draft Report) 

DOD Response: Concur. The Department appreciates the importance of 
Earned Value Management (EVM) as an effective management tool that 
promotes efficient management of programs such as Navy ERP. Navy ERP 
has been implementing "Program-Level" EVM since 2006. During that time, 
the Navy ERP Program Office has received detailed guidance and advice 
from various government and industry groups since Program-Level EVM is 
rare and requires tailoring to achieve compliance with best practices.
A key part of this implementation is the Integrated Baseline Review 
(IBR), which has been scheduled for Navy ERP Release 1.1 in August 
2008. Although this IBR will focus on Release 1.1, it will also include 
the remaining Release 1.0 deployments. The IBR will focus on the health 
of the baseline, assessing schedule and associated resourcing. Navy ERP 
will also conduct IBR on future releases. 

An IBR was not conducted on Release 1.0 because once the rebaselining 
was approved in December 2006, development was essentially complete and 
the release was in the latter stages of testing. However, a Schedule 
Risk Assessment (SRA) of Release 1.0 was conducted in March 2007. The 
main finding was that the Integrated Master Schedule (IMS) was high 
risk for achieving Go-Live due to the lack of integration of the Navy 
ERP schedule with the schedule of Naval Air System Command (NAVAIR), 
the customer. Navy ERP took corrective measures to mitigate this risk 
by developing a cut-over schedule that integrated with NAVAIR's 
schedule. Navy ERP has made it a requirement to incorporate customer 
schedules in all future releases. The corrective measures successfully 
mitigated the risk associated with 1.0 as indicated by the Program 
achieving Go-Live as scheduled. 

Navy ERP follows the direction of the Assistant Secretary of the Navy 
for Research. Development and Acquisition (ASN(RDA)) to work with the 
Navy Center for Earned Value Management (CEVM) to validate that 
management processes meet the intent of the 32 guidelines described in 
the American National Standards Institute/Electronic Industries 
Alliance (ANSI/EIA) standard ANSI/EIA-748. CEVM will also perform 
ongoing surveillance of Navy ERP processes and EVM data, addressing the 
last element of GAO's recommendation. Additionally, as of August 2008, 
Navy ERP is complying with the Under Secretary of Defense for 
Acquisition, Technology, and Logistics (AT&L) memo issued July 11, 2007 
on the implementation of the Central Repository System. This memo 
requires that all Acquisition Category I programs submit monthly cost 
and schedule performance reports via the Defense Cost and Resource 
Center (DCARC) Earned Value Repository. 

Recommendation 3: The GAO recommends that the Secretary of Defense 
direct the Secretary of the Navy, through the appropriate chain of 
command, to ensure that the schedule: (1) includes the logical 
sequencing of all activities; (2) reflects whether all required 
resources will be available when needed; (3) defines a critical path 
that integrates all three releases; (4) allocates reserve for the high-
risk activities on the entire program's critical path; and (5) 
incorporates the results of a schedule risk analysis for all three 
releases, and recalculates program cost and schedule variances to more 
accurately determine a most likely cost and schedule overrun. (Page 
52/GAO Draft Report) 

DOD Response: Partially Concur. The Department's EVM policy recognizes 
that the preparation of an Integrated Master Schedule (IMS) is a best 
practice. The Department concurs with the following elements of the 
recommendation and the Navy ERP Program Office will ensure that the 
schedule: 

* Includes the logical sequencing of all activities: Navy ERP has 
implemented a logically sequenced schedule for Release 1.1 of all 
activities based on experience gained from Navy ERP Release 1.0. 
Improvements have been made and the schedule is continuously monitored 
through weekly schedule metrics. 

* Reflects whether all required resources will be available when 
needed: The Navy ERP IMS has a resource-loaded, time-phased schedule 
for each release that identifies resource allocation and is aligned 
with the Program's approved staffing plan. 

* Allocates reserve for the high-risk activities on the entire 
program's critical path: Navy ERP has actively begun refining the risk 
identification process to incorporate risk and potential impact to the 
program schedule utilizing schedule inputs. As this process matures, 
the result will be used to assist in identify adequate resources 
required in managing integrated program activities across all three 
releases. 

* Incorporates the results of a schedule risk analysis (SRA) for all 
releases: The Results of the SRA for Release 1.1 will be compared to 
the results of the SRA for Release 1.0 and incorporated into the 
variance analysis for cost and schedule to avoid any potential 
overruns. Navy ERP is implementing an estimate at completion (EAC) 
process for quarterly updates beginning in Fiscal Year 2009.
The Department partially concurs with the third element of the 
recommendation, that the schedule defines a critical path that 
integrates all three releases. Navy ERP utilizes a critical path for 
each release. This approach recognizes the size and complexity of the 
Navy's implementation releases and still allows the Navy to remain 
focused on the true dependencies the individual releases have with one 
another. The Program Office will review this recommendation with Navy 
leadership to determine feasibility for Navy [RP. 

Recommendation 4: The GAO recommends that the Secretary of Defense 
direct the Secretary of the Navy, through the appropriate chain of 
command, to ensure that: (1) the plans for mitigating the risks 
associated with converting data from legacy systems to Navy ERP and 
positioning the commands for adopting the new business processes 
embedded in the Navy ERP are re-evaluated in light of the recent 
experience with Naval Air Systems Command (NAVAIR) and adjust 
accordingly; (2) the status and results of these and other mitigation 
plans' implementation are periodically reported to program oversight 
and approval authorities; (3) these authorities ensure that those 
entities responsible for implementing these strategies are held 
accountable for doing so; and (4) each of the risks discussed in this 
report are included in the program's inventory of active risks, and 
managed accordingly. (Page 53/GAO Draft Report) 

DOD Response: Concur. Navy ERP incorporated lessons learned from NAVAIR 
into the Navy ERP Program Command Implementation Guidance. The 
guidance, a copy of which is attached for reference, incorporates key 
lessons learned from prior deployments and provides a detailed overview 
of the implementation process and key information required in 
structuring a Command's implementation team and activities needed for 
success. The Navy ERP Program Command Implementation Guidance also 
identifies critical success factors based on extensive commercial and 
Navy ERP implementation experience and provides a time-phased checklist 
to help focus a Command's resources on the right actions, at the right 
time. It has been reviewed by the Navy System Command Commanders and is 
being used by the Navy Supply System Command (NAVSUP) in preparation 
for the next deployment of Navy ERP Release 1.0. 

The Readiness Assessment checklist in the Navy ERP Program Command 
Implementation Guidance helps to identify risks, and through the Navy 
ERP Risk Program process, elevate the risk appropriately to the right 
level of management to facilitate mitigation planning. In addition to 
those risks identified by the program, Navy ERP will add the risks 
identified by GAO to its risk inventory. Risks and mitigation plans are 
formally captured by the Navy ERP Risk Management Board in its Risk 
Radar tracking tool. 

Furthermore, in light of the experience learned with deployment to 
NAVAIR, Navy ERP has completed a full review and reevaluation of all 
active risks and corresponding mitigation plans. This review included 
two areas noted by GAO in the report: (I) conversion of data from 
legacy systems to Navy ERP and (2) transformation change management to 
position Commands to adopt Navy ERP. 

Since Navy's ERP's deployment of Release 1.0 to NAVAIR, the enterprise 
governance structure that reviews program risks has evolved. Governance 
now includes the Navy Component Acquisition Executive's review process, 
the Navy ERP Senior Integration Board, and Enterprise Risk Assessment 
Methodology (ERAM) assessments. The Navy will work through these 
governance bodies and with other appropriate oversight and approval 
authorities to determine additional reporting mechanisms for risks with 
associated mitigation plans, to ensure commitment and accountability 
across the Navy Enterprise. Direction received from these authorities 
with respect to risk and risk management will be incorporated into the 
Navy ERP Risk Management Program. Through the Navy and Office of the 
Secretary of Defense (OSD) governance bodies. including the Milestone 
Decision Authority (MDA), Deputy Under Secretary of Defense (Business 
Transformation), as well as Navy Enterprise Senior Integration Board 
(NESIB), the Navy ERP Program Manager is held accountable for overall 
program performance, including implementing risk mitigation strategies. 
This aligns to Department of Defense Directive 5000.1, paragraph 3.5, 
which states that the Program Manager is accountable for credible cost, 
schedule, and performance reporting to the MDA. 

[End of section] 

Appendix III: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Randolph C. Hite, (202) 512-3439, or hiter@gao.gov: 

Staff Acknowledgments: 

In addition to the individual named above, key contributors to this 
report were Neelaxi Lakhmani, Assistant Director; Monica Anatalio; 
Harold Brumm; Neil Doherty; Cheryl Dottermusch; Nancy Glover; Mustafa 
Hassan; Michael Holland; Ethan Iczkovitz; Anh Le; Josh Leiling; Emily 
Longcore; Lee McCracken; Madhav Panwar; Karen Richey; Melissa 
Schermerhorn; Karl Seifert; Sushmita Srikanth; Jonathan Ticehurst; and 
Adam Vodraska. 

[End of section] 

Footnotes: 

[1] Business systems are information systems, including financial and 
nonfinancial systems, that support DOD business operations, such as 
civilian personnel, finance, health, logistics, military personnel, 
procurement, and transportation. 

[2] GAO, High-Risk Series: An Update, [hyperlink, 
http://www.gao.gov/cgi-bin/getrpt?GAO-07-310] (Washington, D.C.: 
January 2007). 

[3] The Department of the Navy is a major component of DOD, consisting 
of two uniformed services: the Navy and the Marine Corps. 

[4] This amount is the difference between the September 2007 estimated 
life cycle cost of $2,445.0 million and the September 2003 estimated 
life cycle cost of $1,873.1 million. 

[5] Program-level means that EVM covers all aspects of the program, 
regardless of whether they are performed by the government or the 
contractor. In contrast, EVM has historically been implemented on a 
contract-by-contract basis, and has not included government performed 
work. 

[6] These commands are Naval Air, Naval Sea, Space and Naval Warfare, 
and Naval Supply Systems. 

[7] The defense acquisition system is a framework-based approach that 
is intended to translate mission needs and requirements into stable, 
affordable, and well-managed acquisition programs. 

[8] Congressional defense committees have established reprogramming 
guidelines, including setting dollar thresholds that direct DOD to seek 
the prior approval of the committees before executing the reprogramming 
of appropriated funds. In accordance with these guidelines, DOD's 
financial management regulation provides that DOD does not need 
congressional committee approval when the amount to be reprogrammed 
falls below certain thresholds (referred to as a "below-threshold 
reprogramming"). As relevant here, as of fiscal year 2005, the 
threshold is $10 million or 20 percent of the program's appropriation, 
whichever is less. To fund shortfalls in Navy ERP, DON plans to 
reprogram amounts below this threshold from other programs. 

[9] FOC means that the system has been successfully deployed in all 
intended locations. 

[10] This 2003 estimate, which was prepared to assist in budget 
development and support the Milestone A/B approval, was for 
development, deployment, and sustainment costs in fiscal years 2003 
through 2021. 

[11] According to DOD's acquisition guidebook, an Acquisition Program 
Baseline is a program manager's estimated cost, schedule, and 
performance goals. Goals consist of objective values, which represent 
what the user desires and expects, and threshold values, which 
represent acceptable limits. When the program manager determines a 
current cost, schedule or performance threshold value will not be 
achieved, the MDA must be notified, and a new baseline developed, 
reviewed by decision makers, and, if the program is to continue, 
approved by the MDA. 

[12] According to the August 2004 Acquisition Program Baseline, this 
estimate is for acquisition, operations, and support for fiscal years 
2004 through 2021. 

[13] According to the September 2007 Acquisition Program Baseline, this 
estimate is for acquisition, operations, and support for fiscal years 
2004 through 2023. 

[14] Donald E. Harter, Mayuram S. Krishnan, and Sandra A. Slaughter, 
"Effects of Process Maturity on Quality, Cycle Time, and Effort in 
Software Product Development," Management Science, 46, no. 4 (2000); 
and Bradford K. Clark, "Quantifying the Effects of Process Improvement 
on Effort," IEEE Software (November/December 2000). 

[15] GAO, Information Technology: DOD's Acquisition Policies and 
Guidance Need to Incorporate Additional Best Practices and Controls, 
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-04-722] (Washington, 
D.C.: July 30, 2004). 

[16] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-04-722]. 

[17] See, for example, GAO, DOD Business Transformation: Lack of an 
Integrated Strategy Puts the Army's Asset Visibility System Investments 
at Risk, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-860] 
(Washington, D.C.: July 27, 2007); GAO, Information Technology: DOD 
Needs to Ensure That Navy Marine Corps Intranet Program Is Meeting 
Goals and Satisfying Customers, [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO-07-51] (Washington, D.C.: Dec. 8, 2006); GAO, Defense 
Travel System: Reported Savings Questionable and Implementation 
Challenges Remain, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-
980] (Washington, D.C.: Sept. 26, 2006); GAO, DOD Systems 
Modernization: Uncertain Joint Use and Marginal Expected Value of 
Military Asset Deployment System Warrant Reassessment of Planned 
Investment, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-171] 
(Washington, D.C.: Dec. 15, 2005); and GAO, DOD Systems Modernization: 
Planned Investment in the Navy Tactical Command Support System Needs to 
Be Reassessed, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-
215] (Washington, D.C.: Dec. 5, 2005). 

[18] Ronald W. Reagan National Defense Authorization Act for Fiscal 
Year 2005, Pub. L. No. 108-375, § 332 (2004) (codified at 10 U.S.C. §§ 
186 and 2222). 

[19] Field/tactical refers to Army units that are deployable to 
locations around the world, such as Iraq or Afghanistan. 

[20] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-860]. 

[21] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-215]. 

[22] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-171]. 

[23] GAO, DOD Business Systems Modernization: Navy ERP Adherence to 
Best Business Practices Critical to Avoid Past Failures, [hyperlink, 
http://www.gao.gov/cgi-bin/getrpt?GAO-05-858] (Washington, D.C.: Sept. 
29, 2005). 

[24] Department of Defense Architecture Framework, Version 1.0, Volume 
1 (February 2004); GAO, Information Technology: A Framework for 
Assessing and Improving Enterprise Architecture Management (Version 
1.1), [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-03-584G] 
(Washington, D.C.: Apr. 1, 2003); Chief Information Officer Council, A 
Practical Guide to Federal Enterprise Architecture, Version 1.0 
(February 2001); and Institute of Electrical and Electronics Engineers 
(IEEE), Standard for Recommended Practice for Architectural Description 
of Software-Intensive Systems 1471-2000 (Sept. 21, 2000). 

[25] A well-defined enterprise architecture provides a clear and 
comprehensive picture of an entity, whether it is an organization 
(e.g., a federal department) or a functional or mission area that cuts 
across more than one organization (e.g., personnel management). This 
picture consists of snapshots of both the enterprise's current or "as 
is" environment and its target or "to be" environment, as well as a 
capital investment road map for transitioning from the current to the 
target environment. These snapshots consist of integrated "views," 
which are one or more architecture products that describe, for example, 
the enterprise's business processes and rules; information needs and 
flows among functions, supporting systems, services, and applications; 
and data and technical standards and structures. 

[26] DOD has adopted a federated approach for developing its business 
mission area enterprise architecture, which includes the corporate BEA 
representing the thin layer of DOD-wide corporate architectural 
policies, capabilities, rules, and standards; component architectures 
(e.g., Navy enterprise architecture); and program architectures (e.g., 
Navy ERP architecture). 

[27] See, for example, GAO, Information Technology: FBI Is Taking Steps 
to Develop an Enterprise Architecture, but much Remains to Be 
Accomplished, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-05-363] 
(Washington, D.C.: Sept. 9, 2005); GAO, Homeland Security: Efforts 
Under Way to Develop Enterprise Architecture, but Much Work Remains, 
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-04-777] (Washington, 
D.C.: Aug. 6, 2004); GAO, Information Technology: Architecture Needed 
to Guide NASA's Financial Management Modernization, [hyperlink, 
http://www.gao.gov/cgi-bin/getrpt?GAO-04-43] (Washington, D.C.: Nov. 
21, 2003); GAO, DOD Business Systems Modernization: Important Progress 
Made to Develop Business Enterprise Architecture, but Much Work 
Remains, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-03-1018] 
(Washington, D.C.: Sept. 19, 2003); GAO, Information Technology: DLA 
Should Strengthen Business Systems Modernization Architecture and 
Investment Activities, [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO-01-631] (Washington, D.C.: June 29, 2001); and GAO, 
Information Technology: INS Needs to Better Manage the Development of 
Its Enterprise Architecture, [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO/AIMD-00-212] (Washington, D.C.: Aug. 1, 2000). 

[28] DOD, Business Enterprise Architecture Compliance Guidance (Apr. 
10, 2006). 

[29] Initiated in 2003, GCSS-MC is to modernize the Marine Corps 
logistics systems and thereby provide the decision makers with timely 
and complete logistics information to support the warfighter. Moreover, 
according to program officials, both GCSS-MC and Navy ERP are under the 
leadership of Navy's Program Executive Office for Enterprise 
Information Systems, which is responsible for developing, acquiring, 
and deploying seamless enterprise-wide information technology systems. 

[30] GAO, DOD Business Systems Modernization: Progress in Establishing 
Corporate Management Controls Needs to be Replicated Within Military 
Departments, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-705] 
(Washington, D.C.: May 15, 2008). 

[31] As we recently reported, while the corporate BEA includes 
corporate policies, capabilities, rules, and standards, it is still 
evolving and will continue to add additional details. See [hyperlink, 
http://www.gao.gov/cgi-bin/getrpt?GAO-08-705]. 

[32] GAO, DOD Business Systems Modernization: Military Departments Need 
to Strengthen Management of Enterprise Architecture Programs, 
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-519] (Washington, 
D.C.: May 12, 2008). 

[33] GAO, DOD Business Systems Modernization: Key Navy Programs' 
Compliance with DOD's Federated Business Enterprise Architecture Needs 
to Be Adequately Demonstrated, [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO-08-972] (Washington, D.C.: Aug. 7, 2008). 

[34] Office of Management and Budget, Circular No. A-94: Guidelines and 
Discount Rates for Benefit-Cost Analysis of Federal Programs (Oct. 29, 
1992). 

[35] The benefits estimates for the areas are rounded; therefore, they 
add to more than $8.6 billion. 

[36] For example, the Uniform Accounting Data Process System-Inventory 
Control Points has yearly maintenance costs of $39.3 million. 

[37] These reductions in expected savings represent the costs of 
maintaining the legacy systems during the period in which the systems' 
retirement dates have been delayed. 

[38] OMB, Circular No. A-11, Preparation, Submission, and Execution of 
the Budget (Washington, D.C.: Executive Office of the President, June 
2006); Circular No. A-130 Revised, Management of Federal Information 
Resources (Washington, D.C.: Executive Office of the President, Nov. 
28, 2000); and Capital Programming Guide: Supplement to Circular A-11, 
Part 7, Preparation, Submission, and Execution of the Budget 
(Washington, D.C.: Executive Office of the President, June 2006). 

[39] GAO, Cost Assessment Guide: Best Practices for Estimating and 
Managing Program Costs, Exposure Draft, [hyperlink, 
http://www.gao.gov/cgi-bin/getrpt?GAO-07-1134SP] (Washington, D.C.: 
July 2, 2007). 

[40] The Defense Cost and Resource Center is responsible for collecting 
current and historical major defense acquisition program cost and 
software resource data in a joint service environment and making those 
data available for use by authorized government analysts when 
estimating the cost of ongoing and future government programs. 

[41] GAO, DOD Business Systems Modernization: Key Marine Corps System 
Acquisition Needs to Be Better Justified, Defined, and Managed, 
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-822] (Washington, 
D.C.: July 28, 2008). 

[42] Sensitivity analysis reveals how the cost estimate is affected by 
a change in a single assumption or cost driver while holding all other 
variables constant. 

[43] GAO, Missile Defense: Additional Knowledge Needed in Developing 
System for Intercepting Long-Range Missiles, [hyperlink, 
http://www.gao.gov/cgi-bin/getrpt?GAO-03-600] (Washington, D.C.: Aug. 
21, 2003). 

[44] OMB, Preparation, Submission, and Execution of the Budget, 
Circular A-11 (Washington, D.C.: Executive Office of the President, 
June 2006), part 7, Planning, Budgeting, Acquisition, and Management of 
Capital Assets, sec. 300. [hyperlink, 
http://www.whitehouse.gov/omb/circulars/index.html]. 

[45] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-1134SP]. 

[46] Defense Contract Management Agency, Department of Defense Earned 
Value Management Implementation Guide (Washington, D.C.: October 2006); 
and [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-1134SP]. 

[47] American National Standards Institute (ANSI)/Electronic Industries 
Alliance (EIA) Standard, EVM Systems Standard, ANSI/EIA-748-A-1998 
(R2002), approved May 19, 1998; revised January 2002. 

[48] Defense Contract Management Agency, Department of Defense Earned 
Value Management Implementation Guide (Washington, D.C.: October 2006). 
See also DOD Memorandum: Revision to DOD Earned Value Management Policy 
(Washington, D.C.: Mar. 7, 2005). 

[49] DON established the Center for Earned Value Management in April 
2007 to, among other things, work with program offices to improve the 
accuracy of EVM data and to provide independent assistance to program 
managers when requested. 

[50] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-1134SP]. 

[51] Float time is the amount of time an activity can slip before 
affecting the critical path. 

[52] Department of the Navy, Commander, Operational Test and Evaluation 
Force, Navy Enterprise Resource Planning System Initial Operational 
Test and Evaluation, OT-C1 Final Report to the Chief of Naval 
Operations (Norfolk, VA: June 13, 2008). 

[53] Cost variances compare the earned value of the completed work with 
the actual cost of the work performed. For example, if a contractor 
completed $5 million worth of work, and the work actually cost $6.7 
million, there would be a negative $1.7 million cost variance. Schedule 
variances are also measured in dollars, but they compare the earned 
value of the work actually completed as of a point in time to the value 
of work that was expected to be completed. For example, if a contractor 
completed $5 million worth of work at the end of the month but was 
budgeted to complete $10 million worth of work, there would be a 
negative $5 million schedule variance. 

[54] This figure is the total estimated budget for the program. It 
represents the cumulative value of the budgeted cost of work scheduled 
over the life of the program. 

[55] To make these assessments, we applied earned value analysis 
techniques to data from the program's contract performance reports. 
These techniques compare budget versus actual costs versus project 
status in dollar amounts. For our analysis, we used standard earned 
value formulas to calculate cost and schedule variance and forecast the 
range of cost overrun at completion. 

[56] DOD, Defense Acquisition Guidebook, Version 1.0 (Oct. 17, 2004). 
Software Engineering Institute, Software Acquisition Capability 
Maturity Model® version 1.03, CMU/SEI-2002-TR-010 (Pittsburgh, PA: 
March 2002). 

[57] Department of Defense, Risk Management Guide for DOD Acquisition, 
6th Edition, Version 1.0., [hyperlink, 
http://www.acq.osd.mil/sse/ed/docs/2006-RM-Guide-4Aug06-final-
version.pdf] (accessed March 13, 2008) and Software Engineering 
Institute, CMMI for Acquisition, Version 1.2, CMU/SEI-2007- TR-017 
(Pittsburgh, PA: November 2007). 

[58] The risk management board oversees risk management activities by 
assigning risk owners, approving and directing resources to facilitate 
risk handling strategies, and reviewing risk handling status. 

[59] Risk levels of high, medium or low are assigned using quantitative 
measurements of the probability of the risk occurring and the potential 
impact to the program's cost, schedule, and performance. Based on that 
assessment, a risk level is assigned to represent the risk's 
significance. 

[60] See, for example, GAO, Office of Personnel Management: Retirement 
Systems Modernization Program Faces Numerous Challenges, [hyperlink, 
http://www.gao.gov/cgi-bin/getrpt?GAO-05-237] (Washington, D.C.: Feb. 
28, 2005); and GAO, Information Technology: Customs Automated 
Commercial Environment Program Progressing, but Need for Management 
Improvements Continues, [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO-05-267] (Washington, D.C.: Mar. 14, 2005). 

[61] Ronald W. Reagan National Defense Authorization Act for Fiscal 
Year 2005, Pub. L. No. 108-375, § 332 (2004) (codified at 10 U.S.C. §§ 
186 and 2222). 

[62] Office of Management and Budget, Circular No. A-94: Guidelines and 
Discount Rates for Benefit-Cost Analysis of Federal Program (Oct. 29, 
1992), [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-1134SP]. 

[63] Defense Contract Management Agency, Department of Defense Earned 
Value Management Implementation Guide (Washington, D.C.: Apr. 7, 2005). 
See also DOD Memorandum: Revision to DOD Earned Value Management Policy 
(Washington, D.C.: Mar. 7, 2005). 

[64] The earned value concept is applied as a means of placing a dollar 
value on project status. The techniques we used compared the program's 
budget to actual costs and project status in dollar amounts. We used 
standard earned value formulas to calculate cost and schedule variance 
and forecast the range of cost overrun at program completion. 

[65] American National Standards Institute (ANSI) /Electronic 
Industries Alliance (EIA) EVM Systems Standard, ANSI/EIA-748-A-1998 
(R2002), approved May 19, 1998; revised January 2002. 

[66] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-1134SP]. 

[67] Department of Defense, Risk Management Guide for DOD Acquisition, 
6th Edition, Version 1.0., [hyperlink, 
http://www.acq.osd.mil/sse/ed/docs/2006-RM-Guide-4Aug06-final-
version.pdf] (accessed March 13, 2008) and Software Engineering 
Institute, CMMI for Acquisition, Version 1.2, CMU/SEI-2007-TR-017 
(Pittsburgh, PA: November 2007). 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "E-mail Updates." 

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office: 
441 G Street NW, Room LM: 
Washington, D.C. 20548: 

To order by Phone: 
Voice: (202) 512-6000: 
TDD: (202) 512-2537: 
Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: fraudnet@gao.gov: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Ralph Dawn, Managing Director, dawnr@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: