GAO-09-841, DOD Business Systems Modernization: Navy Implementing a Number of Key Management Controls on Enterprise Resource Planning System, but Improvements Still Needed


This is the accessible text file for GAO report number GAO-09-841 
entitled 'DOD Business Systems Modernization: Navy Implementing a 
Number of Key Management Controls on Enterprise Resource Planning 
System, but Improvements Still Needed' which was released on 
September 15, 2009. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to Congressional Requesters: 

United States Government Accountability Office: 
GAO: 

September 2009: 

DOD Business Systems Modernization: 

Navy Implementing a Number of Key Management Controls on Enterprise 
Resource Planning System, but Improvements Still Needed: 

GAO-09-841: 

GAO Highlights: 

Highlights of GAO-09-841, a report to congressional requesters. 

Why GAO Did This Study: 

The Department of Defense (DOD) has long been challenged in effectively 
implementing key acquisition management controls on its thousands of 
business system investments. For this and other reasons, GAO has 
designated DOD’s business systems modernization efforts as high-risk 
since 1995. One major business system investment is the Navy’s 
Enterprise Resource Planning (ERP) system. Initiated in 2003, it is to 
standardize the Navy’s business processes, such as acquisition and 
financial management. It is being delivered in increments, the first of 
which is to cost about $2.4 billion over its 20-year useful life and be 
fully deployed by fiscal year 2013. To date, the program has 
experienced about $570 million in cost overruns and a 2-year schedule 
delay. GAO was asked to determine whether (1) system testing is being 
effectively managed, (2) system changes are being effectively 
controlled, and (3) independent verification and validation (IV&V) 
activities are being effectively managed. To do this, GAO analyzed 
relevant program documentation, traced random samples of test defects 
and change requests, and interviewed cognizant officials. 

What GAO Found: 

The Navy has largely implemented effective controls on Navy ERP 
associated with system testing and change control. For example, it has 
established a well-defined structure for managing tests, including 
providing for a logical sequence of test events, adequately planning 
key test events, and documenting and reporting test results. In 
addition, it has documented, and is largely following, its change 
request review and approval process, which reflects key aspects of 
relevant guidance, such as having defined roles and responsibilities 
and a hierarchy of control boards. However, important aspects of test 
management and change control have not been fully implemented. 
Specifically, the program’s tool for auditing defect management did not 
always record key data about changes made to the status of identified 
defects. To its credit, the program office recently took steps to 
address this, thereby reducing the risk of defect status errors or 
unauthorized changes. Also, while the program office’s change review 
and approval procedures include important steps, such as considering 
the impact of a change, and program officials told GAO that cost and 
schedule impacts of a change are discussed at control board meetings, 
GAO’s analysis of 60 randomly selected change requests showed no 
evidence that cost and schedule impacts were in fact considered. 
Without such key information, decision-making authorities lack an 
adequate basis for making informed investment decisions, which could 
result in cost overruns and schedule delays. 

The Navy has not effectively managed its IV&V activities, which are 
designed to obtain an unbiased position on whether product and process 
standards are being met. In particular, the Navy has not ensured that 
the IV&V contractor is independent of the products and processes that 
it is reviewing. Specifically, the same contractor responsible for 
performing IV&V of Navy ERP products (e.g., system releases) is also 
responsible for ensuring that system releases are delivered within cost 
and schedule constraints. Because performance of this system 
development and management role makes the contractor potentially unable 
to render impartial assistance to the government in performing the IV&V 
function, there is an inherent conflict of interest. In addition, the 
IV&V agent reports directly and solely to the program manager and not 
to program oversight officials. As GAO has previously reported, the 
IV&V agent should report the findings and associated risks to program 
oversight officials, as well as program management, in order to better 
ensure that the IV&V results are objective and that the officials 
responsible for making program investment decisions are fully informed. 
Furthermore, the contractor has largely not produced the range of IV&V 
deliverables that were contractually required between 2006 and 2008. To 
its credit, the program office recently began requiring the contractor 
to provide assessment reports, as required under the contract, as well 
as formal quarterly reports; the contractor delivered the results of 
the first planned assessment in March 2009. Notwithstanding the recent 
steps that the program office has taken, it nevertheless lacks an 
independent perspective on the program’s products and management 
processes. 

What GAO Recommends: 

GAO is making recommendations to the Secretary of Defense aimed at 
improving the program’s system change request review and approval 
process and its IV&V activities. DOD concurred with the recommendations 
and identified actions that it plans to take. 

View [hyperlink, http://www.gao.gov/products/GAO-09-841] or key 
components. For more information, contact Randolph C. Hite at (202) 512-
3439 or hiter@gao.gov. 

[End of section] 

Contents: 

Letter: 

Background: 

Key Aspects of Navy ERP Testing Have Been Effectively Managed: 

System Changes Have Been Controlled, but Their Cost and Schedule 
Impacts Were Not Sufficiently Considered: 

Navy ERP IV&V Function Is Not Independent and Has Not Been Fully 
Performed: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments: 

Appendix I: Objectives, Scope, and Methodology: 

Appendix II: Comments from the Department of Defense: 

Appendix III: GAO Contact and Staff Acknowledgments: 

Tables: 

Table 1: Navy Systems Commands and Their Responsibilities: 

Table 2: Navy ERP Template 1 Releases: 

Table 3: Organizations Responsible for Navy ERP Oversight and 
Management: 

Table 4: Navy ERP Program Contracts: 

Table 5: Description of the Purpose of Navy ERP Tests: 

Table 6: Navy ERP Testing-Related Organizations and Respective Roles 
and Responsibilities: 

Table 7: Roles and Responsibilities for Change Review and Approval: 

Figures: 

Figure 1: Navy ERP Timeline: 

Figure 2: Navy ERP Life-Cycle Cost Estimates in Fiscal Years 2003, 
2004, and 2007: 

Figure 3: Navy ERP Deployment Schedule: 

Figure 4: Release 1.0 and 1.1 Test Activity Schedule: 

Abbreviations: 

DOD: Department of Defense: 

DON: Department of the Navy: 

ERP: Enterprise Resource Planning: 

FISC: Fleet Industrial Supply Center: 

FOC: full operational capability: 

FOT&E: follow-on operational test and evaluation: 

GCSS-MC: Global Combat Support System--Marine Corps: 

GDIT: General Dynamics Information Technology: 

IOC: initial operational capability: 

IOT&E: initial operational test and evaluation: 

IST: integrated system testing: 

IT: information technology: 

IV&V: independent verification and validation: 

MDA: milestone decision authority: 

NAVAIR: Naval Air Systems Command: 

NAVSEA: Naval Sea Systems Command: 

NAVSUP: Naval Supply Systems Command: 

NTCSS: Naval Tactical Command Support System: 

OT&E: operational test and evaluation: 

SAP: Systems Applications and Products: 

SPAWAR: Space and Naval Warfare Systems Command: 

TEMP: Test and Evaluation Master Plan: 

UAT: user acceptance testing: 

[End of section] 

United States Government Accountability Office:
Washington, DC 20548: 

September 15, 2009: 

The Honorable Evan Bayh: 
Chairman: 
The Honorable Richard Burr: 
Ranking Member: 
Subcommittee on Readiness and Management Support: 
Committee on Armed Services: 
United States Senate: 

The Honorable John Ensign: 
United States Senate: 

For decades, the Department of the Defense (DOD) has been challenged in 
modernizing its timeworn business systems.[Footnote 1] In 1995, we 
designated DOD's business systems modernization program as high-risk, 
and continue to do so today.[Footnote 2] Our reasons include the 
modernization's large size, complexity, and its critical role in 
addressing other high-risk areas, such as overall business 
transformation and financial management. Moreover, we continue to 
report on business system investments that fail to effectively employ 
acquisition management controls and deliver promised benefits and 
capabilities on time and within budget.[Footnote 3] 

Nevertheless, DOD continues to invest billions of dollars in thousands 
of these business systems, 11 of which account for about two-thirds of 
the department's annual spending on business programs. The Navy 
Enterprise Resource Planning (ERP) program is one such program. 
Initiated in 2003, Navy ERP is to standardize the Navy's acquisition, 
financial, program management, plant and wholesale supply, and 
workforce management business processes across its dispersed 
organizational environment. As envisioned, the program consists of a 
series of major increments, the first of which includes three releases 
and is expected to cost approximately $2.4 billion over its 20-year 
life cycle and to be fully operational in fiscal year 2013. We recently 
reported that Navy ERP program management weaknesses had contributed to 
a 2-year schedule delay and about $570 million in cost overruns. 
[Footnote 4] 

As agreed, our objectives were to determine whether (1) system testing 
is being effectively managed, (2) system changes are being effectively 
controlled, and (3) independent verification and validation (IV&V) 
activities are being effectively managed. To accomplish this, we 
analyzed relevant program documentation, such as test management 
documents, individual test plans and procedures and related test 
results and defect reports; system change procedures and specific 
change requests and decisions; change review board minutes; and 
verification and validation plans and contract documents. We also 
observed the use of tools for recording and tracking test defects and 
change requests, including tracing a statistically valid sample of 
transactions through these tools. 

We conducted this performance audit from August 2008 to September 2009, 
in accordance with generally accepted government auditing standards. 
Those standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe that 
the evidence obtained provides a reasonable basis for our findings and 
conclusions based on our audit objectives. Additional details on our 
objectives, scope, and methodology are in appendix I. 

Background: 

The Department of the Navy's (DON) primary mission is to organize, 
train, maintain, and equip combat-ready naval forces capable of winning 
wars, deterring aggression by would-be foes, preserving freedom of the 
seas, and promoting peace and security. Its operating forces, known as 
the fleet, are supported by four systems commands. Table 1 provides a 
brief description of each command's responsibilities. 

Table 1: Navy Systems Commands and Their Responsibilities: 

Systems command: Naval Air Systems Command (NAVAIR); 
Responsibilities: Developing, delivering, and supporting aircraft and 
weapons used by sailors and marines. 

Systems command: Naval Supply Systems Command (NAVSUP); 
Responsibilities: Providing supply, fuel, transportation, and other 
logistics programs. 

Systems command: Space and Naval Warfare Systems Command (SPAWAR); 
Responsibilities: Developing, delivering, and supporting specialized 
command and control technologies, business information technology, and 
space capabilities. 

Systems command: Naval Sea Systems Command (NAVSEA); 
Responsibilities: Acquiring and maintaining the department's ships and 
submarines. 

Source: GAO analysis of DON data. 

[End of table] 

To support the department's mission, these commands perform a variety 
of interrelated and interdependent business functions (e.g., 
acquisition and financial management), relying heavily on business 
systems to do so. In fiscal year 2009, DON's budget for business 
systems and associated infrastructure was about $2.7 billion, of which 
about $2.2 billion was allocated to operations and maintenance of 
existing systems and about $500 million to systems in development and 
modernization. Of the approximately 2,480 business systems that DOD 
reports having, DON accounts for 569, or about 23 percent, of the 
total. Navy ERP is one such system investment. 

Navy ERP: A Brief Description: 

In July 2003, the Assistant Secretary of the Navy for Research, 
Development, and Acquisition established Navy ERP to converge the 
functionality of four pilot systems that were under way at the four 
commands into one system.[Footnote 5] According to DOD, Navy ERP is to 
address the Navy's long-standing problems related to financial 
transparency and asset visibility. Specifically, the program is 
intended to standardize the Navy's acquisition, financial, program 
management, plant and wholesale supply, and workforce management 
business processes across its dispersed organizational components, and 
support about 86,000 users when fully implemented. 

Navy ERP is being developed in a series of increments using the Systems 
Applications and Products (SAP) commercial software package, augmented 
as needed by customized software. SAP consists of multiple, integrated 
functional modules that perform a variety of business-related tasks, 
such as finance and acquisition. The first increment, called Template 
1, is currently the only funded portion of the program and consists of 
three releases (1.0, 1.1, and 1.2).[Footnote 6] Release 1.0, Financial 
and Acquisition, is the largest of the three releases in terms of 
Template 1 functional requirements.[Footnote 7] See table 2 for a 
description of these releases. 

Table 2: Navy ERP Template 1 Releases: 

Release: 1.0 Financial and Acquisition; 
Functionality: 
* General Fund and Navy Working Capital Fund finance applications, such 
as billing, budgeting, and cost planning; 
* Acquisition applications, such as activity-based costing, contract 
awards, and budget exhibits; 
* Workforce management applications, such as personnel administration 
and training, as well as events management. 

Release: 1.1 Wholesale and Retail Supply; 
Functionality: 
* Wholesale applications, such as supply and demand planning, order 
fulfillment, and supply forecasting; 
* Retail supply applications, such as inventory management, supply and 
demand processing, and warehouse management. 

Release: 1.2 Intermediate-Level Maintenance; 
Functionality: 
* Maintenance applications, such as maintenance management, quality 
management, and calibration management. 

Source: GAO analysis of DON data. 

[End of table] 

DON estimates the life-cycle cost for Template 1 to be about $2.4 
billion, including about $1 billion for acquisition and $1.4 billion 
for operations and maintenance. The program office reported that 
approximately $600 million was spent from fiscal year 2004 through 
fiscal year 2008. For fiscal year 2009, about $190 million is planned 
to be spent. 

Program Oversight, Management, and Contractor Roles and 
Responsibilities: 

To acquire and deploy Navy ERP, DON established a program management 
office within the Program Executive Office for Executive Information 
Systems. The program office manages the program's scope and funding and 
is responsible for ensuring that the program meets its key objectives. 
To accomplish this, the program office performs program management 
functions, including testing, change control, and IV&V. In addition, 
various DOD and DON organizations share program oversight and review 
activities. A listing of key entities and their roles and 
responsibilities is provided in table 3. 

Table 3: Organizations Responsible for Navy ERP Oversight and 
Management: 

Entity: Under Secretary of Defense for Acquisition, Technology, and 
Logistics; 
Roles and responsibilities: Serves as the milestone decision authority 
(MDA), which according to DOD, has overall responsibility for the 
program, to include approving the program to proceed through its 
acquisition cycle on the basis of, for example, independent operational 
test evaluation and certification. 

Entity: Assistant Secretary of the Navy, Research, Development, and 
Acquisition; 
Roles and responsibilities: Serves as DON's oversight organization for 
the program, to include enforcement of Under Secretary of Defense for 
Acquisition, Technology, and Logistics policies and procedures. 

Entity: DON, Program Executive Office for Executive Information 
Systems; 
Roles and responsibilities: Oversees a portfolio of large-scale 
projects and programs designed to enable common business processes and 
provide standard capabilities, to include reviewing and approving 
overarching test plans and user acceptance test readiness. 

Entity: Navy ERP Senior Integration Board; 
Roles and responsibilities: Reviews progress in attaining acceptable 
system performance at systems commands, including approving new system 
capabilities. Chaired by the Principal Deputy Assistant Secretary of 
the Navy. 

Entity: Navy ERP Program Management Office; 
Roles and responsibilities: Performs day-to-day program management and 
serves as the single point of accountability for managing the program's 
objectives through development, testing, deployment, and sustainment. 

Source: GAO analysis of DOD data. 

[End of table] 

To deliver system and other program capabilities and to provide program 
management support services, Navy ERP relies on multiple contractors, 
as described in table 4. 

Table 4: Navy ERP Program Contracts: 

Contract: Release 1.0 System Integration; 
Award date: September 2004; 
Completion date: February 2008; 
Contract value: $176 million; 
Awarded to: BearingPoint; 
Purpose: Design and development of release 1.0; training and deployment 
at NAVAIR. 

Contract: Release 1.1 & 1.2 System Integration; 
Award date: June 2007; 
Completion date: September 2011; 
Contract value: $152.9 million; 
Awarded to: IBM; 
Purpose: Design and development of release 1.1 and 1.2. 

Contract: Professional Support Service 1; 
Award date: June 2006; 
Completion date: September 2010; 
Contract value: $163.7 million; 
Awarded to: IBM; 
Purpose: Business process analysis, training, organizational change 
management, and deployment and sustainment support. 

Contract: Professional Support Service 2; 
Award date: June 2006; 
Completion date: September 2010; 
Contract value: $69 million; 
Awarded to: General Dynamics Information Technology; 
Purpose: Support to the government in its oversight of the system 
integrators and other contractors, release management, and IV&V. 

Source: GAO analysis of DON data. 

[End of table] 

Overview of Navy ERP's Status: 

Template 1 of Navy ERP was originally planned to reach full operational 
capability (FOC) in fiscal year 2011, and its original estimated life-
cycle cost was about $1.87 billion.[Footnote 8] The estimate was later 
baselined[Footnote 9] in August 2004 at about $2.0 billion.[Footnote 
10] In December 2006 and again in September 2007, the program was 
rebaselined. FOC is now planned for fiscal year 2013, and the estimated 
life-cycle cost is about $2.4 billion (a 31 percent increase over the 
original estimate).[Footnote 11] 

The program is currently in the production and deployment phase of the 
defense acquisition system, having completed the system development and 
demonstration phase in September 2007.[Footnote 12] This was 17 months 
later than the program's original schedule set in August 2004, but on 
time according to the revised schedule set in December 2006. Changes in 
the program's acquisition phase timeline are depicted in figure 1, and 
life-cycle cost estimates are depicted in figure 2. 

Figure 1: Navy ERP Life-Cycle Cost Estimates in Fiscal Years 2003, 
2004, and 2007: 

[Refer to PDF for image: illustration] 

Phase: Concept refinement and technology development; 
Fiscal year 2003-2004: Program established (activity prior to the 2004 
plan); 

Phase: System development and demonstration; 
Fiscal year 2004-2006: 2004 plan; 
Fiscal year 2004-2008: 2007 plan; 

Phase: Production and deployment; 
Fiscal year 2006-2011: 2004 plan; 
* 2006: Initial Operational Capability; 
* 2011: Full Operational Capability. 
Fiscal year 2007-2013: 2007 plan; 
* 2008: Initial Operational Capability; 
* 2013: Full Operational Capability. 

Source: GAO analysis of DON data. 

[End of figure] 

Figure 2: Navy ERP Life-Cycle Cost Estimates in Fiscal Years 2003, 
2004, and 2007: 

[Refer to PDF for image: vertical bar graph] 

Fiscal year: 2003; 
Navy ERP Cost estimate: $1.87 billion. 

Fiscal year: 2004; 
Navy ERP Cost estimate: $1.99 billion. 

Fiscal year: 2007; 
Navy ERP Cost estimate: $2.44 billion. 

Source: GAO analysis of DON data. 

[End of figure] 

Release 1.0 was deployed at NAVAIR in October 2007, after passing 
developmental testing and evaluation. Initial operational capability 
(IOC) was achieved in May 2008, 22 months later than the baseline 
established in August 2004, and 4 months later than the new baseline 
established in September 2007. According to program documentation, 
these delays were due, in part, to challenges experienced at NAVAIR in 
converting data from legacy systems to run on the new system and 
implementing new business procedures associated with the system. In 
light of the delays at NAVAIR in achieving IOC, the deployment 
schedules for the other commands were revised in 2008. Release 1.0 was 
deployed at NAVSUP in October 2008 as scheduled, but deployment at 
SPAWAR was rescheduled for October 2009, 18 months later than planned, 
and at NAVSEA General Fund in October 2010, and at Navy Working Capital 
Fund in October 2011, each 12 months later than planned. 

Release 1.1 is currently being developed and tested, and is planned to 
be deployed at NAVSUP in February 2010, 7 months later than planned, 
and at the Navy's Fleet and Industrial Supply Centers (FISC)[Footnote 
13] starting in February 2011. Changes in the deployment schedule are 
depicted in figure 3. 

Figure 3: Navy ERP Deployment Schedule: 

[Refer to PDF for image: illustration] 

Release: 1.0 Financial and Acquisition; 
NAVAIR: Late FY 2007 (2007 plan); 
NAVSUP: Late FY 2008 (2007 and 2008 plan); 
SPAWAR: Mid FY 2007 (2007 plan); Early FY 2010 (2008 plan); 
NAVSEA for General Fund: Early FY 2010 (2007 plan); Early FY 2011 (2008 
plan); 
NAVSEA for Working Capital Fund: Early FY 2011 (2007 plan); Early FY 
2012 (2008 plan). 

Release: 1.1 Wholesale and Retail Supply; 
NAVSUP: Late FY 2009 (2007 plan); Mid FY 2010 (2008 plan); 
FISC (first of seven deployments): Mid FY 2011 (2008 plan). 

Source: GAO analysis of DON data. 

[End of figure] 

Prior GAO Reviews of DOD Business System Investments Have Identified IT 
Management Weaknesses: 

We have previously reported that DOD has not effectively managed key 
aspects of a number of business system investments,[Footnote 14] 
including Navy ERP. Among other things, our reviews have identified 
weaknesses in such areas as architectural alignment and informed 
investment decision making, which are the focus of the Fiscal Year 2005 
Defense Authorization Act business system provisions.[Footnote 15] Our 
reviews have also identified weaknesses in other system acquisition and 
investment management areas, such as earned value management,[Footnote 
16] economic justification, risk management, requirements management, 
test management, and IV&V practices. 

In September 2008, we reported that DOD had implemented key information 
technology (IT) management controls on Navy ERP to varying degrees of 
effectiveness.[Footnote 17] For example, the control associated with 
managing system requirements had been effectively implemented, and 
important aspects of other controls had been at least partially 
implemented, including those associated with economically justifying 
investment in the program and proactively managing program risks. 
However, other aspects of these controls, as well as the bulk of what 
was needed to effectively implement earned value management, had not 
been effectively implemented. As a result, the controls that were not 
effectively implemented had, in part, contributed to sizable cost and 
schedule shortfalls. Accordingly, we made recommendations aimed at 
improving cost and schedule estimating, earned value management, and 
risk management. DOD largely agreed with our recommendations. 

In July 2008, we reported that DOD had not implemented key aspects of 
its IT acquisition policies and related guidance on its Global Combat 
Support System-Marine Corps (GCSS-MC) program.[Footnote 18] For 
example, we reported that it had not economically justified its 
investment in GCSS-MC on the basis of reliable estimates of both 
benefits and costs and had not effectively implemented earned value 
management. Moreover, the program office had not adequately managed all 
program risks and had not used key system quality measures. We 
concluded that by not effectively implementing these IT management 
controls, the program was at risk of not delivering a system solution 
that optimally supports corporate mission needs, maximizes capability 
mission performance, and is delivered on time and within budget. 
Accordingly, we made recommendations aimed at strengthening cost 
estimating, schedule estimating, risk management, and system quality 
measurement. The department largely agreed with our recommendations. 

In July 2007, we reported that the Army's approach for investing about 
$5 billion in three related programs--the General Fund Enterprise 
Business System, Global Combat Support System-Army Field/Tactical, and 
Logistics Modernization Program--did not include alignment with the 
Army enterprise architecture or use of a portfolio-based business 
system investment review process.[Footnote 19] Further, the Logistics 
Modernization Program's testing was not adequate and had contributed to 
the Army's inability to resolve operational problems. In addition, the 
Army had not established an IV&V function for any of the three 
programs. Accordingly, we recommended, among other things, use of an 
independent test team and establishment of an IV&V function. DOD agreed 
with the recommendations. 

In December 2005, we reported that DON had not, among other things, 
economically justified its ongoing and planned investment in the Naval 
Tactical Command Support System (NTCSS) and had not adequately 
conducted requirements management and testing activities.[Footnote 20] 
Specifically, requirements were not traceable and developmental testing 
had not identified problems that, subsequently, twice prevented the 
system from passing operational testing. Moreover, DON had not 
effectively performed key measurement, reporting, budgeting, and 
oversight activities. We concluded that DON could not determine whether 
NTCSS, as defined and as being developed, was the right solution to 
meet its strategic business and technological needs. Accordingly, we 
recommended developing the analytical basis necessary to know if 
continued investment in NTCSS represented a prudent use of limited 
resources, and strengthening program management, conditional upon a 
decision to proceed with further investment in the program. The 
department largely agreed with our recommendations. 

In September 2005, we reported that while Navy ERP had the potential to 
address some of DON's financial management weaknesses, it faced 
significant challenges and risks, including developing and implementing 
system interfaces with other systems and converting data from legacy 
systems.[Footnote 21] Also, we reported that the program was not 
capturing quantitative data to assess effectiveness, and had not 
established an IV&V function. We made recommendations to address these 
areas, including having the IV&V agent report directly to program 
oversight bodies, as well as the program manager. DOD generally agreed 
with our recommendations, including that an IV&V function should be 
established. However, it stated that the IV&V team would report 
directly to program management who in turn would inform program 
oversight officials of any significant IV&V results. In response, we 
reiterated the need for the IV&V to be independent of the program and 
stated that performing IV&V activities independently of the development 
and management functions helps to ensure that the results are unbiased 
and based on objective evidence. We also reiterated our support for the 
recommendation that the IV&V reports be provided to the appropriate 
oversight body so that it can determine whether any of the IV&V results 
are significant. We noted that doing so would give added assurance that 
the results were objective and that those responsible for authorizing 
future investments in Navy ERP have the information needed to make 
informed decisions. 

Key Aspects of Navy ERP Testing Have Been Effectively Managed: 

To be effectively managed, testing should be planned and conducted in a 
structured and disciplined fashion. According to DOD and industry 
guidance,[Footnote 22] system testing should be progressive, meaning 
that it should consist of a series of test events that first focus on 
the performance of individual system components, then on the 
performance of integrated system components, followed by system-level 
tests that focus on whether the entire system (or major system 
increments) is acceptable, interoperable with related systems, and 
operationally suitable to users. For this series of related test events 
to be conducted effectively, all test events need to be, among other 
things, governed by a well-defined test management structure and 
adequately planned. Further, the results of each test event need to be 
captured and used to ensure that problems discovered are disclosed and 
corrected. 

Key aspects of Navy ERP testing have been effectively managed. 
Specifically, the program has established an effective test management 
structure, key development events were based on well-defined plans, the 
results of all executed test events were documented, and problems found 
during testing (i.e., test defects) were captured in a test management 
tool and subsequently analyzed, resolved, and disclosed to decision 
makers. Further, while we identified instances in which the tool did 
not contain key data about defects that are needed to ensure that 
unauthorized changes to the status of defects do not occur, the number 
of instances found are not sufficient to conclude that the controls 
were not operating effectively. Notwithstanding the missing data, this 
means that Navy ERP testing has been performed in a manner that 
increases the chances that the system will meet operational needs and 
perform as intended. 

A Well-defined Test Management Structure Has Been Established: 

The program office has established a test management structure that 
satisfies key elements of DOD and industry guidance.[Footnote 23] For 
example, the program has developed a Test and Evaluation Master Plan 
(TEMP) that defines the program's test strategy. As provided for in the 
guidance, this strategy consists of a sequence of tests in a simulated 
environment to verify first that individual system parts meet specified 
requirements (i.e., development testing) and then verify that these 
combined parts perform as intended in an operational environment (i.e., 
operational testing). As we have previously reported,[Footnote 24] such 
a sequencing of test events is an effective approach because it permits 
the source of defects to be isolated sooner, before it is more 
difficult and expensive to address. 

More specifically, the strategy includes a sequence of developmental 
tests for each release consisting of three cycles of integrated system 
testing (IST) followed by user acceptance testing (UAT). Following 
development testing, the sequence of operational tests includes the 
Navy's independent operational test agency conducting initial 
operational test and evaluation (IOT&E) and then follow-on operational 
test and evaluation (FOT&E), as needed, to validate the resolution of 
deficiencies found during IOT&E. See table 5 for a brief description of 
the purpose of each test activity, and figure 4 for the schedule of 
Release 1.0 and 1.1 test activities. 

Table 5: Description of the Purpose of Navy ERP Tests: 

Test: Developmental testing: IST; 
Purpose: To validate that the technical and functional components of 
the system work properly together and operate as specified by the 
requirements. 

Test: Developmental testing: Cycle 1 (Scenario Testing); 
Purpose: To validate chains of business process transactions using 
small scenarios, such as a standard sales order, delivery, and 
invoicing. Also, independent evaluators observe scenario testing in 
preparation for operational test and evaluation. 

Test: Developmental testing: Cycle 2 (Scenario Testing and Conversions 
and Interfaces); 
Purpose: To validate more complex sequences of transactions plus 
customized software. 

Test: Developmental testing: Cycle 3 (Final Integration Testing); 
Purpose: To validate the entire system, including external components. 

Test: Developmental testing: UAT; 
Purpose: To allow the customer to ensure Navy ERP works properly and 
operates as specified by the requirements. 

Test: Operational testing: IOT&E; Purpose: To evaluate the operational 
effectiveness and suitability of the system. 

Test: Operational testing: FOT&E; Purpose: To verify the correction of 
deficiencies identified during IOT&E. 

Source: GAO analysis of DON data. 

[End of table] 

Figure 4: Release 1.0 and 1.1 Test Activity Schedule: 

[Refer to PDF for image: illustration] 

Release 1.0: Developmental testing: 
IST Cycle 1: Duration FY 2007, October-November; 
IST Cycle 2: Duration FY 2007, December-January; 
IST Cycle 3: Duration FY 2007, February-June; 
UAT: Duration FY 2007, July-August; 
Release 1.0: IOT&E: 
at NAVAIR: Duration, FY 2008, November-March; 
Release 1.0: FOT&E: 
at NAVAIR and NAVSUP: Duration, FY 2009, January-April. 

Release 1.1: Developmental testing: 
IST Cycle 1: Duration, FY 2009, January-February; 
IST Cycle 2: Duration, FY 2009, March-April; 
IST Cycle 3: Duration, FY 2009, May-October, 2010; 
UAT: Duration, FY 2010, October-December; 
Release 1.1: IOT&E: 
at NAVSUP: Duration, FY 2010, May-August. 

Source: GAO analysis of DON data. 

[End of figure] 

The TEMP also clearly identifies the roles and responsibilities of key 
Navy ERP testing organizations, as provided for in DOD and industry 
guidance. For example, it describes specific responsibilities of the 
program manager, system integrator, quality assurance/test team lead, 
and independent operational test and evaluation organizations. Table 6 
summarizes the responsibilities of these various test organizations. 

Table 6: Navy ERP Testing-Related Organizations and Respective Roles 
and Responsibilities: 

Testing-related organization: Program manager; 
Responsibilities: Provides overall management and direction of Navy ERP 
test and evaluation; Conducts test readiness reviews; Certifies that 
the program is ready to proceed from developmental to operational 
testing in a developmental test and evaluation report. 

Testing-related organization: System integrator; 
Responsibilities: Supports the execution of integration and user 
acceptance testing, including training system testers and users; 
Reports to the Navy ERP program manager. 

Testing-related organization: Quality assurance/test team lead; 
Responsibilities: Creates the test and evaluation strategy and 
developmental test and evaluation plan; Assists in planning, 
coordinating, and conducting developmental testing and evaluation, and 
reporting the results to the program manager; Conducts integration 
testing. 

Testing-related organization: Operational Test and Evaluation Force; 
Responsibilities: Plans and conducts Navy ERP operational test and 
evaluation (OT&E); Reports results and recommendations to DOD's 
Director, Operational Test and Evaluation; Performs follow-on OT&E to 
verify that deficiencies found during initial OT&E have been resolved. 

Testing-related organization: Joint Interoperability Test Command; 
Responsibilities: Certifies to the Joint Chiefs of Staff that 
interoperability requirements are met; Verifies readiness for 
interoperability to the responsible operational test agency during or 
prior to operational test readiness review. 

Testing-related organization: Office of Director, Operational Test and 
Evaluation; 
Responsibilities: Reviews and approves IOT&E and FOT&E plans; Analyzes 
OT&E results; Provides independent assessment to the MDA. 

Source: GAO analysis of DOD data. 

[End of table] 

Well-defined Plans for Developmental Test Events Were Developed: 

According to relevant guidance,[Footnote 25] test activities should be 
governed by well-defined and approved plans. Among other things, such 
plans are to include a defect triage process, metrics for measuring 
progress in resolving defects, test entrance and exit criteria, and 
test readiness reviews. 

Each developmental test event for Release 1.0 (i.e., each cycle of 
integrated systems testing and user acceptance testing) was based on a 
well-defined test plan. For example, each plan provided for conducting 
daily triage meetings to (1) assign new defects a criticality level 
using documented criteria,[Footnote 26] (2) record new defects and 
update the status of old defects in the test management tool, and (3) 
address other defect and testing issues. Further, each plan included 
defect metrics, such as the number of defects found and corrected and 
their age. In addition, each plan specified that testing was not 
complete until all major defects found during the cycle were resolved, 
and all unresolved defects' impact on the next test event were 
understood. Further, the plans provided for holding test readiness 
reviews to review test results as a condition for proceeding to the 
next event. By ensuring that plans for key development test activities 
include these aspects of effective test planning, the risk of test 
activities not being effectively and efficiently performed is reduced, 
thus increasing the chances that the system will meet operational 
requirements and perform as intended. 

Test Results Were Documented and Reported, but Key Information about 
Changes to the Status of Reported Defects Was Not Always Recorded: 

According to industry guidance[Footnote 27], effective system testing 
includes capturing, analyzing, resolving, and disclosing to decision 
makers the status of problems found during testing (i.e., test 
defects). Further, this guidance states that these results should be 
collected and stored according to defined procedures and placed under 
appropriate levels of control to ensure that any changes to the results 
are fully documented. 

To the program's credit, the relevant testing organizations have 
documented test defects in accordance with defined plans. For example, 
daily triage meetings involving the test team lead, testers, and 
functional experts were held to review each new defect, assign it a 
criticality level, and designate someone responsible for resolving it 
and for monitoring and updating its resolution in the test management 
tool. Further, test readiness reviews were conducted at which entrance 
and exit criteria for each key test event were evaluated before 
proceeding to the next event. As part of these reviews, the program 
office and oversight officials, command representatives, and test 
officials reviewed the results of test events to ensure, among other 
things, that significant defects were closed and that there were no 
unresolved defects that could affect execution of the next test event. 

However, the test management tool did not always contain key data for 
all recorded defects that are needed to ensure that unauthorized 
changes to the status of defects do not occur. According to information 
systems auditing guidelines,[Footnote 28] audit tools should be in 
place to monitor user access to systems to detect possible errors or 
unauthorized changes. For Navy ERP, this was not always the case. 
Specifically, while the tool has the capability to track changes to 
test defects in a history log,[Footnote 29] our analysis of 80 randomly 
selected defects in the tool disclosed two instances in which the tool 
did not record when a change in the defect's status was made or who 
made the change. In addition, our analysis of 12 additional defects 
that were potential anomalies[Footnote 30] disclosed two additional 
instances where the tool did not record when a change was made and who 
made it. While our sample size and results do not support any 
conclusions as to the overall effectiveness of the controls in place 
for recording and tracking test defect status changes, they do show 
that it is possible that changes can be made without a complete audit 
trail surrounding those changes. After we shared our results with 
program officials, they stated that they provided each instance for 
resolution to the vendor responsible for the tracking tool. These 
officials attributed these instances to vendor updates to the tool that 
caused the history settings to default to "off." To address this 
weakness, they added that they are now ensuring that the history logs 
are set correctly after any update to the tool. This addition is a 
positive step because without an effective information system access 
audit tool, the probability of test defect status errors or 
unauthorized changes is increased. 

System Changes Have Been Controlled, but Their Cost and Schedule 
Impacts Were Not Sufficiently Considered: 

Industry best practices and DOD guidance[Footnote 31] recognize the 
importance of system change control when developing and maintaining a 
system. Once the composition of a system is sufficiently defined, a 
baseline configuration is normally established, and changes to that 
baseline are placed under a disciplined change control process to 
ensure that unjustified and unauthorized changes are not introduced. 
Elements of disciplined change control include (1) formally documenting 
a change control process, (2) rigorously adhering to the documented 
process, and (3) adopting objective criteria for considering a proposed 
change, including its estimated cost and schedule impact. 

To its credit, the Navy ERP program has formally documented a change 
control process. Specifically, it has a plan and related procedures 
that include the purpose and scope of the process--to ensure that any 
changes made to the system are properly identified, developed, and 
implemented in a defined and controlled environment. It also is using 
an automated tool to capture and track the disposition of each change 
request. Further, it has defined roles and responsibilities and a 
related decision-making structure for reviewing and approving system 
changes. In this regard, the program has established a hierarchy of 
review and approval boards, including a Configuration Control Board to 
review all changes and a Configuration Management Board to further 
review changes estimated to require more than 100 hours or $25,000 to 
implement. Furthermore, a Navy ERP Senior Integration Board was 
recently established to review and approve requests to add, delete, or 
change the program's requirements. In addition, the change control 
process states that the decisions are to be based on, among others, the 
system engineering and earned value management (i.e., cost and 
schedule) impacts the change will introduce, such as the estimated 
number of work hours that will be required to effect the change. Table 
7 provides a brief description of the decision-making authorities and 
boards and their respective roles and responsibilities. 

Table 7: Roles and Responsibilities for Change Review and Approval: 

Review and approval organizations: Navy ERP Senior Integration Board; 
Roles and responsibilities: Reviews and approves Engineering Change 
Proposals, which are proposed changes that would impact system scope, 
configuration, cost, or schedule by adding, deleting, or changing 
requirements. The board is chaired by the Principal Deputy Assistant 
Secretary of the Navy, Research, Development, and Acquisition. 

Review and approval organizations: Configuration Management Board; 
Roles and responsibilities: Reviews and approves change requests 
requiring more than 100 hours or $25,000 to implement. The board is 
chaired by the program manager and includes representatives from the 
earned value management team (i.e., cost and schedule). 

Review and approval organizations: Configuration Control Board; 
Roles and responsibilities: Reviews all change requests and approves 
those requiring less than 100 hours or $25,000 to implement. The board 
is chaired by the systems engineer and includes representatives from 
the earned value management team (i.e., cost and schedule). 

Review and approval organizations: Engineering Review Board; 
Roles and responsibilities: Ensures change requests are ready to 
proceed to the Configuration Control Board by reviewing and 
recommending changes. This board is facilitated and chaired by the 
systems engineer and the configuration manager to ensure the change 
request documentation is complete. 

Review and approval organizations: Technical Change Control Board; 
Roles and responsibilities: Approves or defers transport change 
requests, which are requests to release changes into the deployed 
system. The board is chaired by the production manager. 

Source: GAO analysis of DON documentation. 

[End of table] 

Navy ERP is largely adhering to its documented change control process. 
Specifically, our review of a random sample of 60 change requests and 
minutes of related board meetings held between May 2006 and April 2009 
showed that the change requests were captured and tracked using an 
automated tool, and they were reviewed and approved by the designated 
decision-making authorities and boards, in accordance with the 
program's documented process. 

However, the program has not sufficiently or consistently considered 
the cost and schedule impacts of proposed changes. Our analysis of the 
random sample of 60 change requests, including our review of related 
board meeting minutes, showed no evidence that cost and schedule 
impacts were identified or that they were considered. Specifically, we 
did not see evidence that the cost and schedule impacts of these change 
requests were assessed. According to program officials, the cost and 
schedule impacts of each change were discussed at control board 
meetings. In addition, they provided two change requests to demonstrate 
this. However, while these change requests did include schedule impact, 
they did not include the anticipated cost impact of proposed changes. 
Rather, these two, as well as those in our random sample, included the 
estimated number of work hours required to implement the change. 
Because the cost of any proposed change depends on other factors 
besides work hours, such as labor rates, the estimated number of work 
hours is not sufficient for considering the cost impact of a change. In 
the absence of verifiable evidence that cost and schedule impacts were 
consistently considered, approval authorities do not appear to have 
been provided key information needed to fully inform their decisions on 
whether or not to approve a change. System changes that are approved 
without a full understanding of their cost and schedule impacts could 
result in unwarranted cost increases and schedule delays. 

Navy ERP IV&V Function Is Not Independent and Has Not Been Fully 
Performed: 

The purpose of IV&V is to independently ensure that program processes 
and products meet quality standards. The use of an IV&V function is 
recognized as an effective practice for large and complex system 
development and acquisition programs, like Navy ERP, as it provides 
objective insight into the program's processes and associated work 
products.[Footnote 32] To be effective, verification and validation 
activities should be performed by an entity that is managerially 
independent of the system development and management processes and 
products that are being reviewed.[Footnote 33] Among other things, such 
independence helps to ensure that the results are unbiased and based on 
objective evidence. 

The Navy has not effectively managed its IV&V function because it has 
not ensured that the contractor performing this function is independent 
of the products and processes that this contractor is reviewing and 
because it has not ensured that the contractor is meeting contractual 
requirements. In June 2006, DON awarded a professional support services 
contract to General Dynamics Information Technology (GDIT), to include 
responsibilities for, among other things, IV&V, program management 
support, and delivery of releases according to cost and schedule 
constraints. According to the program manager, the contractor's IV&V 
function is organizationally separate from, and thus independent of, 
the contractor's Navy ERP system development function. However, the 
subcontractor performing the IV&V function is also performing release 
management. According to the GDIT contract, the release manager is 
responsible for developing and deploying a system release that meets 
operational requirements within the program's cost and schedule 
constraints, but it also states that the IV&V function is responsible 
for supporting the government in its review, approval, and acceptance 
of Navy ERP products (e.g., releases). The contract also states that 
GDIT is eligible for an optional award fee payment based on its 
performance in meeting, among other things, these cost and schedule 
constraints. Because performance of the system development and 
management role makes the contractor potentially unable to render 
impartial assistance to the government in performing the IV&V function, 
the contractor has an inherent conflict of interest relative to meeting 
cost and schedule commitments and disclosing the results of 
verification and validation reviews that may affect its ability to do 
so. 

The IV&V function's lack of independence is amplified by the fact that 
it reports directly and solely to the program manager. As we have 
previously reported,[Footnote 34] the IV&V function should report the 
issues or weaknesses that increase the risks associated with the 
project to program oversight officials, as well as to program 
management, to better ensure that the verification and validation 
results are objective and that the officials responsible for making 
program investment decisions are fully informed. Furthermore, these 
officials, once informed, can ensure that the issues or weaknesses 
reported are promptly addressed. 

Without ensuring sufficient managerial independence, valuable 
information may not reach decision makers, potentially leading to the 
release of a system that does not adequately meet users' needs and 
operate as intended. 

Beyond the IV&V function's lack of independence, the program office has 
not ensured that the subcontractor has produced the range of 
deliverables that were contractually required and defined in the IV&V 
plan. For example, the contract and plan call for weekly and monthly 
reports identifying weaknesses in program processes and recommendations 
for improvement, a work plan for accomplishing IV&V tasks, and 
associated assessment reports that follow the System Engineering Plan 
and program schedule. However, the IV&V contractor has largely not 
delivered these products. Specifically, until recently, it did not 
produce a work plan and only monthly reports were delivered, and these 
reports only list meetings that the IV&V contactor attended and 
documents that it reviewed. They do not, for example, identify program 
weaknesses or provide recommendations for improvement. According to 
program officials, they have relied on oral reports from the 
subcontractor at weekly meetings, and these lessons learned have been 
incorporated into program guidance. According to the contractor, the 
Navy has expended about $1.8 million between June 2006 and September 
2008 for IV&V activities, with an additional $249,000 planned to be 
spent in fiscal year 2009. 

Following our inquiries about an IV&V work plan, the IV&V contractor 
developed such a plan in October 2008, more than 2 years after the 
contract was awarded, that lists program activities and processes to be 
assessed, such as configuration management and testing. While this plan 
does not include time frames for starting and completing these 
assessments, meeting minutes show that the status of assessments has 
been discussed with the program manager during IV&V review meetings. 
The first planned assessment was delivered to the program in March 2009 
and provides recommendations for improving the program's configuration 
management process, such as using the automated tool to produce certain 
reports and enhancing training to understand how the tool is used. 
Further, program officials stated that they have also recently begun 
requiring the contractor to provide formal quarterly reports, the first 
of which was delivered to the program manager in January 2009. Our 
review of this quarterly report shows that it provides recommendations 
for improving the program's risk management process and organizational 
change management strategy. 

Notwithstanding the recent steps that the program office has taken, it 
nevertheless lacks an independent perspective on the program's products 
and management processes. 

Conclusions: 

DOD's successes in delivering large-scale business systems, such as 
Navy ERP, are in large part determined by the extent to which it 
employs the kind of rigorous and disciplined IT management controls 
that are reflected in department policies and related guidance. While 
implementing these controls does not guarantee a successful program, it 
does minimize a program's exposure to risk and thus the likelihood that 
it will fall short of expectations. In the case of Navy ERP, living up 
to expectations is important because the program is large, complex, and 
critical to addressing the department's long-standing problems related 
to financial transparency and asset visibility. 

The Navy ERP program office has largely implemented a range of 
effective controls associated with system testing and change control, 
including acting quickly to address issues with the audit log for its 
test management tool, but more can be done to ensure that the cost and 
schedule impacts of proposed changes are explicitly documented and 
considered when decisions are reached. Moreover, while the program 
office has contracted for IV&V activities, it has not ensured that the 
contractor is independent of the products and processes that it is to 
review and has not held the contractor accountable for producing the 
full range of IV&V deliverables required under the contract. Moreover, 
it has not ensured that its IV&V contractor is accountable to a level 
of management above the program office, as we previously recommended. 
Notwithstanding the program office's considerable effectiveness in how 
it has managed both system testing and change control, these weaknesses 
increase the risk of investing in system changes that are not 
economically justified and unnecessarily limit the value that an IV&V 
agent can bring to a program like Navy ERP. By addressing these 
weaknesses, the department can better ensure that taxpayer dollars are 
wisely and prudently invested. 

Recommendations for Executive Action: 

To strengthen the management of Navy ERP's change control process, we 
recommend that the Secretary of Defense direct the Secretary of the 
Navy, through the appropriate chain of command, to (1) revise the Navy 
ERP procedures for controlling system changes to explicitly require 
that a proposed change's life-cycle cost impact be estimated and 
considered in making change request decisions and (2) capture the cost 
and schedule impacts of each proposed change in the Navy ERP automated 
change control tracking tool. 

To increase the value of Navy ERP IV&V, we recommend that the Secretary 
of Defense direct the Secretary of the Navy, through the appropriate 
chain of command, to (1) stop performance of the IV&V function under 
the existing contract and (2) engage the services of a new IV&V agent 
that is independent of all Navy ERP management, development, testing, 
and deployment activities that it may review. In addition, we reiterate 
our prior recommendation relative to ensuring that the Navy ERP IV&V 
agent report directly to program oversight officials, while 
concurrently sharing IV&V results with the program office. 

[Refer to PDF for image] 

[End of figure] 

Agency Comments: 

In written comments on a draft of this report, signed by the Assistant 
Deputy Chief Management Officer and reprinted in appendix II, the 
department concurred with our recommendations, and stated that it will 
take the appropriate corrective actions within the next 7 months. 

We are sending copies of this report to interested congressional 
committees; the Director, Office of Management and Budget; the 
Congressional Budget Office; and the Secretary of Defense. The report 
also is available at no charge on our Web site at [hyperlink, 
http://www.gao.gov]. 

If you or your staffs have any questions on matters discussed in this 
report, please contact me at (202) 512-3439 or hiter@gao.gov. Contact 
points for our Offices of Congressional Relations and Public Affairs 
may be found on the last page of this report. GAO staff who made major 
contributions to this report are listed in appendix III. 

Signed by: 

Randolph C. Hite: 
Director: 
Information Technology Architecture and Systems Issues: 

[End of section] 

Appendix I: Objectives, Scope, and Methodology: 

Our objectives were to determine whether (1) system testing is being 
effectively managed, (2) system changes are being effectively 
controlled, and (3) independent verification and validation (IV&V) 
activities are being effectively managed for the Navy Enterprise 
Resource Planning (ERP) program. 

To determine if Navy ERP testing is being effectively managed, we 
reviewed relevant documentation, such as the Test and Evaluation Master 
Plan and test reports and compared them with relevant federal and 
related guidance. Further, we reviewed development test plans and 
procedures for each test event and compared them with best practices to 
determine whether well-defined plans were developed. We also examined 
test results and reports, including test readiness review documentation 
and compared them against plans to determine whether they had been 
executed in accordance with the plans. Moreover, to determine the 
extent to which test defect data were being captured, analyzed, and 
reported, we inspected 80 randomly selected defects from a sample of 
2,258 defects in the program's test management system. In addition, we 
reviewed the history logs associated with each of these 80 defects to 
determine whether appropriate levels of control were in place to ensure 
that any changes to the results were fully documented. This sample was 
designed with a 5 percent tolerable error rate at the 95 percent level 
of confidence, so that, if we found 0 problems in our sample, we could 
conclude statistically that the error rate was less than 4 percent. In 
addition, we interviewed cognizant officials, including the program's 
test lead and the Navy's independent operational testers, about their 
roles and responsibilities for test management. 

To determine if Navy ERP changes are being effectively controlled, we 
reviewed relevant program documentation, such as the change control 
policies, plans, and procedures, and compared them with relevant 
federal and industry guidance. Further, to determine the extent to 
which the program is reviewing and approving change requests according 
to its documented plans and procedures, we inspected 60 randomly 
selected change requests in the program's configuration management 
system. In addition, we reviewed the change request forms associated 
with these 60 change requests and related control board meeting minutes 
to determine whether objective criteria for considering a proposed 
change, including estimated cost or schedule impacts, were adopted. In 
addition, we interviewed cognizant officials, including the program 
manager and systems engineer, about their roles and responsibilities 
for reviewing, approving, and tracking change requests. 

To determine if IV&V activities are being effectively managed we 
reviewed Navy ERP's IV&V contract, strategy, and plans and compared 
them with relevant industry guidance. We also analyzed the contractual 
relationships relative to legal standards that govern organizational 
conflict of interest. In addition, we examined IV&V monthly status 
reports, work plans, an assessment report, and a quarterly report, to 
determine the extent to which contract requirements were met. We 
interviewed contractor and program officials about their roles and 
responsibilities for IV&V and to determine the extent to which the 
program's IV&V function is independent. 

We conducted this performance audit at Department of Defense offices in 
the Washington, D.C., metropolitan area; Annapolis, Maryland; and 
Norfolk, Virginia; from August 2008 to September 2009, in accordance 
with generally accepted government auditing standards. Those standards 
require that we plan and perform the audit to obtain sufficient, 
appropriate evidence to provide a reasonable basis for our findings and 
conclusions based on our audit objectives. We believe that the evidence 
obtained provides a reasonable basis for our findings and conclusions 
based on our audit objectives. 

Appendix II: Comments from the Department of Defense: 

Office Of The Deputy Chief Management Officer: 
9010 Defense Pentagon: 
Washington, DC 20301-9010: 

August 21, 2009: 

Mr. Randolph C. Hite: 
Director, Information Technology Architecture and Systems Issues: 
U.S. Government Accountability Office: 
441 G Street, NW: 
Washington, DC 20548: 

Dear Mr. Hite: 

This is the Department of Defense (DoD) response to the GAO draft 
report 09-841, "DOD Business Systems Modernization: Navy Implementing A 
Number Of Key Management Controls On Enterprise Resource Planning 
System, but Improvements Still Needed," dated July 21, 2009 (GAO Code 
310666). 

The Department concurs with all four of GAO's recommendations. The Navy 
Enterprise Resource Planning (ERP) program office will take the 
appropriate corrective actions within the next seven months. Detailed 
responses to each recommendation are attached. 

We appreciate the support of GAO as the Department further advances in 
its business transformation efforts, and look forward to continuing our 
partnership in achieving our shared goals. 

Signed by: 

Elizabeth A. McGrath: 
Assistant Deputy Chief Management Officer: 

Attachment(s): As stated: 

[End of letter] 

GAO Draft Report Dated July 21, 2009: 
GAO-09-841 (GAO Code 310666): 

"DOD Business Systems Modernization: Navy Implementing A Number Of Key 
Management Controls On Enterprise Resource Planning System, But 
Improvements Still Needed" 

Department Of Defense Comments To The GAO Recommendations: 

Recommendation 1: The GAO recommends that the Secretary of Defense 
direct the Secretary of the Navy, through the appropriate chain of 
command, to revise the Navy Enterprise Resource Planning (ERP) 
procedures for controlling system changes to explicitly require that a 
proposed change's life cycle cost impact be estimated and considered in 
making change request decisions. (Page 31/GAO Draft Report). 

DOD Response: Concur. The Navy ERP program office will revise the 
Enterprise Change Request Process and Procedures document to require 
explicitly that a proposed change's life cycle cost impact be estimated 
as part of the change control decision process. Corrective actions to 
address GAO's recommendation will he taken by the beginning of Fiscal 
Year (FY) 2010. 

Recommendation 2: The GAO recommends that the Secretary of Defense 
direct the Secretary of the Navy, through the appropriate chain of 
command, to capture the cost and schedule impacts of each proposed 
change in the Navy ERP automated change control tracking tool. (Page 
31/GAO Draft Report). 

DOD Response: Concur. The Navy ERP program office will update its 
automated change control tracking tool to capture "dollarized" cost 
impacts and better identify schedule impacts of future proposed 
changes. Corrective actions to address GAO's recommendation will he 
taken by the beginning of FY 2010. 

Recommendation 3: The GAO recommends that the Secretary of Defense 
direct the Secretary of the Navy, through the appropriate chain of 
command, to stop performance of the Independent Verification and 
Validation (IV&V) function under the existing contract. (Page 31/GAO 
Draft Report). 

DOD Response: Concur. The Navy ERP program office plans to stop IV&V 
functions under the existing contract at the end of the current fiscal 
year (2009). 

Recommendation 4: The GAO recommends that the Secretary of Defense 
direct the Secretary of the Navy, through the appropriate chain of 
command. to engage the services of a new IV&V agent that is independent 
of all Navy ERP management, development, testing, and deployment 
activities that it may review. (Page 31/GAO Draft Report). 

DOD Response: Concur. The Navy ERP program plans to execute future IV&V 
functions using contract support that is not associated with any of the 
other Navy ERP program activities to ensure there is no conflict of 
interest, real or perceived. Corrective actions to address GAO's 
recommendation will be taken no later than March 2010. 

[End of section] 

Appendix III: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Randolph C. Hite, (202) 512-3439, or hiter@gao.gov: 

Staff Acknowledgments: 

In addition to the individual named above, key contributors to this 
report were Neelaxi Lakhmani, Assistant Director; Monica Anatalio; Carl 
Barden; Neil Doherty; Cheryl Dottermusch; Lee McCracken; Karl Seifert; 
Adam Vodraska; Shaunyce Wallace; and Jeffrey Woodward. 

[End of section] 

Footnotes: 

[1] Business systems are information systems, including financial and 
nonfinancial systems that support DOD business operations, such as 
civilian personnel, finance, health, logistics, military personnel, 
procurement, and transportation. 

[2] GAO, High-Risk Series: An Update, [hyperlink, 
http://www.gao.gov/products/GAO-09-271] (Washington, D.C.: January 
2009). 

[3] See, for example, GAO, DOD Business Systems Modernization: 
Important Management Controls Being Implemented on Major Navy Program, 
but Improvements Needed in Key Areas, [hyperlink, 
http://www.gao.gov/products/GAO-08-896] (Washington, D.C.: Sept. 8, 
2008); DOD Business Systems Modernization: Key Marine Corps System 
Acquisition Needs to Be Better Justified, Defined, and Managed, 
[hyperlink, http://www.gao.gov/products/GAO-08-822] (Washington, D.C.: 
July 28, 2008); and DOD Business Transformation: Lack of an Integrated 
Strategy Puts the Army's Asset Visibility System Investments at Risk, 
[hyperlink, http://www.gao.gov/products/GAO-07-860] (Washington, D.C.: 
July 27, 2007). 

[4] [hyperlink, http://www.gao.gov/products/GAO-08-896]. 

[5] The four pilots are SIGMA, CABRILLO, NEMAIS, and SMART. 

[6] The Navy is considering deleting the third release, Release 1.2, 
from Template 1. 

[7] Release 1.0 accounts for about 56 percent of the requirements; 
Release 1.1, about 33 percent; and Release 1.2, about 10 percent. 

[8] This 2003 estimate, which was prepared to assist in budget 
development and support the Milestone A/B approval, was for 
development, deployment, and sustainment costs in fiscal years 2003 
through 2021. 

[9] According to DOD's acquisition guidebook, an Acquisition Program 
Baseline is a program manager's estimated cost, schedule, and 
performance goals. Goals consist of objective values, which represent 
what the user desires and expects, and threshold values, which 
represent acceptable limits. When the program manager determines that a 
current cost, schedule, or performance threshold value will not be 
achieved, the MDA must be notified, and a new baseline developed, 
reviewed by decision makers and, if the program is to continue, 
approved by the MDA. 

[10] According to the August 2004 Acquisition Program Baseline, this 
estimate is for acquisition, operations, and support for fiscal years 
2004 through 2021. 

[11] According to the September 2007 Acquisition Program Baseline, this 
estimate is for acquisition, operations, and support for fiscal years 
2004 through 2023. 

[12] The defense acquisition system is a framework-based approach that 
is intended to translate mission needs and requirements into stable, 
affordable, and well-managed acquisition programs. It was updated in 
December 2008 and consists of five key program life-cycle phases and 
three related milestone decision points--(1) Materiel Solution Analysis 
(previously Concept Refinement), followed by Milestone A; (2) 
Technology Development, followed by Milestone B; (3) Engineering and 
Manufacturing Development (previously System Development and 
Demonstration), followed by Milestone C; (4) Production and Deployment; 
and (5) Operations and Support. 

[13] Fleet and Industrial Supply Centers are located in San Diego, 
California; Norfolk, Virginia; Jacksonville, Florida; Puget Sound, 
Washington; Pearl Harbor, Hawaii; Yokosuka, Japan; and Sigonella, 
Italy; and provide worldwide logistics services for the Navy. 

[14] See, for example, [hyperlink, 
http://www.gao.gov/products/GAO-08-896]; GAO, DOD Business Systems 
Modernization: Planned Investment in Navy Program to Create Cashless 
Shipboard Environment Needs to be Justified and Better Managed, 
[hyperlink, http://www.gao.gov/products/GAO-08-922] (Washington, D.C.: 
Sept. 8, 2008); [hyperlink, http://www.gao.gov/products/GAO-08-822]; 
[hyperlink, http://www.gao.gov/products/GAO-07-860]; Information 
Technology: DOD Needs to Ensure that Navy Marine Corps Intranet Program 
Is Meeting Goals and Satisfying Customers, [hyperlink, 
http://www.gao.gov/products/GAO-07-51] (Washington, D.C.: Dec. 8, 
2006); DOD Systems Modernization: Planned Investment in the Navy 
Tactical Command Support System Needs to be Reassessed, [hyperlink, 
http://www.gao.gov/products/GAO-06-215] (Washington, D.C.: Dec. 5, 
2005); and DOD Business Systems Modernization: Navy ERP Adherence to 
Best Business Practices Critical to Avoid Past Failures, [hyperlink, 
http://www.gao.gov/products/GAO-05-858] (Washington, D.C.: Sept. 29, 
2005). 

[15] Ronald W. Reagan National Defense Authorization Act for Fiscal 
Year 2005, Pub. L. No. 108-375, Sec. 332 (2004) (codified at 10 U.S.C. 
Sections 186 and 2222). 

[16] Earned value management is a means for measuring actual program 
progress against cost and schedule estimates. 

[17] [hyperlink, http://www.gao.gov/products/GAO-08-896]. 

[18] [hyperlink, http://www.gao.gov/products/GAO-08-822]. 

[19] [hyperlink, http://www.gao.gov/products/GAO-07-860]. 

[20] [hyperlink, http://www.gao.gov/products/GAO-06-215]. 

[21] [hyperlink, http://www.gao.gov/products/GAO-05-858]. 

[22] See, for example, Office of the Under Secretary of Defense for 
Acquisition, Technology, and Logistics, Department of Defense 
Instruction 5000.02 (Arlington, VA: Dec. 2, 2008); Defense Acquisition 
University, Test and Evaluation Management Guide, 5th ed. (Fort 
Belvoir, VA: January 2005); Institute of Electrical and Electronics 
Engineers, Inc., Standard for Software Verification and Validation, 
IEEE Std 1012-2004 (New York, NY: June 8, 2005); Software Engineering 
Institute, Capability Maturity Model Integration for Acquisition, 
version 1.2 (Pittsburgh, PA: May 2008); and GAO, Year 2000 Computing 
Crisis: A Testing Guide, [hyperlink, 
http://www.gao.gov/products/GAO/AIMD-10.1.21] (Washington, D.C.: 
November 1998). 

[23] Office of the Under Secretary of Defense for Acquisition, 
Technology, and Logistics, Department of Defense Instruction 5000.02 
(Arlington, VA: Dec. 2, 2008); Defense Acquisition University, Test and 
Evaluation Management Guide, 5th ed. (Fort Belvoir, VA: January 2005); 
and Institute of Electrical and Electronics Engineers, Inc., Standard 
for Software and System Test Documentation, IEEE Std 829-2008 (New 
York, NY: 2008). 

[24] GAO, Secure Border Initiative: DHS Needs to Address Significant 
Risks in Delivering Key Technology Investment, [hyperlink, 
http://www.gao.gov/products/GAO-08-1086] (Washington, D.C.: Sept. 22, 
2008). 

[25] See, for example, [hyperlink, 
http://www.gao.gov/products/GAO/AIMD-10.1.21]. 

[26] According to program documentation, criticality levels range from 
1 to 5, as follows: 1 is a problem that prevents accomplishment of an 
operational or mission critical capability; 2 is a major technical 
problem with no work-around solution; 3 is a major technical problem 
with a work-around solution; 4 is a minor technical problem; and 5 is 
any other defect, such as a cosmetic problem. 

[27] Institute of Electrical and Electronics Engineers, Inc., Standard 
for Information Technology--Software Life Cycle Processes--
Implementation Considerations, IEEE/EIA Std 12207.2-1997 (New York, NY: 
April 1998) and Software Engineering Institute, Capability Maturity 
Model Integration for Acquisition, version 1.2 (Pittsburgh, PA: May 
2008). 

[28] Information Systems Audit and Control Association, Inc., IS 
Standards, Guidelines and Procedures for Auditing and Control 
Professionals (Rolling Meadows, IL: Jan. 15, 2009). 

[29] According to program documentation, the date a defect is entered 
into the system and the date the status of the defect is changed to 
"closed" are automatically populated. Further, changes to a defect's 
status, including from "new" to "open" and from "open" to "closed;" 
changes to the criticality level; and the user who makes the changes 
are tracked in a defect's history log. 

[30] These anomalies are defects that we found that (1) were attributed 
to integrated system test events, but were not detected until after the 
system was deployed; (2) had a criticality level that was different 
from the level that was reported at a test readiness review; (3) were 
deferred to a later test event or to post-deployment to be verified as 
resolved; or (4) had no criticality level. 

[31] See, for example, Electronics Industries Alliance, National 
Consensus Standard for Configuration Management , ANSI/EIA-649-1998 
(Arlington, VA: August 1998) and Department of Defense, Military 
Handbook: Configuration Management Guidance, MIL-HDBK-61A(SE) 
(Washington, D.C.: Feb. 7, 2001). 

[32] GAO, Homeland Security: U.S. Visitor and Immigration Status 
Indicator Technology Program Planning and Execution Improvements 
Needed, [hyperlink, http://www.gao.gov/products/GAO-09-96] (Washington, 
D.C.: Dec. 12, 2008); Homeland Security: Recommendations to Improve 
Management of Key Border Security Program Need to Be Implemented, 
[hyperlink, http://www.gao.gov/products/GAO-06-296] (Washington, D.C.: 
Feb. 14, 2006); and [hyperlink, 
http://www.gao.gov/products/GAO-05-858]. 

[33] Institute of Electrical and Electronics Engineers, Inc., Standard 
for Software Verification and Validation, IEEE Std 1012-2004 (New York, 
NY: June 8, 2005). 

[34] See [hyperlink, http://www.gao.gov/products/GAO-07-860] and 
[hyperlink, http://www.gao.gov/products/GAO-05-858]. 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "E-mail Updates." 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO’s Web site, 
[hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional 
information. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: fraudnet@gao.gov: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Ralph Dawn, Managing Director, dawnr@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: