This is the accessible text file for GAO report number GAO-06-215 
entitled 'DOD Systems Modernization: Planned Investment in the Naval 
Tactical Command Support System Needs to Be Reassessed' which was 
released on December 6, 2005. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to the Subcommittee on Readiness and Management Support, 
Committee on Armed Services, U.S. Senate: 

December 2005: 

DOD Systems Modernization: 

Planned Investment in the Naval Tactical Command Support System Needs 
to Be Reassessed: 

GAO-06-215: 

GAO Highlights: 

Highlights of GAO-06-215, a report to the Subcommittee on Readiness and 
Management Support, Committee on Armed Services, U.S. Senate: 

Why GAO Did This Study: 

Because it is important that the Department of Defense (DOD) adheres to 
disciplined information technology (IT) acquisition processes to 
successfully modernize its business systems, GAO was asked to determine 
whether the Naval Tactical Command Support System (NTCSS) is being 
managed according to important aspects of DOD’s acquisition policies 
and guidance, as well as other relevant acquisition management best 
practices. NTCSS was started in 1995 to help Navy personnel effectively 
manage ship, submarine, and aircraft support activities. To date, about 
$1 billion has been spent to partially deploy NTCSS to about one-half 
its intended ashore and afloat sites.

What GAO Found: 

The Department of the Navy has not managed its NTCSS program in 
accordance with key aspects of the department’s policies and related 
guidance, including federal and recognized best practice guidance. 
Collectively, these policies and guidance are intended to reasonably 
ensure that investment in a given IT system represents the right 
solution to fill a mission need and, if it is, that acquisition and 
deployment of the system are handled in a manner that maximizes the 
chances of delivering defined system capabilities on time and within 
budget. In the case of NTCSS, neither of these outcomes is being 
realized. Specifically,

* The Navy has not economically justified its ongoing and planned 
investment in NTCSS. Specifically, it (1) has not reliably estimated 
future costs and benefits and (2) has not ensured that independent 
reviews of its economic justification were performed to determine its 
reliability. 

* The Navy has not invested in NTCSS within the context of a well-
defined DOD or Navy enterprise architecture, which is necessary to 
guide and constrain NTCSS in a way that promotes interoperability and 
reduces redundancy with related and dependent systems. 

* The Navy has not effectively performed key measurement, reporting, 
budgeting, and oversight activities. In particular, earned value 
management, which is a means for determining and disclosing actual 
performance against budget and schedule estimates, has not been 
implemented effectively, and oversight entities have not had the 
visibility into the program needed to affect its direction.

* The Navy has not adequately conducted requirements management and 
testing activities. For example, requirements were neither prioritized 
nor traced to related documentation to ensure that the system delivers 
capabilities that meet user needs. This contributed to failures in 
developmental testing that have prevented the latest component of NTCSS 
from passing operational testing twice over the last 4 years.
Reasons the Navy cited for not following policies and guidance ranged 
from their not being applicable to the NTCSS program, to lack of time 
available to apply them, to plans for strengthening system practices 
not being applied retroactively. Nevertheless, the Navy has begun 
taking steps and is considering other steps intended to address some of 
the above problems. Until program management improves, NTCSS will 
remain a risky program. 

What GAO Recommends: 

GAO is making recommendations to the Secretary of Defense to develop 
the analytical basis to determine if continued investment in NTCSS 
represents prudent use of limited resources. GAO is also making 
recommendations to strengthen management of the program, conditional 
upon a decision to proceed with further investment in the program. DOD 
either fully or partially concurred with the recommendations. It also 
stated that while some of GAO’s findings are valid, the overall 
findings understated and misrepresented the program’s level of 
discipline and conformance with applicable guidance and direction.

www.gao.gov/cgi-bin/getrpt?GAO-06-215. 

To view the full product, including the scope and methodology, click on 
the link above. For more information, contact Randolph C. Hite at (202) 
512-3439 or hiter@gao.gov. 

[End of section] 

Contents: 

Letter: 

Results in Brief: 

Background: 

NTCSS Has Not Been Managed in Accordance with DOD and Other Relevant 
System Acquisition and Development Guidance: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendixes: 

Appendix I: Objective, Scope, and Methodology: 

Appendix II: Trouble Reports and Change Proposals Assessment: 

Trouble Reports: 

Change Proposals: 

Appendix III: Earned Value Management Assessment: 

Appendix IV: Comments from the Department of Defense: 

Appendix V: GAO Contact and Staff Acknowledgments: 

Tables: 

Table 1: Legacy Systems and Applications: 

Table 2: Optimized Applications Developed During Stage 2 of the NTCSS 
Program: 

Table 3: Modernized Applications Developed During Stage 3 of the NTCSS 
Program: 

Table 4: Applications in Operation as of April 2005: 

Table 5: NTCSS Budget from FY 1995 through FY 2005: 

Table 6: NTCSS Oversight Roles and Responsibilities: 

Table 7: NTCSS Management and Stakeholder Roles and Responsibilities: 

Table 8: Navy Satisfaction of Cost Estimating Criteria: 

Table 9: Navy Satisfaction of OMB Economic Analysis Criteria: 

Table 10: Threshold Amounts in NTCSS Acquisition Program Baselines: 

Table 11: NTCSS Trouble Report and Change Proposal Priorities: 

Table 12: Navy Satisfaction of EVM Criteria: 

Figures: 

Figure 1: Total Number of Open NTCSS and OOMA Priority 1, 2, and 3 
Trouble Reports

Figure 2: Open NTCSS Priority 1 and 2 Trouble Reports

Figure 3: Open NTCSS Priority 3 Trouble Reports

Figure 4: Open OOMA Priority 1 and 2 Trouble Reports

Figure 5: Open OOMA Priority 3 Trouble Reports

Figure 6: Total Number of Open NTCSS and OOMA Priority 1, 2, and 3 
Change Proposals

Figure 7: Open NTCSS Priority 1 and 2 Change Proposals

Figure 8: Open NTCSS Priority 3 Change Proposals

Figure 9: Open OOMA Priority 1 and 2 Change Proposals

Figure 10: Open OOMA Priority 3 Change Proposals: 

Abbreviations: 

CDA: central design agency: 

CIO: Chief Information Officer: 

DOD: Department of Defense: 

ERP: enterprise resource planning: 

EVM: earned value management: 

IT: information technology: 

MRMS: Maintenance Resource Management System: 

NALCOMIS: Naval Aviation Logistics Command Management Information 
System: 

OMB: Office of Management and Budget: 

OOMA: Optimized Organizational Maintenance Activity: 

NTCSS: Naval Tactical Command Support System: 

PA&E: Office of Program Analysis and Evaluation: 

RIT: Rapid Improvement Team: 

SEI: Software Engineering Institute: 

SNAP: Shipboard Non-Tactical Automated Data Processing Program: 

SW-CMM: Software Capability Maturity Model: 

Letter December 5, 2005: 

The Honorable John Ensign: 
Chairman: 
The Honorable Daniel K. Akaka: 
Ranking Minority Member: 
Subcommittee on Readiness and Management Support: 
Committee on Armed Services: 
United States Senate: 

Because it is so important that the Department of Defense (DOD) adhere 
to disciplined information technology (IT) acquisition processes in 
order to successfully modernize its business systems, you requested 
that we determine whether the department is following its own revised 
policies and guidance for acquiring systems,[Footnote 1] which it 
issued in May 2003. As part of our response to your request, we agreed 
to review the Naval Tactical Command Support System (NTCSS) program. 
NTCSS was started in 1995 and is intended to help Navy personnel 
effectively manage ships, submarines, and aircraft support activities. 
The Navy expects to spend $348 million on NTCSS between fiscal years 
2006 and 2009, for a total of approximately $1.45 billion since program 
inception. 

As agreed, our objective was to determine whether NTCSS is being 
managed according to important aspects of DOD's acquisition policies 
and guidance, as well as other relevant acquisition management best 
practices. We focused on the program's (1) economic justification; (2) 
architectural alignment; (3) project management, including progress 
measurement, progress reporting, funding disclosure, and oversight 
activities; and (4) system development, including requirements 
management and testing. For requirements management and testing, we 
focused on the NTCSS application that is currently being developed, 
known as the Optimized Organizational Maintenance Activity (OOMA). 

We conducted our review from September 2004 through November 2005 in 
accordance with generally accepted government auditing standards. For 
details on our objective, scope, and methodology, see appendix I. 

Results in Brief: 

The Navy has not managed its NTCSS program in accordance with key 
aspects of the department's system acquisition policies and related 
guidance, including federal and recognized best practice guidance. 
Collectively, these policies and guidance are intended to reasonably 
ensure that investment in a given IT system represents the right 
solution to fill a mission need--and if it is, that acquisition and 
deployment of the system are handled in a manner that maximizes the 
chances of delivering defined system capabilities on time and within 
budget. In the case of NTCSS, neither of these outcomes is being 
realized. As a result, the Navy does not currently have a sufficient 
basis for determining whether NTCSS is the right systems solution for 
its aircraft, ship, and submarine tactical command support needs, and 
it has not pursued the proposed solution in the right way, meaning in a 
fashion that increases chances of delivering defined capabilities on 
time and within budget. Key areas in which the Navy did not follow 
relevant policies and guidance are described here. 

* The Navy has not economically justified its ongoing and planned 
investment in NTCSS on the basis of reliable estimates of future costs 
and benefits. The most recent economic justification's cost estimates 
were not reliably derived, and return on investment was not properly 
calculated. In addition, independent reviews of the economic 
justification to determine its reliability did not occur, and the Navy 
has not measured whether already deployed and operating components of 
the system are producing expected value. 

* The Navy has not invested in NTCSS within the context of a well-
defined enterprise architecture, which is an institutional blueprint to 
guide and constrain program investment decisions in a way that promotes 
interoperability and reduces redundancy among related and dependent 
systems. As we recently reported,[Footnote 2] DOD's business enterprise 
architecture does not contain sufficient context (depth and scope of 
operational and technical requirements) to effectively guide and 
constrain business transformation and system modernization efforts. 
Further, the Navy does not yet have a defined architecture, although it 
plans to develop one. Investing in systems, in the absence of an 
enterprise architecture, requires explicit recognition and deliberate 
consideration of the inherent risks to ensure fully informed investment 
decision making. 

* The Navy has not effectively performed key measurement, reporting, 
and oversight activities. In particular, earned value management, which 
is a means for determining and disclosing actual performance against 
budget and schedule estimates, and revising estimates based on 
performance to date has not been implemented effectively. Also, 
complete and current reporting of NTCSS progress and problems in 
meeting cost, schedule, and performance goals has not occurred, leaving 
oversight entities without the information needed to mitigate risks, 
address problems, and take corrective action. In addition, NTCSS 
budgets have not reflected the proper category of appropriated funds 
associated with system development efforts. Further, oversight 
entities' roles and responsibilities have not been fully discharged. 

* The Navy has not adequately conducted requirements management and 
testing activities. For the NTCSS application that is currently under 
development, the Navy has not adequately managed requirements, as 
evidenced by the absence of requirements traceability to system design 
specifications and testing documents, and the lack of prioritization of 
the requirements. The lack of requirements traceability and other 
issues have in turn contributed to problems with developmental testing, 
including the failure of these tests to identify problems that 
subsequently prevented the system from passing operational testing 
twice over the last 4 years. Based on the Navy's data, the recent trend 
in key indicators of system maturity, such as the number and nature of 
reported systems problems and change proposals, shows that problems 
with NTCSS persist and that these problems could involve costly and 
timely rework to address.[Footnote 3] 

Reasons the Navy cited for not following policies and guidance included 
questioning their applicability to the NTCSS program, having 
insufficient time in which to apply them, and believing that plans to 
adopt them were not meant to be applied retroactively. In some cases, 
the Navy did not acknowledge that any deviations from policies and 
guidance had occurred, but in these cases, it has yet to provide us 
with documentation demonstrating that it did adhere to them. 
Collectively, this means that after investing 10 years and $1 billion 
on NTCSS, it is unclear whether the Navy's planned future investment in 
the program is warranted. Even if key uncertainties are addressed and 
it can be demonstrated that NTCSS is the right solution, then the 
manner in which NTCSS is being defined, developed, tested, measured, 
and overseen is also of concern. Accordingly, we are making 
recommendations to the Secretary of Defense aimed at developing the 
basis needed to determine whether continued investment in NTCSS is a 
prudent use of limited departmental resources. We are also making 
recommendations to strengthen management of the program, conditional 
upon a decision to proceed with further investment in the NTCSS 
program. 

The Office of the Assistant Secretary of Defense for Networks and 
Information Integration provided written comments on a draft of the 
report. In its comments, DOD concurred with two of the recommendations 
and partially concurred with the remaining five. DOD also stated that 
while some of our findings are valid, our overall findings 
significantly understated and misrepresented the program's level of 
discipline and conformance with applicable guidance and direction. We 
do not agree. Our report cites numerous instances, supported by 
analyses, where the Navy did not comply with either DOD acquisition 
policies and guidelines or industry best practices. DOD's comments are 
reprinted in their entirety in appendix IV of this report, along with 
our detailed responses to each. 

Background: 

The Navy's primary mission is to organize, train, maintain, and equip 
combat-ready naval forces capable of winning the global war on terror 
and any other armed conflict, deterring aggression by would-be foes, 
preserving freedom of the seas, and promoting peace and security. To 
support this mission, the Navy performs a variety of interrelated and 
interdependent business functions such as logistics and financial 
management. The Navy requested, for fiscal year 2005, about $3.5 
billion to operate, maintain, and modernize its business systems and 
related IT infrastructure that support these business functions. This 
request represents about 27 percent of the $13 billion that DOD 
requested for all of its business systems for fiscal year 2005. Of the 
4,150 business systems that DOD reports in its current inventory, the 
Navy accounts for 2,353, or about 57 percent, of the total. 

In 1995, we designated DOD's business systems modernization efforts as 
a high-risk program and continue to designate it as such today[Footnote 
4] for several reasons, including the department's challenges in 
implementing effective IT investment management structures and 
processes, developing and implementing an enterprise architecture, and 
implementing effective IT system acquisition and development processes. 

NTCSS Genesis and Status Overview: 

In the early 1990s, the Navy employed a variety of IT systems to 
support the management of information, personnel, materials, and funds 
required to maintain and operate ships, submarines, and aircraft. Three 
core systems--each managed by a separate program office--consisting of 
nine major applications, provided this support: (1) the Shipboard Non-
Tactical Automated Data Processing Program (SNAP), managed by the Space 
and Naval Warfare Systems Command; (2) the Naval Aviation Logistics 
Command Management Information System (NALCOMIS), managed by the Naval 
Air Systems Command; and (3) the Maintenance Resource Management System 
(MRMS), managed by the Naval Sea Systems Command. See table 1 for a 
description of these three legacy systems and a list of their 
respective applications. 

Table 1: Legacy Systems and Applications: 

Legacy system: SNAP systems; SNAP I; SNAP II; 
Description: Manages systems for maintenance, supply, and financial 
operations at the organizational and intermediate levels.[A]; Manages 
medical and dental services, pay and personnel administration, food 
service, retail sales and service, training programs, technical data 
storage and retrieval, support and test equipment, and other mission 
support-related areas at the organizational level; SNAP I was developed 
for the Navy's larger ships, marine aviation logistics squadrons,[B] 
training sites, and selected activities ashore; SNAP II provides the 
same functionality as SNAP I, but it was developed for use on smaller 
ships and submarines. SNAP II was also modified to use microcomputers 
as the computing platforms when it is deployed on ships with 
constricted physical space; this version is known as MicroSNAP; 
Application: SNAP I: 
* Shipboard Uniform Automated Data Processing System; 
* Organizational Maintenance Management System; 
* Administration Data Management I; SNAP II: 
* Supply and Financial Management; 
* Organizational Maintenance Management System II Maintenance Data 
System; 
* Administration Data Management II. 

Legacy system: NALCOMIS; 
Description: Supports day-to-day aircraft maintenance and related 
material maintenance functionality both at sea and ashore; Provides the 
initial maintenance response when a problem is reported--including 
aircraft component troubleshooting, servicing, inspection, and removal 
and replacement at the organizational level; Supports, at the 
intermediate maintenance level, the repair of components after 
defective parts have been removed from an aircraft and sent to a 
central location to be refurbished; 
Application: 
* NALCOMIS Organizational Maintenance Activity; 
* NALCOMIS Intermediate Maintenance Activity. 

Legacy system: MRMS; 
Description: Supports intermediate-level ship and submarine maintenance 
at ashore facilities by providing management information such as 
planning, scheduling, workload forecasting, work progression, 
production control, productivity analysis, and resource management; 
Application: 
* Maintenance Resource Management System. 

Source: Navy. 

[A] The "organizational" level is the first stage of aircraft 
maintenance activity that is performed on individual planes and 
involves the upkeep and servicing of the aircraft at the location where 
it is deployed, such as a ship. Components or parts that cannot be 
repaired at the organizational level are removed from the plane and 
sent to a central location for repair. This second stage of maintenance 
is known as the "intermediate" level, and it normally occurs on land. 
If the defective part cannot be fixed at the intermediate level, it is 
then sent to a third stage of maintenance, known as the "depot" level, 
which is not in the scope of the NTCSS program. 

[B] Marine aviation logistics squadrons are groups of planes that are 
land-based but that can be deployed on an aircraft carrier for a 
specific mission. When the mission is completed, these planes return to 
their land base. 

[End of table] 

In 1992, we recommended that the Navy merge the management of all 
shipboard nontactical programs under a single command that would have 
authority and control over funding and development.[Footnote 5] In 
1993, the Navy developed a strategy to do so. In 1994, the Navy also 
identified a number of problems with the three legacy systems. 
Specifically, the Navy determined that (1) the individual systems did 
not consistently handle increasing workloads and provide the 
flexibility to meet changing operational demands; (2) the systems' 
software architectures were ineffective and inefficient; (3) the 
hardware was outdated, slow, expensive to maintain, and nonstandard; 
and (4) the systems could not support modernized applications. 

To address these concerns, the Navy initiated the NTCSS program in 1995 
to enhance the combat readiness of ships, submarines, and aircraft. To 
accomplish this, NTCSS was to provide unit commanding officers and 
crews with information about, for example, maintenance activities, 
parts inventories, finances, technical manuals and drawings, and 
personnel. According to the Navy, it spent approximately $1.1 billion 
for NTCSS from its inception through fiscal year 2005 and expects to 
spend another $348 million between fiscal years 2006 and 2009, for a 
total of approximately $1.45 billion. 

The Navy defined a three-stage acquisition process for NTCSS. 

Stage 1: Purpose was to replace hardware in order to establish a common 
infrastructure across all force-level ships, unit-level ships, aviation 
squadrons, Naval air stations, marine aviation logistics squadrons, and 
other maintenance activities--both at sea and ashore.[Footnote 6] 
During this stage, software and business processes were not to be 
changed. This phase was begun in 1994 under the legacy SNAP and 
NALCOMIS programs and, according to program officials, it is 
fundamentally complete--although technology refresh or replacement 
activities are still occurring. 

Stage 2: Purpose was to provide the functionality of the legacy systems 
software with more efficient, more easily maintained software and to 
eliminate functional overlap among the systems. This stage was to 
involve software technology modernization but no changes in software 
functionality or business processes. Existing legacy systems used flat 
files and hierarchical databases, which were to be converted to 
relational databases, and the existing application code was to be 
rewritten using modern software languages. A common hardware and 
systems software environment was also to be implemented, and 
functionality found in eight of the nine legacy applications was to be 
consolidated and rewritten as four new NTCSS applications. Development 
of these four applications began in 1995 and was reportedly completed 
in 2000. This stage was known as NTCSS Optimization. See table 2 for a 
description of the functionality of these new applications. 

Stage 3: Purpose was to improve NTCSS's functionality by implementing 
business process improvements. According to Navy officials, this stage 
is known as NTCSS Modernization and, to date, includes two efforts: (1) 
replace the last legacy application and (2) create a Web-enabled 
version of the three unit-level Optimized NTCSS applications that were 
developed under Stage 2. See table 3 for a description of the 
functionality of these business process improvements. 

Table 2: Optimized Applications Developed During Stage 2 of the NTCSS 
Program: 

NTCSS Optimized applications: Relational Supply; 
Description: Supports supply chain management, inventory management, 
and financial management processes; Provides Navy personnel with access 
to the supply support functions they perform most often--ordering, 
receiving, and issuing necessary supplies and materials; maintaining 
financial records; and reconciling supply, inventory, and financial 
records with the Navy's shore infrastructure; 
Status: Operational, as of September 1998, on large force-level ships, 
smaller unit-level ships, and at air stations and marine aviation 
logistics squadrons.[A]. 

NTCSS Optimized applications: Organizational Maintenance Management 
System--Next Generation; 
Description: Assists shipboard personnel in planning, scheduling, 
reporting, and tracking maintenance and related logistics support 
actions; Maintains online lists of maintenance actions to be performed, 
parts required to maintain shipboard equipment, and parts carried 
onboard ship to support maintenance actions; Interfaces with Relational 
Supply to requisition parts that are not onboard; 
Status: Operational, as of September 1998, primarily on large force-
level ships and smaller unit-level ships. 

NTCSS Optimized applications: Relational Administration Data 
Management; 
Description: Automates the management of personnel awards and 
decorations, work assignments, and berthing assignments; 
Status: Operational, as of April 2000, on large force-level ships, 
smaller unit-level ships, and at air stations and marine aviation 
logistics squadrons. 

NTCSS Optimized applications: Optimized Intermediate Maintenance 
Activities; 
Description: Provides online intermediate-level aviation maintenance, 
configuration, and logistics management support; Interfaces with other 
major integrated logistics support systems within the Naval aviation 
community; 
Status: Operational, as of April 2000, at force-level ships and at air 
stations and marine aviation logistics squadrons. 

Source: Navy. 

[A] Relational Supply is also in use at additional sites that are not a 
part of the NTCSS program. 

[End of table] 

Table 3: Modernized Applications Developed During Stage 3 of the NTCSS 
Program: 

NTCSS modernized applications: Optimized Organizational Maintenance 
Activity (OOMA); 
Description: Is to support day-to-day maintenance management tools for 
aviation squadrons and other organizational-level maintenance 
activities; Is to provide the foundation for achieving a completely 
automated maintenance environment, such as a single point of data 
entry, automated and assisted pilot and maintenance debrief, online 
diagnostics, structural life prognostics,[A] interactive electronic 
technical manuals, and forecasting and tracking of maintenance 
schedules; 
Status: Initiated in 1999, withdrawn from operational testing[B] in 
April 2001 when it became clear that it would fail. Failed operational 
testing again in May 2004. Scheduled for third operational test in the 
third quarter of fiscal year 2006; Fielded at 77 sites as of June 2005. 

NTCSS modernized applications: eNTCSS; 
Description: Was to provide a Web-enabled version of NTCSS, and allow 
users to access the three unit-level Optimized applications from any 
workstation on a ship's local area network via a standard Web browser 
and to execute work activities in a Web-server environment; 
Status: Initiated in 2001. Cancelled in April 2004; Fielded on one 
submarine and scheduled to be fielded on one more; Is to be replaced 
with the Optimized applications, but a date has yet to be determined. 

Source: Navy. 

[A] According to the U.S. Marine Corps Logistics Directorate, 
structural life prognostics is defined as the ability to reliably 
predict the remaining useful life of mechanical or structural 
components, within an actionable time period, and within acceptable 
confidence limits. 

[B] According to the DOD Defense Acquisition Guidebook, the primary 
purpose of operational test and evaluation is for representative users 
to evaluate systems in a realistic environment in order to determine 
whether these systems are operationally effective and suitable for 
their intended use before production or deployment. 

[End of table] 

As of April 2005, legacy applications were still in use at 51 percent 
of the Navy's 659 sites. These 659 sites either have legacy, Optimized, 
or modernized applications. Table 4 shows the distribution of the 
legacy, Optimized, and modernized applications. 

Table 4: Applications in Operation as of April 2005: 

Legacy applications: 

Applications: SNAP I[A, B]; 
Number of sites: 10. 

Applications: SNAP II[A, B, C]; 
Number of sites: 68. 

Applications: MicroSNAP; 
Number of sites: 32. 

Applications: NALCOMIS Organizational Maintenance Activity[D]; 
Number of sites: 214. 

Applications: NALCOMIS Intermediate Maintenance Activity[B]; 
Number of sites: 10. 

Applications: Maintenance Resource Management System[E]; 
Number of sites: 2. 

Applications: Subtotal; 
Number of sites: 336; 
Percentage of total: 51. 

Optimized applications[F]: 

Applications: Relational Supply[C], Organizational Maintenance 
Management System-Next Generation, Relational Administration Data 
Management, Optimized Intermediate Maintenance Activities. 

Applications: Subtotal; 
Number of sites: 229; 
Percentage of total: 35. 

Modernized applications: 

Applications: Optimized Organizational Maintenance Activity; 
Number of sites: 93. 

Applications: eNTCSS; 
Number of sites: 1. 

Applications: Subtotal; 
Number of sites: 94; 
Percentage of total: 14. 

Applications: Total; 
Number of sites: 659; 
Percentage of total: 100. 

Source: Navy. 

[A] SNAP I and SNAP II are each composed of three different legacy 
applications (see table 1). 

[B] The Navy plans to decommission some of the ships that use these 
applications and upgrade the remaining ships to NTCSS Optimized 
applications. 

[C] This application also is in use at additional sites that are not a 
part of the NTCSS program. 

[D] The functionality included in this application is to be replaced in 
the future by Optimized Organizational Maintenance Activity. 

[E] The Navy plans to incorporate this functionality into 
Organizational Maintenance Management System-Next Generation at a 
future date. 

[F] These four applications are deployed as a single software package 
at all 229 sites. 

[End of table] 

According to Navy officials, about $1.1 billion was spent on NTCSS 
between 1995 and 2005. This includes about $1 billion on NTCSS 
Optimized applications[Footnote 7] and $91 million on OOMA and eNTCSS. 
Table 5 shows NTCSS's budget totals from the time the program began in 
fiscal year 1995 through fiscal year 2005. 

Table 5: NTCSS Budget from FY 1995 through FY 2005: 

Dollars in thousands: 

NTCSS Optimized; 
FY 95: 83,537; 
FY 96: 69,794; 
FY 97: 69,075; 
FY 98: 123,469; 
FY 99: 119,822; 
FY 00: 91,053; 
FY 01: 95,322; 
FY 02: 95,549; 
FY 03: 82,708; 
FY 04: 108,087; 
FY 05: 71,926; 
Total: 1,010,342. 

OOMA; 
FY 96: 920; 
FY 97: 700; 
FY 98: 983; 
FY 99: 4,724; 
FY 00: 16,527; 
FY 01: 20,854; 
FY 02: 14,920; 
FY 03: 3,981; 
FY 04: 2,871; 
FY 05: 13,291; 
Total: 79,771. 

eNTCSS; 
FY 01: 5,000; 
FY 02: 5,309; 
FY 03: 985; 
Total: 11,294. 

Total; 
FY 95: 83,537; 
FY 96: 70,714; 
FY 97: 69,775; 
FY 98: 124,452; 
FY 99: 124,546; 
FY 00: 107,580; 
FY 01: 121,176; 
FY 02: 115,778; 
FY 03: 87,674; 
FY 04: 110,958; 
FY 05: 85,217; 
Total: 1,101,407. 

Source: Navy. 

[End of table] 

NTCSS Oversight and Management Roles and Responsibilities: 

A number of Navy and DOD organizations are involved in overseeing and 
managing the NTCSS program. Table 6 lists the organizations involved in 
NTCSS oversight and their respective roles and responsibilities. 

Table 6: NTCSS Oversight Roles and Responsibilities: 

Oversight entity: Deputy Assistant Secretary of the Navy for Command, 
Control, Communication, Computers and Intelligence, and Space; 
Roles and responsibilities: Currently serves as the milestone decision 
authority. Assigned overall responsibility for the NTCSS program; 
approves the program to proceed through its acquisition cycle on the 
basis of a review of key documents, such as an acquisition plan, an 
independently evaluated life cycle cost-and-benefits estimate, 
Acquisition Program Baseline documents, and Defense Acquisition 
Executive Summary reports. 

Oversight entity: Program Executive Office for Command, Control, 
Communication, Computers and Intelligence, and Space; Space and Naval 
Warfare Systems Command; 
Roles and responsibilities: Serves as the program executive office. 
Assigned overall responsibility for NTCSS program oversight; reviews 
the component cost analysis, acquisition strategy, and Acquisition 
Program Baseline prior to approval by the milestone decision authority. 

Oversight entity: Department of Navy Chief Information Officer; 
Roles and responsibilities: Reviews the acquisition program during the 
department's planning, programming, budgeting, and execution processes 
to ensure that the program's goals are achievable and executable; 
ensures conformance to appropriation law, financial management 
regulations, and Navy, DOD, and federal IT policies in several areas 
(e.g., security, architecture, and investment management); works 
closely with the program office during milestone review assessments. 

Oversight entity: Assistant Secretary of the Navy, Research Development 
and Acquisition, Chief Engineer; 
Roles and responsibilities: Ensures system compliance with 
architectural standards and promotes interoperability of the Navy's 
systems. 

Oversight entity: Office of the Secretary of Defense, Office of the 
Director for Program Analysis and Evaluation; 
Roles and responsibilities: Verifies and validates the reliability of 
cost and benefit estimates found in economic analyses and provides its 
results to the milestone decision authority. 

Oversight entity: Naval Cost Analysis Division; 
Roles and responsibilities: Performs independent cost estimates, 
maintains cost analysis tools, and focuses on cost analysis policy and 
oversight. 

Oversight entity: Executive Steering Committee; Members are 
representatives from: Office of the Chief of Naval Operations for 
Material Readiness and Logistics Operations (Chairman); Commander in 
Chief, U.S. Atlantic Fleet; Commander in Chief, U.S. Pacific Fleet; 
Commandant of the Marine Corps; and; Program Executive Office for 
Command, Control, Communication, Computers and Intelligence, and Space; 
Roles and responsibilities: Establishes priorities for NTCSS 
development and implementation and for defining long-term architectural 
goals; meets after regularly scheduled NTCSS meetings (e.g., 
Requirements Integrated Product Team meetings and Forum meetings).[A]. 

Source: Navy. 

[A] The Requirements Integrated Product Team is chartered to collect 
and analyze users' requirements, input these requirements into the 
NTCSS requirements management process, and provide recommendations to 
the program office on these requirements. The Forum brings together 
stakeholders and acquisition and development personnel to (1) discuss 
issues and requirements related to current and future system readiness, 
(2) develop specific action items and recommendations that will result 
in improved program products and services to the Fleet, and (3) 
facilitate key decisions by senior program leadership at Executive 
Steering Committee meetings. 

[End of table] 

There have been three milestone decision authorities for NTCSS since 
the program was begun. Initially, the milestone decision authority was 
in the Office of the Assistant Secretary of Defense for Networks and 
Information Integration/Chief Information Officer. In July 1999, this 
authority was delegated to the Assistant Secretary of the Navy for 
Research, Development, and Acquisition, who then delegated oversight 
authority to Deputy Assistant Secretary of the Navy for Command, 
Control, Communication, Computers and Intelligence, and Space in March 
2000. 

Table 7 lists the organizations involved in NTCSS management and their 
respective roles and responsibilities. 

Table 7: NTCSS Management and Stakeholder Roles and Responsibilities: 

Entity: Program Manager, Warfare; Space and Naval Warfare Systems 
Command; 
Roles and responsibilities: Serves as the program office. Assigned 
responsibility for day-to-day program management of NTCSS and, as such, 
is the single point of accountability for managing the program's 
objectives through development, production, and sustainment. Manages 
cost, schedule, and performance reporting. Prepares and updates the 
acquisition strategy, component cost analysis, and acquisition program 
baselines. Coordinates all testing activities in coordination with 
requirements. 

Entity: Space and Naval Warfare Systems Command, Systems Center 
Norfolk; 
Roles and responsibilities: Serves as the central design agency. 
Assigned responsibility for software development, including application 
design, development, and testing activities. Responsible for managing 
trouble reports and change proposals.[A] Manages Space and Naval 
Warfare Systems Command, Systems Center Norfolk Detachment San Diego, 
which installs the initial NTCSS systems on ships, submarines, and at 
land sites and performs subsequent on-site software maintenance. 

Entity: Space and Naval Warfare Systems Command, Systems Center 
Charleston; 
Roles and responsibilities: Serves as the in-service engineering 
activity. Provides engineering support and installs and integrates 
hardware. 

Entity: Office of the Chief of Naval Operations for Material Readiness 
and Logistics Operations; 
Roles and responsibilities: Serves as the program and resource sponsor. 
Balances user requirements with available resources. Works with users 
to ensure that operational and functional requirements are prioritized 
correctly and are supported. Addresses various issues pertaining to 
Navy policy, requirements, resources, and schedules. 

Entity: Functional Managers; Includes representatives from: Naval Sea 
Systems Command; Naval Supply Systems Command; Naval Air Systems 
Command; and; Commander in Chief, U.S. Atlantic Fleet; 
Roles and responsibilities: Represent the system users. Participate in 
the process of establishing functional requirements for input into the 
change management and system design processes. Prepare test plans and 
test analysis reports to support functional certification of software. 

Source: Navy. 

[A] Navy officials provided data regarding trouble reports and change 
proposals for the Optimized and modernized NTCSS applications. For 
details see appendix II. 

[End of table] 

NTCSS Participation in DOD's Rapid Improvement Team Pilot: 

In 2001, the DOD Chief Information Officer and the Undersecretary of 
Defense for Acquisition, Technology, and Logistics chartered a pilot 
project aimed at saving time by significantly reducing the reporting 
and oversight requirements. The ultimate goal was to enable the 
acquisition process to deliver mission-effective IT systems within 18 
months. Known as the Rapid Improvement Team (RIT) for IT Acquisition 
Management Transformation, the pilot was to cover a 2-year period from 
January 1, 2002, through December 31, 2003. Nine programs from the 
military services participated in the pilot. NTCSS was selected to 
participate in the pilot by its milestone decision authority due to its 
longevity and because of its perceived low risk, stability, and 
compliance with IT management best practices. It was also believed that 
little system development remained to be done. NTCSS began 
participating in the RIT pilot in October 2002. 

The RIT pilot relieved the program office of the normal acquisition 
process activities, such as preplanned, formal milestone decision 
reviews or briefings, and it granted the program office the authority 
to pass key milestones once it determined that established requirements 
had been met. This streamlined approach was considered possible because 
all information related to these requirements was to be continually 
updated and available to oversight organizations and stakeholders via a 
RIT Web site. More specifically, the program office was to update the 
Web site monthly via a set of electronic forms with the kind of data 
that were traditionally found in DOD oversight documents. The program 
office was also to use the Web site to input key acquisition documents 
(e.g., acquisition plans, economic analyses, requirements documents and 
test plans) in an electronic library. In turn, the milestone decision 
authority and other oversight organizations were to review these data 
on at least a monthly basis and directly retrieve any acquisition 
documents to be reviewed from the library. No response from the 
milestone decision authority would indicate implicit approval of the 
program data. Although the formal RIT pilot ended in December 2003, 
program officials told us that they continued to operate using the RIT 
pilot's procedures and continued to update program information on the 
Web site through December 2004. 

According to a memorandum issued by the Office of the Assistant 
Secretary of Defense for Networks and Information Integration/Chief 
Information Officer and the Undersecretary of Defense for Acquisition, 
Technology, and Logistics, the principal output of the pilot would be a 
blueprint for IT acquisition that is transferable to other systems. A 
report summarizing the results of the entire RIT pilot program was 
published in April 2005.[Footnote 8] This report concluded that (1) by 
instituting risk-based governance, the milestone decision authority can 
be assigned to an organization subordinate to the Office of the 
Secretary of Defense without adding unacceptable risk to the investment 
process and (2) the success of risk-based governance and cycle time 
reduction is predicated on the adoption of net-centricity[Footnote 9] 
by the business community. 

Prior Review Identified Strengths and Weaknesses in DOD's Acquisition 
Policies and Guidance: 

In July 2004, we reported[Footnote 10] that DOD's revised systems 
acquisition policies and guidance incorporated many best practices for 
acquiring business systems, such as (1) justifying system investments 
economically, on the basis of costs, benefits, and risks, and (2) 
continually measuring an acquisition's performance, cost, and schedule 
against approved baselines. However, the revised policies and guidance 
did not incorporate a number of other best practices, particularly 
those associated with acquiring commercial component-based business 
systems, and DOD did not have documented plans for incorporating these 
additional best practices into its policies. We also reported that the 
department's revised acquisition policies did not include sufficient 
controls to ensure that military services and defense agencies would 
appropriately follow these practices. We concluded that, until these 
additional best practices were incorporated into DOD's acquisition 
policies and guidance, there was increased risk that system 
acquisitions would not deliver planned capabilities and benefits on 
time and within budget and increased risk that an organization will not 
adopt and use best practices that were defined. Accordingly, we made 14 
recommendations to the Secretary of Defense that were aimed at 
strengthening DOD's acquisition policy and guidance by including 
additional IT systems acquisition best practices and controls for 
ensuring that these best practices were followed. DOD agreed with most 
of our recommendations and has since issued additional system 
acquisition guidance.[Footnote 11] 

NTCSS Has Not Been Managed in Accordance with DOD and Other Relevant 
System Acquisition and Development Guidance: 

DOD system acquisition and development policies and guidance, along 
with other federal and best practices guidance, provide an effective 
framework within which to manage IT business system programs and 
investments, like NTCSS. Proper implementation of this framework can 
minimize program risks and better ensure that system investments 
deliver promised capabilities and benefits on time and within budget. 
The Navy has not managed NTCSS in accordance with many key aspects of 
these policies and guidance. For example, the Navy has not economically 
justified its investment in NTCSS on the basis of cost and benefits. It 
has not invested in NTCSS within the context of a well-defined 
enterprise architecture. Further, the Navy has not effectively 
performed key measurement, reporting, and oversight activities, and has 
not adequately conducted requirements management and testing 
activities. Reasons the Navy cited for not following policies and 
guidance included questioning their applicability to the NTCSS program, 
having insufficient time in which to apply them, and believing that 
plans to adopt them were not meant to be applied retroactively. In some 
cases, the Navy did not acknowledge that any deviations from policies 
and guidance had occurred but, in these cases, it has yet to provide us 
with documentation demonstrating that it did adhere to them. As a 
result, the Navy does not currently have a sufficient basis for 
determining whether NTCSS is the right system solution for its tactical 
command support needs, and it has not pursued the proposed solution in 
a way that increases the likelihood of delivering defined capabilities 
on time and within budget. 

The Navy Has Not Economically Justified Investment in NTCSS on the 
Basis of Costs and Benefits: 

The decision to invest in any system should be based on reliable 
analyses of estimated system costs and expected benefits over the life 
of the program. DOD policy requires such analyses, and other relevant 
acquisition management practices provide guidance on how these analyses 
should be prepared. However, the current economic analysis for the 
NTCSS program does not meet this guidance. Additionally, the analysis 
was not independently reviewed in accordance with DOD guidance. 
Finally, contrary to DOD policy and relevant acquisition management 
practices, the Navy has not demonstrated that NTCSS Optimized 
applications are producing expected benefits. Without such reliable 
analyses, an organization cannot adequately know that a given system 
investment is justified. 

The Latest NTCSS Cost Estimate Was Not Derived Reliably: 

According to DOD guidance,[Footnote 12] the cost estimates used to 
economically justify an investment should be reasonable, traceable, and 
based on realistic assumptions. Our research shows that a reliable cost 
estimate should meet nine specific criteria developed by Carnegie 
Mellon University's Software Engineering Institute (SEI),[Footnote 13] 
such as appropriately sizing the task being estimated and identifying 
and explaining estimate assumptions. 

In March 2004, the NTCSS program office prepared its fifth NTCSS 
economic analysis. This analysis examined the costs associated with 
three alternative NTCSS hardware, software, operating system and data 
base management configurations, and was to be used to inform decisions 
about system development and implementation. The analysis did include 
estimated costs for each alternative. However, it did not include 
measurable, quantifiable benefits for each alternative. Rather, it 
included only qualitative benefits. Further, the cost estimates used in 
this analysis did not meet six of the nine criteria associated with 
reliable cost estimates. For example, while the estimate's purpose was 
stated in writing, the system life cycle used was 6 years rather than 
the 10 years recommended. Also, documentation showing that the costs 
were based on data from the program's demonstrated accomplishments has 
yet to be provided to us, and the assumptions used to create the cost 
estimate were not identified and explained. See table 8 for the results 
of our analyses relative to each of the nine criteria. 

Table 8: Navy Satisfaction of Cost Estimating Criteria: 

Criterion: The objectives of the program are stated in writing; 
Explanation: The objectives of the program should be clearly and 
concisely stated for the cost estimator to use; 
Criterion met[A]: Yes; 
GAO analysis: The objective of the program was clearly stated. 

Criterion: The life cycle to which the estimate applies is clearly 
defined; 
Explanation: The life cycle should be clearly defined to ensure that 
the full cost of the program--that is, all direct and indirect costs 
for planning, procurement, operations and maintenance, and disposal--
are captured. For investments such as NTCSS, the life cycle should 
cover 10 years past full operational capability of the system.[B]; 
Criterion met[A]: No; 
GAO analysis: The life cycle was not clearly defined to ensure that the 
full cost of the program is included. The life cycle defined was 6 
years past full operational capability, instead of the full 10 years 
defined in the DOD guidance. 

Criterion: The task has been appropriately sized; 
Explanation: An appropriate sizing metric should be used in the 
development of the estimate, such as the amount of software to be 
developed and the amount of software to be revised; 
Criterion met[A]: Yes; 
GAO analysis: The method used in the model lends itself to being 
appropriately sized. 

Criterion: The estimated cost and schedule are consistent with 
demonstrated accomplishments on other projects; 
Explanation: Estimates should be validated by relating them back to 
demonstrated and documented performance on completed projects; 
Criterion met[A]: No; 
GAO analysis: No documentation was provided to show the use of 
historical data to produce the estimate. 

Criterion: A written summary of parameter values and their rationales 
accompany the estimate; 
Explanation: If a parametric equation was used to generate the 
estimate, the parameters that feed the equation should be provided, 
along with an explanation of why they were chosen; 
Criterion met[A]: No; 
GAO analysis: The model used undocumented values as the source of the 
estimate for multiple elements. 

Criterion: Assumptions have been identified and explained; 
Explanation: Accurate assumptions regarding issues such as schedule, 
quantity, technology, development processes, manufacturing techniques, 
software language, etc., should be understood and documented; 
Criterion met[A]: No; 
GAO analysis: Any assumptions used in the model were not identified. 

Criterion: A structured process, such as a template or format, has been 
used to ensure that key factors have not been overlooked; 
Explanation: A work breakdown structure or similar structure that 
organizes, defines, and graphically displays the individual work units 
to be performed should be used. The structure should be revised over 
time as more information becomes known about the work to be performed; 
Criterion met[A]: Yes; 
GAO analysis: A work breakdown structure was provided and included all 
the standards elements. 

Criterion: Uncertainties in parameter values have been identified and 
quantified; 
Explanation: For all major cost drivers, an uncertainty analysis should 
be performed to recognize and reflect the risk associated with the cost 
estimate; 
Criterion met[A]: No; 
GAO analysis: No risk analysis was documented in the estimate. 

Criterion: If more than one cost model or estimating approach has been 
used, any differences in the results have been analyzed and explained; 
Explanation: The primary methodology or cost model results should be 
compared with any secondary methodology (for example, cross-checks) to 
ensure consistency; 
Criterion met[A]: No; 
GAO analysis: No secondary model was discussed in the estimate 
documentation. 

Sources: SEI criteria, DOD guidance, and GAO analysis of Navy data. 

[A] "Yes" means that the program provided documentation demonstrating 
satisfaction of the criterion. "Partially" means that the program 
provided documentation demonstrating satisfaction of part of the 
criterion. "No" means that the program has yet to provide documentation 
demonstrating satisfaction of the criterion. 

[B] DOD, DOD Automated Information System (AIS) Economic Analysis (EA) 
Guide, May 1, 1995. 

[End of table] 

Program officials told us that they did not develop the 2004 cost 
estimate in accordance with all of the SEI cost estimating criteria 
because they had only a month to complete the economic analysis. By not 
following practices associated with reliable estimates, the Navy has 
decided on a course of action that is not based on one of the key 
ingredients to sound and prudent decision making--a reliable estimate 
of system life cycle costs. Among other things, this means that the 
investment decision made by the Navy has not been adequately justified 
and, that to the extent that program budgets were based on cost 
estimates, the likelihood of funding shortfalls and inadequate funding 
reserves is increased. 

The Latest NTCSS Economic Analysis Did Not Meet Key Federal Guidance: 

According to Office of Management and Budget (OMB) guidance,[Footnote 
14] economic analyses should meet certain criteria to be considered 
reasonable, such as comparing alternatives on the basis of net present 
value and conducting an uncertainty analysis of costs and benefits. 

The latest NTCSS economic analysis, prepared in March 2004, identified 
potential costs and benefits from three alternative NTCSS hardware, 
software, operating system, and data base management configurations. 
However, the analysis provided only monetized costs for each 
alternative. It did not provide monetized benefits. Further, the 
analysis did not meet five of eight OMB criteria. For example, the 
alternatives were not compared on the basis of their net present 
values, an appropriate interest rate was not used to discount the net 
present values, and the uncertainty associated with the cost estimates 
was not disclosed and used in the analysis. See table 9 for the results 
of our analyses relative to each of the eight criteria. 

Table 9: Navy Satisfaction of OMB Economic Analysis Criteria: 

Criterion: The economic analysis clearly explained why the investment 
was needed; 
Explanation: The economic analysis should clearly explain the reason 
why the investment is needed, i.e., why the status quo alternative is 
unacceptable; 
Criterion met[A]: Yes; 
GAO analysis: The economic analysis explained why the status quo 
alternative was not viable. 

Criterion: At least two alternatives to the status quo were considered; 
Explanation: At least two meaningful alternatives to the status quo 
should be examined to help ensure that the alternative chosen was not 
preselected; 
Criterion met[A]: Yes; 
GAO analysis: Three alternatives to the status quo were considered. 

Criterion: The general rationale for the inclusion of each alternative 
was discussed; 
Explanation: The general rationale for the inclusion of each 
alternative should be discussed to enable reviewers of the analysis to 
gain an understanding of the context for the selection of one 
alternative over the others; 
Criterion met[A]: Yes; 
GAO analysis: The rationale for each alternative was discussed. 

Criterion: The quality of the cost estimate for each alternative was 
reasonable; 
Explanation: The quality of the cost estimate of each alternative 
should be complete and reasonable for a net present value to be 
accurate. One measure of a cost estimate's reasonableness is its 
satisfaction of earlier cited SEI criteria; 
Criterion met[A]: No; 
GAO analysis: The cost estimates were not complete and did not meet a 
majority of the SEI criteria. 

Criterion: The quality of the benefits to be realized from each 
alternative was reasonable; 
Explanation: The quality of the benefit estimate of each alternative 
should be complete and reasonable for a net present value to be 
calculable and accurate; 
Criterion met[A]: No; 
GAO analysis: Monetized estimates of benefits were not provided, and no 
explanation was given as to why these estimates were not provided. 

Criterion: Alternatives were compared on the basis of net present 
value; 
Explanation: The net present value should be calculated because it 
consistently results in the selection of the alternative with the 
greatest benefit net of cost; 
Criterion met[A]: No; 
GAO analysis: The economic analysis stated that all costs and benefits 
were expressed in undiscounted constant fiscal year 2004 dollars; 
however, monetized benefits were not reported in the economic analysis. 
As a result, the net present value was not calculated. 

Criterion: The proper discount rate used for calculating each 
alternative's overall net present value should be used; 
Explanation: OMB Circular A-94 is the general guidance for conducting 
cost-benefit analyses for federal government programs and provides 
specific guidance on the discount rates to be used in evaluating those 
programs whose benefits and costs are distributed over time; 
Criterion met[A]: No; 
GAO analysis: Since all dollar amounts are expressed in undiscounted 
constant fiscal year 2004 dollars, the discount rate used in the 
economic analysis is, by default, zero. The discount rates provided by 
OMB Circular No. A-94 are all positive (i.e., greater than zero). 

Criterion: An uncertainty analysis of costs and benefits was included; 
Explanation: Estimates of benefits and costs are typically uncertain 
because of imprecision in both underlying data and modeling 
assumptions. Because such uncertainty is basic to virtually any cost-
benefit analysis, its effects should be analyzed and reported; 
Criterion met[A]: No; 
GAO analysis: No uncertainty analysis for the overall reported costs 
was included. 

Sources: OMB guidance and GAO analysis of Navy data. 

[A] "Yes" means that the program provided documentation demonstrating 
satisfaction of the criterion. "Partially" means that the program 
provided documentation demonstrating satisfaction of part of the 
criterion. "No" means that the program has yet to provide documentation 
demonstrating satisfaction of the criterion. 

[End of table] 

Program officials told us that they did not adhere to the OMB criteria 
because they had only a month to complete the economic analysis and, 
therefore, did not have the time necessary to comply with it. By not 
following established OMB guidance, the reliability of the latest NTCSS 
economic analysis is questionable. This further increases the risk that 
the Navy is following a course of action that will not produce the 
expected return on investment. 

The Latest NTCSS Economic Analysis Was Not Independently Reviewed: 

DOD guidance[Footnote 15] states that economic analyses and cost 
estimates should be independently reviewed and assessed. In this 
regard, the Office of Program Analysis and Evaluation, located in the 
Office of the Secretary of Defense, is responsible for verifying and 
validating the reliability of economic analyses and providing the 
results to the milestone decision authority; the Naval Cost Analysis 
Division is responsible for preparing independent cost estimates. 

However, neither of these offices reviewed the most recent economic 
analysis for NTCSS. An official from the Office of Program Analysis and 
Evaluation told us that this office did not review the 2004 economic 
analysis because, once NTCSS entered the RIT Pilot, the program office 
no longer provided documentation needed to review the analysis. 
Officials from the Naval Cost Analysis Division also stated that they 
did not review the estimates in this economic analysis. According to 
officials from this office, they are only required to review cost 
estimates that are prepared for milestone reviews, and staffing 
limitations do not permit them to review all cost estimates. 

By not having the economic analysis reviewed by independent parties, 
the Navy has no independent verification that the estimates of life 
cycle costs and benefits are reasonable and traceable, that the cost 
estimates are built on realistic program and schedule assumptions, or 
that the return on investment calculation is valid. This casts further 
doubt on the reliability of the economic analysis the Navy has used to 
justify its ongoing investment in NTCSS. 

The Navy Has Yet to Measure Whether Actual Benefits Have Accrued from 
Deployed NTCSS Capabilities: 

The Clinger-Cohen Act of 1996 and OMB guidance[Footnote 16] emphasize 
the need to develop information to ensure that IT projects are actually 
contributing to tangible, observable improvements in mission 
performance. DOD guidance[Footnote 17] also requires that analyses be 
conducted to validate estimated benefits and measure the extent to 
which desired outcomes have been achieved. To this end, agencies should 
define and collect metrics to determine whether expected benefits are 
being achieved and modify subsequent applications and investments to 
reflect the lessons learned. 

However, the Navy has yet to measure whether NTCSS Optimized 
applications are actually producing expected benefits commensurate with 
actual costs. For example, in 1999 the Navy projected that deploying 
the NTCSS Optimized applications would result in reduced costs 
associated with NTCSS maintenance, training, and other support 
activities. However, the Navy does not know the extent to which NTCSS 
Optimized applications are meeting these expectations--even though 
these applications have been deployed to 229 user sites since 1998--
because metrics to demonstrate that these expectations have been met 
have not been defined and collected. 

Program officials and officials representing the milestone decision 
authority stated that the Navy is not required to measure actual 
accrual of benefits because DOD guidance to do so was not yet in effect 
when the NTCSS Optimized applications were deployed, and there was no 
explicit requirement to apply this guidance retroactively. Program 
officials also stated that it will not be possible to measure actual 
return-on-investment for the already deployed NTCSS Optimized 
applications until the entire NTCSS system is deployed and operational. 
Similarly, an official with the milestone decision authority stated 
that actual NTCSS return-on-investment has not yet been measured. 

Because it is not measuring whether cost and benefit projections are 
being met, the Navy lacks important information that it will need to 
inform future economic analyses and investment decisions. 

The Navy Recently Decided to Prepare a Benefits Assessment: 

In February 2005, officials from the Office of the Chief of Naval 
Operations for Material Readiness and Logistics Operations[Footnote 18] 
and representatives from key user organizations questioned whether 
NTCSS can cost effectively meet users' future needs. Initially this 
office tasked the program office to develop a new economic analysis to 
determine whether to continue investing in NTCSS or in some other 
system solution, such as the Navy enterprise resource planning (ERP) 
program.[Footnote 19] In November 2005, officials from the Office of 
the Chief of Naval Operations for Material Readiness and Logistics 
Operations stated that they were no longer planning to develop a new 
economic analysis but planning to conduct a benefits assessment to 
evaluate changing NTCSS to some solution to enable the system to 
perform future ashore activities. These officials acknowledged that 
this assessment will be less than the initially planned economic 
analysis in that it will exclude any analysis of costs and alternative 
solutions. However, they also acknowledged that DOD policy and guidance 
does not address benefits assessments as a recognized acquisition 
program document. They stated that this assessment will be prepared for 
inclusion in the 2006 budget submission. 

Without knowing the extent to which NTCSS Optimized applications are 
meeting cost and benefit expectations, the Navy is not in a position to 
make informed, and thus justified, decisions on whether and how to 
proceed with the program. Such a situation introduces a serious risk 
that the Navy will not be able to demonstrate whether NTCSS is cost-
effective until it has already spent hundreds of millions of dollars 
more on the NTCSS Optimized applications and OOMA. 

The Navy Has Not Defined and Developed NTCSS within the Context of an 
Enterprise Architecture: 

DOD policy and guidance,[Footnote 20] as well as federal and best 
practice guidance,[Footnote 21] recognize the importance of investing 
in IT business systems within the context of an enterprise 
architecture. Our research and experience in reviewing federal agencies 
shows that not doing so often results in systems that are duplicative, 
not well integrated, unnecessarily costly to interface and maintain, 
and do not optimally support mission outcomes.[Footnote 22] NTCSS has 
not been defined and developed in the context of a DOD or Navy 
enterprise architecture because a well-defined version of either has 
not existed to guide and constrain the program, and meaningful analysis 
showing how NTCSS aligns to evolving DOD and Navy architecture efforts 
was not produced. This means that the Navy does not have a sufficient 
basis for knowing if NTCSS, as defined, properly fits within the 
context of future DOD and Navy business operational and technological 
environments. 

More specifically, a well-defined enterprise architecture provides a 
clear and comprehensive picture of an entity, whether it is an 
organization (e.g., a federal department) or a functional or mission 
area that cuts across more than one organization (e.g., personnel 
management). This picture consists of snapshots of both the 
enterprise's current or "As Is" environment and its target or "To Be" 
environment, as well as a capital investment road map for transitioning 
from the current to the target environment. These snapshots consist of 
integrated "views," which are one or more architecture products that 
describe, for example, the enterprise's business processes and rules; 
information needs and flows among functions; supporting systems, 
services, and applications; and data and technical standards and 
structures. GAO has promoted the use of architectures to guide and 
constrain systems modernization, recognizing them as a crucial means to 
a challenging goal: agency operational structures that are optimally 
defined in both the business and technological environments. 

DOD has long operated without a well-defined enterprise architecture 
for its business environment. In 2001, we first reported that DOD did 
not have such an architecture and recommended that it develop one to 
guide and constrain IT business systems, like NTCSS.[Footnote 23] Over 
the next 4 years, we reported that DOD's architecture development 
efforts were not resulting in the kind of business enterprise 
architecture that could effectively guide and constrain business system 
investments,[Footnote 24] largely because the department did not have 
in place the architecture management structures and processes described 
in federal guidance. In particular, we most recently reported in July 
2005[Footnote 25] that despite spending about $318 million producing 
eight versions of its architecture, DOD's latest version still did not 
have, for example, a clearly defined purpose that could be linked to 
the department's goals and objectives and a description of the "As Is" 
environment and a transition plan. Further, we reported that the 
description of the "To Be" environment was still missing important 
content (depth and scope) relative to, for example, the actual systems 
to be developed or acquired to support future business operations and 
the physical infrastructure (e.g., hardware and software) that would be 
needed to support the business systems. Over the last several years, we 
have also reported that DOD's efforts for determining whether ongoing 
investments were aligned to its evolving architecture were not 
documented and independently verifiable.[Footnote 26] On September 28, 
2005, DOD issued the next version of its business enterprise 
architecture,[Footnote 27] which we are required to review, along with 
other things such as the department's efforts to review certain 
investments' alignment with the architecture, pursuant to the Fiscal 
Year 2005 National Defense Authorization Act.[Footnote 28] 

The Navy has also not had an enterprise architecture to guide and 
constrain its IT system investments. For example, in February 2002 and 
November 2003, we reported that while the Navy was developing an 
enterprise architecture, the architecture products were not complete 
and they were not, for example, under configuration 
management.[Footnote 29] Since that time, the Navy has yet to develop 
an enterprise architecture. In response to our request for the latest 
version of its architecture, the Assistant Secretary of the Navy, 
Research Development and Acquisition, Chief Engineer, provided us 
documentation that describes high-level principles or goals that the 
Navy wants to achieve, such as systems interoperability. However, most 
of the critical products that an enterprise architecture should include 
were not provided, such as (1) a data dictionary, which is a repository 
of standard data definitions for applications; (2) a logical database 
model that provides the data structures that support information flows 
and that provides the basis for developing the schemas for designing, 
building, and maintaining the existing physical databases; and (3) an 
analysis of the gaps between the baseline and target architecture for 
business processes, information/data, and services/application systems 
to define missing and needed capabilities. According to the Deputy 
Assistant Secretary of the Navy for Command, Control, Communication, 
Computers and Intelligence, and Space, the Navy does not have an 
enterprise architecture. However, these officials stated that the Navy 
recognizes the importance of developing and using one and is taking 
steps to do so. They did not have a time frame as to when this would be 
accomplished, however. 

In addition, NTCSS program officials told us that the system has been 
assessed against DOD's business enterprise architecture, and based on 
this assessment, the system is aligned. However, our analysis of the 
alignment documentation showed while NTCSS could be mapped to several 
enterprise architecture elements (e.g., strategic goals and 
organizational roles), it was not mapped to other important elements 
(e.g., technical standards and data model). Moreover, as previously 
discussed, the version of the enterprise architecture used to assess 
alignment lacked utility and did not provide a sufficient basis for 
making informed investment decisions. 

These officials stated that they have not yet assessed the system 
against the Navy's architecture because (1) the architecture has yet to 
be sufficiently developed and (2) compliance with this architecture may 
not be required. 

Without having a well-defined architecture to set the institutional 
context within which a given investment like NTCSS must fit and taking 
proactive and verifiable steps to understand the extent to which the 
system as it is defined fits within this context, misalignments can 
occur that can introduce redundancies and incompatibilities and that 
can produce inefficiencies and require costly and time consuming rework 
to fix. In the case of NTCSS, this could be a problem because of the 
Navy's ongoing investment in its ERP program.[Footnote 30] As we 
recently reported,[Footnote 31] this program is intended to provide 
functionality in such areas as supply and workforce management for 
ashore activities, which is functionality similar to that of NTCSS for 
afloat activities. However, both programs have proceeded without a 
common, institutional frame of reference (i.e., enterprise 
architecture) that can be used to effectively manage their 
relationships and dependencies. Our research and experience in 
reviewing federal agencies shows that managing such relationships on a 
program to program basis is untenable and has proven unsuccessful. This 
is why the inherent risks associated with investing in systems in the 
absence of a well-defined architecture need to be explicitly disclosed 
and deliberately evaluated in order to make a well-informed investment 
decision. 

Key Program Management and Oversight Activities Have Not Been 
Effectively Performed: 

Key aspects of effective program management include reliable progress 
measurement and reporting, appropriate budgeting, and meaningful 
oversight. DOD policy requires such activities, and DOD and other 
industry best practices provide guidance on how these activities should 
be conducted. However, these activities have not been effectively 
performed on the NTCSS program. Specifically, the Navy has not 
adequately measured progress against planned cost and scheduled work 
commitments, fulfilled defined reporting requirements, properly 
budgeted for expenditures, and conducted meaningful program oversight. 
As a result, opportunities for proactive program intervention and 
actions to address risks and problems were missed, allowing the program 
to proceed largely unchecked. 

The Navy is Not Adequately Measuring Progress Against Planned Cost and 
Scheduled Work Commitments: 

Measuring and reporting progress against cost and schedule commitments 
is a vital element of effective program management. DOD policy and 
guidance recognize this by requiring the use of earned value 
management, and describing how it is to be performed. The NTCSS program 
has elected to use earned value management; however, it is not doing so 
effectively. As a result, the program, as well as Navy and DOD 
oversight authorities, have not had access to the kind of reliable and 
timely information they need to make informed decisions. 

DOD Has Adopted Industry Standards for Earned Value Management: 

According to DOD policy and guidance,[Footnote 32] program offices 
should obtain data from contractors and central design agencies on work 
progress, and these data should relate cost, schedule, and technical 
accomplishments. Moreover, the guidance states that these data should 
be valid, timely, and auditable. The tool that many DOD entities, 
including the NTCSS's program office and its central design agency, use 
to obtain and report these data is known as earned value management 
(EVM). Through EVM, program offices and others can determine a 
contractor's or central design agency's ability to perform work within 
cost and schedule estimates. It does so by examining variances between 
the actual cost and time to perform work tasks and the 
budgeted/estimated cost and time to perform the tasks. 

In 1996, DOD adopted industry guidance[Footnote 33] that identifies 32 
criteria that a reliable EVM system should meet. The 32 criteria are 
organized into five categories: organization, planning and budgeting, 
accounting, analysis and management reports, and revisions and data 
maintenance (see app. III for the 32 criteria). As we previously 
reported,[Footnote 34] EVM offers many benefits when done properly. In 
particular, it is a means to measure performance and serves as an early 
warning system for deviations from plans. It therefore enables a 
program office to mitigate the risk of cost and schedule overruns. 

NTCSS Has Not Effectively Implemented EVM: 

The EVM system that NTCSS has implemented to measure program 
performance does not provide the kind of reliable and timely data 
needed to effectively identify and mitigate risks. According to the 
NTCSS central design agency's self-assessment of its earned value 
management system, 17 of the 32 industry best practice criteria are not 
being satisfied by the EVM system it has implemented. For example, the 
central design agency reported that the system cannot (1) establish and 
maintain a budget baseline against which program performance can be 
measured over time, (2) identify management reserves in case of 
contingencies, (3) record all indirect costs[Footnote 35] that will be 
allocated to the work, (4) summarize data elements and associated 
variances through the work breakdown structure to support management 
needs, and (5) develop revised estimates of cost at completion based on 
performance to date. 

Beyond this self-assessment, our review showed that 29 of the 32 
criteria were not satisfied. For example, the system does not (1) 
provide for the integration of planning, scheduling, budgeting, work 
authorization, and cost accumulation management process; (2) identify 
physical products, milestones, technical performance goals, or other 
indicators used to measure progress; (3) reconcile current budgets to 
prior budgets in terms of changes to the authorized work and internal 
replanning; and (4) control retroactive changes to records. See 
appendix III for the Navy's complete self-assessment and our full 
analysis of the extent to which the 32 criteria are satisfied. 

Officials with the program office and the central design agency stated 
that although they chose to use EVM, they are not required by DOD 
policy to do so and, therefore, do not have to comply with the 32 
criteria. These officials stated that one reason they are not required 
to use it is because the program office and the central design agency 
are part of the same organization (the Space and Naval Warfare Systems 
Command) and thus a formal contract or written agreement between them 
does not exist. They also stated that although the program as a whole 
exceeds dollar thresholds for which EVM is required,[Footnote 36] they 
have chosen to break the program into smaller projects managed on a 
fiscal year basis, and none of these projects individually exceeds 
either the new or old DOD policy thresholds that would require the use 
of EVM. 

We do not agree that the absence of a contractual relationship or the 
decomposition of the program into small, fiscal year-based projects is 
a valid reason for not effectively implementing EVM. DOD and OMB 
guidance require that the Navy base programmatic decisions on reliable 
analyses of estimated system's costs and expected benefits over the 
life of the program. The program office chose to use EVM as a means to 
satisfy these requirements and to measure progress and identify 
potential problems early, so that they could be effectively addressed. 
To accomplish this, EVM must be performed correctly. By not 
implementing it correctly on NTCSS, the Navy is losing an opportunity 
to gain the kind of visibility into program progress needed to identify 
problems and risks early and better ensure program success. Moreover, 
by tracking individual projects on a yearly basis the program office 
cannot adequately understand the status of the NTCSS program as a 
whole, which hinders its ability to accurately forecast program costs 
at completion and provide realistic schedule projections. In short, 
without reliable, timely, and auditable EVM data, the program office 
cannot adequately manage technical, cost, and schedule risks and 
problems. 

Two NTCSS Projects Illustrate How EVM Has Been Poorly Implemented: 

Two of the individual NTCSS projects for which EVM activities were 
reportedly being performed are (1) 2004 OOMA software development and 
(2) 2004 NTCSS hardware installation and integration (for both OOMA and 
Optimized NTCSS). For the OOMA software project, EVM was performed by 
the central design agency and for the NTCSS hardware project it was 
performed by the Space and Naval Warfare Systems Command Systems 
Center, Charleston. On both projects, we found several examples of 
ineffective EVM implementation, including the following: 

* An integrated baseline review was not conducted for either of the 
projects. According to DOD guidance and best practices, an integrated 
baseline review should be conducted as needed throughout the life of a 
program to ensure that the baseline for tracking cost, technical, and 
schedule status reflects (1) all tasks in the statement of work, (2) 
adequate resources in terms of staff and materials to complete the 
tasks, and (3) integration of the tasks into a well-defined schedule. 
Further, program managers are to use cost performance reports that have 
been validated by an integrated baseline review. Without verifying the 
baseline, monthly cost performance reporting, which is to track against 
a set budget and schedule, does not have sufficient meaning or 
validity. 

* The estimate at completion for the 2004 OOMA software project, which 
is a forecast value expressed in dollars representing the final 
projected costs of the project when all work is completed, showed a 
negative cost for a 6-month period (November 2003 to April 2004). When 
EVM is properly implemented, this amount should include all work 
completed and always be a positive number. The negative estimate at 
completion for this project would mean that the central design agency 
had incurred a savings rather than spending money, even though during 
that time more than $1.7 million had been spent. 

* The schedule performance index for the OOMA software project, which 
is to reflect the critical relationship between the actual work 
performed versus the costs expended to accomplish the work, showed 
program performance during a time when the program office stated no 
work was being performed. Specifically, the reports showed the schedule 
performance fluctuating between $0.21 worth of work performed for every 
dollar spent to more than $3.75 worth of work performed for every 
dollar spent during a time that the program office claims all work was 
halted. Perfect performance would indicate schedule indices equal to 
1.0 at best (i.e., for every dollar spent there was 100 percent of the 
schedule achieved). 

* The estimate at completion for the OOMA hardware installation project 
showed that almost $1 million in installation costs had been removed 
from the total sunk costs, but no reason for doing so was provided in 
the cost performance report. 

* The cost and schedule indices for the OOMA hardware installation 
project showed improbably high program performance during a time when 
the installation schedules and installation budget had been drastically 
cut because OOMA software failed operational testing. Specifically, the 
reports between March 2004 and July 2004 showed the current cost 
performance fluctuating between $0.07 worth of work performed for every 
dollar spent to $8.48 worth of work performed for every dollar spent. 

Navy officials cited several reasons for these shortcomings. For the 
software project, program officials stated that prior to the 
operational testing of OOMA in 2003, the central design agency's 
implementation of EVM was primitive at best and that the resulting data 
were not usable. They also stated that after the project failed 
operational testing, they did not see the value in rebaselining the 
project and thus all EVM analysis was halted. They did, however, 
continue to invest in OOMA. For the hardware installation project, a 
Charleston Center official responsible for developing the installation 
reports stated that there were problems with collecting actual costs 
because the individuals responsible for doing the work were covered by 
other contracts, and there was no way to ensure that the costs were 
being reported consistently. Regarding the approximately $1 million in 
installation costs that were removed from the total sunk costs, this 
official stated that these costs were erroneously charged to this 
project and were thus removed because they were not part of the 
original plan. 

Ineffective implementation of EVM, as occurred on these two projects, 
precludes NTCSS program officials from having reliable and timely 
information about actual program status and does not provide these 
officials with a sound basis for making informed program decisions. 

The Navy Has Not Adequately Reported NTCSS's Progress and Problems: 

One essential aspect of effective program management is complete and 
current reporting by the program office to oversight organizations 
responsible for making decisions regarding the program's future. DOD 
policy recognizes this, stating that the program office is accountable 
for providing credible schedule, performance, and cost reporting 
information to the milestone decision authority. Officials from the 
NTCSS milestone decision authority told us that they relied on the 
program office to fully disclose progress against, and deviations from, 
program cost, schedule, and performance goals. However, the program 
office has not reported consistently or reliably on the program's 
progress and, as a result, has not fully disclosed program status to 
Navy and DOD oversight authorities who are responsible for making 
proper investment decisions. 

Navy Reporting Requirements for NTCSS Have Changed over the Last 
Several Years: 

Since program inception, NTCSS requirements for reporting cost, 
schedule, and performance information have changed. Prior to October 
2002, the program office was required to comply with applicable DOD 
acquisition policies and guidance.[Footnote 37] This guidance generally 
required the program office to provide oversight organizations with the 
following three key reports: 

* The Acquisition Program Baseline, which describes the program's cost, 
schedule, and performance goals. This baseline document is to be 
developed when the program is initiated, and it is to be updated for 
each milestone review. Within 90 days of a program breach,[Footnote 38] 
unless the program is back within its baseline goals, a new Acquisition 
Program Baseline is to be prepared by the program office and approved 
by the milestone decision authority. 

* The Program Deviation Report, which is to be prepared when the 
program office identifies deviations from the approved Acquisition 
Program Baseline goals. More specifically, when the program office has 
reason to believe that a program breach will occur, it is to 
immediately notify the milestone decision authority. Within 30 days, 
the program office is to inform the milestone decision authority of the 
reason for the deviation and the actions it considers necessary to 
bring the program back within baseline goals. 

* The Defense Acquisition Executive Summary, which is prepared to 
inform the milestone decision authority on the program's progress 
against cost, schedule, and performance goals reflected in the 
Acquisition Program Baseline. Prepared quarterly, the summary is 
designed to provide an early warning to the DOD Chief Information 
Officer (CIO) and the milestone decision authority by identifying 
existing and potential program problems and describing mitigating 
actions that have been taken. 

Between October 2002 and December 2004, the reporting requirements for 
the program changed.[Footnote 39] As previously discussed, NTCSS was 
selected by its milestone decision authority to participate in the RIT 
pilot, which was aimed at saving time in the acquisition management 
process by reducing traditional DOD reporting and oversight 
requirements, while still adhering to DOD acquisition guidance. Under 
the RIT pilot, the program office was required to prepare the following 
two monthly electronic reports: 

* The Monthly Acquisition Program Review, which was to assess the 
current health of the program on a monthly basis in such areas as cost 
and schedule performance, testing, funding, and contracting. This 
report was broken into eight parts. According to the program office, 
the main part for NTCSS was the Program Manager Assessment. 

* The Smart Chart, which was to address risks for different projects 
within the program, including a description of the risk, actions taken 
to address the risk, and recommendations for further actions. The Smart 
Chart was also to contain any updates to the Acquisition Program 
Baseline. 

In short, the RIT reporting was to provide the same information 
reported via the traditional acquisition baseline and the summary 
report, but it was to be more frequent (monthly versus quarterly) and 
use a different format (electronic versus paper). In addition, under 
the RIT pilot, certain acquisition documents, such as acquisition 
plans, economic analyses, requirements documents, and test plans, were 
to be posted to the RIT Web site's electronic library rather than sent 
in hard copy to the program's stakeholders. 

In December 2004, the program office and the milestone decision 
authority agreed to discontinue use of the RIT pilot procedures. In 
January 2005, the reporting requirements reverted to the acquisition 
policies and procedures as prescribed in the updated DOD 5000 series. 
Currently, the program office is required to prepare the summary report 
quarterly and the acquisition baseline as needed. Also, in January 
2005, the Navy required the program office to begin making entries into 
the Dashboard. The Dashboard, like the summary report, is prepared by 
the program office on a quarterly basis for the milestone decision 
authority and is to provide an assessment of the program in such areas 
as cost, schedule, and performance characteristics. 

The Navy Has Not Satisfied All NTCSS Reporting Requirements: 

The program office did not comply with the reporting requirements that 
were in effect during the 27 months of the RIT pilot. Specifically: 

* The Smart Chart was not updated for 19 of the 27 months. 
Specifically, the data were updated eight times between October 2002 
and November 2003; the data were not updated after November 2003. 

* The Program Manager Assessment was not updated for 11 of the 27 
months. In addition, the updates were not always made in a timely 
manner. For the 16 months that were updated, 7 were done after the 
month had ended, and most of these updates were a month late. 

* Of the 15 essential acquisition documents that the program office 
committed to entering in the RIT electronic library, 10 were not 
entered. For example, the most recent economic analysis and the test 
and evaluation master plan for OOMA were not entered. 

* The Program Deviation Report and Acquisition Program Baseline were 
not prepared in a timely manner. Specifically, in April 2004, the 
acquisition of eNTCSS was cancelled and, in May 2004, OOMA did not pass 
operational testing--two events that caused the related cost and 
schedule thresholds in the Acquisition Program Baseline to be breached. 
While program officials had notified the milestone decision authority 
of these events via (1) e-mail, (2) entries into the Program Manager 
Assessment on the RIT Web site, and (3) briefings, the program office 
did not prepare a Program Deviation Report until about 15 months later. 
Moreover, this deviation report addressed only the OOMA failure, not 
the cancellation of eNTCSS and reprogramming of unexpended eNTCSS 
funding. In addition, program officials have yet to provide us with a 
new Acquisition Program Baseline to reflect the program breach or 
documentation showing that this revised baseline has been approved by 
the milestone decision authority. 

For the DOD and Navy reporting requirements in effect since January 
2005, the Navy has satisfied some, but not all, of the reporting 
requirements. For example, the program office has prepared the 
Dashboard reports quarterly as required. However, it has not prepared 
the Defense Acquisition Executive Summary quarterly as required; the 
first report was not prepared until June 2005--6 months after the 
requirement resumed and the report was due. 

Program officials provided various reasons why the required program 
reporting has not occurred. In the case of the Smart Charts and the 
Program Manager Assessment reports, a contractor supporting the 
Assistant Program Manager stated that the data may have been entered 
into the Web site but not properly saved. Regarding the posting of 
documents into the electronic library, an official from the milestone 
decision authority stated that there was no documentation from the 
Office of the Assistant Secretary of Defense for Networks and 
Information Integration/Chief Information Officer that directed which, 
if any, acquisition documents were to be entered into the RIT Web site. 
Similarly, a contractor supporting the Assistant Program Manager stated 
that the folders in the electronic library were established by the Army 
and thus the Navy was not required to use them. However, our review of 
documentation provided by the program office shows that it clearly 
states which specific documents should be included in the electronic 
library. Regarding the delay in preparation of the Program Deviation 
Report and subsequent Acquisition Program Baseline revision, a 
contractor supporting the Assistant Program Manager stated that a new 
baseline should have been prepared sooner, but that this reporting was 
delayed due to the uncertainty of which reporting methods to use after 
the end of the formal RIT pilot. 

Officials representing the milestone decision authority stated that 
they relied on program office reporting on program status and progress, 
and that they expected the program office to inform them if the program 
exceeded its cost, schedule, and performance thresholds. Without 
adequate reporting, oversight officials were not positioned to 
effectively execute their roles and responsibilities. 

The Navy Has Not Properly Budgeted for NTCSS: 

In September 1999, the Navy Comptroller issued guidance directing 
program offices to review their budgets and identify efforts that were 
being improperly funded and to take the steps necessary to realign 
these funds to "Research, Development, Test and Evaluation" as quickly 
as possible. Further, DOD Financial Management Regulation[Footnote 40] 
requires that IT development, test, and evaluation requirements 
generally be funded in the "Research, Development, Test and Evaluation" 
appropriations. More specifically it states that, "The Research, 
Development, Test and Evaluation funds should be used to develop major 
upgrades increasing the performance envelope of existing systems, 
purchase test articles, and conduct developmental testing and/or 
initial operational test and evaluation prior to system acceptance." 
Similarly, Navy financial management policy[Footnote 41] states that, 
"All costs associated with software development/modification efforts 
that provide a new capability or expand the capability of the current 
software program (i.e., expand the performance envelope) are funded in 
the Research, Development, Test and Evaluation appropriation."[Footnote 
42] 

However, this has not occurred. Since 1997, the program office has not 
identified "Research, Development, Test and Evaluation" funds in five 
of its seven Acquisition Program Baseline documents, three of which 
were prepared after the guidance was issued by the Comptroller of the 
Navy. Instead, the Navy funded these activities primarily out of the 
"Operations and Maintenance," "Other Procurement," and "Ship 
Construction" appropriations. (See table 10.) 

Table 10: Threshold Amounts in NTCSS Acquisition Program Baselines: 

Dollars in thousands. 

Revision 0; 
Date prepared: March 1997; 
Operations and maintenance: $182,986; 
Other procurement: $199,636; 
Ship construction: $11,683; 
Research, development, test and evaluation: $0. 

Revision 1; 
Date prepared: March 1998; 
Operations and maintenance: $257,542; 
Other procurement: $303,565; 
Ship construction: $23,836; 
Research, development, test and evaluation: $3,026. 

Revision 2; 
Date prepared: December 1998; 
Operations and maintenance: $223,370; 
Other procurement: $285,550; 
Ship construction: $18,220; 
Research, development, test and evaluation: $0. 

Revision 3; 
Date prepared: January 2001; 
Operations and maintenance: $276,100; 
Other procurement: $382,000; 
Ship construction: $27,300; 
Research, development, test and evaluation: $0. 

Revision 4; 
Date prepared: January 2003; 
Operations and maintenance: $276,100; 
Other procurement: $382,000; 
Ship construction: $27,300; 
Research, development, test and evaluation: $0. 

Revision 5; 
Date prepared: July 2003; 
Operations and maintenance: $276,100; 
Other procurement: $382,000; 
Ship construction: $27,300; 
Research, development, test and evaluation: $0. 

Revision 6; 
Date prepared: January 2004; 
Operations and maintenance: $376,400; 
Other procurement: $346,600; 
Ship construction: $25,700; 
Research, development, test and evaluation: $29,800. 

Source: Navy. 

[End of table] 

Program officials agreed that they have funded NTCSS development 
activities, such as those associated with OOMA, out of the "Operation 
and Maintenance" appropriation rather than the "Research, Development, 
Test and Evaluation" appropriation. A contractor supporting the 
Assistant Program Manager stated that, although they were aware of the 
Comptroller of the Navy's budget guidance, the program office chose not 
to comply because program officials believed in 1999 that the OOMA 
application, which had been under development for 3 years, would pass 
developmental testing and operational testing by 2001. As a result, 
program officials determined that the effort required to reprogram 
funding from the "Operation and Maintenance" appropriation into the 
"Research, Development, Test and Evaluation" appropriation was not 
warranted. Further, the official stated that although OOMA did not pass 
operational testing in 2001, the program office did not fund OOMA with 
"Research, Development, Test and Evaluation" funds until 2004 because 
it continued to consider OOMA as being close to becoming operational. 

The lack of proper budgeting for "Research, Development, Test and 
Evaluation" funding has given oversight authorities the misleading 
impression that NTCSS development activities were completed and that 
the system was fully operational. Specifically, officials from the 
Office of the Assistant Secretary of Defense for Networks and 
Information Integration/Chief Information Officer, which was the 
original NTCSS milestone decision authority, stated that since most of 
the "Research, Development, Test and Evaluation" funding appeared to 
have been spent, they concluded that the development portion of NTCSS 
was essentially complete. As a result, these officials stated that they 
had considered taking NTCSS off of the list of programs subject to 
oversight reviews. However, after 9 years and over $79 million in 
expenditures, the OOMA application still has not passed operational 
testing and thus is still in development. 

Navy Oversight of NTCSS Has Not Been Adequate: 

DOD and Navy policies task a number of organizations with oversight of 
IT system acquisition and development programs. For example, DOD policy 
states that a milestone decision authority has overall program 
responsibility. In addition, the Navy Chief Information Officer is 
responsible for reviewing programs at certain points in the acquisition 
cycle. Finally, the NTCSS Executive Steering Committee is responsible 
for monitoring the near-term development and evolution of the NTCSS 
program. However, effective oversight by these entities has not 
occurred. As a result, opportunities to address long-standing program 
weaknesses have been missed, and the program has been allowed to 
proceed virtually unchecked. 

The Milestone Decision Authority Has Not Adequately Overseen the 
Program: 

DOD acquisition policy[Footnote 43] states that a milestone decision 
authority is the designated individual with overall responsibility for 
a program and is to ensure accountability and maximize credibility in 
program cost, schedule, and performance reporting. In this role, the 
milestone decision authority is responsible for reviewing the program 
throughout its acquisition life cycle, including: (1) whenever the 
program reaches a milestone decision point; (2) whenever cost, 
schedule, or performance goals are baselined or must be changed; and 
(3) periodically through review of management information such as that 
found in the Defense Acquisition Executive Summary reports. 

However, the Navy milestone decision authority[Footnote 44] has not 
conducted such reviews. Specifically: 

* The NTCSS program has not reached a milestone decision point in over 
5 years. The last such milestone was in April 2000 when the final two 
NTCSS Optimized applications became operational. The next scheduled 
milestone was to be in 2001, but because OOMA operational testing was 
stopped and has yet to be successfully completed, a milestone decision 
point has yet to occur. As a result, there has not been a triggering 
event that would cause the milestone decision authority to formally 
review the program or any of its projects. We discussed the state of 
NTCSS in March 2005 with the milestone decision authority's 
representatives. In July 2005, the authority was briefed by the program 
office. According to program officials, this was the first formal 
program review to occur since termination of the RIT pilot in December 
2003. These officials also stated that quarterly acquisition team 
meetings have since resumed--with the first meeting having occurred in 
September 2005 and the next scheduled for December 2005--to prepare for 
the next milestone review of OOMA. 

* The program office notified the milestone decision authority in April 
and June 2004 that OOMA failed operational testing and that eNTCSS was 
cancelled via e-mail, entries into the Program Manager Assessment on 
the RIT Web site, and briefings. According to officials with the 
milestone decision authority, they followed up with the program office 
and provided guidance; however, these events did not trigger a formal 
program review. 

* The milestone decision authority did not contact the program office 
to inquire as to the reason why monthly reports were not being prepared 
as agreed to after the formal RIT pilot had ended. For example, Smart 
Charts were not prepared after November 2003. However, according to 
milestone decision authority officials, they did not seek an 
explanation from the program office as to why. Milestone decision 
authority officials told us that they were relying on the Dashboard 
report in order to stay informed on the program's progress. However, 
they did not require the program office to begin preparing the 
Dashboard report until January 2005. 

According to DOD and Navy officials, including officials from the 
Office of the Assistant Secretary of Defense for Networks and 
Information Integration/Chief Information Officer, the Navy milestone 
decision authority, and the program office, NTCSS participation in the 
RIT pilot resulted in disruption of normal oversight activities, which 
have yet to be fully restored. They added that compounding this is the 
fact that the Navy's milestone decision authority's staffing has been 
reduced in recent years. According to these officials, approximately 2 
years ago the number of full time staff in the Office of the Deputy 
Assistant Secretary of the Navy for Command, Control, Communication, 
Computers and Intelligence, and Space was reduced from 16 to 6 people, 
and these 6 are responsible for reviewing approximately 60 acquisition 
programs. The officials stated that, given the large number of programs 
and limited staffing, they are unable to fully perform oversight 
activities so they have increasingly relied on the program executive 
office's assistance to perform detailed oversight of this program. 
Without adequate oversight by the milestone decision authority, the 
NTCSS program has been allowed to proceed despite the program 
weaknesses discussed in this report. 

Other Navy Organizations Have Not Conducted Program Oversight: 

While the milestone decision authority is the main program oversight 
entity, two other Navy organizations have oversight responsibilities. 
However, these entities also have not performed effective oversight of 
the program. Specifically, 

* Department of Navy CIO is responsible for reviewing programs at 
certain points in the acquisition cycle to ensure, among other things, 
that program goals are achievable and executable and that the program 
is providing value (i.e., producing a positive return-on-investment). 
Navy CIO officials stated that they have overseen NTCSS primarily by 
reviewing the Capital Investment Reports[Footnote 45] prepared by the 
program office. They stated that they have not performed any proactive 
activities to verify and validate the program's status and progress. 
Instead, they rely on information in the Capital Investment Reports, 
such as economic justification; budget information by appropriation 
type; and cost, schedule, progress, and status. However, as was 
discussed previously, the program office does not have or has not 
reported reliable information on these topics. 

* The NTCSS Executive Steering Committee is responsible for 
establishing priorities for NTCSS development and implementation and 
determining the strategic direction of the program. Among other things, 
it is to meet immediately following each major NTCSS program meeting. 
However, it has not met since December 2002, even though the program 
office convened both a Requirements Integrated Product Team meeting and 
a Forum meeting in February 2005. Further, during this period, major 
setbacks occurred on the program, including the failure of OOMA to pass 
operational testing and the cancellation of eNTCSS, which were issues 
that affected the direction of the program and its priorities and thus 
were consistent with the committee's charter. Program officials agreed 
that the Executive Steering Committee has not formally convened during 
this time frame. However, program officials stated that members of the 
committee informally met to discuss and provide advice regarding OOMA 
concerns, and Navy officials higher than the Executive Steering 
Committee made the decision to cancel eNTCSS. Therefore, these 
officials stated there was no need to formally convene an Executive 
Steering Committee meeting. Program officials stated that the Executive 
Steering Committee will be meeting in January 2006. 

NTCSS Requirements and Test Management Weaknesses Have Contributed to 
Deployment Delays and System Quality Problems: 

As we have previously reported,[Footnote 46] the effectiveness of the 
processes used to develop a system is a reliable predictor of the 
quality of the system products produced. Two key system development 
processes are requirements development and management and test 
management. For the NTCSS application currently under development, we 
found weaknesses with both of these process areas. While improvements 
are planned, until they are implemented effectively, the risk of 
continued NTCSS cost, schedule, and performance shortfalls persists. 

The Navy Has Not Adequately Managed Requirements for the NTCSS 
Application Currently Under Development: 

Well-defined requirements can be viewed as a cornerstone of effective 
system development and implementation. Accordingly, DOD guidance and 
industry best practices recognize effective requirements development 
and management as an essential system development and acquisition 
management process. For the NTCSS application that is currently under 
development--OOMA--the Navy has not adequately managed its 732 
requirements, as evidenced by a lack of requirements traceability and 
prioritization. NTCSS program officials told us that NTCSS requirements 
development practices have historically been poor, but that 
improvements are under way. Without effective requirements management, 
it is likely that the Navy's challenges to date in developing NTCSS 
applications that meet user needs on time and on schedule will 
continue. 

Requirements for OOMA Release 4.10 Were Not Traced: 

DOD guidance and industry best practices also recognize the importance 
of requirements traceability.[Footnote 47] The purpose of requirements 
traceability is to ensure that the finished product is compliant with 
the requirements. To do this, the system documentation should be 
consistent and thus complete, allowing for requirements traceability. 
Requirements traceability involves both the alignment and consistency 
backward to system documentation and forward to system design and test 
documentation. 

OOMA release 4.10 requirements were not traced to an Operational 
Requirements Document. According to DOD guidance,[Footnote 48] an 
Operational Requirements Document translates nonsystem-specific 
statements of a needed operational capability into a set of validated 
and prioritized user requirements. However, the Navy did not develop an 
Operational Requirements Document for NTCSS. As a result, the Navy did 
not take a basic validation step to ensure that the requirements to 
which it designed and built the application were complete and correct. 
In addition, release 4.10 requirements were not always traceable to 
associated system specifications. Specifically, we were unable to trace 
215 requirements found in the system segment specification to the 
requirements listed in the requirements checklist. Requirements should 
also be traced to test cases, but the program office has yet to provide 
us with the developmental test cases used to test the OOMA release 4.10 
so that we could verify this traceability. 

Program officials acknowledged that release 4.10 requirements were not 
traceable but that improvements are planned for the next OOMA release. 
We found that 97 percent of the OOMA release 5.0 requirements found in 
the system segment specification were traceable to the requirements 
listed in the requirements checklist. However, these documents have yet 
to be approved. Requirements should also be traced to test cases, but 
the program office has yet to provide us with the developmental test 
cases used to test the OOMA release 5.0 so that we could verify this 
traceability. Without this traceability, the Navy has not had a 
sufficient basis for knowing that the scope of its development efforts, 
including testing, provides adequate assurance that applications will 
perform as intended. 

Requirements for OOMA Release 4.10 Were Not Prioritized: 

According to published best practices guidance,[Footnote 49] any 
project with resource limitations should establish the relative 
priorities of the requested features or requirements. Prioritization 
helps the project office resolve conflicts, make trade-off decisions 
among competing requirements, and helps to ensure that the delivered 
system will be operationally suitable. 

However, OOMA's approximately 732 requirements have never been 
prioritized, and a program official told us that they are all 
considered to be equally important. This means, for example, that a 
requirement that dictates basic application functionality (e.g., if 
text can be entered on a particular screen) is as important as a 
requirement addressing safety issues that, if not met, could result in 
the loss of an aircraft or even a life. 

This lack of requirements prioritization contributed to release 4.10 
passing developmental testing but failing operational testing. (See 
later section of this report for a detailed discussion of OOMA 
testing.) A developmental testing threshold that the Navy set for 
release 4.10 was that each requirement was to be tested, and 95 percent 
of the requirements had to pass in order for the application to proceed 
to operational testing. For developmental testing of the OOMA release 
4.10, 97 percent of the requirements passed. Of the 3 percent of the 
requirements that failed this test, some of these deficiencies 
seriously impacted squadron level operations. Further, for operational 
testing of OOMA release 4.10, 96 percent of the requirements passed. 
However, the remaining 4 percent contained significant defects. 
Specifically, the release provided inconsistent and inaccurate flight 
and usage hours, as well as incorrect aircraft usage records. According 
to the Navy's independent operational test organization, these 
deficiencies impacted aircraft and component time-based inspection 
cycles and thus were the basis for the system failing operational 
testing. The Navy has yet to provide evidence that the requirements 
have been prioritized for the OOMA release 5.0. 

The Navy's Developmental Testing for OOMA Has Not Been Effective, but 
Improvements Planned: 

Both DOD policy and relevant guidance recognize that effective testing 
is an essential component of system development or acquisition 
programs. Generally, testing can be viewed as consisting of two major 
phases--a developmental phase in which tests are performed to ensure 
that defined system requirements and specifications are met and an 
operational phase that includes tests to determine if the system meets 
user needs and is suitable in an operational environment. The OOMA 
application has failed operational testing twice over the last 4 years 
reportedly because of deficiencies in developmental testing. Program 
officials attributed developmental testing deficiencies to poor 
software development practices, such as the earlier discussed 
requirements development problems. These testing deficiencies can also 
be attributed to incomplete testing documentation. Without effective 
developmental testing, there is an increased risk that application 
problems will be detected later in the system life cycle when they are 
more expensive and difficult to fix. 

Navy Operational Testing Organization Reported That Developmental 
Testing Has Failed to Identify Problems: 

According to DOD guidance and recognized best practices,[Footnote 50] 
the purpose of developmental testing is to provide objective evidence 
that the product (e.g., software module, application, system) satisfies 
defined requirements and performs as intended. Successful completion of 
developmental testing provides the basis for proceeding into 
operational testing to determine whether the integrated product (e.g., 
application, system, system of systems) performs as intended in an 
operational or real-world setting. 

OOMA operational testing results over the last 4 years show that the 
program office's developmental testing efforts have not been effective 
in identifying critical product problems. In particular, the 
application has failed operational testing twice during this time frame 
and, according to an official in the office of the Director of Navy 
Test and Evaluation and Technology Requirements, the failures occurred 
in operational testing because they were not identified during 
developmental testing. More specifically, 

* In March 2001, the program office certified that OOMA release 3.25 
had passed developmental testing and was ready for operational testing. 
However, 1 month into a scheduled 3-month operational test, the 
decision was made to cease further testing because of significant 
problems with system reliability, data transfer between the application 
and the database, and user training on the application. As a result, 
the program office decertified this release, and the Navy's independent 
test organization recommended discontinuing OOMA deployment. 

* Using results from the failed operational test, the central design 
agency developed release 4.0. In February and March 2002, developmental 
testing of this release was conducted. Test results showed that the 
application was not ready for operational testing because it did not 
satisfy key functional requirements. Subsequently, the central design 
agency incorporated software fixes in release 4.10. In August and 
September 2002, developmental testing was conducted on this release 
and, while a number of deficiencies were verified as fixed, additional 
corrections were needed. From January to June 2003, developmental 
testing was again conducted on OOMA release 4.10. 

* From August 2002 to April 2003, the Naval Audit Service[Footnote 51] 
reviewed OOMA and reported several problems that would affect the 
application's readiness for operational testing. For example, it 
reported that controls to prevent unauthorized access were not in 
place, Privacy Act information was not adequately protected, and backup 
and recovery procedures were not in place. It also reported that the 
program had not adopted and implemented a risk-based system life cycle 
management approach. According to the report, these weaknesses could 
compromise safety, affect planning, and distort readiness reporting if 
OOMA was implemented throughout the Navy. 

* In June 2003, the program office certified OOMA release 4.10 as 
having passed developmental testing and being ready for operational 
testing. The Navy's independent operational test organization 
subsequently conducted testing from August to December 2003 and, in May 
2004,[Footnote 52] this organization concluded that OOMA was not 
operationally effective or suitable and thus it again failed 
operational testing. In particular, the operational testing results 
showed that the application was incorrectly calculating flight and 
component usage hours--defects, which according to an official in the 
office of the Director of Navy Test and Evaluation and Technology 
Requirements, could have resulted in the loss of aircraft or life. The 
Assistant Program Manager also told us that release 4.10 did not 
address all of the deficiencies reported by the Naval Audit Service. 

For about a year, the central design agency has been developing and 
testing OOMA release 5.0 to fix the problems found in the prior 
version. The program office expects that this release will be certified 
as ready for operational testing sometime between April and June 2006. 
In preparation for operational testing, the Navy's independent 
operational test organization has been observing OOMA 5.0 developmental 
testing. A memo from this organization states that this release is an 
improvement over the previous releases. 

According to Navy officials, including the NTCSS Assistant Program 
Manager and the official responsible for OOMA developmental testing, 
previous application development practices were poor, which led to 
testing problems. Specifically, they cited poor requirements 
definitions, poor documentation, and concurrent development of 
application releases as examples. Further, Navy officials stated that 
the central design agency has not had a developmental testing lab to 
facilitate effective testing of application components and their 
integration. To address the poor development practices, program 
officials told us that they are in the process of implementing a new 
system life cycle management process that they said incorporates 
industry best practices, including those related to testing. However, 
the program office has yet to provide us with information defining how 
the practices in this plan will be implemented. To address the need for 
a developmental testing lab, the Naval Air Systems Command organization 
representing NTCSS users recently created a lab to strengthen the 
program's developmental testing capability. According to officials 
associated with the lab, they are finding defects that the central 
design agency should have found. 

It is important that the NTCSS program improve its developmental 
testing. Without effective developmental testing, there is an increased 
risk that system application problems will be detected late in the 
system life cycle, such as during operational testing. Generally, 
problems discovered late in the cycle are more expensive and difficult 
to fix than those discovered early. 

Developmental Test Documentation Has Not Been Adequate, but 
Improvements Planned: 

To be effective, testing should be approached in a rigorous and 
disciplined fashion. One aspect of such testing is developing and using 
various testing documentation. DOD policy, guidance, and related best 
practices[Footnote 53] state that such documentation includes a test 
and evaluation master plan for the program, as well as documentation 
that is system product (e.g., module, application, system) and test 
type (e.g., integration, stress, regression, developmental) specific. 
This documentation includes approved test plans, test procedures and 
cases, and test results. According to DOD and other guidance, test 
plans should include, among other things, objectives, responsibilities, 
resources (tools, people, and facilities), schedules, and performance 
and exit criteria; test procedures should include detailed test 
scenarios, test events, steps, inputs, and expected outputs that are 
traced back to requirements. Test results include the test scenarios 
that passed and failed, assessments of deviations from test plans, and 
the extent to which requirements have been met. 

The NTCSS test and evaluation master plan identified, among other 
things, three phases of developmental testing for OOMA release 4.10. 
However, key test documentation for each of these phases was not 
produced. Specifically, 

* For the first phase, a test report was produced that contained 
detailed information on test results, but the program office has yet to 
provide us with a test plan or test procedures. 

* For the second phase, a test report was produced but it only 
contained the number of defects found (organized by severity) and did 
not include any other information on test results. Moreover, the 
program office has yet to provide us with a test plan or test 
procedures. 

* For the third phase, both a test plan and test report were produced, 
and the plan included the test purpose and objectives, schedule, 
responsibilities, and people resources, while the test report described 
test issues and contained detailed test results. However, the program 
office has yet to provide us with test procedures. 

According to Navy officials, including the Assistant Program Manager 
and officials responsible for developmental testing, the previously 
mentioned poor application development practices contributed to the 
absence of testing documentation. To address these poor practices, the 
program has developed a system life cycle plan that they said 
incorporates industry best practices, including those associated with 
testing documentation. However, the program has yet to provide us with 
plans defining how these practices will be implemented. Moreover, while 
the plan contains a recommended list of testing documents (e.g., test 
plan, test procedures, and test results report), our review of OOMA 
release 5.0 developmental testing documentation shows that not all the 
documentation is being prepared. Specifically, available documentation 
included an updated test and evaluation master plan and two test 
reports. Documentation not yet provided to us included test procedures, 
which would include documentation tracing test cases to requirements. 

The lack of a full set of developmental test documentation is a 
problem. Without such documentation, the adequacy and reliability of 
developmental testing cannot be substantiated, and thus the quality of 
the associated system products is in doubt. 

Central Design Agency Reports Management Improvements are Under Way: 

In an effort to improve its performance on NTCSS and other programs, 
central design agency officials told us that they chose to undergo an 
SEI Capability Maturity Model Software Capability Appraisal in July and 
August 2005. Carnegie Mellon University's SEI, recognized for its 
expertise in software and system processes, has developed the 
Capability Maturity Model™ for Software (SW-CMM)[Footnote 54] to 
provide guidance on how to gain control of their processes for 
developing and maintaining software and how to evolve toward a culture 
of software engineering and management excellence. 

In brief, SW-CMM calls for assessing different process areas--clusters 
of related activities such as project planning, requirements 
management, and quality assurance--by determining whether key practices 
are implemented and whether overarching goals are satisfied. Successful 
implementation of these practices and satisfaction of these goals 
result in the achievement of successive maturity levels. SW-CMM 
maturity levels range from 1 to 5, with level 1 meaning that the 
process is either characterized as ad hoc and occasionally even chaotic 
with few processes defined and success depending on individual effort; 
level 2 meaning that the process is repeatable; level 3 meaning that 
the process is defined; level 4 meaning that the process is managed; 
and level 5 meaning that the process is optimized. 

According to the central design agency they achieved a maturity rating 
of level 3 against the SW-CMM based on 13 process areas, including 
requirements management, software project planning, software project 
tracking and oversight, subcontract management, software quality 
assurance, software configuration management, organizational process 
focus, organizational process definition, training program, integrated 
software management, software product engineering, intergroup 
coordination, and peer reviews. Further, we were told that NTCSS was 
one of the programs included in the review. However, we have yet to 
receive the appraisal report to determine the extent to which the 
appraisal addressed the weaknesses discussed in this report. 
Nevertheless, our research has shown that properly performing such 
appraisals can be a useful starting point for making software and 
system related development improvements. 

Conclusions: 

It is unclear whether the Navy's planned investment in NTCSS is 
warranted. Of particular concern is the absence of reliable analysis 
showing that further investment will produce future mission benefits 
commensurate with estimated costs, as well as the void in information 
concerning whether the deployed and operational components of NTCSS are 
actually producing expected value. Compounding this uncertainty is the 
inherent risk of defining and developing NTCSS outside the context of 
either a well-defined DOD or Navy enterprise architecture. Without this 
information, the Navy cannot determine whether NTCSS as defined, and as 
being developed, is the right solution to meet its strategic business 
and technological needs. 

Even if these uncertainties were to be addressed, and the Navy had the 
data needed to demonstrate that NTCSS plans are the right course of 
action, then the manner in which NTCSS is being defined, developed, 
tested, measured, and overseen would still be of concern. While any one 
of the concerns that we found is troubling, their combination subjects 
the program to an unacceptably high risk of failure. These effects are 
being realized on NTCSS, as evidenced by the cancellation of one system 
component and the repeated failure of another key component to pass 
testing. 

It is extremely important that Navy and DOD authorities responsible and 
accountable for ensuring prudent use of limited resources reassess 
whether allowing NTCSS to continue as planned is warranted. It is also 
important that the decision on how to proceed be based on reliable data 
about program cost, benefits, risk, and status. 

Recommendations for Executive Action: 

We recommend that the Secretary of Defense direct the Secretary of the 
Navy to determine if continued investment in NTCSS, as planned, 
represents a prudent use of the department's limited resources. To 
accomplish this, the Secretary of the Navy should direct the program 
office to take the following three actions: 

* collaborate with the Office of the Assistant Secretary of Defense for 
Networks and Information Integration/Chief Information Officer, the 
Office of Program Analysis and Evaluation, and the Naval Cost Analysis 
Division to prepare a reliable economic analysis that encompasses all 
viable alternatives, including the Navy's recent enterprise resource 
planning program; 

* ensure that development of this economic analysis (1) complies with 
cost estimating best practices, including recognition of costs to 
resolve open trouble reports and change proposals, and relevant OMB 
cost benefit guidance and (2) incorporates available data on whether 
deployed NTCSS capabilities are actually producing benefits; and: 

* collaborate with the Undersecretary of Defense for Acquisition, 
Technology, and Logistics and the Under Secretary of Defense 
(Comptroller) to ensure that NTCSS is adequately aligned with evolving 
DOD and Navy enterprise architectures. 

In addition, we recommend that the Secretary of Defense direct the 
Secretary of the Navy to present the results of these analyses to the 
Deputy Secretary of Defense, or his designee, and seek a departmental 
decision on how best to proceed with the program. Until this is done, 
we recommend that the Secretary of Defense direct the Secretary of the 
Navy to halt further deployment of NTCSS and to limit future investment 
in already deployed applications to essential operation and maintenance 
activities and only developmental activities deemed essential to 
national security needs. 

If--based on reliable data--a decision is made to continue the NTCSS 
program, we recommend that the Secretary of Defense direct the 
Secretary of the Navy to ensure that the following two actions are 
taken: 

* the NTCSS program implements effective program management activities, 
including earned value management, requirements development and 
management, and test management; and: 

* key stakeholders, such as the central design agency and the 
developmental testing organization, have the people, processes, and 
tools to effectively execute their respective roles and 
responsibilities. 

Finally, we recommend that Secretary of Defense reestablish the Office 
of the Assistant Secretary of Defense for Networks and Information 
Integration/Chief Information Officer as the milestone decision 
authority and direct the Secretary of the Navy to take steps to ensure 
that Navy oversight entities fulfill their roles and responsibilities 
on NTCSS, including ensuring that reliable program reporting occurs and 
is acted upon. 

Agency Comments and Our Evaluation: 

In its written comments on our draft report, signed by the Deputy to 
the Assistant Secretary of Defense for Networks and Information 
Integration (Command, Control, Communications, Intelligence, 
Surveillance, and Reconnaissance and Information Technology 
Acquisition) and reprinted in appendix IV along with our detailed 
responses, DOD stated that some of our findings are valid. For example, 
it acknowledged that NTCSS was defined and implemented without a 
complete and formal enterprise architecture. However, it also commented 
that our overall findings significantly understated and misrepresented 
the program's level of discipline and conformance with applicable 
guidance and direction. The department added that NTCSS "has proven to 
be the right solution to meet the Navy's strategic business and 
technological needs," and that sound program management practices are 
in place and improving. 

Neither DOD's comment about our overall findings nor its claims about 
NTCSS being the right solution and being effectively managed are 
adequately supported, as evidenced by the numerous factual instances 
that we site in the report where the Navy did not comply with either 
DOD acquisition policies and guidelines or industry best practices. 
Specifically, the report shows that the program's latest economic 
analysis did not provide the Navy a reliable basis upon which to make 
investment decisions. For example, the analysis did not include 
measurable, quantifiable benefits for each alternative, and the cost 
estimates did not meet six of the nine criteria associated with 
reliable cost estimates. The analysis also was not independently 
reviewed in accordance with DOD guidance and the Navy had yet to 
demonstrate that already deployed NTCSS Optimized applications are 
producing expected benefits. We appropriately concluded that the Navy 
does not know whether the program as defined is the right solution to 
meet the Navy's strategic business and technological needs. 

With respect to our recommendations, DOD fully concurred with two of 
the recommendations and partially concurred with the remaining five 
recommendations. The five areas of disagreement, DOD's basis for its 
disagreement, and our response to DOD's position follow. 

First, DOD stated that it does not see merit in conducting a formal 
economic analysis for the NTCSS program that would address all viable 
alternatives because, at this late stage, NTCSS is a "very mature 
program," and the final application (OOMA) is about to be fielded. 
Further, DOD said it saw no merit in seeking Office of Program Analysis 
and Evaluation (PA&E) review of the economic analysis. Instead, it said 
that it will "coordinate" with PA&E in analyzing the relationship of 
NTCSS with other programs that may provide similar functionality and 
"brief the results" to selected stakeholders. 

We do not agree that NTCSS is a "very mature program." In particular, 
the Navy still plans to spend in fiscal years 2006 through 2009 an 
additional $348 million, which is approximately one-third of what has 
been spent on the program to date. Further, there is no evidence to 
support the claim that the OOMA application is about to be fielded. 
OOMA has failed operational testing twice and is not yet fully 
developed or tested despite the Navy's initial plan to field it in 
2001. In addition, the Navy's stated intention to develop an economic 
analysis for OOMA only and then, separately, prepare an "analysis to 
determine the relationship" of NTCSS and other alternative programs is 
not consistent with guidance and best practice, which advocate basing 
such analyses on the full scope of the planned investment. In addition, 
the proposal to limit key stakeholders' involvement in developing the 
economic justification to "coordinating" and "briefing would be 
inappropriate." These stakeholders have specific expertise and roles 
relative to economically justifying system investments that should be 
exploited. Until it conducts a complete and disciplined analysis of the 
entire NTCSS program (reviewed and approved by PA&E and the Naval Cost 
Analysis Division) and provides this analysis to all key stakeholders, 
the Navy's investment decisions will continue to be made without 
complete and reliable data. 

Second, the department stated that further deployment of NTCSS should 
not be limited at this time. Nevertheless, it stated that it will use 
the results of the analysis referred to above that depicts NTCSS's 
relationship with other programs to provide appropriate direction to 
the program. We do not agree that development should not be limited and 
would note that the department's own comment acknowledges the need to 
decide on an appropriate direction for the program. In our view, 
prudent use of taxpayer resources warrant both a reliable economic 
analysis that can be used to inform any decision on this direction and 
fiscal restraint to investing until an informed decision can be made. 

Third, DOD said that the Navy does not need to be directed to ensure 
that effective program management activities are implemented because it 
is continuously improving program management activities. Further, DOD 
stated that, although it is not required to implement an earned value 
management system because the individual projects do not meet the 
dollar threshold and there are no formal contract deliverables, it is 
nevertheless adhering to the 32 earned value management criteria set 
forth in applicable standards. The department added that it intends to 
have the Navy Inspector General conduct a separate study to further 
ensure that the program is using the best program management 
activities. 

We do not agree with these comments. In particular, neither during our 
review nor in its comments did the Navy provide evidence that it has 
implemented effective program management activities or has improvements 
under way. As we state in our report, neither the decomposition of the 
program into small, fiscal year-based projects nor the absence of a 
contractual relationship is a valid reason for not effectively 
implementing earned value management. Further, the Navy's earned value 
management self-assessment showed that it had not adhered to 17 of the 
32 earned value management standards. Without reliable, timely, and 
auditable earned value management data, the program office cannot 
adequately manage technical, cost, and schedule risks and problems. 

Fourth, the department stated that key stakeholders of the NTCSS 
program have the necessary people, processes, and tools to effectively 
execute their respective roles and responsibilities, noting in 
particular that the central design agency has demonstrated its 
competency and capability and was certified as SW-CMM maturity level 3. 
Nevertheless, the department agreed to address this recommendation. 

We support the Navy's stated commitment to address this recommendation. 
In addition, we would note that DOD's comment that stakeholders have 
the resources they need is not consistent with statements from 
stakeholders during our review who said that there were manpower and 
resource shortfalls that affected the oversight and execution of 
program activities. Further, despite the Navy's statement that the 
central design agency achieved SW-CMM maturity level 3, no 
documentation supporting this statement, such as appraisal reports, 
were provided. Furthermore, Navy officials told us that the central 
design agency did not have a development testing lab and was therefore 
unable to effectively execute testing activities. 

Fifth, DOD stated that it is "premature" to reestablish the DOD Chief 
Information Officer as the milestone decision authority as NTCSS 
development is over 95 percent complete. Instead, it stated that 
existing oversight entities would ensure that effective program 
management and reporting was occurring. 

We do not agree that elevating the milestone decision authority at this 
time is premature based on the statement that the program is 95 percent 
complete. For programs that have not been developed using industry best 
practices and technical and management discipline, which is the case 
for NTCSS, such claims of being essentially complete have historically 
proven inaccurate because they are not grounded in reliable performance 
data. Moreover, the Navy still plans to spend $348 million on NTCSS 
over the next three fiscal years. Finally, as stated in our report, the 
current milestone decision authority has allowed the program to operate 
unchecked although a major application has repeatedly failed 
operational testing, and another application was cancelled. 

We are sending copies of this report to interested congressional 
committees; the Director, Office of Management and Budget; the 
Secretary of Defense; the Deputy Secretary of Defense; the Under 
Secretary of Defense for Acquisition, Technology and Logistics; the 
Under Secretary of Defense (Comptroller); the Assistant Secretary of 
Defense (Networks and Information Integration)/Chief Information 
Officer; the Deputy Assistant Secretary of the Navy for Command, 
Control, Communication, Computers and Intelligence, and Space; the 
Program Executive Office for Command, Control, Communication, Computers 
and Intelligence, and Space within the Space and Naval Warfare Systems 
Command; the Department of the Navy Chief Information Officer; and the 
Office of the Chief of Naval Operations for Material Readiness and 
Logistics Operations. This report will also be available at no charge 
on our Web site at [Hyperlink, http://www.gao.gov]. 

If you or your staff have any questions on matters discussed in this 
report, please contact me at (202) 512-3439 or [Hyperlink, 
hiter@gao.gov]. Contact points for our Offices of Congressional 
Relations and Public Affairs may be found on the last page of this 
report. GAO staff who made major contributions to this report are 
listed in appendix V. 

Signed by: 

Randolph C. Hite: 
Director, Information Technology Architecture and Systems Issues: 

[End of section] 

Appendixes: 

Appendix I: Objective, Scope, and Methodology: 

Our objective was to determine whether the Naval Tactical Command 
Support System (NTCSS) is being managed according to important aspects 
of the Department of Defense's (DOD) acquisition policies and guidance, 
as well as other relevant acquisition management best practices. To 
accomplish our objective, we focused on the program's (1) economic 
justification; (2) architectural alignment; (3) program management, 
namely progress measurement and reporting, funding disclosure, and 
oversight; and (4) key system development activities, namely 
requirements development and management, test management, and system 
maturity indicators. For requirements and test management, we focused 
on the one NTCSS application that is currently being acquired, known as 
the Optimized Organizational Maintenance Activity (OOMA). 

To determine whether the Navy has economically justified its investment 
in NTCSS, we reviewed the latest economic analysis to determine the 
basis for the cost and benefit estimates and net present value 
calculations. This included evaluating the analysis against DOD and 
Office of Management and Budget (OMB) guidance, as well as relevant 
best practices.[Footnote 55] It also included interviewing program 
officials, including the Assistant Program Manager; the office of the 
Deputy Assistant Secretary of the Navy for Command, Control, 
Communication, Computers and Intelligence, and Space; the Office of 
Program Analysis and Evaluation; and the Naval Cost Analysis Division 
as to their respective roles, responsibilities, and actual efforts in 
developing and/or reviewing the economic analysis. In addition, we also 
interviewed the Assistant Program Manager and the office of the Deputy 
Assistant Secretary of the Navy for Command, Control, Communication, 
Computers and Intelligence, and Space about the purpose and use of the 
analysis for managing the Navy's investment in the NTCSS program 
including the extent to which measures and metrics showed that 
projected benefits in the economic analysis were actually being 
realized. 

To determine whether the Navy has aligned NTCSS to either the DOD 
business enterprise architecture[Footnote 56] or a Navy architecture, 
we relied on our prior reports addressing DOD and Navy architecture 
development and implementation efforts, a memo and analysis results on 
NTCSS's compliance with the business enterprise architecture, and 
documents on the Navy's architecture efforts. We also interviewed Navy 
officials from the program office; the office of the Deputy Assistant 
Secretary of the Navy for Command, Control, Communication, Computers 
and Intelligence, and Space; the office of the Navy Research, 
Development, and Acquisition Chief Engineer; and the Office of the 
Assistant Secretary of Defense for Networks and Information 
Integration/Chief Information Officer about DOD and Navy architecture 
efforts and NTCSS's alignment to them. 

To determine whether the Navy was effectively measuring, reporting, and 
overseeing the program, we did the following: 

* We first asked the central design agency to self-assess their 
satisfaction of 32 best practice criteria regarding their earned value 
management system. Using the results of their self-assessment to target 
our analysis, we then assessed those aspects of the earned value 
management system the self-assessment reported as meeting the criteria, 
by comparing the documentation with relevant DOD guidance and best 
practices.[Footnote 57] We selected these two projects as case studies 
to determine the degree to which earned value management was being 
implemented. The two projects selected were (1) 2004 OOMA software 
project and (2) 2004 NTCSS hardware installation and integration (for 
both OOMA and Optimized NTCSS). We selected these two because they were 
the projects for which Navy provided us the most earned value 
management related documentation. To understand the Navy's reasons why 
they were not performing certain elements of earned value management, 
we interviewed officials including the Assistant Program Manager, and 
officials at the central design agency in Norfolk and the in service 
engineering agency in Charleston. 

* To assess reporting capabilities, we reviewed program documentation 
such as Acquisition Program Baselines, program deviation reports, and 
Defense Acquisition Executive Summary reports. We also reviewed 
information and documentation on the Rapid Improvement Team pilot Web 
site including a report that assesses the current health of the program 
on a monthly basis and a report that address risks for different 
projects within the program. 

* To assess compliance with budget policies and guidance, we compared 
NTCSS budget documentation with DOD and Navy financial management 
policies and guidance. 

* To assess oversight of the program, we interviewed the program 
manager, milestone decision authority, functional sponsor, Navy Chief 
Information Officer, and a representative of the program's executive 
steering committee. 

* To determine whether the Navy was effectively managing key system 
development activities, namely requirements management, testing, and 
system maturity indicators, we did the following: 

* To assess requirements development and management capabilities, we 
reviewed program documentation such as the official list of 
requirements and system specifications, and evaluated them against 
relevant best practices[Footnote 58] for several characteristics 
including traceability and prioritization. We attempted to trace 
requirements to both higher level documents and lower level 
specifications. We also attended the NTCSS Forum where requirements 
were gathered and discussed. We interviewed Navy officials such as the 
Assistant Program Manager, Commanding Officer and Executive Director of 
the central design agency, and the OOMA Functional Manager to discuss 
their roles and responsibilities for developing and managing 
requirements. 

* To assess test management, we reviewed program documentation such as 
the test and evaluation master plan, test plans, test reports, and 
guidance. We then compared these documents with DOD guidance and best 
practices and focused on the effectiveness of developmental testing and 
the adequacy of developmental testing documentation.[Footnote 59] Our 
review: 

* also included an audit report prepared by the Naval Audit 
Service[Footnote 60] and a test report prepared by Navy's independent 
operational test organization.[Footnote 61] We interviewed Navy 
officials such as the Assistant Program Manager, Commanding Officer and 
Executive Director of the central design agency, OOMA Functional 
Manager, and an official in the office of the Director of Navy Test and 
Evaluation and Technology Requirements to discuss their roles and 
responsibilities for test management. 

We did not independently validate information on the program's cost and 
budget or the number of trouble reports and change proposals. 

We conducted our work at DOD headquarters in Arlington, Virginia; at 
Space and Naval Warfare Center, San Diego, California; Space and Naval 
Warfare Systems Center, Norfolk, Virginia; and Naval Air Systems 
Command in Patuxent River, Maryland. We performed our work from 
September 2004 through November 2005 in accordance with generally 
accepted government auditing standards. 

[End of section] 

Appendix II: Trouble Reports and Change Proposals Assessment: 

One indicator of system quality, and thus the effectiveness of the 
development activities used to produce system products, is the volume 
and significance of system problems and change proposals. For the Naval 
Tactical Command Support System (NTCSS), trouble reports are prepared 
to document system defects, and change proposals are prepared to 
introduce additional system functionality. Priority levels are assigned 
to trouble reports and change proposals, with 1 being the most critical 
and 5 being the least critical. Table 11 defines the 5 priority levels. 

Table 11: NTCSS Trouble Report and Change Proposal Priorities: 

Priority level: Priority 1; 
Definition: Prevents the accomplishment of an operational or mission-
essential capability; and jeopardizes safety or security. 

Priority level: Priority 2; 
Definition: Adversely affects the accomplishment of an operational or 
mission-essential capability, and no work-around solution is available. 

Priority level: Priority 3; 
Definition: Adversely affects the accomplishment of an operational or 
mission-essential capability, but a work-around solution is available. 

Priority level: Priority 4; 
Definition: Results in user/operator inconvenience or annoyance but 
does not affect a required operational or mission-essential capability. 

Priority level: Priority 5; 
Definition: Any other effect. 

Source: Navy. 

[End of table] 

Available data on the number and significance of open trouble reports 
and change proposals over the last 2 years do not demonstrate that 
NTCSS overall is a high-quality system that is delivering promised or 
expected capabilities. Specifically, the data shows that hundreds of 
open (yet to be resolved) trouble reports and change proposals have 
continued to affect the system. 

Trouble Reports: 

The total number of NTCSS priority 1, 2, and 3 trouble reports have 
stayed about the same over the last 2 years--totaling about 700. Of 
this total, NTCSS priority 1 and 2 trouble reports have decreased by 
117, with priority 1 trouble reports being virtually eliminated. While 
this is movement in a positive direction, about 300 priority 2 trouble 
reports still remain open and these by definition are adversely 
affecting accomplishment of an operational or mission-essential 
capability. (See figs. 1 and 2.) 

Figure 1: Total Number of Open NTCSS and OOMA Priority 1, 2, and 3 
Trouble Reports: 

[See PDF for image] 

[End of figure] 

Figure 2: Open NTCSS Priority 1 and 2 Trouble Reports: 

[See PDF for image] 

[End of figure] 

Further, open priority 3 trouble reports have increased during this 
time to about 250 and, given that priority 3s require work-arounds, 
they decrease system capability and performance. Neither the number of 
priority 2 trouble reports, which continue to be in the hundreds, nor 
the upward trend in priority 3 trouble reports are indicative of a 
maturing, high-quality system. (See fig. 3.) 

Figure 3: Open NTCSS Priority 3 Trouble Reports: 

[See PDF for image] 

[End of figure] 

With respect to the OOMA application in particular, the trend in the 
volume of significant trouble reports shows that this application is 
particularly problematic. Specifically, while priority 1 OOMA open 
trouble reports have been virtually eliminated, the number of open 
priority 2 OOMA trouble reports has risen significantly from 12 to 90 
in about the last 2 years. (See fig. 4.) 

Figure 4: Open OOMA Priority 1 and 2 Trouble Reports: 

[See PDF for image] 

[End of figure] 

Moreover, the number of open OOMA priority 3 trouble reports has not 
significantly declined over the last 2 years, with these remaining at 
roughly 160. (See fig. 5.) 

Figure 5: Open OOMA Priority 3 Trouble Reports: 

[See PDF for image] 

[End of figure] 

Change Proposals: 

The picture for NTCSS change proposals is similar to that for trouble 
reports. Specifically, the total number of open NTCSS priority 1, 2, 
and 3 change proposals has increased over the last 2 years--going from 
about 325 to 425. Of this total, NTCSS priority 2 change proposals have 
increased by 72, with 247 priority 2 proposals still being open. (See 
figs. 6 and 7.) 

Figure 6: Total Number of Open NTCSS and OOMA Priority 1, 2, and 3 
Change Proposals: 

[See PDF for image] 

[End of figure] 

Figure 7: Open NTCSS Priority 1 and 2 Change Proposals: 

[See PDF for image] 

[End of figure] 

Further, NTCSS priority 3 change proposals have increased during this 
time to about 81, and given that priority 3 change proposals require 
current work-arounds, this is not a positive trend. (See fig. 8.) 

Figure 8: Open NTCSS Priority 3 Change Proposals: 

[See PDF for image] 

[End of figure] 

With respect to OOMA specifically, the number of open priority 2 change 
proposals has risen slightly from 7 to 12. (See fig. 9.) Similarly, the 
number of open priority 3 change proposals has also increased somewhat 
from 78 to 97. (See fig. 10.) While the number of priority 2 change 
proposals is not large, the trend in these, as well as the trend in the 
more significant number of priority 3 change proposals, is not 
consistent with those of a maturing system. 

Figure 9: Open OOMA Priority 1 and 2 Change Proposals: 

[See PDF for image] 

[End of figure] 

Figure 10: Open OOMA Priority 3 Change Proposals: 

[See PDF for image] 

[End of figure] 

[End of section] 

Appendix III: Earned Value Management Assessment: 

Earned value management (EVM) guidance was developed by the American 
National Standards Institute/Electronic Industries Alliance.[Footnote 
62] This guidance identifies 32 criteria that reliable EVM systems 
should meet. The 32 criteria are organized into the following five 
categories: 

* Organization: Activities that define the scope of the effort and 
assign responsibilities for the work; 

* Planning and budgeting: Activities for planning, scheduling, 
budgeting, and authorizing the work; 

* Accounting: Activities to accumulate the costs of work and material 
needed to complete the work; 

* Analysis: Activities to compare budgeted, performed, and actual 
costs; analyze variances; and develop estimates of final costs; and: 

* Revisions and data maintenance: Activities to incorporate internal 
and external changes to the scheduled, budgeted, and authorized work. 

NTCSS central design agency (CDA) officials provided a self-assessment 
of their compliance with each of the criteria, reporting that they met 
15 of the 32 criteria (see table 12). Using the results of their self-
assessment to target our analysis, we then assessed those aspects of 
the EVM system the self-assessment reported as meeting the criteria, by 
comparing the documentation with relevant Department of Defense (DOD) 
guidance and best practices.[Footnote 63] Our assessment indicates that 
the NTCSS program satisfied two, and partially satisfied one, of the 32 
criteria (see table 12).[Footnote 64] 

Table 12: Navy Satisfaction of EVM Criteria: 

Organization: 

Criteria[A]: Define the authorized work elements for the program. A 
work breakdown structure, tailored for effective internal management 
control, is commonly used in this process; 
Definitions: The work breakdown structure is a direct representation of 
the work scope in the project, documenting the hierarchy and 
description of tasks to be performed and the relationship to the 
product deliverables. The work breakdown structure breaks down all 
authorized work scope into appropriate elements for planning, 
budgeting, scheduling, cost accounting, work authorization, measuring 
progress, and management control. It also ensures the statement of work 
is entirely captured and allows for integration of technical, schedule, 
and cost information; 
Self-assessment: Yes; 
GAO assessment: Yes; 
GAO analysis: The EVM reports for the OOMA software development project 
and the NTCSS hardware installation project had a work breakdown 
structure. 

Criteria[A]: Identify the program organizational breakdown structure, 
including the major subcontractors responsible for accomplishing the 
authorized work, and define the organizational elements in which work 
will be planned and controlled; 
Definitions: The organizational structure identifies the organization 
responsible for each segment of work, including subcontracted and intra-
organizational effort. In order to meet this guideline, objective 
evidence requires a work breakdown structure intersection with an 
organizational breakdown structure; 
Self-assessment: Yes; 
GAO assessment: No; 
GAO analysis: CDA officials have yet to provide documentation to 
demonstrate satisfaction of this criterion. Such documentation includes 
an organizational breakdown structure with detail regarding 
subcontractors. 

Criteria[A]: Provide for the integration of the company's planning, 
scheduling, budgeting, work authorization, and cost accumulation 
processes with each other and, as appropriate, the program work 
breakdown structure and the program organizational structure; 
Definitions: The integration of planning, scheduling, budgeting, work 
authorization, and cost accumulation management processes provides the 
capability for establishing the performance measurement baseline, 
identifying work progress, and collecting of actual costs for 
management analysis and corrective actions; 
Self-assessment: Yes; 
GAO assessment: No; 
GAO analysis: The CDA has yet to provide documentation to demonstrate 
satisfaction of this criterion. Such documentation includes copies of 
master, intermediate, and detailed schedules; operational schedules; 
control account plans; performance reports by work breakdown structure 
and organizational breakdown structure; responsibility assignment 
matrix; statement of work; work authorization documents; and work 
breakdown structure and organizational breakdown structure 
documentation. 

Criteria[A]: Identify the company organization or function responsible 
for controlling overhead (indirect costs); 
Definitions: Visibility into direct and indirect costs is essential for 
successful management of a project. Therefore, project managers should 
clearly identify managers who are responsible for controlling indirect 
costs, including overhead, burden, general and administrative costs, 
and who has authority to approve expenditure of resources. They should 
also document the process for management and control of indirect costs; 
Self-assessment: No; 
GAO assessment: No; 
GAO analysis: We did not analyze this criterion because it was self-
assessed by the CDA as not being met. 

Criteria[A]: Provide for integration of the program work breakdown 
structure and the program organizational structure in a manner that 
permits cost and schedule performance measurement by elements of either 
or both structures, as needed; 
Definitions: The integration of the work breakdown structure and 
organizational breakdown structure establishes where the performance 
measurement necessary for project management is performed. This 
intersection results in designation of a focal point for management 
control (the control account manager). It is also the initiation point 
for work authorization, performance management, and performance 
measurement. The control account manager identifies the plan for work 
task accomplishment, including defining the effort required, cost 
elements (labor, material, etc.), and the resources required to do the 
job; 
Self-assessment: No; 
GAO assessment: No; 
GAO analysis: We did not analyze this criterion because it was self-
assessed by the CDA as not being met. 

Planning and budgeting: 

Criteria[A]: Schedule the authorized work in a manner that describes 
the sequence of work and identifies significant task interdependencies 
required to meet the program requirements; 
Definitions: The scheduling of authorized work facilitates effective 
planning, reporting, and forecasting, which is critical to the success 
of all projects. An integrated network scheduling system has distinct 
tasks that can be summarized by work breakdown structure and 
organizational breakdown structure identifiers to track progress and 
measure performance; 
Self-assessment: Yes; 
GAO assessment: Yes; 
GAO analysis: Detailed schedule documents for both projects describe 
the sequence and interdependence of work relative to project 
requirements. 

Criteria[A]: Identify physical products, milestones, technical 
performance goals, or other indicators that will be used to measure 
progress; 
Definitions: Objective indicators enable measurement of work 
accomplished, thereby allowing accurate comparison to planned work. 
Meaningful performance metrics enable better management insight and 
decision making, allowing maximum time for management action to keep 
the project on plan; 
Self-assessment: Yes; 
GAO assessment: No; 
GAO analysis: The metrics in the NTCSS hardware installation project 
reports contained unexpectedly and unrealistically large improvements 
in performance that were not explained. In addition, the program office 
told us that the measurement data for the OOMA software project is 
distorted due to numerous baseline changes and requirements changes. 
Satisfying this criterion requires valid data. 

Criteria[A]: Establish and maintain a time-phased budget baseline, at 
the control account level, against which program performance can be 
measured. Budget for far-term efforts may be held in higher-level 
accounts until an appropriate time for allocation at the control 
account level. Initial budgets established for performance measurement 
will be based on either internal management goals or the external 
customer negotiated target cost, including estimates for authorized but 
undefinitized work. On government contracts, if an over-target baseline 
is used for performance measurement reporting purposes, prior 
notification must be provided to the customer; 
Definitions: The assignment of budgets to scheduled segments of work 
produces a plan against which actual performance can be compared. This 
is called the performance measurement baseline. The establishment, 
maintenance, and use of the performance measurement baseline are 
indispensable to effective program management; 
Self-assessment: No; 
GAO assessment: No; 
GAO analysis: We did not analyze this criterion because it was self-
assessed by the CDA as not being met. 

Criteria[A]: Establish budgets for authorized work with identification 
of significant cost elements (e.g., labor and material) as needed for 
internal management and for control of subcontractors; 
Definitions: An essential part of project planning and establishing a 
performance measurement baseline is the establishment of budgets for 
all work authorized. Identification of the budget cost elements 
documents the required resources and integrates the work scope with the 
performing organization; 
Self-assessment: No; 
GAO assessment: No; 
GAO analysis: We did not analyze this criterion because it was self-
assessed by the CDA as not being met. 

Criteria[A]: To the extent it is practical to identify the authorized 
work in discrete work packages, establish budgets for this work in 
terms of dollars, hours, or other measurable units. Where the entire 
control account is not subdivided into work packages, identify the far-
term effort in larger planning packages for budget and scheduling 
purposes; 
Definitions: The effort contained within a control account is 
distributed into either work packages or planning packages. Work 
packages are single tasks, assigned to a performing organization for 
completion, and should be natural subdivisions of control account 
effort resulting in a definable end product or event. Budgets 
established at the work package level provide the detail for effective 
execution of the baseline plan. This approach provides meaningful 
product or management-oriented events for performance measurement; 
Self-assessment: Yes; 
GAO assessment: No; 
GAO analysis: The CDA has yet to provide documentation to demonstrate 
satisfaction of this criterion. Such documentation includes control 
account plans divided into work and planning packages, or control 
account schedules and time-phased budgets. 

Criteria[A]: Provide that the sum of all work package budgets, plus 
planning package budgets within a control account, equals the control 
account budget; 
Definitions: The integrity of the performance measurement baseline is 
maintained when the budget of the control account equals the sum of its 
work and planning package budgets. This prevents duplicate recording of 
budgets; 
Self-assessment: No; 
GAO assessment: No; 
GAO analysis: We did not analyze this criterion because it was self-
assessed by the CDA as not being met. 

Criteria[A]: Identify and control the level of effort activity by time-
phased budgets established for this purpose. Only that effort that is 
unmeasurable or for which measurement is impractical may be classified 
as level of effort; 
Definitions: Meaningful events are critical for performance 
measurement. Measurement of level of effort activity provides no 
visibility into actual performance. Level of effort activity is defined 
as having no measurable output or product at the work package level 
and, therefore, must be limited to avoid distorting project performance 
data; 
Self-assessment: No; 
GAO assessment: No; 
GAO analysis: We did not analyze this criterion because it was self-
assessed by the CDA as not being met. 

Criteria[A]: Establish overhead budgets for each significant 
organizational component of the company for expenses that will become 
indirect costs. Reflect in the program budgets, at the appropriate 
level, the amounts in overhead accounts that are planned to be 
allocated to the program as indirect costs; 
Definitions: Indirect costs are for common activities that cannot be 
specifically identified with a particular project or activity and 
should typically be budgeted and controlled separately at the 
functional or organization manager level. It is important to have an 
indirect budgeting and forecasting process because indirect costs 
account for a major portion of the cost of any project. As such, the 
budgetary control and management of this category cannot be overlooked 
or minimized; 
Self-assessment: No; 
GAO assessment: No; 
GAO analysis: We did not analyze this criterion because it was self-
assessed by the CDA as not being met. 

Criteria[A]: Identify management reserves and undistributed budget; 
Definitions: Project managers need to realize the performance 
measurement baseline planning process contains risk and identify a 
management reserve contingency for unplanned activity within the 
project scope; 
Self-assessment: No; 
GAO assessment: No; 
GAO analysis: We did not analyze this criterion because it was self-
assessed by the CDA as not being met. 

Criteria[A]: Provide that the program target cost goal is reconciled 
with the sum of all internal program budgets and management reserves; 
Definitions: A project baseline that reflects the common agreement 
between the two parties provides a common reference point for progress 
assessment. It provides recognition of contractual requirements and 
precludes unauthorized changes to the performance measurement baseline; 
Self-assessment: No; 
GAO assessment: No; 
GAO analysis: We did not analyze this criterion because it was self-
assessed by the CDA as not being met. 

Accounting considerations: 

Criteria[A]: Record direct costs in a manner consistent with the 
budgets in a formal system controlled by the general books of account; 
Definitions: A project cost-charging structure established in the 
accounting system ensures that actual direct costs are accumulated and 
reported in a manner consistent with the way the work is planned and 
budgeted; 
Self-assessment: No; 
GAO assessment: No; 
GAO analysis: We did not analyze this criterion because it was self-
assessed by the CDA as not being met. 

Criteria[A]: When a work breakdown structure is used, summarize direct 
costs from control accounts into the work breakdown structure without 
allocation of a single control account to two or more work breakdown 
structure elements; 
Definitions: Actual costs need to be available at all levels of the 
work breakdown structure to support project management with performance 
measurement data. Cost collection accounts mapped to the work breakdown 
structure ensure performance measurement data integrity; 
Self-assessment: No; 
GAO assessment: No; 
GAO analysis: We did not analyze this criterion because it was self-
assessed by the CDA as not being met. 

Criteria[A]: Summarize direct costs from the control accounts into the 
contractor's organizational elements without allocation of a single 
control account to two or more organizational elements; 
Definitions: To ensure performance measurement data integrity, actual 
costs need to be available at all levels of the organizational 
breakdown structure; 
Self-assessment: No; 
GAO assessment: No; 
GAO analysis: We did not analyze this criterion because it was self-
assessed by the CDA as not being met. 

Criteria[A]: Record all indirect costs that will be allocated to the 
project; 
Definitions: All indirect costs should be recorded in the accounting 
system. Allocating indirect costs to the appropriate direct costs 
assures that all projects benefiting from indirect costs receive their 
fair share; 
Self-assessment: No; 
GAO assessment: No; 
GAO analysis: We did not analyze this criterion because it was self-
assessed by the CDA as not being met. 

Criteria[A]: Identify unit costs, equivalent unit costs, or lot costs 
when needed; 
Definitions: A manufacturing accounting system capable of isolating 
unit and lot costs in a production environment allows the flexibility 
to plan, measure performance, and forecast in a more efficient way when 
there are multiple projects in the same production line; 
Self-assessment: Yes; 
GAO assessment: No; 
GAO analysis: The CDA has not yet provided documentation to demonstrate 
satisfaction of this criterion. Such documentation includes a 
manufacturing resource planning project cost collection structure or an 
enterprise resource planning system that supports the identification of 
unit costs, equivalent unit costs, or lot costs when needed including 
differentiation of work in process. 

Criteria[A]: For EVM, the material accounting system will provide (1) 
accurate cost accumulation and assignment of costs to control accounts 
in a manner consistent with the budgets using recognized, acceptable, 
costing techniques; (2) cost performance measurement at the point in 
time most suitable for the category of material involved, but no 
earlier than the time of progress payments or actual receipt of 
material; and (3) full accountability of all material purchased for the 
program, including the residual inventory; 
Definitions: Material items consumed in the production of project 
deliverables are accounted for and progress is measured at the point 
most closely aligned to the actual consumption. Material accounting 
systems should adhere to these three characteristics: (1) the material 
accounting system provides full accountability and effective 
measurement of all material purchased; (2) material costs should be 
accurately charged to control accounts using recognized, acceptable 
costing techniques; and (3) when necessary, the use of estimated actual 
costs to ensure accurate performance measurement should be used; 
Self-assessment: No; 
GAO assessment: No; 
GAO analysis: We did not analyze this criterion because it was self-
assessed by the CDA as not being met. 

Analysis and management reports: 

Criteria[A]: At least on a monthly basis, generate the following 
information at the control account and other levels as necessary for 
management control using actual cost data from, or reconcilable with, 
the accounting system: (1) comparison of the amount of planned budget 
and the amount of budget earned for work accomplished (this comparison 
provides the schedule variance) and (2) comparison of the amount of the 
budget earned and the actual (applied where appropriate) direct costs 
for the same work (this comparison provides the cost variance); 
Definitions: Visibility into project performance helps the project 
manager focus resources on those areas in need of attention. Accurate 
and reliable EVM data supports management control needs by allowing the 
project manager to identify root causes for variances and establish 
actions to minimize impact at the control account level; 
Self-assessment: Yes; 
GAO assessment: No; 
GAO analysis: In order to produce reliable and accurate variance 
reports, many of the aforementioned criteria that our analysis showed 
that the CDA did not perform must be satisfied. Therefore, this 
criterion is not being satisfied. 

Criteria[A]: Identify, at least monthly, the significant differences 
between both planned and actual schedule performance and planned and 
actual cost performance and provide the reasons for the variances in 
the detail needed by program management; 
Definitions: The analysis of deviations from plan for both schedule and 
cost at least monthly provides management at all levels the ability to 
rapidly and effectively implement corrective actions with an 
understanding of the project risk and causes of risk; 
Self-assessment: Yes; 
GAO assessment: No; 
GAO analysis: The metrics in the NTCSS hardware installation project 
reports contained unexpectedly and unrealistically large improvements 
in performance that were not explained. In addition, the program office 
told us that the measurement data for the OOMA software project is 
distorted due to numerous baseline changes and requirements changes. 
Satisfying this criterion requires valid data. 

Criteria[A]: Identify budgeted and applied (or actual) indirect costs 
at the level and frequency needed by management for effective control, 
along with the reasons for any significant variances; 
Definitions: Ongoing indirect cost analysis provides visibility into 
potential indirect cost overruns and the opportunity to develop and 
implement management action plans to meet project objectives; 
Self-assessment: No; 
GAO assessment: No; 
GAO analysis: We did not analyze this criterion because it was self-
assessed by the CDA as not being met. 

Criteria[A]: Summarize the data elements and associated variances 
through the program organization and/or work breakdown structure to 
support management needs and any customer reporting specified in the 
project; 
Definitions: Variances provide an understanding of the conditions, 
allowing the project manager to properly allocate available resources 
to mitigate project risk. They also identify significant problem areas 
from all levels of the organization and project scope of work, derived 
from the same data sources. Thus, variances provide valuable management 
information; 
Self-assessment: No; 
GAO assessment: No; 
GAO analysis: We did not analyze this criterion because it was self-
assessed by the CDA as not being met. 

Criteria[A]: Implement managerial actions taken as the result of earned 
value information; 
Definitions: Earned value data must be utilized by all levels of 
management for effective project execution. Because of this, the data 
produced by the EVM system must be available to managers on a timely 
basis and must be of sufficient quality to ensure that effective 
management decisions can be made as a result of its analysis; 
Self-assessment: Yes; 
GAO assessment: No; 
GAO analysis: The metrics in the NTCSS hardware installation project 
reports contained unexpectedly and unrealistically large improvements 
in performance that were not explained. In addition, the program office 
told us that the measurement data for the OOMA software project is 
distorted due to numerous baseline changes and requirements changes. 
Satisfying this criterion requires valid data. 

Criteria[A]: Develop revised estimates of cost at completion based on 
performance to date, commitment values for material, and estimates of 
future conditions. Compare this information with the performance 
measurement baseline to identify variances at completion important to 
company management and any applicable customer reporting requirements, 
including statements of funding requirements; 
Definitions: Estimates at completion based on predictive performance 
measures increase the probability that the project can be executed 
within the reported estimates at completion. When estimates at 
completions are analyzed at least monthly and updated as required, the 
robustness of the financial reporting requirements is enhanced, thereby 
reducing the potential for surprises. Monthly estimates at completion 
reviews are essential for management decisions including the planning 
of project future funding requirements; 
Self-assessment: No; 
GAO assessment: No; 
GAO analysis: We did not analyze this criterion because it was self-
assessed by the CDA as not being met. 

Revisions and data maintenance: 

Criteria[A]: Incorporate authorized changes in a timely manner, 
recording the effects of such changes in budgets and schedules. In the 
directed effort prior to negotiation of a change, base such revisions 
on the amount estimated and budgeted to the program organizations; 
Definitions: The incorporation of authorized changes in a timely manner 
maintains the integrity of the performance measurement baseline and 
thus its effectiveness as a baseline against which to manage and 
control performance; 
Self-assessment: Yes; 
GAO assessment: No; 
GAO analysis: The CDA has yet to provide documentation to demonstrate 
satisfaction of this criterion. Such documentation includes change 
control logs and work authorization documents. 

Criteria[A]: Reconcile current budgets to prior budgets in terms of 
changes to the authorized work and internal replanning in the detail 
needed by management for effective control; 
Definitions: Budget changes should be controlled and understood in 
terms of scope, resources, and schedule, and that budgets should 
reflect current levels of authorized work. Furthermore, budget 
revisions should be traceable to authorized contractual targets and 
control account budgets; 
Self-assessment: Yes; 
GAO assessment: No; 
GAO analysis: The CDA has yet to provide documentation to demonstrate 
satisfaction of this criterion. Such documentation includes change 
documents or change control logs. 

Criteria[A]: Control retroactive changes to records pertaining to work 
performed that would change previously reported amounts for actual 
costs, earned value, or budgets. Adjustments should be made only for 
correction of errors, routine accounting adjustments, effects of 
customer or management directed changes, or to improve the baseline 
integrity and accuracy of performance measurement data; 
Definitions: Retroactive changes to the baseline may mask variance 
trends and prevent use of the performance data to project estimates of 
cost and schedule at completion. Retroactive budget adjustments may 
delay visibility of overall project variance from plan, thus reducing 
the alternatives available to managers for project redirection or 
termination; 
Self-assessment: Yes; 
GAO assessment: No; 
GAO analysis: The CDA has yet to provide documentation to demonstrate 
satisfaction of this criterion. Such documentation includes change 
control logs or approved retroactive change controls. 

Criteria[A]: Prevent revisions to the program budget except for 
authorized changes; 
Definitions: Changes made outside the authorized baseline control 
processes compromise the integrity of performance trend data and delay 
visibility into overall project variance from plan; 
Self-assessment: Yes; 
GAO assessment: No; 
GAO analysis: The CDA has yet to provide documentation to demonstrate 
satisfaction of this criterion. Such documentation includes change 
control logs, control accounts, and work package plans. 

Criteria[A]: Document changes to the performance measurement baseline; 
Definitions: By ensuring that budget and schedule revisions are 
documented and traceable, the integrity of the performance measurement 
baseline is maintained and can be verified. The performance measurement 
baseline should reflect the most current plan for accomplishing the 
effort. Authorized changes should be quickly recorded in the system and 
incorporated into all relevant planning. Planning and authorization 
documents must also be updated accordingly prior to commencement of new 
work; 
Self-assessment: Yes; 
GAO assessment: Partial; 
GAO analysis: We were provided documentation showing eight baseline 
changes for the NTCSS hardware installation project. However, the 
program office told us that the EVM data for the OOMA software project 
is distorted due to numerous baseline changes and requirements changes. 

Criteria[A]: Number satisfied; 
Self-assessment: 15; 
GAO assessment: 2. 

Criteria[A]: Number partially satisfied; 
Self-assessment: 0; 
GAO assessment: 1. 

Criteria[A]: Number not satisfied; 
Self-assessment: 17; 
GAO assessment: 29. 

Criteria[A]: Total; 
Self-assessment: 32; 
GAO assessment: 32. 

Sources: Navy CDA self-assessment and GAO analysis of Navy provided 
data. 

[A] Based on the National Defense Industrial Association Program 
Management Systems Committee Intent Guide (January 2005). 

[End of table] 

[End of section] 

Appendix IV: Comments from the Department of Defense: 

OFFICE OF THE ASSISTANT SECRETARY OF DEFENSE: 
NETWORKS AND INFORMATION INTEGRATION: 
6000 DEFENSE PENTAGON: 
WASHINGTON, DC 20301-6000: 

NOV 23 2005: 

Mr. Randolph C. Hite: 
Director, Information Technology Architecture and Systems Issues: 
U.S. General Accounting Office: 
441 G Street, N.W.: 
Washington, D.C. 20548: 

Dear Mr. Hite: 

This is the Department of Defense (DoD) response to the GAO draft 
report (06-215), "DoD SYSTEMS MODERNIZATION: PLANNED INVESTMENT IN THE 
NAVAL TACTICAL COMMAND SUPPORT SYSTEM NEEDS TO BE REASSESSED," dated 
November 3, 2005 (GAO Code 310287). 

We appreciate the opportunity to comment on the draft report and the 
time your staff afforded us during their preparation of the report. 

As acquisition milestone decision authority for NTCSS has been 
delegated to the Navy since 1999, the Navy provided significant 
analysis and comment regarding the report. The Navy recognizes that 
some of the report's findings are valid but believes that the overall 
findings significantly understate and misrepresent the level of 
discipline and conformance with applicable guidance and direction 
achieved by the NTCSS Program. They also believe that although NTCSS 
was defined and implemented without a complete and formal enterprise 
architecture, it has proven to be the right solution to meet Navy's 
strategic business and technological needs; and that sound management 
practices are in place and improving. The Navy Acquisition Executive 
particularly emphasizes that no manpower or resource shortfalls have 
hampered oversight or execution of the NTCSS program. 

Rather than providing detailed comments on the report's findings, we 
have chosen to focus our response on the report's recommendations. Our 
reply to each recommendation is enclosed. Our point of contact is Lt 
Col Alex Holder at 703-602-2720, ext 123. 

Sincerely, 

Signed by: 

John R. Landon: 
Deputy to the ASD(NII) (C3ISR & IT Acquisition): 

Enclosure: As stated: 

DoD Comments to GAO draft report (06-215), "DOD SYSTEMS MODERNIZATION: 
PLANNED INVESTMENT IN THE NAVAL TACTICAL COMMAND SUPPORT SYSTEM NEEDS 
TO BE REASSESSED," dated November 3, 2005 (GAO Code 310287): 

RECOMMENDATION 1: The GAO recommended that the Secretary of Defense 
direct the Secretary of the Navy to determine if continued investment 
in Naval Tactical Command Support System (NTCSS) as planned represents 
a prudent use of the department's limited resources. To accomplish 
this, the Secretary of the Navy should direct the program office to 
collaborate with the Office of the Assistant Secretary of Defense for 
Networks and Information Integration/Chief Information Officer, the 
Office of Program Analysis and Evaluation, and the Naval Cost Analysis 
Division to prepare a reliable economic analysis (EA) that encompasses 
all viable alternatives, including the Navy's recent enterprise 
resource planning (ERP) program. (p. 60/GAO Draft Report): 

DoD RESPONSE: Partially concur. 

As planned for some time, the NTCSS program office will continue to 
collaborate with the Naval Cost Analysis Division to conduct a reliable 
EA to support the Optimized Organizational Maintenance Activity (OOMA) 
fielding decision. However, at this late phase of this very mature 
program, when the final application is about to be fielded, we do not 
see merit in conducting a formal EA that would address all viable 
alternatives, or in seeking Office of Secretary of Defense (OSD) Office 
of Program Analysis and Evaluation (PA&E) review of the EA. 

However, we will direct the Navy to (1) conduct, in coordination with 
OSD PA&E, an analysis to determine the relationship of NTCSS, the Navy 
ERP Program and other programs that may provide similar functionality 
and (2) brief the results of that analysis to the Networks and 
Information Integration (NII) Overarching Integrated Product Team 
(OIPT) within 120 days. 

The NTCSS Program Office has conducted three EAs to support the 
program's milestone decisions. These EAs were conducted in accordance 
with the DoD 5000 series and OMB Circulars A-94 and A-11. 

The 1997 and 1999 EAs focused on NTCSS Optimization (less Optimized 
Organizational Maintenance Activity (OOMA)). 

The 2004 EA focused on OOMA, the final NTCSS application. 

* These EAs were independently reviewed by the Office of the Secretary 
of Defense (Program Analysis and Evaluation) and the (then) Navy Center 
for Cost Analysis, both of whom found that the EAs supported the 
milestone decision being requested. 

The OOMA 2004 EA was conducted in March 2004 and concurred upon by the 
Navy Cost Analysis Division (NCAD) in support of the upcoming Fielding 
Decision for the Optimized Organizational Maintenance Activity (OOMA) 
application. This OOMA EA is being updated in collaboration with NCAD 
to reflect Program plans established pursuant to upcoming OOMA version 
831-01.05.00 Follow-on Operational Testing and Fielding Decision. 

The NTCSS EAs did not address the Navy's Enterprise Resource Planning 
(ERP) Program, and the Navy ERP Program did not exist when the original 
NTCSS AoA was conducted. 

In addition to EAs supporting milestone decision points, NTCSS has 
historically used Business Case Analyses (BCA) to evaluate investment 
decisions, including two conducted in 2001 and one in 2004. The 2004 
Fielding Alternatives BCA was conducted by the Program Office at the 
request of Chief of Naval Operations for Material Readiness and 
Logistics Operations (OPNAV N4) in order to assess POM06 funding 
decisions and plan a way-ahead for modernizing naval afloat logistics 
systems. The 2004 Fielding Alternatives BCA was conducted largely 
consistent with the procedures outlined in OMB Circulars A-94 and A-11 
and the Federal Chief Information Officers Council, Best Practices 
Committee report, "Value Measuring Methodology (VMM), How to Guide." 

RECOMMENDATION 2: The GAO recommended that the Secretary of Defense 
direct the Secretary of the Navy to determine if continued investment 
in NTCSS as planned represents a prudent use of the department's 
limited resources. To accomplish this, the Secretary of the Navy should 
direct the program office to ensure that development of this economic 
analysis (1) complies with cost estimating best practices, including 
recognition of costs to resolve open trouble reports and change 
proposals, and relevant OMB cost benefit guidance and (2) incorporates 
available data on whether deployed NTCSS capabilities are actually 
producing benefits. (p. 60/GAO Draft Report): 

DoD RESPONSE: Concur. 

As stated above, the 2004 OOMA EA is being updated in collaboration 
with NCAD to reflect program plans established pursuant to upcoming 
Follow-on Operational Testing of OOMA version 831-01.05.00 in support 
of the OOMA Fielding Decision. As was the case with the 1997 and 1999 
NTCSS Optimized EAs, this OOMA EA is being compiled in compliance with 
cost and benefit guidance and best practices of the DoD 5000 series, 
OMB Circular A-94 and OMB Circular A-11 as recommended in the GAO 
report. 

Software Trouble Reports and Change Proposals are prioritized by the 
NTCSS Requirements Integrated Product Team in accordance with Program 
Office policy and available funding, and, if approved for 
implementation, will be slated for subsequent maintenance updates to 
the application baseline. 

In accordance with DoDI 5000.2 and current DoD CIO guidance on Clinger-
Cohen Act (CCA) Compliance Certification, and to demonstrate the 
benefits realized by NTCSS, outcome measures of effectiveness (MOEs) 
and a post implementation review (PIR) plan will be developed for the 
deployed NTCSS capabilities and for the OOMA Fielding Decision. The 
OOMA PIR will be conducted 6 to 12 months after the OOMA Fielding 
Decision. Results of these reviews will be reported to the appropriate 
Navy and DoD stakeholders. 

RECOMMENDATION 3: The GAO recommended that the Secretary of Defense 
direct the Secretary of the Navy to determine if continued investment 
in NTCSS as planned represents a prudent use of the department's 
limited resources. To accomplish this, the Secretary of the Navy should 
direct the program office to collaborate with the Under Secretary of 
Defense for Acquisition, Technology, and Logistics and the Under 
Secretary of Defense (Comptroller) to ensure that NTCSS is adequately 
aligned with evolving DoD and Navy enterprise Architecture. (p. 60/GAO 
Draft Report): 

DoD RESPONSE: Concur. 

DoD concurs that the NTCSS program should be adequately aligned with 
the evolving DoD and Navy Enterprise Architecture. The NTCSS program 
was reviewed in FY05 and FY06 as part of the Secretary of Defense 
Business Management Modernization Program (BMMP). As part of these 
annual reviews, the program was evaluated against and found to be 
aligned with the evolving DoD and Navy Enterprise Architecture. 
Ultimately, these reviews led to Under Secretary of Defense 
(Comptroller) certification and (in FY06) Defense Business Systems 
Management Committee (DBSMC) approval for system modernization 
investment. On a continuing basis, the NTCSS Program Office works to 
ensure that it is properly aligned to the currently defined DoD and 
Navy business architecture. 

As part of the BMMP review in FY05, the Program Office worked in 
conjunction with OPNAV N4 and the Office of the Deputy Undersecretary 
of Defense Logistics and Materiel Readiness within the Office of the 
Under Secretary of Defense for Acquisition, Technology, and Logistics 
(USD(AT&L)). In FY06, the Program Office worked closely with the Navy 
Investment Review Board (IRB) that was established as part of the BMW 
process. To ensure architecture alignment, the NTCSS program will 
collaborate with the new Business Transformation Agency (BTA). The BTA 
reports to the DBSMC through the USD(AT&L) and manages the DoD IRB 
process. 

RECOMMENDATION 4: The GAO recommended that the Secretary of Defense 
direct the Secretary of the Navy to present the result of the analysis 
outlined in Recommendation 1 to the Deputy Secretary of Defense, or his 
designee, and seek a departmental decision on how to best proceed with 
the program. Until this is done, the GAO recommended that the Secretary 
of Defense direct the Secretary of the Navy to halt further deployment 
of NTCSS and to limit future investment in already deployed 
applications to essential operations and maintenance activities and 
only developmental activities deemed essential to national security 
needs. (p. 61/GAO Draft Report): 

DoD RESPONSE: Partially concur. 

We do not agree that further deployment of NTCSS should be limited at 
this time. However, we will direct the Navy to work with OSD(PA&E) to 
conduct the analysis described in our response to Recommendation 1 
above, and the Navy will present the results of their analysis to the 
NII OIPT within 120 days. Appropriate direction will be given to the 
program, based on the results of that analysis. 

RECOMMENDATION 5: The GAO recommended that if based on reliable data, a 
decision is made to continue the NTCSS program, the Secretary of 
Defense direct the Secretary of the Navy to ensure that the NTCSS 
program implements effective program management activities, including 
earned value management, requirements development and management, and 
test management. (p. 61/GAO Draft Report): 

DoD RESPONSE: Partially Concur: 

Agree that the NTCSS program should implement effective program 
management activities but do not agree that the Secretary of Defense 
needs to give direction to the Secretary of the Navy regarding this 
issue. 

The Navy's position is as follows: 

The NTCSS Program has implemented and is continuously improving program 
management activities, including earned value management, requirements 
development and management, and test management in accordance with 
applicable directives and industry best practices. 

The Program Office has instituted a tailored approach to formal 
reporting and extends performance measurement principles to the Central 
Design Activity (CDA) even though the projects fall below the dollar 
threshold requiring use of an EVMS and despite the absence of formal 
contract deliverables. Projects selected for earned value application 
are based on risk level, resource limitations, and the level of 
management oversight required. The value of the individual projects 
contained in NTCSS did not warrant the CDA having a certified EVMS but 
adherence to the 32 criteria set forth in ANSI/EIA standards are 
actively applied. The CDA has set internal definitions on which 
project(s) warrant earned value reporting and these are specified in 
the CDA Software Measurement Plan. 

* To further ensure that the program is using the best program 
management practices, the Navy Acquisition Executive will ask the Navy 
Inspector General to conduct a study to see where improvement could be 
made in NTCSS program management activities. 

RECOMMENDATION 6: The GAO recommended that if based on reliable data, a 
decision is made to continue the NTCSS program, the Secretary of 
Defense direct the Secretary of the Navy to [ensure that] key 
stakeholders, such as the central design agency and the developmental 
testing organization, have the people, processes, and tools to 
effectively execute their respective roles and responsibilities. (p. 
61/GAO Draft Report): 

DoD RESPONSE: Partially Concur. 

We concur with the GAO that all acquisition programs should have the 
necessary people, processes and tools to effectively execute their 
respective roles and responsibilities. It is the Navy's position that 
key stakeholders of the NTCSS program do, in fact, have the people, 
processes and tools to effectively execute their respective roles and 
responsibilities. For example, SPAWARSYSCEN Norfolk, the Central Design 
Activity, demonstrated their competency and capability and was 
certified CMM-SW Maturity Level 3. 

A review of this area will be part of the NII OIPT meeting discussed 
under Recommendation 1. A determination of whether further action is 
needed will be made by the OIPT at that time. 

RECOMMENDATION 7: The GAO recommended that the Secretary of Defense 
reestablish the Office of the Assistant Secretary of Defense for 
Networks and Information Integration/Chief Information Officer as the 
milestone decision authority, and direct the Secretary of the Navy to 
take steps to ensure that Navy oversight entities fulfill their roles 
and responsibilities on NTCSS, including ensuring that reliable program 
reporting occurs and is acted upon. (p. 62/GAO Draft Report): 

DoD RESPONSE: Partially Concur: 

We believe it is premature to reestablish the ASD(NII)/DoD CIO as the 
milestone decision authority for NTCSS, given the fact that over 95% of 
the development work is complete. However, we will revisit that 
decision based on the results of the analysis discussed in our replies 
to Recommendations 1, 4 and 6 above. 

The NII OIPT Leader and the Navy Acquisition Executive will ensure that 
Navy oversight entities fulfill their roles and responsibilities on 
NTCSS, including ensuring that reliable program reporting occurs and is 
acted upon. Currently, NTCSS is overseen by: 

* PEO (C4I and Space); 

* SPAWAR Comptroller and NCAD; 

* Domain Functional Managers (FAM process); 

* OPNAV N4; 

* NTCSS Executive Steering Committee; 

* DASN (C4I and Space); 

* NII OIPT Leader (through the review of quarterly Defense Acquisition 
Executive Summaries): 

NTCSS will continue to comply with all statutory and regulatory 
reporting requirements, including: 

* Navy CIO and DoD CIO Clinger-Cohen Act certification for the OOMA 
full-rate production decision; 

* All reports and documents required for each Milestone Decision. 

The ASN (RDA) DASHBOARD: 

The PEO (C4I and Space) Acquisition Management Office Database: 

The following are GAO's comments on the Department of Defense's letter 
dated November 23, 2005. 

GAO Comments: 

1. See the Agency Comments and Our Evaluation section of this report. 

2. We disagree. Our report contains numerous instances where the Navy 
did not comply with either DOD acquisition policies and guidelines or 
industry best practices, in the areas of (1) economic justification; 
(2) architectural alignment; (3) project management, including progress 
measurement and reporting, funding disclosure, and oversight 
activities; and (4) system development, including requirements 
management and testing. Moreover, the Navy has not provided any 
evidence to demonstrate that our report is incorrect with respect to 
the level of program discipline and conformance with applicable policy 
and guidance in the areas that we reviewed. 

3. We disagree. Knowing that NTCSS is the right solution to meet the 
Navy's strategic business and technological needs would require that a 
frame of reference articulating these needs be available as a point of 
comparison. Such a frame of reference is an enterprise architecture. 
However, the Navy stated the system was defined and implemented without 
a complete and formal enterprise architecture. Our experience with 
federal agencies has shown that investing in an information technology 
solution without defining the solution in the context of an 
architecture often results in systems that are duplicative, not well 
integrated, and unnecessarily costly to maintain and interface. In 
addition, in February 2005, key program stakeholders and 
representatives from user organizations questioned whether NTCSS as 
defined was the right solution to cost effectively meet users' needs. 
At that time, program officials stated their intent to develop a new 
economic analysis to gather the information needed to determine whether 
to continue investing in NTCSS. In November 2005, program officials 
told us that they no longer planned to develop this economic analysis. 
Without a well-defined architecture and a reliable economic analysis, 
the Navy cannot be sure that NTCSS is the right solution. 

4. See comment 2. 

5. We acknowledge DOD's comment but would note that it is contrary to 
statements made to us during the audit. For example, officials with the 
milestone decision authority stated that, due to staffing reductions, 
the office was unable to fully perform oversight activities and has had 
to delegate completion of these activities. Also, Naval Cost Analysis 
Division officials stated that they only review cost estimates that are 
prepared for milestone reviews because staffing limitations do not 
permit them to review all cost estimates. Further, Navy officials 
stated that the central design agency was unable to effectively execute 
testing activities because it did not have a development testing lab. 

6. We disagree with this approach because its scope is narrower than 
our recommendation. Specifically, we recommended that the Navy develop 
a reliable economic analysis of the NTCSS program that includes all 
viable alternatives, including the Navy's Enterprise Resource Planning 
program. DOD acquisition policy and guidance provide detailed 
instructions on how economic analyses should be performed to obtain 
information that is critical for decisions regarding investments of 
scarce resources. Without such information, Navy risks that its 
continued investment in the system may not be justified. 

7. We disagree. With respect to the statement that NTCSS is a "very 
mature program," NTCSS has been under development for about 10 years at 
a cost of about $1.1 billion, and the Navy plans to spend an additional 
$348 million between fiscal years 2006 and 2009. Further, as appendix 
II of our report shows, there are hundreds of open trouble reports and 
change proposals that need to be addressed before the system can 
deliver promised or expected capabilities. In addition, should the OOMA 
application pass operational testing and be fielded, there are over 200 
sites where the necessary hardware must be installed and related 
training must occur. These two efforts will require a significant 
investment of time and resources, and it is therefore critical that the 
Navy ensure that NTCSS is the proper system before investing additional 
funds. With respect to the statement that "the final application is 
about to fielded," there is no evidence to support this. Since its 
originally planned fielding date of 2001, OOMA has failed operational 
testing twice, and the application is still under development. 
Therefore, it is premature to assert that the application will soon 
pass developmental and follow-on operational testing. 

8. See comment 6. Further, we disagree with the proposal to limit key 
stakeholders' involvement in developing the economic justification to 
"coordinating" and "briefing." These stakeholders have specific 
expertise and roles relative to economically justifying system 
investments that should be exploited. Until it conducts a complete and 
disciplined analysis of the entire NTCSS program (reviewed and approved 
by the Office of Program Analysis and Evaluation and the Naval Cost 
Analysis Division) and provides this analysis to all key stakeholders, 
the Navy's investment decisions will continue to be made without 
complete and reliable data. 

9. We disagree. As discussed in our report, the 2004 economic analysis 
did not adhere to five of eight criteria elements contained in the 
Office of Management and Budget Circulars A-94 and A-11. 

10. We disagree. The 2004 economic analysis that the Navy provided us 
focused on three fielding alternatives for the NTCSS program, not just 
the OOMA application. The Navy did not provide a 2004 economic analysis 
for just OOMA as the final NTCSS application. 

11. We disagree. As stated in our report, officials from the Office of 
Program Analysis and Evaluation and the Naval Cost Analysis Division 
told us that they did not review the 2004 NTCSS economic analysis. 

12. See comment 10. 

13. We agree that the Navy ERP program did not exist when the original 
NTCSS analysis of alternatives was conducted. However, the Navy ERP 
program was initiated in 1998 and therefore did exist when the Navy 
conducted subsequent analysis of alternatives. 

14. See comment 9. 

15. We do not question whether these annual reviews occurred and what 
resulted from them. However, the point in our report is that NTCSS has 
not been defined and developed in the context of a DOD or Navy 
enterprise architecture because a well-defined version of either has 
not existed to guide and constrain the program. As a result, meaningful 
analysis showing how NTCSS aligns to evolving DOD and Navy architecture 
efforts could not be produced. This means that the Navy does not have a 
sufficient basis for knowing if NTCSS, as defined, properly fits within 
the context of future DOD and Navy business operational and 
technological environments. 

16. We disagree. Our recommendation to limit further deployment of 
NTCSS is a way of ensuring that the Navy takes a "strategic pause" 
while it takes the time to ensure that decisions regarding future 
investment are made using reliable information, which our report shows 
has not historically been the case. As long as the Navy is not 
appropriately limiting work on NTCSS, it is continuing to invest 
resources without having justified doing so. 

17. See comment 6. 

18. See comment 2. 

19. We disagree. As we state in our report, neither the decomposition 
of the program into small, fiscal year-based projects nor the absence 
of a contractual relationship is a valid reason for not effectively 
implementing earned value management. Without reliable, timely, and 
auditable earned value management data, the program office cannot 
adequately manage technical, cost, and schedule risks and problems. 

20. We disagree. The Navy's own self-assessment of compliance with the 
32 criteria, detailed in appendix III of our report, showed that 17 of 
these criteria were not being satisfied. Further, our assessment showed 
that the Navy did not satisfy 29 of the 32 criteria, and program 
officials did not provide any evidence to refute the results of our 
assessment. 

21. The Navy did not provide us with a copy of the CDA Software 
Measurement Plan. 

22. See comment 5. Further, the Navy's position that "key stakeholders 
of the NTCSS program do, in fact, have the people, processes and tools 
to effectively execute their respective roles and responsibilities," is 
not consistent with its comment that this area will be part of a 
planned review. 

23. We disagree. Although the Navy states that the program is 95 
percent complete, it still plans to spend $348 million over the next 
three fiscal years, which is approximately 32 percent of what has been 
spent on the program to date. In addition, because the Navy lacks 
disciplined acquisition management practices, as discussed in our 
report, including earned value management, we question how it is able 
to reliably determine what percentage of the work has been completed 
and the percentage that remains to be done. As stated in our report, 
the current milestone decision authority has allowed the program to 
proceed while a major application repeatedly failed operational 
testing, and another application was cancelled. In addition, the Navy 
stated its intent to revisit the need to change milestone decision 
authority. 

[End of section] 

Appendix V: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Randolph C. Hite (202) 512-3439 or [Hyperlink, hiter@gao.gov]. 

Staff Acknowledgments: 

In addition to the contact named above, Cynthia Jackson, Assistant 
Director; Harold J. Brumm; Calvin L. H. Chang; Jennifer K. Echard; 
Joanne Fiorino; Neelaxi Lakhmani; Freda Paintsil; Jamelyn Payan; Karen 
Richey; Dr. Karl Seifert; Andrea Smith; and Dr. Rona B. Stillman made 
key contributions to this report. 

(310287): 

FOOTNOTES 

[1] DOD, Department of Defense Directive Number 5000.1, The Defense 
Acquisition System (May 12, 2003); Department of Defense Instruction 
Number 5000.2, Operation of the Defense Acquisition System (May 12, 
2003); Interim Defense Acquisition Guidebook (Oct. 30, 2002). 

[2] GAO, DOD Business Systems Modernization: Long-standing Weaknesses 
in Enterprise Architecture Development Need to Be Addressed, GAO-05-702 
(Washington, D.C. July 22, 2005). 

[3] We did not independently validate these data. 

[4] GAO, High-Risk Series: An Update, GAO-05-207 (Washington, D.C. 
January 2005). 

[5] GAO, ADP Procurement: Prompt Navy Action Can Reduce Risks to SNAP 
III Implementation, GAO/IMTEC-92-69 (Washington, D.C. Sept. 29, 1992). 

[6] Force-level ships include large ships, such as aircraft carriers 
and submarine tenders. Unit-level ships include command ships, hospital 
ships, other auxiliary and support ships, and submarines. Aviation 
squadrons are groups of planes that are always based on a specific 
aircraft carrier. Naval air stations and Marine aviation logistics 
squadrons are groups of planes that are land-based. The Naval air 
stations support land-based planes that are not deployed to ships. The 
Marine aviation logistics squadrons can be deployed on an aircraft 
carrier for a specific mission and, when the mission is completed, 
these planes return to their land base. 

[7] According to program officials, in addition to development and 
support of NTCSS Optimized applications, this amount includes legacy 
application support, shore-based legacy application procurements and 
installations, and Space and Naval Warfare Systems Command civilian 
salaries. 

[8] DOD, Blueprint for Establishing Risk-based Governance of IT 
Investments in a Net-centric Department of Defense (Apr. 13, 2005). 

[9] Net-centricity is a robust, globally interconnected network 
environment (including infrastructure, systems, processes, and people) 
in which data is shared in real time and seamlessly among users, 
applications and platforms. Net-centricity enables transformation by 
allowing applications to share data and services more effectively and 
flexibly, thereby allowing more agile, effective business practices to 
be used at reduced cost. 

[10] GAO, Information Technology: DOD's Acquisition Policies and 
Guidance Need to Incorporate Additional Best Practices and Controls, 
GAO-04-722 (Washington, D.C. July 30, 2004). 

[11] DOD, Defense Acquisition Guidebook, Version 1.0 (Oct. 17, 2004). 

[12] DOD, Defense Acquisition Guidebook, Version 1.0 (Oct. 17, 2004). 

[13] Carnegie Mellon University's SEI is a government-funded research 
organization that is widely considered an authority on software 
implementation. The checklist used is CMU/SEI-95-SR-004, A Manager's 
Checklist for Validating Software Cost and Schedule Estimates, January 
1995. SEI developed these checklists to help evaluate software costs 
and schedule. However, SEI states that these checklists are equally 
applicable to hardware and systems engineering projects. 

[14] Office of Management and Budget, Circular No. A-94: Guidelines and 
Discount Rates for Benefit-Cost Analysis of Federal Programs (Oct. 29, 
1992); and Circular No. A-11: Planning, Budgeting, Acquisition and 
Management of Capital Assets (June 21, 2005). 

[15] DOD, Defense Acquisition Guidebook, Version 1.0 (Oct. 17, 2004). 

[16] Clinger-Cohen Act of 1996, 40 U.S.C. sections 11101-11704, and 
Office of Management and Budget (OMB) Circular No. A-130, Management of 
Federal Information Resources (Nov. 30, 2000). 

[17] DOD, Defense Acquisition Guidebook, Version 1.0 (Oct. 17, 2004). 

[18] The Office of the Chief of Naval Operations for Material Readiness 
and Logistics Operations serves as the program and resource sponsor for 
the NTCSS program in order to balance user requirements with available 
resources. See table 7: NTCSS Management and Stakeholder Roles and 
Responsibilities. 

[19] ERP is an automated system using commercial off-the-shelf software 
consisting of multiple, integrated functional modules that perform a 
variety of business-related tasks such as payroll, general ledger 
accounting, and supply chain management. In August 2002, the Assistant 
Secretary of the Navy for Research, Development, and Acquisition 
established a Navy-wide ERP program to converge four ERP pilot programs 
that had been ongoing since 1998. 

[20] DOD, Department of Defense Directive Number 5000.1, The Defense 
Acquisition System (May 12, 2003) and Department of Defense 
Architecture Framework, Version 1.0, Volume 1 (February 2004). 

[21] See, for example, Clinger-Cohen Act of 1996, 40 U.S.C. §§ 11312 
and 11315(b)(2); E-Government Act of 2002, Pub. L. No. 107-347 (Dec. 
17, 2002); GAO, Information Technology: A Framework for Assessing and 
Improving Enterprise Architecture Management (Version 1.1), GAO-03-584G 
(Washington, D.C. April 2003); Chief Information Officer Council, A 
Practical Guide to Federal Enterprise Architecture, Version 1.0 
(February 2001); and Institute of Electrical and Electronics Engineers, 
Standard for Recommended Practice for Architectural Description of 
Software-Intensive Systems 1471-2000 (Sept. 21, 2000). 

[22] See, for example, GAO, Homeland Security: Efforts Under Way to 
Develop Enterprise Architecture, but Much Work Remains, GAO-04-777 
(Washington, D.C. Aug. 6, 2004); DOD Business Systems Modernization: 
Limited Progress in Development of Business Enterprise Architecture and 
Oversight of Information Technology Investments, GAO-04-731R 
(Washington, D.C. May 17, 2004); Information Technology: Architecture 
Needed to Guide NASA's Financial Management Modernization, GAO-04-43 
(Washington, D.C. Nov. 21, 2003); DOD Business Systems Modernization: 
Important Progress Made to Develop Business Enterprise Architecture, 
but Much Work Remains, GAO-03-1018 (Washington, D.C. Sept. 19, 2003); 
Business Systems Modernization: Summary of GAO's Assessment of the 
Department of Defense's Initial Business Enterprise Architecture, GAO-
03-877R (Washington, D.C. July 7, 2003); Information Technology: DLA 
Should Strengthen Business Systems Modernization Architecture and 
Investment Activities, GAO-01-631 (Washington, D.C. June 29, 2001); and 
Information Technology: INS Needs to Better Manage the Development of 
Its Enterprise Architecture, GAO/AIMD-00-212 (Washington, D.C. Aug. 1, 
2000). 

[23] GAO, Information Technology: Architecture Needed to Guide 
Modernization of DOD's Financial Operations, GAO-01-525 (Washington, 
D.C. May 17, 2001). 

[24] GAO, DOD Business Systems Modernization: Improvements to 
Enterprise Architecture Development and Implementation Efforts Needed, 
GAO-03-458 (Washington, D.C. Feb. 28, 2003); Information Technology: 
Observations on Department of Defense's Draft Enterprise Architecture, 
GAO-03-571R (Washington, D.C. Mar. 28, 2003); GAO-03-877R; GAO-03-1018; 
GAO-04-731R. 

[25] GAO-05-702. 

[26] GAO, DOD Business Systems Modernization: Billions Being Invested 
without Adequate Oversight, GAO-05-381 (Washington, D.C. Apr. 29, 
2005). 

[27] The Under Secretary of Defense for Acquisition, Technology, and 
Logistics and the Under Secretary of Defense (Comptroller) are 
responsible for overseeing the development of DOD's business enterprise 
architecture. 

[28] Ronald W. Reagan National Defense Authorization Act for Fiscal 
Year 2005, Pub. L. No. 108-375, § 332, 118 Stat. 1811, 1851-1856, (Oct. 
28, 2004) (codified in part at 10 U.S.C. § 2222). 

[29] GAO, Information Technology: Enterprise Architecture Use across 
the Federal Government Can Be Improved, GAO-02-6 (Washington, D.C. Feb. 
19, 2002); and Information Technology: Leadership Remains Key to 
Agencies Making Progress on Enterprise Architecture Efforts, GAO-04-40 
(Washington, D.C. Nov. 17, 2003). 

[30] GAO, DOD Business Systems Modernization: Navy ERP Adherence to 
Best Practices Critical to Avoid Past Failures, GAO-05-858 (Washington, 
D.C. Sept. 29, 2005). 

[31] GAO-05-858. 

[32] DOD, Department of Defense Instruction Number 5000.2, Operation of 
the Defense Acquisition System (May 12, 2003) and Defense Acquisition 
Guidebook, Version 1.0 (Oct. 17, 2004). 

[33] American National Standards Institute (ANSI)/Electronic Industries 
Alliance (EIA) EVM System Standard (ANSI/EIA-748-98), Chapter 2 (May 
19, 1998). 

[34] GAO, Missile Defense: Additional Knowledge Needed in Developing 
System for Intercepting Long-Range Missiles, GAO-03-600 (Washington, 
D.C. Aug. 21, 2003). 

[35] Indirect costs are also known as "burden" or overhead costs. All 
organizations have indirect costs, which may include, for example, the 
cost of an office building, its depreciation, fringe benefits, office 
furniture, supplies, computers, vacations, sick pay, and telephone 
costs. By omitting indirect costs, NTCSS is understating the true 
program costs. 

[36] Before April 2005, DOD policy required the use of EVM and the use 
of integrated baseline reviews for programs with (1) contracts or 
agreements for research and development or test and evaluations over 
$73 million or (2) procurement or operations and maintenance contracts 
over $315 million (both in fiscal year 2000 constant dollars). Since 
April 2005, DOD now requires the use of EVM for all cost or incentive 
contracts over $20 million. 

[37] DOD, Department of Defense Directive Number 5000.1, The Defense 
Acquisition System (Oct. 23, 2000) (current version dated May 12, 
2003); DOD Instruction Number 5000.2, Operation of the Defense 
Acquisition System (Apr. 5, 2002) (current version dated May 12, 2003); 
and DOD 5000.2-R, Mandatory Procedures for Major Defense Acquisition 
Programs (MDAPS) and Major Automated Information System (MAIS) 
Acquisition Programs (Apr. 5, 2002) (canceled, replaced by DOD Defense 
Acquisition Guidebook [Oct. 17, 2004]). 

[38] A program breach occurs when the program office has reason to 
believe that a cost, schedule or performance goal, as documented in an 
Acquisition Program Baseline, will not be reached. 

[39] NTCSS participated in the formal RIT pilot program between October 
2002 and December 2003, when the pilot ended. However the program 
office, with agreement from the milestone decision authority, continued 
to use the RIT pilot procedures until December 2004. 

[40] DOD Financial Management Regulation 7000.14-R, (FMR) Vol. 2A, 
Chap. 1, section 010213 (June 2004). 

[41] Navy Financial Management Policy Manual, NAVSO P-1000, section 
075371.2.a (December 2002). 

[42] In some circumstances, software modernization costs under $250,000 
may be considered "expenses," and funded with "Operation and 
Maintenance" appropriations. (DOD Financial Management Regulation 
7000.14-R, (FMR) Vol. 2A, Chap. 1, section 010212 [June 2004]). The 
threshold in the current Navy guidance is $100,000. (Navy Financial 
Management Policy Manual, NAVSO P-1000, section 075371. [December 
2002]). 

[43] DOD, Department of Defense Directive Number 5000.1, The Defense 
Acquisition System (May 12, 2003). 

[44] There have been three milestone decision authorities for NTCSS 
since the program was begun. Initially, the milestone decision 
authority was in the Office of the Assistant Secretary of Defense for 
Networks and Information Integration/Chief Information Officer. In July 
1999, this authority was delegated to the Assistant Secretary of the 
Navy for Research, Development and Acquisition, who then delegated 
oversight authority to Deputy Assistant Secretary of the Navy for 
Command, Control, Communication, Computers and Intelligence, and Space 
in March 2000. 

[45] Capital Investment Reports, also known as Exhibit 300s, are 
prepared annually by DOD for each major IT initiative and submitted to 
OMB. 

[46] GAO, Customs Service Modernization: Serious Management and 
Technical Weaknesses Must Be Corrected, GAO/AIMD-99-41 (Washington, 
D.C. Feb. 26, 1999). 

[47] DOD, Defense Acquisition Guidebook, Version 1.0 (Oct. 17, 2004). 
Software Engineering Institute, Software Acquisition Capability 
Maturity Model® version 1.03, CMU/SEI-2002-TR-010 (Pittsburgh, PA: 
March 2002). 

[48] Defense Acquisition University, Test and Evaluation Management 
Guide, Fourth Edition (November 2001). 

[49] Software Engineering Institute, Issues in Requirements 
Elicitation, CMU/SEI-92-TR-12 (Pittsburgh, PA: September 2002). 

[50] Software Engineering Institute, Software Acquisition Capability 
Maturity Model® version 1.03, CMU/SEI-2002-TR-010 (Pittsburgh, PA: 
March 2002); and Defense Acquisition University, Test and Evaluation 
Management Guide, Fourth Edition (November 2001). 

[51] Naval Audit Service, Audit Report Reliability and Validity of the 
Optimized Naval Aviation Logistics Command Management Information 
System, July 22, 2003. NAVAUDSVC P-7520.1, N2003-0060. 

[52] Naval Aviation Logistics Command Management Information System 
(NALCOMIS) Optimization for Organizational Maintenance Activities 
(OOMA) Follow-on Operational Test and Evaluation OT-IIIA Report to the 
Chief of Naval Operations, May 7, 2004, Commander, Operational Test and 
Evaluation. 

[53] Software Engineering Institute, Software Acquisition Capability 
Maturity Model® version 1.03, CMU/SEI-2002-TR-010 (Pittsburgh, PA: 
March 2002); Defense Acquisition University, Test and Evaluation 
Management Guide, Fourth Edition (November 2001); and DOD Instruction 
Number 5000.2, Operation of the Defense Acquisition System (Apr. 5, 
2002) (current version dated May 12, 2003). 

[54] CMM®, Capability Maturity Model, and Capability Maturity Modeling 
are registered in the U.S. Patent and Trademark Office. 

[55] DOD, Defense Acquisition Guidebook, Version 1.0 (Oct. 17, 2004). 
Office of Management and Budget, Circular No. A-94: Guidelines and 
Discount Rates for Benefit-Cost Analysis of Federal Programs, October 
29, 1992; and Circular No. A-11: Planning, Budgeting, Acquisition and 
Management of Capital Assets, June 21, 2005. Software Engineering 
Institute, A Manager's Checklist for Validating Software Cost and 
Schedule Estimates, CMU/SEI-95-SR-004 (Pittsburgh, PA. January 1995). 

[56] GAO-05-702; GAO-02-6; and GAO-04-40. 

[57] DOD, Defense Acquisition Guidebook, Version 1.0 (Oct. 17, 2004); 
and American National Standards Institute (ANSI)/Electronic Industries 
Alliance (EIA) EVM System Standard (ANSI/EIA-748-98), Chapter 2 (May 
19, 1998). 

[58] Software Engineering Institute, Software Acquisition Capability 
Maturity Model® Version 1.03, CMU/SEI-2002-TR-010 (Pittsburgh, PA: 
March 2002). 

[59] Software Engineering Institute, Software Acquisition Capability 
Maturity Model® Version 1.03, CMU/SEI-2002-TR-010 (Pittsburgh, PA: 
March 2002). Defense Acquisition University, Test and Evaluation 
Management Guide, Fourth Edition (November 2001). 

[60] Naval Audit Service, Audit Report Reliability and Validity of the 
Optimized Naval Aviation Logistics Command Management Information 
System, July 22, 2003. NAVAUDSVC P-7520.1, N2003-0060. 

[61] Naval Aviation Logistics Command Management Information System 
(NALCOMIS) Optimization for Organizational Maintenance Activities 
(OOMA) Follow-on Operational Test and Evaluation OT-IIIA Report to the 
Chief of Naval Operations, May 7, 2004, Commander, Operational Test and 
Evaluation. 

[62] American National Standards Institute (ANSI)/Electronic Industries 
Alliance (EIA) EVM System Standard (ANSI/EIA-748-98), Chapter 2 (May 
19, 1998). 

[63] DOD, Defense Acquisition Guidebook, Version 1.0 (Oct. 17, 2004); 
and American National Standards Institute (ANSI)/Electronic Industries 
Alliance (EIA) EVM System Standard (ANSI/EIA-748-98), Chapter 2 (May 
19, 1998). 

[64] "Yes" means that the program provided documentation demonstrating 
satisfaction of the criterion. "Partially" means that the program 
provided documentation demonstrating satisfaction of part of the 
criterion. "No" means that the program has yet to provide documentation 
demonstrating satisfaction of the criterion. 

GAO's Mission: 

The Government Accountability Office, the investigative arm of 
Congress, exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO's commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics. 

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as "Today's Reports," on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order 
GAO Products" heading. 

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office 

441 G Street NW, Room LM 

Washington, D.C. 20548: 

To order by Phone: 

Voice: (202) 512-6000: 

TDD: (202) 512-2537: 

Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm 

E-mail: fraudnet@gao.gov 

Automated answering system: (800) 424-5454 or (202) 512-7470: 

Public Affairs: 

Jeff Nelligan, managing director, 

NelliganJ@gao.gov 

(202) 512-4800 

U.S. Government Accountability Office, 

441 G Street NW, Room 7149 

Washington, D.C. 20548: