This is the accessible text file for GAO report number GAO-03-458 
entitled 'DOD Business Systems Modernization: Improvements to 
Enterprise Architecture Development and Implementation Efforts Needed' 
which was released on February 28, 2003.



This text file was formatted by the U.S. General Accounting Office 

(GAO) to be accessible to users with visual impairments, as part of a 

longer term project to improve GAO products’ accessibility. Every 

attempt has been made to maintain the structural and data integrity of 

the original printed product. Accessibility features, such as text 

descriptions of tables, consecutively numbered footnotes placed at the 

end of the file, and the text of agency comment letters, are provided 

but may not exactly duplicate the presentation or format of the printed 

version. The portable document format (PDF) file is an exact electronic 

replica of the printed version. We welcome your feedback. Please E-mail 

your comments regarding the contents or accessibility features of this 

document to Webmaster@gao.gov.



Report to the Chairman and Ranking Minority Member, Subcommittee on 

Readiness and Management Support, Committee on Armed Services, U.S. 

Senate:



February 2003:



DOD Business Systems Modernization:



Improvements to Enterprise Architecture Development and Implementation 

Efforts Needed:



GAO-03-458:



GAO Highlights: 



Highlights of GAO-03-458, a report to the Chairman and Ranking Minority 

Member, Subcommittee on Readiness and Management Support, Committee on 
Armed 

Services, U.S. Senate:



February 2003:



DOD Business Systems Modernization:



Improvements to Enterprise Architecture Development and Implementation 

Efforts Needed:



Why GAO Did This Study:



The Department of Defense (DOD) is developing an enterprise 
architecture, 

or corporate modernization blueprint, to guide and constrain its 
ongoing and 

planned business system investments. GAO was asked to review DOD’s 
processes 

and controls for developing the enterprise architecture and ensuring 
that 

ongoing IT investments are consistent with its enterprise architecture 

development efforts.



What GAO Found:



DOD has undertaken a challenging and ambitious task to, within 1 year, 
develop 

a departmentwide blueprint for modernizing its over 1,700 time-worn, 
inefficient, 

and nonintegrated business processes and supporting information 
technology (IT) 

assets.  Such a blueprint, commonly called an enterprise architecture, 
is an 

essential modernization management tool. We support the Secretary of 
Defense’s 

decision to develop an architecture and the department’s goal of 
acquiring systems 

that provide timely, reliable, and relevant information.



Successfully doing so requires the application of effective enterprise 

architecture and IT investment management processes and controls. While 
DOD is 

following some of these enterprise architecture practices, it is not 
following 

others, in part because it is focused on meeting its ambitious 
schedule. More 

specifically, with respect to developing the architecture, DOD has yet 
to (1) 

establish the requisite architecture development governance structure 
and process 

controls needed to ensure that ownership of and accountability for the 

architecture are vested with senior leaders across the department, (2) 
clearly 

communicate to intended architecture stakeholders the purpose, scope, 
and 

approach to developing the initial and subsequent versions of the 
architecture, 

and their roles and responsibilities, and (3) define and implement an 
independent 

quality assurance process. Until it follows these practices, DOD 
increases the 

risk of developing an architecture that will be limited in scope, be 
resisted by 

those responsible for implementing it, and will not support effective 
systems 

modernization.



DOD has taken initial steps aimed at improving its management of 
ongoing 

business system investments.  However, DOD has yet to establish the 
necessary 

departmental investment governance structure and process controls 
needed to 

adequately align ongoing investments with its architectural goals and 
direction. 

Instead, DOD continues to allow its component organizations to make 
their own 

parochial investment decisions, following different approaches and 
criteria. 

This stovepiped decision-making process has contributed to the 
department’s 

current complex, error-prone environment of over 1,700 systems.  In 
particular, 

DOD has not established and applied common investment criteria to its 
ongoing 

IT system projects using a hierarchy of investment review and funding 
decision-

making bodies, each composed of representatives from across the 
department. DOD 

also has not yet conducted a comprehensive review of its ongoing IT 
investments 

to ensure that they are consistent with its architecture development 
efforts. 

Until it takes these steps, DOD will likely continue to lack effective 
control 

over the billions of dollars it is currently spending on IT projects.



What GAO Recommends: 



To assist DOD in successfully developing an enterprise architecture and 
using 

it to gain control over its ongoing business system investments, we are 
making 

recommendations to the Secretary of Defense to ensure that DOD (1) 
expands its 

use of effective architecture development processes and controls and 
(2) 

strengthens controls over its ongoing business systems investments.  



DOD concurred with our recommendations and described recently 
completed, ongoing, 

and planned efforts to address them.



www.gao.gov/cgi-bin/getrpt?GAO-03-458 To view the full report, 
including 

the scope and methodology, click on the link above. For more 
information, 

contact Gregory Kutz, (202) 512-9095 (kutzg@gao.gov) or Randolph Hite, 

(202) 512-3439 (hiter@gao.gov).



Contents: 



Letter:



Recommendations for Executive Action:



Agency Comments and Our Evaluation:



Appendixes:



Appendix I: Briefing to Subcommittee Staff: 



Appendix II: Comments from the Under Secretary of Defense:



Appendix III: GAO Contacts and Staff Acknowledgements:



GAO Contacts:



Acknowledgments:



This is a work of the U.S. Government and is not subject to copyright 

protection in the United States. It may be reproduced and distributed 
in its

entirety without further permission from GAO. It may contain 
copyrighted 

graphics, images or other materials. Permission from the copyrighted 

materials separately from GAO’s product. 



Letter February 28, 2003:



The Honorable John Ensign 

Chairman

The Honorable Daniel K. Akaka

Ranking Minority Member 

Subcommittee on Readiness and Management Support

Committee on Armed Services

United States Senate:



In May 2001,[Footnote 1] we reported that the Department of Defense 

(DOD) had neither an enterprise architecture for its financial and 

financial-related business operations, nor the management structure, 

processes, and controls in place to effectively develop and implement 

one. In September 2002, the Secretary of Defense designated improving 

financial management operations, which include not only finance and 

accounting but also business areas such as logistics, acquisition, and 

personnel management, as 1 of the department’s top 10 priorities. In 

addition, the Secretary established a program to develop and implement 

an enterprise architecture. In response to your request, we determined 

whether DOD is (1) following effective processes and controls in 

developing its enterprise architecture and (2) ensuring that ongoing 

information technology (IT) investments are consistent with its 

enterprise architecture development efforts.



On January 31, 2003, we briefed your offices on the results of this 

review and the recommendations we are making to the Secretary of 

Defense. This report transmits those briefing materials, including our 

scope and methodology, as appendix I.



DOD has undertaken a challenging and ambitious task to, within 1 year, 

develop a departmentwide enterprise architecture (blueprint) for 

modernizing its business operations and systems. We support DOD’s goals 

of developing an architecture to guide and constrain its modernization 

efforts and acquiring systems that provide timely, reliable, and 

relevant information. Toward these goals, DOD has taken a number of 

positive steps, including:



* designating improving financial management operations as 1 of its top 

10 priorities;



* establishing a program office responsible for managing the enterprise 

architecture development effort;



* capturing key data needed to develop the “As Is” architecture, such 

as documenting its inventory of over 1,700 business systems; and:



* requiring DOD Comptroller review and approval of IT investments that 

meet certain criteria.



Our May 2001[Footnote 2] report provided a number of fundamental steps 

on how DOD should approach the development of its enterprise 

architecture. At that time, we had also recommended that the department 

limit business system investments until the enterprise architecture is 

developed. While DOD has taken some positive actions, as specified 

above, the department has yet to implement some of our recommendations 

and certain best practices for developing and implementing the 

architecture. The following discussion summarizes those key practices 

DOD has yet to employ.



Architecture Development:



Successful architecture development requires the application of proven 

management practices. Thus far, DOD has not implemented certain 

practices. Specifically, DOD has yet to:



* establish the requisite architecture development governance structure 

needed to ensure that ownership of and accountability for the 

architecture is vested with senior leaders across the department;



* develop and implement a strategy to effectively communicate the 

purpose and scope, approach to, and roles and responsibilities of 

stakeholders in developing the enterprise architecture; and:



* fully define and implement an independent quality assurance process.



Not implementing these practices increases DOD’s risk of developing an 

architecture that will be limited in scope, be resisted by those 

responsible for implementing it, and will not support effective systems 

modernization. DOD recognizes the need to follow these practices, and 

attributes its delays in doing so to tight schedule demands, 

unawareness of certain best practices, and competing resource 

priorities. Among other things, it plans to strengthen the architecture 

governance and management structure.



Architecture Implementation and Control of Ongoing Investments:



Ensuring that ongoing IT investments are consistent with DOD’s 

architecture development efforts requires the application of proven 

investment management practices. To date, DOD has not implemented 

certain practices. Specifically, DOD has yet to establish an investment 

management governance structure that includes:



* a hierarchy of investment review boards composed of representatives 

from across the department who are assigned investment selection and 

control responsibilities based on project threshold criteria;



* a standard set of investment review and decision-making criteria for 

use by all boards, including criteria to ensure architectural 

compliance and consistency; and:



* a specified, near-term date by which ongoing investments have to be 

subjected to this investment review process, and by which decisions 

should be made as to whether to proceed with each investment.



Until the investment management governance structure is established, 

DOD component organizations will continue to make their own parochial 

investment decisions, following different approaches and criteria. As 

we have previously reported,[Footnote 3] this stovepiped decision-

making process has contributed to the department’s current complex, 

error-prone systems environment. This deeply embedded cultural 

resistance to a more holistic decision-making process is a substantial 

risk to successful development and implementation of the enterprise 

architecture. DOD’s leadership plans to strengthen its governance and 

oversight over ongoing IT investments.



Recommendations for Executive Action:



To assist DOD in its efforts to effectively develop and implement an 

enterprise architecture, and guide and constrain its business system 

investments, and to address the problems discussed during the briefing, 

we reiterate the recommendations that we made in our May 2001 

report[Footnote 4] that DOD has yet to implement. In addition, we 

recommend that the Secretary of Defense ensure that:



* the enterprise architecture executive committee members are 

singularly and collectively made explicitly accountable to the 

Secretary for delivery of the enterprise architecture, including 

approval of each version of the architecture;



* the enterprise architecture program is supported by a proactive 

marketing and communication program; and:



* the quality assurance function (1) includes the review of adherence 

to process standards and reliability of reported program performance, 

(2) is made independent of the program management function, and (3) is 

not performed by subject matter experts involved in the development of 

key architecture products.



Additionally, we recommend that the Secretary gain control over ongoing 

IT investments by:



* establishing a hierarchy of investment review boards, each 

responsible and accountable for selecting and controlling investments 

that meet defined threshold criteria, and each composed of the 

appropriate level of executive representatives, depending on the 

threshold criteria, from across the department;



* establishing a standard set of criteria to include (1) alignment and 

consistency with the DOD enterprise architecture and (2) our open 

recommendations governing limitations in business system investments 

pending development of the architecture; and:



* directing these boards to immediately apply these criteria in 

completing reviews of all ongoing IT investments, and to not fund 

investments that do not meet these criteria unless they are otherwise 

justified by explicit criteria waivers.



Agency Comments and Our Evaluation:



In written comments on a draft of this report (see appendix II), the 

Under Secretary of Defense (Comptroller) stated that the department 

concurred with our recommendations and described recently completed, 

ongoing, and planned efforts to address them. For example, the 

department stated that it is currently developing a new architecture 

development and implementation governance structure and a marketing and 

communication strategy to facilitate ongoing development activities. 

Additionally, the department stated that it is providing for the 

independence of its quality assurance function by organizationally 

moving it under the Director of Business Modernization and Systems 

Integration. DOD stated that it would also require the reporting of 

quality assurance information to the architecture Executive Steering 

Committee. We did not verify or evaluate the extent to which the 

efforts described in DOD’s comments will address our recommendation.



Regarding the scope of quality assurance reviews, the department stated 

that it has established and implemented a quality assurance function 

that includes review of architecture products, program performance, and 

architecture development process standards. We agree that the quality 

assurance function includes review of architecture products. However, 

neither during our review nor in its comments did the department 

provide documentary evidence to support that its quality assurance 

reviews address program performance and adherence to architecture 

development process standards. Further, as stated in our report, 

quality assurance function officials told us that they were never 

tasked to perform such reviews.



We are sending copies of this report to the Chairmen and Ranking 

Minority Members of other Senate and House committees and subcommittees 

that have jurisdiction and oversight responsibilities for the 

Department of Defense. We are also sending copies to the Secretary of 

Defense; the Secretary of the Army; the Secretary of the Navy; the 

Secretary of the Air Force; the Under Secretary of Defense 

(Comptroller); the Under Secretary of Defense (Acquisition, Technology, 

and Logistics); the Under Secretary of Defense (Personnel & Readiness); 

the Assistant Secretary of Defense (Command, Control, Communications, 

and Intelligence); the Director of the Defense Finance and Accounting 

Service; and the Director of the Office of Management and Budget. 

Copies will also be available at no charge on our Web site at 

www.gao.gov.



Should you or your staff have any questions on matters discussed in 

this report, please contact us at (202) 512-9095 or (202) 512-3439, 

respectively. We can also be reached by e-mail at kutzg@gao.gov or 

hiter@gao.gov. GAO contacts and key contributors to this report are 

listed in appendix III.



Gregory D. Kutz

Director, Financial Management and Assurance:



Signed by Gregory D. Kutz:



Randolph C. Hite

Director, Information Technology Architecture

and System Issues:



Signed by Randolph C. Hite:



[End of section]



Appendixes:



Appendix I: Briefing to Subcommittee Staff:



[See PDF for image] 



[End of figure] 



[End of section]



Appendix II: Comments from the Under Secretary of Defense:



UNDER SECRETARY OF DEFENSE:



1100 DEFENSE PENTAGON WASHINGTON, DC 20301-1100:



COMPTROLLER:



FEB 27 2003:



Mr. Gregory Kurz Director:



Financial Management and Assurance United States General Accounting 

Office Washington, DC 20548:



Dear Mr. Katz,



My original reply of February 21, 2003, is rescinded. This is the 

Department of Defense (DoD) response to the most recent GAO draft 

report, “DoD Business Systems Modernization: Improvements to Enterprise 

Architecture Development and Implementation Efforts Needed,” dated 

February 7, 2003, (GAO Code 192070/GAO-03-458).



We concur in the recommendations of this draft report. The DoD comments 

to the draft GAO recommendations are enclosed. My point of contact for 

this matter is Mr. Steven Worton, Director for Business Modernization 

and Systems Integration. Mr. Worton may be contacted by e-mail at 

wortons@ osd.pentagon.mil or by telephone at (703) 607-3370.



Signed by Dov S. Zakheim:



Enclosure: As stated:



DoD Comments to GAO Draft Report: “DoD Business Systems Modernization: 

Improvements to Enterprise Architecture Development and Implementation 

Efforts Needed,” dated February 7, 2003, (GAO Code 192070/GAO-03-458):



GAO Recommendation 1: The GAO recommended that the Secretary of Defense 

ensure that the enterprise architecture executive committee members are 

singularly and collectively made explicitly accountable to the 

Secretary for delivery of the enterprise architecture, including 

approval of each version of the architecture.



DoD Comment to GAO Recommendation 1: Concur. The Department currently 

is developing a governance structure that will include detailed 

responsibilities for the enterprise architecture (EA) executive 

committee, the Financial Management Modernization Program (FMMP) 

Steering Committee, the FMMP Domain Owners, and the FMMP Domain Owners’ 

Lead Agents. Included in these detailed responsibilities will be the 

stipulation that the EA executive committee members are singularly and 

collectively accountable to the Secretary for delivery of the 

enterprise architecture. The proposed governance structure is in the 

review and coordination phase. We plan for the governance structure to 

be approved in March 2003.



GAO Recommendation 2: The GAO recommended that the Secretary of Defense 

ensure that the enterprise architecture program is supported by a 

proactive marketing and communication program.



DoD Response to GAO Recommendation 2: Concur. The Department agrees 

that a proactive marketing and communication program is essential to 

eliciting best participation by the cross-functional DoD 

representatives managing development of the EA. In this regard, the 

Department awarded a change management contract in December 2002, for 

the purpose of developing a change and communication management 

strategy and plan, obtaining the input and advice of the EA’s 

stakeholders, and communicating the anticipated changes to various 

levels and stakeholders within the Department and to the public. We 

plan on the change and communication management strategy and plan being 

completed by April 30, 2003. This plan is an ongoing process which will 

be further refined by the Domain Owners and adjusted as necessary.



GAO Recommendation 3: The GAO recommended that the Secretary of Defense 

ensure that the quality assurance function: 1) includes the review of 

adherence to process standards and reliability of reported program 

performance, 2) is made independent of the program management function, 

and 3) is not performed by subject matter experts involved in the 

development of key architecture products.



DoD Response to GAO Recommendation 3: Concur. The Department has 

established and implemented a quality assurance (QA) process that 

includes architecture products, program performance using earned value 

management (EVM), and process standards. The Program Office will 

document the QA process currently being performed by April 30, 2003.



Further, we have a contractor providing independent verification and 

validation (IV&V) assessment of all contractual deliverables, including 

the architecture products.



We agree that, currently, the quality assurance (QA) function is not 

independent of the program management function, but is independent of 

the architecture development group. The QA function was moved from the 

architecture group in January 2003 and reports directly to the 

Director, Business Modernization and Systems Integration. The QA staff 

is not involved in architecture development processes. We will begin 

reporting Quality Assurance information to the Executive Steering 

Committee to mitigate this factor beginning March 2003.



GAO Recommendation 4: The GAO recommended that the Secretary of Defense 

gain control over ongoing IT investments by establishing a hierarchy of 

investment review boards, each responsible and accountable for 

selecting and controlling investments that meet defined threshold 

criteria, and each composed of appropriate level of executive 

representatives, depending on the threshold criteria, from across the 

Department.



DoD Response to GAO Recommendation 4: Concur. The Governance Strategy 

contains a requirement for an investment review board (IRB). The IRB 

will utilize a “portfolio management” approach. They will consider both 

new procurements and current system modifications. The details of the 

operations and structure of the IRB are planned to be completed March 

31, 2003. We welcome any input GAO can provide and will invite GAO to 

these meetings.



GAO Recommendation 5: The GAO recommended that the Secretary of Defense 

gain control over ongoing IT investments by establishing a standard set 

of criteria, to include 1) alignment and consistency with the DoD 

enterprise architecture, and 2) GAO’s prior recommendations governing 

limitations in business system investments pending development of the 

architecture.



DoD Response to GAO Recommendation 5: Concur. DoD has taken a variety 

of steps to limit IT investments that may be inconsistent with the 

architecture. First, we have issued guidance for use by the FMMP 

Systems Review team in the system life cycle milestone decision 

process. This guidance includes standards for EA compliance, economic 

and business case analysis, return on investment requirements, and 

compliance with the “Federal Financial Management Improvement Act of 

1996.” Secondly, the USD(C) has issued two memoranda that outline 

system review requirements. One memorandum relates to Enterprise 

Resource Planning (ERP) initiatives (August 21, 2002) and one 

memorandum relates to new system development or modifications to 

existing systems (October 12, 2002). These memoranda stipulate USD(C) 

certification and approval requirements. Finally, the Governance 

Structure will incorporate an IRB with defined processes.



GAO Recommendation 6: The GAO recommended that the Secretary of Defense 

gain control over ongoing IT investments by directing investment review 

boards to immediately apply the GAO defined criteria in completing 

reviews of all ongoing Information Technology (IT) investments, and to 

not fund investments that do not meet these criteria unless they are 

otherwise justified by explicit criteria waivers.



DoD Response to GAO Recommendation 6: Concur. We intend that the 

actions of the FMMP System Review team and the Governance Strategy will 

lead to an improved control over IT investments.



[End of section]



Appendix III: GAO Contacts and Staff Acknowledgments:



GAO Contacts:



Cynthia Jackson, (202) 512-5086

Jenniffer Wilson, (202) 512-9192:



Acknowledgments:



In addition to the individuals named above, key contributors to this 

report included Johnny Bowen, Teressa Broadie-Gardner, Christopher 

DePerro, Eric Essig, Brian Johnson, Neelaxi Lakhmani, John Ledford, 

Evelyn Logue, Mai Nguyen, Sanford Reigle, Darby Smith, Stacey Smith, Al 

Steiner, and Randolph Tekeley.



FOOTNOTES



[1] U.S. General Accounting Office, Information Technology: 

Architecture Needed to Guide Modernization of DOD’s Financial 

Operations, GAO-01-525 (Washington, D.C.: May 17, 2001).



[2] GAO-01-525.



[3] U.S. General Accounting Office, DOD Financial Management: Important 

Steps Underway But Reform Will Require a Long-term Commitment, GAO-02-

784T (Washington, D.C.: June 4, 2002).



[4] GAO-01-525.



GAO’s Mission:



The General Accounting Office, the investigative arm of Congress, 

exists to support Congress in meeting its constitutional 

responsibilities and to help improve the performance and accountability 

of the federal government for the American people. GAO examines the use 

of public funds; evaluates federal programs and policies; and provides 

analyses, recommendations, and other assistance to help Congress make 

informed oversight, policy, and funding decisions. GAO’s commitment to 

good government is reflected in its core values of accountability, 

integrity, and reliability.



Obtaining Copies of GAO Reports and Testimony:



The fastest and easiest way to obtain copies of GAO documents at no 

cost is through the Internet. GAO’s Web site ( www.gao.gov ) contains 

abstracts and full-text files of current reports and testimony and an 

expanding archive of older products. The Web site features a search 

engine to help you locate documents using key words and phrases. You 

can print these documents in their entirety, including charts and other 

graphics.



Each day, GAO issues a list of newly released reports, testimony, and 

correspondence. GAO posts this list, known as “Today’s Reports,” on its 

Web site daily. The list contains links to the full-text document 

files. To have GAO e-mail this list to you every afternoon, go to 

www.gao.gov and select “Subscribe to daily E-mail alert for newly 

released products” under the GAO Reports heading.



Order by Mail or Phone:



The first copy of each printed report is free. Additional copies are $2 

each. A check or money order should be made out to the Superintendent 

of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 

more copies mailed to a single address are discounted 25 percent. 

Orders should be sent to:



U.S. General Accounting Office



441 G Street NW,



Room LM Washington,



D.C. 20548:



To order by Phone: 	



	Voice: (202) 512-6000:



	TDD: (202) 512-2537:



	Fax: (202) 512-6061:



To Report Fraud, Waste, and Abuse in Federal Programs:



Contact:



Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov



Automated answering system: (800) 424-5454 or (202) 512-7470:



Public Affairs:



Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S.



General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C.



20548: