This is the accessible text file for GAO report number GAO-05-189 
entitled 'DOD Systems Modernization: Management of Integrated Military 
Human Capital Program Needs Additional Improvements' which was released 
on February 11, 2005.

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov.

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately.

Report to the Secretary of Defense:

United States Government Accountability Office:

GAO:

February 2005:

DOD Systems Modernization:

Management of Integrated Military Human Capital Program Needs 
Additional Improvements:

GAO-05-189:

GAO Highlights:

Highlights of GAO-05-189, a report to the Secretary of Defense: 

Why GAO Did This Study:

The Department of Defense (DOD) has long-standing problems with its 
information technology (IT) systems supporting military personnel and 
pay. To address these problems, DOD initiated the Defense Integrated 
Military Human Resources System (DIMHRS) program, which is to provide a 
joint, integrated, standardized military personnel and pay system 
across all military components. In November 2004, DOD accepted the 
design for the first of three phases, DIMHRS (Personnel/Pay). GAO 
reviewed DODís management of the requirements definition for the system 
as well as the programís management structure.

What GAO Found:

DOD faces significant management challenges with DIMHRS, a major system 
acquisition program that is expected to lead to major changes in the 
processing of military personnel and pay. To its credit, DOD has begun 
taking steps to ensure that the requirements and the design for the 
first phase of the program are consistent with each other by tracing 
backward and forward between the detailed requirements and the system 
design, and it did obtain formal user acceptance of the DIMHRS 
(Personnel/Pay) high-level requirements. However, it has not obtained 
user acceptance of the detailed requirements. Furthermore, it has not 
ensured that the detailed requirements are complete and understandable. 
For example, requirements for the interfaces between DIMHRS (Personnel/
Pay) and existing systems have not yet been fully defined because DOD 
has not yet determined how many legacy systems will be partially 
replaced and thus require modification. Furthermore, DOD is still 
determining whether the data requirements provided to the contractor 
for system design are complete. Finally, an estimated 77 percent of the 
detailed requirements are difficult to understand, based on GAOís 
review of a random sample of the requirements documentation. These 
challenges increase the risk that the delivered system capabilities 
will not fully meet the usersí needs. 

Moreover, although DIMHRS (Personnel/Pay) is to be an integrated 
system, its development is not being governed by integrated tools and 
approaches, such as an integrated program management structure, 
enterprise architecture, and master schedule. Furthermore, while DOD is 
appropriately attempting to maximize the use of commercial, off-the 
shelf (COTS) products in building the new system, it has not adequately 
followed some important best practices associated with COTS-based 
system acquisitions. For example, DODís program plan/schedule does not 
adequately recognize the needs of end-user organizations for the time 
and resources to integrate DIMHRS (Personnel/Pay) with their respective 
legacy systems and to prepare their workforces for the organizational 
changes that the system will introduce. 

DODís requirements definition challenges and shortcomings in program 
governance can be attributed to a number of causes, including the 
programís overly schedule-driven approach and DODís difficulty in 
overcoming its long-standing cultural resistance to departmentwide 
solutions. Unless these challenges are addressed, the risk is increased 
that the system will not provide expected capabilities and benefits on 
time and within budget. Given the limitations in some DOD componentsí 
ability to accurately pay military personnel, it is vital that these 
risks be addressed swiftly and effectively.

What GAO Recommends:

To assist DOD and increase its chances of successfully delivering 
DIMHRS (Personnel/Pay), GAO is making recommendations to the Secretary 
of Defense to strengthen DODís requirements-management processes and 
adopt an integrated approach to program management. In commenting on a 
draft of this report, DOD agreed or partially agreed with three of 
GAOís recommendations and partially disagreed with the remaining three.
The department added that it agreed with the general thrust of all the 
recommendations but believed that it was already performing some. GAO 
supports DODís commitment to follow the management principles that the 
recommendations espouse.

www.gao.gov/cgi-bin/getrpt?GAO-05-189.

To view the full product, including the scope and methodology, click on 
the link above. For more information, contact Randolph C. Hite at 202 
512-3439 or hiter@gao.gov.

[End of section]

Contents:

Letter:

DOD's Management of the DIMHRS (Personnel/Pay) Requirements Definition 
Has Recently Improved, but Key Aspects of Requirements Definition 
Remain a Challenge:

DOD Does Not Have a Well-Integrated Management Structure for DIMHRS 
(Personnel/Pay) and Is Not Following All Relevant Supporting 
Acquisition Management Processes:

Conclusions:

Recommendations for Executive Action:

Agency Comments and Our Evaluation:

Appendix I: Briefing to Department of Defense Officials, November 30, 
2004:

Appendix II: Comments from the Department of Defense:

Appendix III: GAO Contact and Staff Acknowledgments:

Abbreviations:

BEA: Business Enterprise Architecture:

CMM: Capability Maturity Model:

COTS: commercial, off-the-shelf:

DFAS: Defense Finance and Accounting Service:

DIMHRS: Defense Integrated Military Human Resources System:

DOD: Department of Defense:

GAO: Government Accountability Office:

IEEE: Institute of Electrical and Electronics Engineers:

IT: information technology:

JFMIP: Joint Financial Management Improvement Program:

JPMO: Joint Program Management Office:

JR&IO: Joint Requirements and Integration Office:

NII: Networks and Information Integration:

ORD: Operational Requirements Document:

P&R: Personnel and Readiness:

PEO-IT: Program Executive Office, Information Technology:

PMO: Program Management Office:

PSA: Principal Staff Assistant:

SEI: Software Engineering Institute:

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. However, because 
this work may contain copyrighted images or other material, permission 
from the copyright holder may be necessary if you wish to reproduce 
this material separately.

United States Government Accountability Office:

Washington, DC 20548:

February 11, 2005:

The Honorable Donald H. Rumsfeld: 
The Secretary of Defense:

Dear Mr. Secretary:

As we first reported in 1993, the Department of Defense (DOD) has had 
serious problems with military personnel and pay systems, including 
shortcomings in its ability to properly pay military personnel and to 
monitor and track them to, from, and within their duty 
stations.[Footnote 1] We recently reported that such problems continue 
today, particularly for Army Reserve and National Guard troops serving 
in Iraq and Afghanistan.[Footnote 2] These long-standing problems can 
be attributed, in part, to:

* hundreds of supporting information technology (IT) systems, many of 
which perform the same tasks and store duplicate data;

* the need for manual data reconciliation, correction, and entry across 
these nonintegrated systems; and:

* the large number of data translations and system interfaces.

To address these problems, DOD initiated the Defense Integrated 
Military Human Resources System (DIMHRS) program, which is to provide a 
joint, integrated system that is standardized across all military 
components. The system is to be Web-based and maximize the use of 
commercial, off-the-shelf (COTS) software. DOD plans to acquire and 
deploy DIMHRS in three phases:

* DIMHRS (Personnel/Pay)--military personnel hiring, promotion, 
retirement, etc., and pay;

* DIMHRS (Manpower)--workforce planning, analysis, utilization, etc; 
and:

* DIMHRS (Training).

DOD accepted the design of the first system phase--DIMHRS (Personnel/
Pay)--in November 2004 and is now proceeding with development of the 
system. Deployment to the Army and the Defense Finance and Accounting 
Service (DFAS) is to begin in the second quarter of fiscal year 2006, 
followed by deployment to the Air Force, Navy, and Marine Corps.

The first phase, DIMHRS (Personnel/Pay), is intended to focus 
particularly on:

* providing joint-theater commanders with accurate and timely human 
capital information;

* providing active service members, reservists, and National Guard 
members with timely and accurate pay and benefits, including those 
performing in theaters of operation or combat; and:

* providing an integrated military personnel and payroll system that 
uses standard data definitions across all services and service 
components, thereby reducing multiple data entries, system maintenance, 
pay discrepancies, and reconciliations of personnel and pay 
information.

The system design for DIMHRS (Personnel/Pay) is based on requirements 
defined to reflect a new, joint approach to processing military 
personnel and pay; this new approach is expected to result in major 
changes to current business processes.

Given the importance of DIMHRS (Personnel/Pay) to DOD's ability to 
manage military personnel and pay, we initiated a review, under the 
authority of the Comptroller General, of the management of the DIMHRS 
(Personnel/Pay) acquisition. Our objectives were to determine:

* whether DOD has effective processes in place for managing the 
definition of the requirements for DIMHRS (Personnel/Pay) and:

* whether DOD has established an integrated program management 
structure for DIMHRS (Personnel/Pay) and is following effective 
processes for acquiring a system based on commercial software 
components.

On November 30, 2004, we briefed officials from DOD's Joint 
Requirements and Integration Office, the Navy Program Executive Office 
(IT), and DOD's Joint Program Management Office on the results of our 
review. This report transmits the briefing. The full briefing, 
including our scope and methodology, is reprinted as appendix I. We 
performed our work from January through November 2004, in accordance 
with generally accepted government auditing standards.

DOD's Management of the DIMHRS (Personnel/Pay) Requirements Definition 
Has Recently Improved, but Key Aspects of Requirements Definition 
Remain a Challenge:

DOD has not effectively managed important aspects of the requirements 
for DIMHRS (Personnel/Pay) to ensure that they are complete, correct, 
and unambiguous. Requirements are the foundation for designing, 
developing, and testing a system. Incorrect or incomplete requirements 
have been commonly identified as a cause of systems that do not meet 
their cost, schedule, or performance goals. Disciplined processes and 
controls for the definition and management of requirements are defined 
in published models and guides, such as the Capability Maturity Models 
developed by Carnegie Mellon University's Software Engineering 
Institute and standards developed by the Institute of Electrical and 
Electronics Engineers.

DOD's management of DIMHRS (Personnel/Pay) requirements had several 
shortcomings:

* First, DOD did not initially ensure that the system requirements and 
system design were aligned. DOD required the contractor to base the 
system design only on high-level (more general) requirements, providing 
detailed requirements to the contractor for information only. However, 
according to the program office, the system design should be based on 
the detailed requirements, and following our inquiries, the program 
office began tracing backward and forward between the detailed 
requirements and the system design to ensure consistency. Among other 
things, DOD is analyzing financial system standards to ensure that all 
applicable standards are included in the requirements and design for 
DIMHRS (Personnel/Pay). Without consistency between requirements and 
design, the risk is increased that the developed and deployed system 
will not fully satisfy financial system standards and users' needs.

* Second, DOD did not ensure that the detailed requirements include 
important content and that they are clear and unambiguous. The 
requirements for the interfaces between DIMHRS (Personnel/Pay) and 
existing systems are not yet complete because DOD has not yet 
determined the extent to which legacy systems will be replaced and thus 
require modification in order to interact with the new system. 
Furthermore, DOD is still determining whether the data requirements 
provided to the contractor for system design are complete. Finally, 
about 77 percent of the detailed requirements are difficult to 
understand, based on our review of a random sample of the requirements 
documentation.[Footnote 3] Our review showed that this documentation 
did not consistently provide a clear explanation of the relationships 
among the parts of each requirement (business rules; information 
requirements; and references to regulations, laws, standards, and so 
on) or adequately identify the sources of data required for 
computations. If requirements are not complete and clear, their 
implementation in the system is not likely to meet users' needs.

* Third, DOD has not obtained user acceptance of the detailed 
requirements. As we have pointed out, when business process changes are 
planned, users' needs and expectations must be addressed, or users may 
not accept change, which can jeopardize the effort.[Footnote 4] One way 
to ensure and demonstrate user acceptance of requirements is to obtain 
sign-off on the requirements by end-user representatives. However, 
although the DIMHRS (Personnel/Pay) program obtained the user 
organizations' formal acceptance of the high-level requirements, the 
process used to define the detailed requirements has not resulted in 
such acknowledgment of agreement on the requirements. Program officials 
stated that gaining formal agreement from some of the user 
organizations would delay the program and be impractical because of end 
users' reluctance to accept a set of joint requirements that requires 
end users to make major changes in their current ways of processing 
military personnel and pay actions. We have previously observed this 
challenge[Footnote 5] and have stated that DOD's organizational 
structure and embedded culture work against efforts to modernize 
business processes and implement corporate information systems such as 
DIMHRS (Personnel/Pay) across component lines. Nevertheless, not 
attempting to obtain agreement on DIMHRS (Personnel/Pay) requirements 
increases the risk that users will not accept and use the developed and 
deployed system, and that later system rework will be required to make 
it function as intended DOD-wide and achieve stated military human 
capital management outcomes.

According to DIMHRS (Personnel/Pay) officials, a number of actions have 
been taken to reduce the risk that users will not accept the system, 
including conducting numerous focus groups, workshops, demonstrations, 
and presentations explaining how the DIMHRS (Personnel/Pay) software 
product could address DOD's existing personnel/pay problems.

However, DIMHRS (Personnel/Pay) officials stated that support for the 
system by the services' executives is mixed. For example, the officials 
said that (1) Army executives are committed to implementing and using 
the DIMHRS (Personnel/Pay) system because they believe it will address 
many problems that the Army currently faces; (2) Air Force officials 
generally support the system but say they do not yet know whether the 
system will meet all their needs; and (3) Navy and Marine Corps 
executives are not as supportive because they are not fully convinced 
that DIMHRS (Personnel/Pay) will be an improvement over their existing 
systems.

The shortcomings in DOD's efforts to effectively manage DIMHRS 
(Personnel/Pay) requirements are attributable to a number of causes, 
including the program's overly schedule-driven approach and the 
difficulty of overcoming DOD's long-standing cultural resistance to 
departmentwide solutions. These shortcomings leave DOD without adequate 
assurance that the requirements will accurately reflect the end users' 
needs and that the resulting system design is reflective of validated 
requirements that will fully meet DOD's needs.

DOD Does Not Have a Well-Integrated Management Structure for DIMHRS 
(Personnel/Pay) and Is Not Following All Relevant Supporting 
Acquisition Management Processes:

DOD does not have a well-integrated structure for managing DIMHRS 
(Personnel/Pay), which DOD has described as an integrated program, 
and it is not following some key supporting processes for acquiring 
COTS-based business systems.

* Program responsibility, accountability, and authority are diffused. 
Leading organizations ensure that programs are structured to ensure 
that a single entity has clear authority, responsibility, and 
accountability for the program. For DIMHRS (Personnel/Pay), these are 
spread among three key stakeholder groups whose respective chains of 
command do not meet at any point below the Secretary and Deputy 
Secretary of Defense levels. Responsibility for requirements definition 
rests with a joint requirements development office, which is 
accountable through one chain of command. Responsibility for system 
acquisition rests with the program office, which is accountable through 
another chain of command. Responsibility for preparing for transition 
to the new system rests with the end-user organizations--11 major DOD 
components reporting through five different chains of command. This is 
consistent with our earlier observation that DOD's organizational 
structure and embedded culture have not adequately accommodated an 
integrated, departmentwide approach to joint systems.[Footnote 6] 
Without a DOD-wide integrated governance structure for a joint, 
integrated program like DIMHRS (Personnel/Pay), the risk is increased 
that the program will not produce an integrated set of outcomes.

* The system has not been defined and designed according to a DOD-wide 
integrated enterprise architecture.[Footnote 7] In accordance with the 
National Defense Authorization Act for Fiscal Year 2003,[Footnote 8] 
DOD has been developing a departmentwide Business Enterprise 
Architecture (BEA), and it has been reviewing some programs, such as 
DIMHRS (Personnel/Pay) with proposed obligations of funds greater than 
$1 million, for consistency with the BEA. In April 2003, the DOD 
Comptroller certified DIMHRS (Personnel/Pay) to be consistent with the 
BEA on the basis of the program manager's commitment that the yet-to-
be-developed system would be designed to be consistent with the yet-to-
be-developed architecture. To follow through on this commitment, DOD 
included a requirement in the DIMHRS (Personnel/Pay) contract that the 
systems specification be compatible with the emerging BEA.

DIMHRS (Personnel/Pay) officials recognize that the April 2003 
architectural certification is preliminary and stated that DIMHRS 
(Personnel/Pay) will undergo another certification before the system 
deployment decision. By that time, however, lengthy and costly design 
and development work will have been completed. The real value in having 
and using an architecture is knowing during system definition, design, 
and development what the larger blueprint for the enterprise is, so 
that these can be guided and constrained by this frame of reference. 
Aligning to the architecture after the system is designed would require 
expensive system rework to address any inconsistencies with the 
architecture.

* Program stakeholders' activities have not been managed according to a 
DIMHRS (Personnel/Pay)-integrated master plan/schedule. An effective 
master plan/schedule should allow for the proper scheduling and 
sequencing of activities and tasks, allocation of resources, 
preparation of budgets, assignment of personnel, and criteria for 
measuring progress. However, the DIMHRS (Personnel/Pay) program plan/
schedule is based on the contractor's and program office's activities 
and does not include all the activities that end-user organizations 
must perform to prepare for DIMHRS (Personnel/Pay), such as the 
redesign of legacy systems and interfaces, business process 
reengineering, and workforce change management. Without a true master 
plan/schedule of activities that includes all DOD program stakeholders, 
the risk increases that key and dependent events, activities, and tasks 
will not be performed as needed, which in turn increases the risk of 
schedule slippage and program goal shortfalls.

* Some, but not all, best practices associated with acquiring COTS-
based business systems are being followed. An example of a best 
practice that DOD is following is to discourage the modification of 
commercial software components without thorough justification; DOD's 
contract includes award fees that give the contractor incentives to, 
among other things, minimize the customization of the COTS software. An 
example of a best practice that DOD is not following is to ensure that 
plans and schedules explicitly provide for preparing users for the new 
business processes associated with the commercial components. DOD does 
not have an integrated program plan/schedule that provides for end-user 
organization activities that are associated with preparing users for 
the changes that the system will introduce. Because it is not following 
all best practices associated with acquiring COTS-based systems, DOD is 
increasing the risk that DIMHRS (Personnel/Pay) will not be 
successfully implemented and effectively used.

DOD's efforts to employ an integrated program management approach have 
not been effective for a number of reasons, including DOD's long-
standing cultural resistance to departmentwide solutions. Without an 
integrated approach and effective processes for managing a program that 
is intended to be an integrated solution that maximizes the use of 
commercially available software products, DOD increases the risk that 
the program will not meet cost, schedule, capability, and outcome 
goals.

Conclusions:

The importance of DIMHRS (Personnel/Pay) to DOD's ability to manage 
military personnel and pay services demands that the department employ 
effective processes and governance structures in defining, designing, 
developing, and deploying the system to maximize its chances of 
success. For DIMHRS (Personnel/Pay), however, DOD did not initially 
perform important requirements-development steps, and the detailed 
system requirements are missing important content. DOD has begun to 
remedy these omissions by taking actions such as tracing among 
requirements documents and system design documents to ensure alignment, 
but user organizations' acceptance of requirements has not occurred. 
Moreover, although DIMHRS (Personnel/Pay) is to be an integrated 
system, it is not being governed by integrated tools and approaches, 
such as an integrated program management structure, integrated DOD 
business enterprise architecture, and an integrated master plan/
schedule.

Furthermore, while DOD is appropriately attempting to maximize the use 
of COTS products in building DIMHRS (Personnel/Pay) and is following 
some best practices for developing COTS-based systems, others are not 
being followed.

The absence of the full complement of effective processes and 
structures related to each of these areas can be attributed to a number 
of causes, including the program's overly schedule-driven approach and 
the difficulty of overcoming DOD's long-standing cultural resistance to 
departmentwide solutions. Effectively addressing these shortcomings is 
essential because they introduce unnecessary risks that reduce the 
chances of accomplishing DIMHRS (Personnel/Pay) goals on time and 
within budget.

It is critical that DOD carefully consider the risks caused by each of 
these areas of concern and that it appropriately strengthen its 
management processes, structures, and plans to effectively minimize 
these risks. To do less undermines the chances of timely and successful 
completion of the program.

Recommendations for Executive Action:

To assist DOD in strengthening its program management processes, 
structures, and plans and thereby increase its chances of successfully 
delivering DIMHRS (Personnel/Pay), we recommend that you direct the 
Assistant Secretary (Networks and Information Integration), the Under 
Secretary (Personnel and Readiness), and the Under Secretary 
(Comptroller), in collaboration with the leadership of the military 
services and DFAS, to take the following six actions to jointly ensure 
an integrated, coordinated, and risk-based approach to all DIMHRS 
(Personnel/Pay) definition, design, development, and deployment 
activities. At a minimum, this should include:

* ensuring that joint system requirements are complete and correct, and 
that they are acceptable to user organizations;

* establishing a DOD-wide integrated governance structure for DIMHRS 
(Personnel/Pay) (1) that vests an executive-level organization or 
entity representing the interests of all program stakeholders--
including the Joint Requirements and Integration Office, the Joint 
Program Management Office, the services, and DFAS--with responsibility, 
accountability, and authority for the entire DIMHRS (Personnel/Pay) 
program and (2) that ensures that all stakeholder interests and 
positions are appropriately heard and considered during program reviews 
and before key program decisions;

* ensuring that the degree of consistency between DIMHRS (Personnel/
Pay) and the evolving DOD-wide business enterprise architecture is 
continuously analyzed and that material inconsistencies between the 
two, both potential and actual, are disclosed at all program reviews 
and decision points and in program budget submissions, along with any 
associated system risks and steps to mitigate these risks;

* developing and implementing a DOD-wide, integrated master plan/
schedule of activities that extends to all DOD program stakeholders;

* ensuring that all relevant acquisition management best practices 
associated with COTS-based systems are appropriately followed; and:

* ensuring that an event-driven, risk-based approach that adequately 
considers factors other than the contract schedule continues to be used 
in managing DIMHRS (Personnel/Pay).

Agency Comments and Our Evaluation:

In written comments on a draft of this report (reprinted in app. II), 
the Under Secretary of Defense for Personnel and Readiness stated that 
DOD largely agrees with the thrust of our recommendations, and that it 
is already following, to the extent practicable, the kind of 
acquisition best practices embodied in them. The department also made 
two overall comments about the report and provided a number of detailed 
comments pertaining to five of our six recommendations.

The first overall comment was that our espousal of certain system 
acquisition management best practices resulted in incongruity among our 
recommendations. In particular, DOD indicated that our recognition that 
DOD is appropriately limiting modification of COTS products (a best 
practice) is incongruous with our recommendation that requirements be 
acceptable to user organizations (another best practice). It further 
stated that if it acted on all comments that it received on 
requirements from all sources, as it suggested we were recommending, 
then this would result in excessive modification to the COTS product. 
We do not agree with DOD's points; we suggest that a careful reading of 
our recommendations would show that the department has not correctly 
interpreted and characterized those recommendations that pertain to 
this overall comment. Specifically, our report does not recommend that 
DOD act on all comments obtained from all sources, regardless of the 
impact and consequences of doing so. Rather, the report contains 
complementary recommendations for ensuring that the system requirements 
are acceptable to user organizations and discouraging changes to the 
COTS product unless the life-cycle costs and benefits justify making 
them. In short, our recommendations concerning system requirements are 
intended to provide DOD with the principles and rules that it should 
apply in executing a requirements-acceptance process that permits all 
stakeholder interests and positions to be heard, considered, and 
resolved in the context of what makes economic sense. While DOD's 
comments note that a process was followed to screen out user inputs 
that, for example, necessitated changes to the COTS product, this 
process did not provide for the effective resolution of such inputs, as 
shown in our report by certain user organizations' comments: 
specifically, that their involvement in defining detailed requirements 
was limited, that their comments on these requirements were not fully 
resolved, and that they were not willing to sign off on the 
requirements as sufficient to meet their needs. This lack of resolution 
is important because not attempting to obtain some level of stakeholder 
acceptance of requirements increases the risk that the system will not 
adequately meet users' needs, that users will not adopt the system, and 
that later system rework will be required to rectify this situation.

The second overall comment was that the department was already 
employing acquisition management best practices, to the extent 
practicable, and that the management process for the program is 
innovative and groundbreaking for DOD, going far beyond what is 
required by the department's regulations. For example, the department 
commented that the system-requirements documentation far exceeds that 
which has been available for any other system effort. We do not dispute 
DOD's comment about efforts on this system relative to other system 
acquisitions, because our review's objectives and approach did not 
extend to comparing DIMHRS (Personnel/Pay) with other DOD acquisitions. 
However, our review did address DOD's use of key acquisition management 
best practices on DIMHRS (Personnel/Pay), and in this regard we support 
the department's recognition of the importance of these practices. In 
our report, we have provided a balanced message by recognizing 
instances where best practices were being followed, such as when DOD 
began tracing detailed system requirements to the system design 
following inquiries that we made during the course of our review. 
However, we do not agree that at the time we concluded our work DOD was 
following all relevant and practicable best practices; examples of 
these practices are cited in our report and discussed below in our 
response to DOD's detailed comments on our individual recommendations.

In its comments specific to our six recommendations, the department 
agreed without further comment with one recommendation (to develop and 
implement a DOD-wide, integrated master plan/schedule of activities 
that extends to all DOD program stakeholders). In addition, it either 
partially agreed or partially disagreed with our other five 
recommendations, and it provided detailed comments on each. Generally, 
DOD's areas of disagreement relate to its view that it is already 
performing the activities that we recommend.

* DOD partially concurred with our recommendation to ensure that joint 
system requirements are complete and correct and acceptable to user 
organizations. In this regard, DOD stated that it has already taken 
great pains to ensure that the requirements are complete and correct, 
although its comments stated that this assurance has occurred "to the 
extent that any documentation [of requirements] this massive can be 
correct." It also stated that the requirements are fully traceable to 
the system design, and that the high-level requirements were validated 
in accordance with DOD regulations. It added that it has taken various 
steps to gain users' acceptance of the system, including a change 
management process, briefings, and prototype demonstrations.

We do not disagree that DOD has taken important steps to meet the goals 
of requirements completeness and correctness. Likewise, we do not 
disagree that since receiving our draft report for comment, the 
department might have completed the important requirements-to-design 
traceability steps that it began in response to our inquiries, which we 
describe in our report. However, DOD's comments contain no evidence to 
show that it has addressed the limitations in the requirements' 
completeness and correctness that we cite in the report, such as those 
relating to the interface and data requirements, and they do not 
address the understandability issues we found relative to 77 percent of 
the detailed requirements. Moreover, DOD stated in its comments that 
its latest program review revealed 606 business process comments and 17 
interface comments that it deemed noncritical, although it noted that 
they were still being analyzed.

We also do not disagree that DOD has taken steps to gain user 
acceptance of the system. However, they did not gain acceptance of the 
detailed requirements that the system is to be designed to meet, which 
is the focus of our recommendation. As we point out in the report, not 
attempting to obtain agreement on the detailed requirements increases 
the risk that users will not adopt the system as developed and 
deployed, and that later system rework will be needed to address this.

* DOD partially concurred with our recommendation that it continuously 
analyze the degree of consistency between DIMHRS (Personnel/Pay) and 
the evolving DOD-wide BEA so that the risks of material inconsistencies 
are understood and addressed. In doing so, DOD stated that the DIMHRS 
(Personnel/Pay) requirements comprise the military personnel and pay 
portion of the architecture and that as one of the first major systems 
developed using all the principles of this architecture, DIMHRS 
(Personnel/Pay) is and will remain fully consistent with it. We do not 
agree with DOD's comments that the system is consistent with the BEA. 
As we state in our report, DOD could not provide us with documented, 
verifiable analysis demonstrating this consistency and forming the 
basis for the DOD Comptroller's April 2003 certification of this 
consistency. Rather, we were told that this certification was based on 
the DIMHRS (Personnel/Pay) program manager's stated commitment to be 
consistent at some future point. However, as we note in our report, the 
real value of an architecture is that it provides the necessary context 
for guiding and constraining system investments in a way that promotes 
interoperability and minimizes overlap and duplication. Without it, 
expensive system rework is likely to be needed to achieve these 
outcomes. As we also note in our report, the absence of verifiable 
analysis of DIMHRS (Personnel/Pay) architectural compliance was in part 
due to the state of the BEA, which we have reported as not being well-
defined and missing important content.[Footnote 9] Recognizing this, as 
well as the pressing need for DIMHRS's (Personnel/Pay) promised 
capabilities, our recommendation calls for ongoing analysis of DIMHRS 
(Personnel/Pay) and the BEA to understand the risks of designing and 
developing the system outside the context of a well-defined 
architecture.

* DOD partially disagreed with our recommendation that it establish a 
DOD-wide governance structure in which responsibility, accountability, 
and authority for the entire program are vested in an executive-level 
organization or entity representing the interests of all program 
stakeholders. In doing so, the department described the roles, 
responsibilities, and authorities for various program stakeholders; 
however, it did not explain its reason for not agreeing with the 
recommendation, and only one of its comments bears relevance to our 
recommendation. Specifically, it commented that the Under Secretary of 
Defense (Personnel and Readiness) has full responsibility and 
accountability for the program. We do not agree. As we state in our 
report, DIMHRS (Personnel/Pay) is a DOD-wide program involving three 
distinct stakeholder groups whose respective chains of command do not 
meet at any point below the Secretary and Deputy Secretary of Defense 
levels. Thus, we concluded that responsibility, accountability, and 
authority for the program are diffused, with responsibility for 
developing functional requirements resting with the Joint Requirements 
and Integration Office, responsibility for system acquisition resting 
with the Joint Program Management Office, and responsibility for 
preparing for the transition to DIMHRS (Personnel/Pay) resting with 11 
major end-user organizations. Under this structure, only the Joint 
Requirements and Integration Office is accountable to the Under 
Secretary, and the two distinct other stakeholder groups are not 
accountable to the Under Secretary. This means, as we state in the 
report, that no single DOD entity is positioned to exercise continuous 
leadership and direction over the entire program.

* The department also partially disagreed with our recommendation to 
follow all relevant acquisition management best practices associated 
with COTS-based systems. According to DOD's comments, all of these best 
practices are currently being followed, including the three that we 
cite in our report as not being followed: (1) ensuring that plans 
explicitly provide for preparing users for the impact that the business 
processes embedded in the commercial components will have on their 
respective roles and responsibilities, (2) proactively managing the 
introduction and adoption of changes to how users will be expected to 
use the system to execute their jobs, and (3) ensuring that project 
plans explicitly provide for the necessary time and resources for 
integrating commercial components with legacy systems. In this regard, 
the department stated that the DIMHRS (Personnel/Pay) program had 
documented every change in current practices and policies that will be 
required for the military services, as well as future practices and 
policies, and that these were fully vetted through the functional user 
community. It also described a number of activities that it has 
undertaken to prepare and train users in the COTS product and other 
aspects of DIMHRS (Personnel/Pay). We do not dispute whether DOD has 
performed activities intended to facilitate the implementation of 
DIMHRS (Personnel/Pay). However, the best practices that we identified 
as not being followed, which form the basis of our recommendation, are 
focused on effectively planning for the full complement of activities 
that are needed to prepare an organization for the institutional and 
individual changes that COTS-based system solutions introduce. Such 
planning is intended to ensure, among other things, that key change 
management activities, including the dependencies among these 
activities, are defined and agreed to by stakeholders, including 
ensuring that adequate resources and realistic time frames are 
established to accomplish them. In this regard, DOD agreed in its 
comments that it does not have an integrated master plan/schedule for 
the program, which is an essential tool for capturing the results of 
the proactive change management planning that the best practices and 
our recommendation advocate. Both published research and our experience 
in evaluating the acquisition and implementation of COTS-based system 
solutions show that the absence of well-planned, proactive 
organizational and individual change management efforts can cause these 
system efforts to fail.

* The department partially disagreed with our last recommendation to 
adopt a more event-driven, risk-based approach to managing DIMHRS 
(Personnel/Pay) that adequately considers factors other than the 
contract schedule, stating that it is currently using an event-driven, 
risk-based approach and revising the schedule when necessary. We 
support DOD's comment, as it indicates that DOD has decided to begin 
following such an approach. However, during the course of our work this 
was not the case. For example, we observed at that time that the DIMHRS 
(Personnel/Pay) program intended to accelerate its deployment schedule 
to meet an externally imposed deadline and that it was not until we 
raised concerns about the associated risks of doing so, as well as the 
absence of effective strategies to mitigate these risks, in an earlier 
draft of the briefing included in this report, that the department 
changed its plans. Also during the course of our work, we observed that 
program activities were truncated or performed concurrently in order to 
meet established deadlines. For example, as we describe in our report, 
data requirements (which are derived from higher-level information 
needs) were provided to the contractor before information needs were 
fully defined because the contractor needed these data requirements to 
complete the system design on schedule. It was this kind of focus on 
schedule that led to our recommendation to adopt a more event-driven, 
risk-based approach. However, in light of DOD's comment that it intends 
to do so, we have slightly modified our recommendation to recognize 
this decision.

All our recommendations are aimed at reducing the risk of failure on 
this important program, which we and DOD agree is critical to the 
department's ability to effectively manage military personnel and pay. 
Furthermore, DOD's comments show that it agrees with us on the 
importance of taking an approach to the program that is based on the 
kinds of management processes and structures that we recommend, and the 
department appears committed to following such an approach. Following 
our recommendations will help the department to do so and thereby avoid 
unnecessary risks. As we state in the report, careful consideration of 
the areas of concern that we raise is critical to improving the chances 
of timely and successful completion of the program.

We are sending copies of this report to the House and Senate Armed 
Services and Appropriations Committees; the House Committee on 
Government Reform; the Senate Committee on Governmental Affairs; and 
the Director, Office of Management and Budget. In addition, the report 
will be available at no charge on the GAO Web site at http://
www.gao.gov.

Should you or your offices have any questions on the matters discussed 
in this report, please contact Randolph Hite at (202) 512-3439 or 
Gregory C. Wilshusen at (202) 512-3317; we can also be reached by e-
mail at hiter@gao.gov or wilshuseng@gao.gov. Other contacts and key 
contributors to this report are listed in appendix III.

Sincerely yours,

Signed by: 

Randolph C. Hite: 
Director, Information Technology Architecture and Systems Issues:

Gregory C. Wilshusen: 
Acting Director, Defense Capabilities and Management Issues:

[End of section]

Appendix I: Briefing to Department of Defense Officials, November 30, 
2004:

DOD Systems Modernization: Management of Integrated Military Human 
Capital Program Needs Additional Improvements:

Briefing to Department of Defense Officials:

November 30, 2004:

Overview:

Introduction:

Objectives:

Results in Brief:

Background:

Findings:

* Requirements Management:

* Program Management Structure and Processes Conclusions:

Recommendations:

Attachment I: Objectives, Scope, and Methodology:

Introduction:

As we have previously reported, the Department of Defense (DOD) faces 
serious military personnel and payroll-processing problems, dating back 
to the early 1990s. [NOTE 1] According to DOD, these problems affect 
its ability to properly pay military personnel and to monitor and track 
them to, from, and within their duty stations. We recently reported 
that these problems continue today, particularly for Army Reserve and 
National Guard troops serving in Iraq and Afghanistan.[NOTE 2]

These problems can be traced, in part, to such causes as:

* hundreds of supporting information technology (IT) systems, many of 
which perform the same tasks and store duplicate data;

* the need for manual data reconciliation, correction, and entry across 
these nonintegrated systems; and:

* the large number of data translations and system interfaces.

The Defense Integrated Military Human Resources System (DIMHRS 
(Personnel/Pay)) is intended to address these problems, with particular 
focus on:

* providing joint-theater commanders with accurate and timely human 
capital information;

* providing active service members, reservists, and National Guard 
members with timely and accurate pay and benefits, especially when they 
are performing in theaters of operation or combat; and:

* providing an integrated military personnel and payroll system that 
uses standard data definitions across all services and service 
components, thereby reducing multiple data entries, system maintenance, 
pay discrepancies, and reconciliations of personnel and pay 
information.

Among other things, the new system is also intended to support DOD's 
efforts to produce accurate and complete financial statements.

DOD plans to acquire and deploy DIMHRS in three phases:

* DIMHRS (Personnel/Pay)-military personnel hiring, promotion, 
retirement, etc., and pay;

* DIMHRS (Manpower)-workforce planning, analysis, utilization, etc.; 
and:

* DIMHRS (Training).

DOD accepted the design of the first system phase-DIMHRS (Personnel/
Pay)-in November 2004 and is now proceeding with development of this 
system phase. Deployment to the Army and the Defense Finance and 
Accounting Service (DFAS) is to begin in the second quarter of fiscal 
year 2006, followed by deployment to the Air Force, Navy, and Marine 
Corps.

Objectives:

In view of the significance of the DIMHRS program to DOD's ability to 
manage military personnel and pay services, we initiated a review under 
the authority of the Comptroller General. Our objectives were to 
determine:

1. whether DOD has effective processes in place for managing the 
definition of the requirements for DIMHRS (Personnel/Pay) and:

2. whether DOD has established an integrated program management 
structure for DIMHRS (Personnel/Pay) and is following effective 
processes for acquiring a system based on commercial software 
components.

To accomplish our objectives, we interviewed officials from relevant 
organizations, analyzed program management documentation and 
activities, and reviewed relevant DOD analyses. Further details on our 
scope and methodology are given in attachment I to this appendix.

Our work was performed from January through November 2004, in 
accordance with generally accepted government auditing standards.

Results in Brief: Objective 1: Requirements Management:

DOD's management of the DIMHRS (Personnel/Pay) requirements definition 
has recently improved, but key aspects of requirements definition 
remain a challenge. In particular, DOD has begun taking steps to ensure 
that the system requirements and the system design are consistent with 
each other. However, DOD:

* has not ensured that the detailed requirements are complete and 
understandable and:

* has not obtained user acceptance of the detailed requirements.

The requirements definition challenges are attributable to a number of 
causes including the program's overly schedule-driven approach and the 
difficulty of overcoming DOD's long-standing cultural resistance to 
departmentwide solutions.

These challenges increase the risk that the delivered system's 
capabilities will not fully meet DOD's needs.

Results in Brief: Objective 2: Management Structure and Processes:

DOD does not have a well-integrated management structure for DIMHRS 
(Personnel/Pay) and is not following all relevant supporting 
acquisition management processes. In particular,

* program responsibility, accountability, and authority are diffused;

* the system has not been defined and designed according to a DOD-wide 
integrated enterprise architecture;

* program stakeholders' activities have not been managed according to a 
master plan/schedule that integrates all stakeholder activities; and:

* the program is following some, but not all, best practices associated 
with acquiring business systems based on commercially available 
software.

Without an integrated approach and effective processes for managing a 
program that is intended to be an integrated solution, DOD has 
increased the risk that the program will not meet cost, schedule, 
capability, and outcome goals.

Results in Brief: Recommendations:

To assist DOD in effectively managing DIMHRS (Personnel/Pay), we are 
making six recommendations to the Secretary of Defense aimed at 
ensuring that DOD follows an integrated, coordinated, and risk-based 
program approach and thereby increases its chances of successfully 
delivering DIMHRS (Personnel/Pay).

Background:

Program Chronology:

June 1996. Defense Science Board Task Force on Military Personnel 
Information Management recommended that DOD "move to a single, all-
service and all-component, fully integrated personnel and pay system 
with common core software."

July 1997 Deputy Secretary of Defense established the Joint 
Requirements and Integration Office (JR&IO) and assigned it 
responsibility for functional oversight of DIMHRS and appointed the 
Navy as the "acquisition agent."

February 1998. The Under Secretary of Defense for Personnel and 
Readiness (P&R) and the Assistant Secretary of Defense for Networks and 
Information Integration (NII) [NOTE 3] approved the DIMHRS (Personnel/
Pay) program to proceed into the concept refinement phase (i.e., 
alternatives analysis).

May 1999. DOD's Institute for Defense Analysis analyzed alternative 
systems solutions, reaffirmed the decision to base DIMHRS (Personnel/
Pay) on commercial, off-the-shelf (COTS) software, and made a number of 
suggestions, including the need to favor the use of COTS "out of the 
box" to hold down costs.

October 2000. Assistant Secretary of Defense (NII) approved the program 
to proceed into the technology development phase (i.e., requirements 
definition).

March 2001. The Navy selected a commercial software product to serve as 
the foundation for DIMHRS (Personnel/Pay) application software.

May 2002. The Navy issued a request for proposals for development and 
integration of DIMHRS (Personnel/Pay).

June 2002. DOD responded to congressional questions about the DIMHRS 
(Personnel/Pay) program management structure and requests for DIMHRS 
(Personnel/Pay) funding from multiple DOD organizations.

September 2002. The Navy awarded five contracts and began a competitive 
acquisition process to select one of the five to develop and integrate 
DIMHRS (Personnel/Pay).

December 2002. The five contractors submitted final proposals.

May 2003. Assistant Secretary of Defense (NII) approved the program to 
proceed into the system development and demonstration phase (i.e., 
software development, integration, testing, etc.).

September 2003. The Navy awarded the development and integration 
contract.

March 2004. DOD established a baseline version of the detailed 
requirements for DIMHRS (Personnel/Pay) and provided it to the 
development and integration contractor for use in designing DIMHRS 
(Personnel/Pay).

November 2004. DOD accepted the contractor's design of DIMHRS 
(Personnel/Pay) and authorized the contractor to proceed with 
development of the system.

DIMHRS (Personnel/Pay) Description:

The DIMHRS (Personnel/Pay) system is to be Web based and maximize the 
use of COTS software. The contract includes award fees that give the 
contractor incentives to, among other things, meet the contract 
schedule and minimize customization of the COTS software.

DIMHRS (Personnel/Pay) was designed in four parts. DOD has accepted all 
four parts and authorized the contractor to develop the system in 
accordance with the accepted design.

According to DOD officials, DIMHRS (Personnel/Pay) will be the largest 
personnel system in the world in terms of the number of people it 
serves and transactions it processes.

Background:

Roles and Responsibilities of DIMHRS (Personnel/Pay) Stakeholders:

Stakeholder: Joint Requirements and Integration Office;
Roles and responsibilities: Developing DOD integrated user 
requirements; providing expertise related to personnel/pay functional 
area.

Stakeholder: Joint Program Management Office (JPMO);
Roles and responsibilities: Acquiring the system, including managing 
the system development contractor, accepting the design; and testing 
and deploying the system.

Stakeholder: End users[A];
Roles and responsibilities: Assisting JR&IO in developing integrated 
requirements; assisting JPMO in addressing technical issues; taking 
other actions necessary for transition.

Stakeholder: DIMHRS management offices for 4 services and DFAS;
Roles and responsibilities: Assisting JR&IO and JPMO in developing 
DIMHRS (Personnel/Pay); managing transition activities for own chain of 
command, including modifying existing systems and interfaces.

Stakeholder: Under Sec. of Defense (P&R);
Roles and responsibilities: Overseeing personnel/pay functional area; 
resolving issues that cannot be resolved by the Executive Steering 
Committee.

Stakeholder: Executive Steering Committee;
Roles and responsibilities: Monitoring the program, resolving issues, 
and advising the Under Secretary (this is a stakeholder executive-level 
committee).

Stakeholder: Joint Integration Group;
Roles and responsibilities: Monitoring the program and resolving issues 
(this is a group of user representatives).

Stakeholder: Assist. Sec. of Defense (NII);
Roles and responsibilities: Designated Milestone Decision Authority.

Stakeholder: Assist. Sec. Navy (Research, Development, and 
Acquisition);
Roles and responsibilities: Establishing acquisition policies and 
procedures in accordance with DOD directives and guidelines and for 
chartering the Program Executive Office, Information Technology (PEO-
IT).

Stakeholder: PEO-IT;
Roles and responsibilities: Providing programmatic and technical 
direction to JPMO.

Sources: GAO and DOD.

[A] End users come from 11 organizations: 4 services, 4 reserve units, 
2 National Guard components (Air Force and Army), and DFAS.

[End of table]

Costs:

DOD does not currently have a total life-cycle cost estimate for DIMHRS 
(Personnel/Pay). According to JPMO officials, JPMO's costs through 
fiscal year 2003 and projected through fiscal year 2009 are about $601 
million:

* $244.7 million obligated during fiscal years 1998 through 2003 and:

* $356.6 million required for fiscal years 2004 through 2009.

However, JR&IO and JPMO officials stated that these amounts do not 
include user organization costs; JPMO originally estimated these costs 
to be about $350 million, but it is now reevaluating these and other 
cost estimates as part of its efforts to update the program's economic 
analysis.

Additionally, the officials stated that the $601 million does not 
include JR& O's actual and estimated costs of $153 million through 
fiscal year 2009 for requirements definition activities, business 
process reengineering planning, enterprise architecture development, 
and other activities pertaining to management and analysis of the human 
resources domain. According to JR&IO officials, this $153 million 
consists of $72.5 million obligated during fiscal years 1998 through 
2003 and $80.4 million required for fiscal years 2004 through 2009.

Benefits:

DOD estimated that after DIMHRS (Personnel/Pay) is fully operational, 
including implementation of associated business process improvements 
and termination of legacy systems, the system will address DOD's long-
standing military personnel and payroll processing problems and result 
in productivity improvements and reduced IT operating and maintenance 
costs. [NOTE 4]

Objective 1: Requirements Management:

DOD's management of the DIMHRS (Personnel/Pay) requirements definition 
has recently improved, but key aspects of requirements definition 
remain a challenge. Requirements are the foundation for designing, 
developing, and testing a system. Our experience indicates that 
incorrect or incomplete requirements are a common cause of systems not 
meeting their cost, schedule, or performance goals. Disciplined 
processes and controls for defining and managing requirements are 
defined in published models and guides, such as the Capability Maturity 
Models developed by Carnegie Mellon University's Software Engineering 
Institute (SEI), and standards developed by the Institute of Electrical 
and Electronics Engineers (IEEE).

In managing DIMHRS (Personnel/Pay) requirements, DOD has begun taking 
steps to ensure that the system requirements and the system design are 
consistent with each other. However, DOD:

* has not ensured that the detailed requirements are complete and 
understandable and:

* has not obtained user acceptance of the detailed requirements.

These challenges increase the risk that the delivered system's 
capabilities will not fully meet DOD's needs.

Objective 1: Requirements Management Traceability:

DOD has recently begun performing important steps to better ensure 
alignment among high-level requirements, detailed requirements, and the 
system design. According to SEI guidance, requirements should be 
completely and correctly incorporated into the system design. As we and 
others have previously reported, addressing requirements issues during 
requirements definition and system design is considerably less 
expensive and quicker than fixing problems during and after 
development. [NOTE 5]

Objective 1: Requirements Management Traceability:

An accepted way of ensuring the complete and accurate incorporation of 
requirements is to trace between levels of requirements and design 
documentation:

High-level requirements: Operational Requirements Document (ORD).

Detailed user requirements: E.g., business rules and information 
requirements.

System design.

In system development, traceability is the degree to which a 
relationship is established between two or more products of the system 
development process.

Traceability allows the user to follow the life of the requirement both 
forward and backward through system documentation from origin through 
implementation. Traceability is critical to understanding the 
parentage, interconnections, and dependencies among the individual 
requirements. This information in turn is critical to understanding the 
impact when a requirement is changed or deleted.

The DIMHRS (Personnel/Pay) ORD defines the high-level capabilities for 
satisfying DOD's mission needs.

* The ORD lists the functional processes, information needs, and 
performance parameters that the system is to support; an example of a 
functional process is "promote enlisted member personnel."

The detailed requirements include "use cases," which are detailed 
descriptions of the activities that the system and the end users must 
perform and data needed to accomplish these activities.

* For example, the functional process "promote enlisted personnel" 
includes multiple use cases, such as "record enlisted member's 
eligibility for promotion." Each use case includes (1) business rules 
describing the processing steps for accomplishing the use case, such as 
steps for determining which members meet the time in grade/time in 
service requirement for promotion; (2) references to the applicable 
statutes, policies, guidance, or regulations that govern the use case; 
and (3) a list of the information needed to perform the use case, such 
as each person's rank, occupation code, and promotion recommendation.

In addition, the use cases incorporate process improvements that are to 
introduce efficiencies and to standardize personnel and pay processing 
DOD-wide.

However, when DOD accepted the first two parts of the system design, it 
had not traced between the detailed requirements and the design. 
Rather, DOD required the contractor to base the system design only on 
the high-level requirements defined in the ORD, and DOD provided 
detailed requirements for information only purposes.

According to JPMO officials, the contract was written in this way to 
provide the contractor with maximum flexibility to design the system 
according to the capabilities of the COTS product and thereby reduce 
system development and maintenance costs. Nonetheless, JR&IO officials 
also stated that the detailed requirements for DIMHRS (Personnel/Pay) 
should be the basis of the system design.

In addition, DOD did not, until recently, trace between ORD 
requirements and detailed requirements. As a consequence, we told DOD 
during the course of our review that relevant financial and accounting 
standards were missing from the detailed requirements, even though they 
had been included in the ORD.

According to the ORD, the system will comply with requirements for 
personnel and payroll feeder systems contained in the Federal Financial 
Management Improvement Act of 1996, the Chief Financial Officers Act of 
1990, the Federal Managers' Financial Integrity Act of 1982, and the 
most current Joint Financial Management Improvement Program (JFMIP) 
Program Management Office (PMO) requirements. [NOTE 6] The details that 
correspond to these statutory and regulatory requirements are found in 
the JFMIP PMO standard for human resources and payroll financial 
systems. [NOTE 7]

* The JFMIP PMO standard is important because it contains requirements 
associated with producing accurate and complete payroll data for the 
financial statements, which is relevant to DOD's efforts to obtain 
"clean," or "unqualified," audit opinions on its financial statements. 
[NOTE 8]

* An example of the JFMIP PMO requirements is functionality to process 
prior period, current, and future period pay actions, on the basis of 
effective dates.

* After we told DOD of the missing requirements, JR&IO officials 
undertook an analysis of the 196 JFMIP PMO human resources and payroll 
requirements [NOTE 9] and have stated that this analysis has allowed 
them to ensure that DIMHRS (Personnel/Pay) will meet 170 of the 196 
requirements and that the remaining requirements are not applicable to 
military human resource and payroll systems. They also stated that all 
applicable requirements are now documented in the DIMHRS (Personnel/
Pay) detailed requirements database.

* According to JPMO officials, tracing the detailed financial 
requirements to the design was not done sooner because the COTS 
software product being used is certified as JFMIP PMO compliant. 
However, JFMIP PMO certification extends only to the core financial 
module of this software (i.e., general ledger, funds control, accounts 
receivable, accounts payable, cost management, and reporting); it does 
not include the two modules used for DIMHRS (Personnel/Pay)-human 
resources and payroll.

In addition, as we have reported, even if JFMIP PMO had certified the 
human resources and payroll modules of the COTS software product, 
certification by itself does not ensure that systems based on this 
software will be compliant with the goals of the Federal Financial 
Management Improvement Act, as JFMIP has made clear, and does not 
ensure that systems based on this software will provide reliable, 
useful, and timely data for day-to-day management. [NOTE 10] Other 
important factors affecting compliance with federal financial 
management system requirements and the effectiveness of an implemented 
COTS system include how the software package has been configured to 
work in the agency's environment, whether any customization is made to 
the software, the success of converting data from legacy systems to new 
systems, and the quality of transaction data in the feeder systems.

To their credit, JR&IO and JPMO officials have begun tracing between 
the detailed requirements and the design, including the financial 
standards. As of late November 2004, they told us that this tracing had 
identified about 630 discrepancies that may require modification to the 
detailed requirements or the design. They stated that they plan to 
complete this tracing by the end of February 2005.

Until DOD completes tracing both backward (from the design back to the 
detailed requirements and the ORD) and forward (from the ORD forward to 
the detailed requirements and the design), the risk is increased that 
the requirements and design are not complete and correct.

Objective 1: Requirements Management: Content of Requirements:

Detailed requirements are missing important content and are difficult 
to understand. According to SEI, requirements should be complete, 
correct, clear, and understandable; IEEE standards state that 
requirements should be communicated in a structured manner to ensure 
that the customers (i.e., end users) and the system's developers reach 
a common understanding of them.

For DIMHRS (Personnel/Pay), certain requirements are missing from the 
detailed requirements. Specifically, the interface requirements remain 
incomplete, and questions exist as to the completeness of the data 
requirements. Finally, some of the use cases that provide the detailed 
requirements are unclear and ambiguous, making them difficult to 
understand. Each of these three areas is discussed in greater detail 
below.

If requirements are not complete and clear, their implementation in the 
system design may not meet users' needs, and it will be unnecessarily 
difficult for DOD to test the system effectively and determine whether 
system requirements have been met.

First, the requirements for the interfaces between DIMHRS (Personnel/
Pay) and existing systems are not yet complete. According to SEI, 
requirements for internal and external interfaces should be 
sufficiently defined to permit these interfaces to be designed and 
interfacing systems to be modified.

For example, DIMHRS (Personnel/Pay) will be required to interface with 
DOD's accounting systems and other systems, such as DOD's travel 
system, either by providing DIMHRS (Personnel/Pay) data for these 
systems or by receiving accounting data from them. DIMHRS (Personnel/
Pay) interfaces must also be designed to ensure compliance with 
applicable JFMIP PMO financial system requirements and applicable 
federal accounting standards. These interface requirements must be 
completed before the DIMHRS (Personnel/Pay) system can be fully 
deployed.

To complete the interface requirements, officials representing JPMO and 
the user organizations' DIMHRS offices told us that they must identify 
which of the legacy systems will be partially replaced, and thus will 
require modification in order to interface with the new system. JPMO 
officials stated that although DOD accepted the system design in 
November 2004, a significant amount of work remained to fully address 
DIMHRS (Personnel/Pay) interface issues by the user organizations.

Second, the data requirements initially provided to the contractor for 
the system design had not been aligned with the users' information 
needs that were included in the detailed requirements. According to 
SEI, the data required to meet users' information needs must be defined 
so that the system can be properly designed and developed. However, 
JR&IO officials told us that they had not fully defined information 
needs when users were asked to identify the data requirements, along 
with the legacy systems that are the best sources of the required data. 
The contractor needed these data requirements to complete the system 
design on schedule.

* DOD recently began comparing the data requirements provided to the 
contractor with the users' information needs developed by JR&IO. JR&IO 
officials stated that they expect to complete this work in February 
2005. Until this task is completed, DOD will not know whether revisions 
will be needed to the system design to ensure that users' information 
needs are met and that the correct data are later migrated to the new 
system.

* JPMO officials also stated that when DOD accepted the system design 
in November 2004, a significant amount of work remained for the user 
organizations to fully address DIMHRS (Personnel/Pay) data issues.

Third, some of the detailed requirements are unclear and ambiguous, 
making them difficult to understand. According to SEI, requirements 
should be clear and understandable. When requirements are ambiguous, 
their meaning is open to varying interpretations, which increases the 
risk that the implementation of the requirements will not meet users' 
needs.

* We reviewed a random sample of the documentation for 40 of 424 use 
cases (detailed requirements): 20 of 284 pay use cases and 20 of 140 
personnel use cases. Our review showed that this documentation did not 
consistently provide a clear explanation of the relationships among the 
parts of each requirement (business rules, information requirements, 
and references) or adequately identify the sources of data required for 
computations.

* Based on our sample, an estimated 22 percent of use cases cite "P&R 
guidance" (that is, guidance from the Office of the Under Secretary for 
P&R) as a reference to support the need for a business rule, either 
alone or along with references to DOD policies. [NOTE 11] According to 
JR&IO officials, this note indicates that the business rule includes 
steps not currently required by DOD's policies, which have been added 
either to take advantage of "out-of-the-box" COTS capabilities or to 
implement a best practice. However, when P&R guidance is cited, the use 
cases do not explain whether an out-of-the-box capability or best 
practice is intended.

* In addition, when both P&R guidance and existing policies are cited, 
the use case does not explain which rules are based on P&R guidance and 
which are based on existing policies. These ambiguities make it 
difficult for stakeholders to understand the business rule and its 
rationale. (This point is further discussed in the following section on 
end users' acceptance of requirements.) According to JR&IO officials, 
such ambiguity was resolved via communication between JR&IO and JPMO 
officials followed by JPMO officials' communicating with the 
contractor.

* Estimates of the extent of use case problems and associated 
confidence intervals are summarized on the next slide.

Extent of Problems with Use Cases:

Problem description: Information requirements are provided, but the 
business rules do not provide enough detail to determine what 
information requirement is to be used for each business rule;
Use cases with problems: Number: Pay: 8;
Use cases with problems: Number: Personnel: 20;
Use cases with problems: Estimated percentage: 60%;
Use cases with problems: 95% confidence levels [A]: 44-74%.

Problem description: Business rule contains a computation but does not 
include a source for the data needed for making the computation;
Use cases with problems: Number: Pay: 8;
Use cases with problems: Number: Personnel: 4;
Use cases with problems: Estimated percentage: 33%;
Use cases with problems: 95% confidence levels [A]: 19-51%.

Problem description: Reference to "P&R guidance" does not clearly 
explain the rationale behind the business rule;
Use cases with problems: Number: Pay: 0;
Use cases with problems: Number: Personnel: 13;
Use cases with problems: Estimated percentage: 22%;
Use cases with problems: 95% confidence levels [A]: 15-29%.

Problem description: Use cases that had one or more of the above 
problems;
Use cases with problems: Number: Pay: 13;
Use cases with problems: Number: Personnel: 20;
Use cases with problems: Estimated percentage: 77%;
Use cases with problems: 95% confidence levels [A]: 60-89%.

Source: GAO.

[A] Ninety five percent confidence interval of the percentage of total 
use cases exhibiting the indicated problem.

[End of table]

JR&IO officials agreed that the clarity of the use cases could be 
improved but stated that the use cases provide a greater level of 
detail than DOD normally provides for a COTS-based system. JR&IO 
officials added that they developed the use cases to support the design 
and development of the system rather than to communicate the detailed 
requirements to the user organizations. In this regard, officials 
representing the DIMHRS (Personnel/Pay) development and integration 
contractor stated that the use cases are providing useful information 
for designing the DIMHRS (Personnel/Pay) system.

Objective 1: Requirements Management: End User Acceptance:

DOD has not obtained user acceptance of detailed requirements. 
According to SEI, users' needs and expectations must be used in 
defining requirements. Furthermore, according to our guidance, when 
business process changes are planned, users' needs and expectations 
must be addressed, or users may not accept change, which can jeopardize 
the effort. [NOTE 12] One way to ensure and demonstrate user acceptance 
of requirements is to obtain sign-off on the requirements by authorized 
end-user representatives.

To its credit, JR&IO obtained the user organizations' formal acceptance 
of the ORD. However, the process used to define the detailed 
requirements (specifically, the use cases) has not resulted in user 
acceptance.

* End-user representatives stated that their involvement in the 
definition of the use cases was limited.

* End users' comments on the use cases were not fully resolved.

* End users are not being asked to approve the detailed requirements.

Each of these three issues is discussed in greater detail below.

First, JR&IO developed the use cases with the help of contractors and 
representative end users (personnel and payroll specialists) from the 
end-users' stakeholder organizations. However, according to these 
representatives, their role in defining the use cases was limited. They 
stated that they principally performed research, identified references, 
and explained how their legacy environments currently process personnel 
and pay transactions and that they had limited influence in deciding 
the content of the use cases.

In response, JR&IO officials stated that many of the user 
representatives were lower-level personnel who were not empowered to 
represent their components in making decisions about requirements.

Second, to obtain users' comments on the use cases, JR&IO provided the 
end-users' organizations with the use cases and other documentation for 
comment, but this process did not resolve all of the comments.

* An initial set of use cases was reviewed by hundreds of individual 
end users (personnel and payroll specialists), resulting in thousands 
of comments.

* Around October 2003, JR&IO provided a second set of use cases to the 
end users (as well as to the development and integration contractor), 
including modifications to reflect changes suggested by users based on 
their first set of comments. The end users provided about 7,000 
comments on the second set of use cases.

* In March 2004, JR&IO established a baseline version of the use cases 
and provided this version to the development and integration contractor 
in order to meet the contractor's schedule. At that time, JR&IO had 
concurred with about 400 of the 7,000 comments and modified the use 
cases in response, but it had not completed its analysis of all 7,000 
comments.

* JR&IO then established a change control process for making further 
modifications to the baseline use cases. According to JR&IO officials, 
as of the end of October 2004, 703 change requests had been submitted: 
163 had been approved, 48 had been disapproved, and the remaining 492 
were still under review.

* According to end-user representatives from each of the services, the 
use cases were difficult to understand because they were shared in a 
piecemeal fashion and did not include sufficient detail. Furthermore, 
they said that JR&IO responses to comments were generally brief and 
often did not provide sufficient explanation-for example, "The Business 
Rule captures this requirement. Action: No change required..." and "The 
comment is out of scope. Action: No action required." As a result, the 
end-user representatives stated that they often did not understand the 
reasoning behind the decisions.

* JR&IO officials stated that users might have had difficulty with 
understanding the use cases because they were defined in terms of what 
processes the system would perform as opposed to how the processes 
would be performed. This approach is consistent with best practices (as 
discussed later in this briefing) and with JR&IO's stated intention to 
discourage the definition of processes in terms of existing systems and 
processes, as well as to allow the development of reengineered joint 
processes using the native capabilities of the COTS software to the 
maximum extent. However, best practices also require that explanations 
of business rules and their rationale be complete and understandable by 
end users.

* JR&IO officials further stated that many of the end users' comments 
were not substantive (e.g., a minor change needed in the citation of a 
regulation); many were duplicative, and some addressed issues outside 
of personnel/pay functionality, such as training and manpower. In 
addition, most were not prioritized.

Third, JR&IO does not intend to gain formal agreement on the detailed 
requirements from the end-users' organizations, although it did obtain 
such agreement on the ORD. JR&IO officials stated that gaining formal 
agreement from some of the users' organizations would delay the program 
and be impractical because of some user organizations allegiance to 
their legacy systems and processes.

We observed this challenge in a previous review, where we stated that 
DOD's organizational structure and embedded culture work against 
efforts to modernize business processes and implement corporate 
information systems such as DIMHRS (Personnel/Pay) across component 
lines. [NOTE 13]

Our prior work has also shown that proactively preparing end users for 
the business process and role changes embedded in COTS products is an 
acquisition management best practice. [NOTE 14]

Nevertheless, not attempting to obtain agreement on DIMHRS (Personnel/
Pay) requirements increases the risk that users will not accept and use 
the developed and deployed system, and that later system rework will be 
required to make it function as intended and achieve stated military 
human capital management outcomes.

Officials representing the DIMHRS offices are not in full agreement 
with JR&IO officials on the state of the requirements, as the following 
slides show.

Officials representing each of the DIMHRS management offices (Army, Air 
Force, Navy, Marine Corps, and DFAS) stated that their organizations 
are not currently willing to sign off on the DIMHRS (Personnel/Pay) 
detailed requirements as being sufficient to meet their organizations' 
military personnel and pay needs. These officials stated that they do 
not yet know what the gaps are between the functionality provided by 
their current systems and the functionality to be provided by DIMHRS 
(Personnel/Pay).

* Officials representing the Army's DIMHRS office stated that they do 
not yet know whether the requirements are adequate to enable the Army 
to replace a number of its existing systems with DIMHRS (Personnel/
Pay).

* Officials representing the Air Force's DIMHRS office stated that the 
requirements are defined at a very high level and are subject to 
interpretation, and as a result, they are unable to determine whether 
DIMHRS (Personnel/Pay) will meet all the Air Force's requirements.

* Officials representing the Navy's DIMHRS office stated that their 
concerns about the adequacy of the detailed requirements relate 
principally to the issue of not knowing which of the functions provided 
by the Navy's legacy systems will not be provided by DIMHRS (Personnel/
Pay).

* Officials representing the Marine Corps' DIMHRS office stated that 
they do not believe that the requirements adequately address service 
specificity or the automation of manual processes. These officials 
stated that the Marine Corps has not been able to determine what 
functionality the system contains versus the functionality contained in 
its legacy systems. They stated that this information is needed to 
enable them to modify legacy systems to ensure that needed functions 
continue to be provided to service members until these functions are 
incorporated into DIMHRS (Personnel/Pay).

* Officials representing DFAS's DIMHRS office stated that they do not 
believe the requirements adequately address a number of pay, 
accounting, and personnel issues.

According to JR&IO officials:

* DIMHRS (Personnel/Pay) will provide all the functionality provided by 
the services' and DFAS' legacy personnel and pay systems. JR&IO stated 
that the defined requirements provide enough information to determine 
what the system will do but acknowledged that understanding exactly how 
the system would perform its functions was not possible until the 
system was fully designed.

* Owners of end users' legacy systems are generally not supportive of 
DIMHRS (Personnel/Pay) because they want to preserve their autonomy in 
the development and control of their own systems.

* Support for the system by the services' executives is mixed. For 
example, JR&IO said that (1) Army executives are committed to 
implementing and using the DIMHRS (Personnel/Pay) system because they 
believe it will address many problems that the Army currently faces, 
(2) Air Force officials are generally supportive of the system but say 
they do not yet know whether the system will meet all their needs, and 
(3) Navy and Marine Corps executives are not as supportive because they 
are not fully convinced that DIMHRS (Personnel/Pay) will be an 
improvement over their existing systems.

* A number of actions have been taken to reduce the risk that users 
will not accept the system, including conducting numerous focus groups, 
workshops, demonstrations, and presentations explaining how the DIMHRS 
(Personnel/Pay) system could address DOD's existing personnel/pay 
problems.

Although JPMO officials told us that a consensus of the users' 
organizations agreed with the decision to accept the contractor's 
design of DIMHRS (Personnel/Pay) in November 2004, they submitted 391 
comments and issues on the design. JPMO officials stated that they 
expect to resolve these comments and issues by the end of February 
2005.

Objective 2: Program Management Structure and Processes:

DOD does not have a well-integrated management structure for DIMHRS 
(Personnel/Pay) and is not following all relevant supporting 
acquisition management processes. In particular,

* program responsibility, accountability, and authority are diffused;

* the system has not been defined and designed according to a DOD-wide 
integrated enterprise architecture; [NOTE 15]

* program stakeholders' activities have not been managed according to a 
master plan/schedule that integrates all stakeholder activities; and:

* the program is following some, but not all, best practices associated 
with acquiring business systems based on commercially available 
software.

Without a well-integrated approach and effective processes for managing 
a program that is intended to be an integrated solution, DOD has 
increased the risk that the program will not meet cost, schedule, 
capability, and outcome goals.

GAO Objective 2: Program Management Structure and Processes: 
Governance:

Program responsibility, accountability, and authority are diffused. 
Research shows that leading organizations ensure that programs are 
structured to ensure that assigned authorities, responsibilities, and 
accountabilities are clear and aligned under the continuous leadership 
and direction of a single entity. For DIMHRS (Personnel/Pay), these 
areas are spread among three key stakeholder groups whose respective 
chains of command do not meet at any point below the Secretary and 
Deputy Secretary of Defense level.

* Responsibility for requirements definition rests with JR&IO, which is 
accountable through one chain of command.

* Responsibility for system acquisition rests with JPMO, which is 
accountable through another chain of command.

* Responsibility for preparing for transition to DIMHRS (Personnel/Pay) 
rests with the end users' organizations-11 major DOD components 
reporting through five different chains of command.

The organization chart on the next slide shows the chain of command and 
the coordination relationships among the primary DIMHRS (Personnel/Pay) 
stakeholder groups.

[See PDF for image]

[End of figure]

As the chart also shows, the services and DFAS have DIMHRS (Personnel/
Pay) management offices to assist JR&IO and JPMO and to represent their 
respective end-user communities (the pay and personnel specialists). In 
addition, various coordination and advisory bodies have been 
established.

The three primary stakeholder groups (JR&IO, JPMO, and the end users) 
are accountable to three different groups of executives.

* JR&IO is ultimately accountable to the Under Secretary for P&R, who 
is the department's Principal Staff Assistant (PSA) [NOTE 16] for 
personnel and compensation and is responsible for oversight of the 
DIMHRS (Personnel/Pay) program from a functional perspective.

* JPMO is ultimately accountable to the Assistant Secretary of Defense 
(NII), who is the DIMHRS (Personnel/Pay) program's Milestone Decision 
Authority (i.e., responsible for authorizing the program to move from 
one acquisition phase into the next) and is responsible for the program 
from an acquisition perspective. [NOTE 17]

* The end users are ultimately accountable to the Offices of the 
Secretaries of the Army, Air Force, and Navy (in coordination with the 
Commandant of the Marine Corps) and the DOD Comptroller, who have 
ultimate responsibility for implementing and using DIMHRS (Personnel/
Pay).

Under this management structure, no single DOD entity is positioned to 
exercise continuous program leadership and direction (i.e., has 
responsibility, accountability, and authority (including budget 
authority) over the entire DIMHRS (Personnel/Pay) program). This is 
consistent with our observation in a previous review that DOD's 
organizational structure and embedded culture have not adequately 
accommodated an integrated, departmentwide approach to joint systems. 
[NOTE 18]

Although no stakeholder organization has continuous programwide 
oversight purview and visibility, the DIMHRS (Personnel/Pay) Executive 
Steering Committee is made up of representatives of each of the 
entities that has ultimate responsibility for the program.

According to DOD, the committee monitors the program, resolves issues 
that are brought before it, and advises the Under Secretary (P&R). 
[NOTE 19] It meets quarterly or when assembled by the chair-the Deputy 
Under Secretary of Defense for Program Integration-who reports to the 
Under Secretary (P&R).

According to JR&IO officials, the diffusion of program accountability, 
responsibility, and authority will be reduced in fiscal year 2005, when 
funding for both JR&IO and JPMO will be consolidated and centrally 
managed by JR&IO. [NOTE 20] However, the end-user organizations will 
continue to separately control their respective funds.

* For example, officials at the Army's DIMHRS (Personnel/Pay) 
management office estimated that the Army's DIMHRS (Personnel/Pay) 
funding needs will range from $27 million to $43 million a year, but 
they said that the Army is unlikely to fund the program at that level 
because of other priorities.

Without a DOD-wide integrated governance structure that vests an 
executive-level organization or entity representing the interests of 
all program stakeholders with responsibility, accountability, and 
authority for a joint or integrated program like DIMHRS (Personnel/
Pay), DOD runs the risk that the program will not produce an integrated 
set of outcomes.

GAO Objective 2: Program Management Structure and Processes: Enterprise 
Architecture:

The DIMHRS (Personnel/Pay) system has not been aligned with a DOD-wide 
integrated enterprise architecture. An enterprise architecture is a 
blueprint for guiding and constraining the definition and 
implementation of programs in a way that supports strategic plans and 
promotes integration, interoperability, and optimization of mission 
performance. This blueprint serves as the common frame of reference to 
inform program operational and technological decisionmaking relative 
to, for example, what functions will be performed where, by whom, and 
using what information. [NOTE 21] Successful organizations have used 
architectures to effectively transform business operations and 
supporting systems. Moreover, the National Defense Authorization Act 
for Fiscal Year 2003 directed DOD to develop and implement a 
departmentwide Business Enterprise Architecture (BEA) to guide and 
constrain its business modernization efforts. [NOTE 22]

Acquiring and implementing DIMHRS (Personnel/Pay) without an enterprise 
architecture increases the risk that DOD will make a substantial 
investment in system solutions that will not be consistent with its 
eventual blueprint for business operational and technological change.

Recognizing this, the Deputy Secretary of Defense issued a memorandum 
in March 2004 requiring the development and implementation of 
architectures for each of DOD's six business domains, including the 
human resources domain. These business domains, according to the 
department's modernization program, are delegated the "authority, 
responsibility, and accountability ... for their respective business 
areas" for implementing business transformation, [NOTE 23] including 
the following:

* "Leading the business transformation within the Domain."

* "Managing its respective [system investment] portfolio to ensure 
implementation of and compliance with the Business Enterprise 
Architecture (BEA) and transition plan."

* "Assisting in the extension of the [departmentwide] BEA" for the 
domain.

The Under Secretary (P&R), who is the domain owner for the human 
resources domain, assigned JR&IO the responsibility for extending the 
BEA for the human resources domain. According to JR&IO officials, the 
development of the human resources portion of the BEA is being done 
concurrently with the acquisition and deployment of DIMHRS (Personnel/
Pay).

Recognizing the importance of managing the concurrency of such 
activities and ensuring that DOD's ongoing investments are pursued 
within the context of its evolving BEA, the National Defense 
Authorization Act for Fiscal Year 2003 also required that system 
improvements with proposed obligations of funds greater than $1 million 
be reviewed to determine if they are consistent with the BEA.

To satisfy this requirement, JPMO officials presented the DOD Office of 
the Comptroller, which is developing the BEA, with information on 
DIMHRS (Personnel/Pay) compliance with version 1.0 of the BEA in April 
2003. However, according to our review of the information used by JPMO 
in April 2003 to obtain an architectural compliance determination, this 
information did not include a documented, verifiable analysis 
demonstrating such compliance. In the absence of such analysis, the 
JPMO program manager instead made commitments that DIMHRS (Personnel/
Pay) would be consistent with the architecture. On the basis of this 
commitment, the DOD Comptroller certified in April 2003 that DIMHRS 
(Personnel/Pay) is consistent with the BEA. Later, JPMO included in the 
DIMHRS (Personnel/Pay) contract a requirement that the systems 
specification be compatible with the emerging BEA.

According to JR&IO officials, the April 2003 architectural 
certification is preliminary, and further certification is needed. They 
stated that DIMHRS (Personnel/Pay) will undergo another certification 
before the system deployment decision.

By this time, however, lengthy and costly DIMHRS (Personnel/Pay) design 
and development work will be completed. The real value in having and 
using an architecture is knowing, at the time that extensive system 
definition, design, and development are occurring, what the larger 
blueprint for the enterprise is, so that definition, design, and 
development can be guided and constrained by this frame of reference. 
Aligning to the architecture after the system is designed could also 
require expensive system rework to address any inconsistencies with the 
architecture.

The absence of verifiable analysis of compliance was in part due to the 
state of completeness of BEA version 1.0. As we reported in September 
2003, this version was missing key content, including sufficient depth 
and detail to effectively guide and constrain system investments. [NOTE 
24] Since then, DOD has issued other versions of the BEA. However, we 
reported in May 2004 that version 2.0 of the BEA still did not include 
many of the key elements of a well-defined architecture. [NOTE 25] For 
example, the "to be" environment did not provide sufficient descriptive 
content related to future business operations and supporting technology 
to permit effective acquisition and implementation of system solutions 
and associated operational change. DOD since issued versions 2.2 and 
2.2.1 in July and August 2004, respectively.

GAO Objective 2: Program Management Structure and Processes: Integrated 
Master Plan/Schedule:

DIMHRS (Personnel/Pay) program stakeholder activities are not being 
managed according to an integrated master plan/schedule. IEEE standards 
state that a master plan/schedule should be prepared and updated 
throughout the system's life cycle to establish key events, activities, 
and tasks across the program, including dependencies and relationships 
among them. A properly designed master plan/schedule should allow for 
the proper scheduling and sequencing of activities and tasks, 
allocation of resources, preparation of budgets, assignment of 
personnel, and criteria for measuring progress.

In contrast, the DIMHRS (Personnel/Pay) program plan/schedule is based 
on the contractor's and JPMO's activities and does not include all the 
activities that end-user organizations must perform to prepare for 
DIMHRS (Personnel/Pay), such as:

* cleaning legacy data, preparing legacy systems for interfacing, and 
acquiring and installing the necessary infrastructure;

* making organizational changes and business process improvements 
associated with DIMHRS (Personnel/Pay), such as revising the duties 
that are now performed by pay specialists and personnel specialists; 
and:

* making revisions to their regulations to ensure consistency with the 
reengineered business rules designed into DIMHRS (Personnel/Pay) that 
differ from existing DOD or service rules and policies (e.g., those 
noted earlier in this briefing as being in accordance with "P&R 
guidance" in the use cases).

With a plan/schedule that focuses on the contractor's and JPMO's 
activities and does not extend to all DOD program stakeholders' 
activities, the risk increases that key and dependent events, 
activities, and tasks will not be performed as needed, which in turn 
increases the risk of schedule slippage and program goal shortfalls.

GAO Objective 2: Program Management Structure and Processes: Policies 
and Guidance:

Some, but not all, key practices associated with acquiring COTS-based 
business systems are being followed. According to SEI, the quality of 
the processes and practices followed in acquiring software-intensive 
systems greatly influences the quality of the systems produced. 
Moreover, acquiring custom-developed system solutions is sufficiently 
different from acquiring COTS-based systems that adherence to certain 
practices unique to the latter is key to their success. We recently 
reported on key acquisition management best practices associated with 
COTS-based systems, [NOTE 26] including:

* discouraging the modification of commercial components and allowing 
it only if justified by a thorough analysis of the life-cycle costs and 
benefits,

* explicitly evaluating systems integration contractors on their 
ability to implement commercial components,

* centrally controlling modification or upgrades to deployed versions 
of system components and precluding users from making unilateral 
changes to releases,

Objective 2: Program Management Structure and Processes: GAO Policies 
and Guidance:

* ensuring that plans explicitly provide for preparing users for the 
impact that the business processes embedded in the commercial 
components will have on their respective roles and responsibilities,

* proactively managing the introduction and adoption of changes to how 
users will be expected to use the system to execute their jobs, and:

* ensuring that project plans explicitly provide for the necessary time 
and resources for integrating commercial components with legacy 
systems.

To its credit, DOD is following the first three of these practices for 
DIMHRS (Personnel/Pay), but it is not following the last three. For 
example, program officials told us that they expected the contractor to 
base the system design on the high-level requirements defined in the 
ORD as a way to maximize the contractor's ability to leverage the COTS 
product. Furthermore, the contract includes award fees that give the 
contractor incentives to, among other things, minimize customization of 
the COTS software.

However, DOD does not have an integrated program plan/schedule that 
provides for end-user organization activities that are associated with 
preparing users for the operational and role-based changes that the 
system will introduce, such as the need to revise the duties that are 
now performed by pay specialists and personnel specialists. 
Furthermore, DOD's program plans do not recognize the end-user 
organizations' time and resource needs associated with integrating 
DIMHRS with their respective legacy systems, and JPMO is not actively 
managing these end-user operational changes. Although JR&IO officials 
told us that some planning has occurred to position end users for 
DIMHRS (Personnel/Pay) changes, officials representing the DIMHRS 
offices in the services and DFAS stated that these plans do not 
adequately address the above areas.

By not following all relevant best practices associated with acquiring 
COTS-based systems, DOD is increasing the risk that DIMHRS (Personnel/
Pay) will not be successfully implemented and effectively used.

Conclusions:

The importance of DIMHRS (Personnel/Pay) to DOD's ability to manage 
military personnel and pay services demands that the department employ 
effective processes and structures in defining, designing, developing, 
and deploying the system to maximize its chances of success. For DIMHRS 
(Personnel/Pay), however, DOD did not initially perform important 
requirements development steps, and the detailed system requirements 
are missing important content. DOD has begun to remedy these omissions 
by taking actions such as tracing among requirements documents and 
system design documents to ensure alignment, but user organizations' 
acceptance of requirements has not occurred. Moreover, although DIMHRS 
(Personnel/Pay) is to be an integrated system, it is not being governed 
by integrated tools and approaches, such as an integrated program 
management structure, integrated DOD business enterprise architecture, 
and an integrated master plan/schedule.

Furthermore, while DOD is appropriately attempting to maximize the use 
of COTS products in building DIMHRS (Personnel/Pay) and is following 
some best practices for developing COTS-based systems, others are not 
being followed.

The absence of the full complement of effective processes and 
structures related to each of these areas can be attributed to a number 
of causes, including the program's overly schedule-driven approach and 
the difficulty of overcoming DOD's long-standing cultural resistance to 
departmentwide solutions. Effectively addressing these shortcomings is 
essential because they introduce unnecessary risks that reduce the 
chances of accomplishing DIMHRS (Personnel/Pay) goals on time and 
within budget.

It is critical that DOD carefully consider the risks caused by each of 
these areas of concern and that it appropriately strengthen its 
management processes, structures, and plans to effectively minimize 
these risks. To do less undermines the program's chances of timely and 
successful completion.

Recommendations:

To assist DOD in strengthening its program management processes, 
structures, and plans and thereby increase its chances of successfully 
delivering DIMHRS (Personnel/Pay), we recommend that the Secretary of 
Defense direct the Assistant Secretary (NII), the Under Secretary 
(P&R), and the Under Secretary (Comptroller), in collaboration with the 
leadership of the military services and DFAS, to jointly ensure an 
integrated, coordinated, and risk-based approach to all DIMHRS 
(Personnel/Pay) definition, design, development, and deployment 
activities. At a minimum, this should include:

* ensuring that joint system requirements are complete and correct and 
are acceptable to users' organizations;

* establishing a DOD-wide integrated governance structure for DIMHRS 
that vests an executive-level organization or entity representing the 
interests of all program stakeholders, including JR&IO, JPMO, the 
services, and DFAS, with responsibility, accountability, and authority 
for the entire DIMHRS (Personnel/Pay) program, and ensuring that all 
stakeholder interests and positions are appropriately heard and 
considered during program reviews and before key program decisions;

* ensuring that the degree of consistency between DIMHRS and the 
evolving DOD-wide business enterprise architecture is continuously 
analyzed, and that material inconsistencies between the two, both 
potential and actual, are disclosed at all program reviews and decision 
points and in budget submissions for the program, along with any 
associated system risks and steps to mitigate these risks;

* developing and implementing a DOD-wide, integrated master plan/
schedule of activities that extends to all DOD program stakeholders;

* ensuring that all relevant acquisition management best practices 
associated with COTS-based systems are appropriately followed; and:

* adopting a more event-driven, risk-based approach to managing DIMHRS 
that adequately considers factors other than the contract schedule.

Attachment I: Objectives, Scope, and Methodology:

Our objectives were to determine:

1. whether the Department of Defense (DOD) has effective management 
processes in place for managing the definition of the requirements for 
the Defense Integrated Military Human Resources System (DIMHRS 
(Personnel/Pay)) and:

2. whether DOD has established an integrated program management 
structure for DIMHRS (Personnel/Pay) and is following effective 
processes for acquiring a system based on commercial software 
components.

To determine industry and government best practices and regulations for 
effective requirements definition and management, we evaluated criteria 
from the Capability Maturity Models (CMM) developed by Carnegie Mellon 
University's Software Engineering Institute and standards developed by 
the Institute of Electrical and Electronics Engineers (IEEE), as well 
as the DOD 5000 series and other applicable DOD policies and 
regulations, federal accounting standards, and prior GAO reports and 
best practices guidance.

To evaluate requirements management efforts, we:

* interviewed officials with the Joint Requirements and Integration 
Office (JR&IO), which is responsible for requirements definition; 
officials from the Joint Program Management Office (JPMO), which is 
responsible for system acquisition and deployment; officials in each of 
the DIMHRS offices for each of the four services (Army, Air Force, 
Navy, and Marine Corps) and the Defense Finance and Accounting Service 
(DFAS); officials in DOD's Business Management Modernization Program, 
which is responsible for Business Enterprise Architecture oversight; 
and officials from supporting contractors;

* attended requirements definition, review, and change meetings, 
including meetings of the Joint Integration Group and the Functional 
Requirements Review Board, as well as a critical design review;

* obtained written responses to questions from these various offices 
and other entities;

* reviewed DIMHRS (Personnel/Pay) planning documentation, milestone 
review materials, requirements and design documentation, contracts, and 
briefings;

* discussed the use cases with selected personnel and pay specialists 
and reviewed end users' written comments to JR&IO on the use cases;

* estimated the extent of several problems by evaluating the clarity 
and understandability of use cases in a probability sample [NOTE 27] of 
20 of 284 pay use cases and 20 of 140 personnel use cases; and:

* compared requirements management activities with relevant industry 
and government guidance and requirements, including CMM and IEEE, and 
the DOD 5000 series and Joint Chiefs of Staff regulations.

To evaluate program management structures and processes, we:

* interviewed officials from JR&IO, JPMO, and the DIMHRS offices for 
each of the services and DFAS;

* analyzed DOD and DIMHRS (Personnel/Pay) program management and 
process management documentation and activities, including charters, 
process descriptions, budgets, and program plans, etc.; and:

* reviewed relevant analysis supporting program decisions, such as 
economic justification and architectural alignment.

We determined that the data used in this report are generally reliable 
for the purposes for which we used them. For DOD-provided data, we have 
made appropriate attribution to indicate the data's source.

We performed our work at DOD headquarters; JR&IO, in the Washington, 
D.C., area; JPMO in New Orleans, Louisiana; the Army's, Navy's, Air 
Force's, and Marine Corps' DIMHRS offices; and DFAS's offices in the 
Washington, D.C., area.

This work was performed from January through November 2004 in 
accordance with generally accepted government auditing standards.

NOTES:

[1] GAO, Financial Management. Defense's System for Army Military 
Payroll Is Unreliable, GAO/AIMD-93-32 (Washington, D.C.: Sept. 30, 
1993).

[2] GAO, Military Pay. Army Reserve Soldiers Mobilized to Active Duty 
Experienced Significant Pay Problems, GAO-04-911 (Washington, D. C.: 
Aug. 20, 2004), and Military Pay. Army National Guard Personnel 
Mobilized to Active Duty Experienced Significant Pay Problems, GAO-04-
89 (Washington, D.C.: Nov. 13, 2003).

[3] At that time, this executive's title was Assistant Secretary of 
Defense for Command, Control, Communications, and Intelligence.

[4] According to DOD, mission performance improvements, not cost 
savings, are the rationale for DIMHRS (Personnel/Pay).

[5] See, for example, Karl E. Wiegers, Software Requirements (1999), 
p.15, and GAO, DOD Business Systems Modernization: Billions Continue to 
Be Invested with Inadequate Management Oversight and Accountability, 
GAO-04-615 (Washington, D.C.: May 27, 2004).

[6] JFMIP is a cooperative undertaking of the Treasury Department, 
Office of Management and Budget, Office of Personnel Management, GAO, 
and others to improve financial management practices in government. The 
PMO, managed by the Executive Director of the JFMIP, using funds 
provided by the Chief Financial Officers Council agencies, is 
responsible for the testing and certification of COTS core financial 
systems for use by federal agencies and coordinating the development 
and publication of functional requirements for financial management 
systems.

[7] JFMIP, "JFMIP Human Resources & Payroll Systems Requirements," 
JFMIP SR-99-5 (April 1999).

[8] A clean, or unqualified, opinion is given when an auditor deems the 
financial statements to be accurate and complete, with no qualifying 
statements.

[9] JFMIP SR-99-5. According to the JFMIP PMO, these requirements are 
not applicable to noncivilian human resources and payroll systems, such 
as military personnel and foreign national systems. However, DOD 
administratively requires compliance with these JFMIP PMO requirements 
in DFAS's Guide to Federal Requirements for Financial Management 
Systems, DFAS 7900.4-G Version 4.1.1 (December 2002). The DFAS Guide 
also states that "Users must exercise their own knowledge and judgment 
of the differences between military and civilian personnel/payroll 
systems in applying these requirements to the different systems."

[10] GAO, Financial Management. Improved Financial Systems Are Key to 
FFMIA Compliance, GAO-05-20 (Washington, D.C.: Oct. 1, 2004), and 
Business Modernization: NASA's Integrated Financial Management Program 
Does Not Fully Address Agency's External Reporting Issues, GAO-04-151 
(Washington, D.C.: Nov. 21, 2003).

[11] This estimate is a weighted average of the sample results for the 
two categories of use cases shown in the table on slide 34. A weighted 
average is used because the population of pay use cases was sampled at 
a rate different from the population of personnel use cases.

[12] GAO, Business Process Reengineering Assessment Guide Version 3, 
GAO/AIMD-10.1.15 (Washington, D.C.: May 1997).

[13] GAO, Defense IRM. Poor Implementation of Management Controls Has 
Put Migration Strategy at Risk, GAO/AIMD-98-5 (Washington, D.C.: Oct. 
20, 1997).

[14] GAO, Information Technology. DOD's Acquisition Policies and 
Guidance Need to Incorporate Additional Best Practices and Controls, 
GAO-04-722 (Washington, D.C.: July 30, 2004).

[15] JPMO officials stated that the system has not been defined and 
designed according to a DOD-wide integrated enterprise architecture 
because the enterprise architecture is not complete.

[16] The PSA is the executive-level manager responsible for the 
management of defined functions within DOD.

[17] According to JR&IO officials, the separation of the functional and 
acquisition lines of authority is a normal DOD practice.

[18] See, for example, GAO/AIMD-98-5.

[19] DIMHRS (Pers/Pay) Report to Congress (June 2002).

[20] According to JR&IO officials, DOD requested consolidated JR&IO and 
JPMO funding for DIMHRS (Personnel/Pay) for fiscal year 2005.

[21] GAO, Information Technology. A Framework for Assessing and 
Improving Enterprise Architecture Management (Version 1.1), Executive 
Guide, GAO-03-584G (Washington, D.C.: April 2003).

[22] Bob Stump National Defense Authorization Act for Fiscal Year 2003, 
Pub. L. No. 107-314, section 1004, 116 Stat. 2458, 2629-2631 (Dec. 2, 
2002).

[23] DOD Business Management Modernization Program, Governance 
Approach. http://www.dod.mil/comptroller/bmmp/pages/govern_dod.html.

[24] GAO, DOD Business Systems Modernization: Important Progress Made 
to Develop Business Enterprise Architecture, but Much Work Remains, 
GAO-03-1018 (Washington, D.C.: Sept. 19, 2003).

[25] GAO, DOD Business Systems Modernization: Limited Progress in 
Development of Business Enterprise Architecture and Oversight of 
Information Technology Investments, GAO-04-731R (Washington, D.C., May 
17, 2004).

[26] GAO-04-722.

[27] A different probability sample of use cases could produce 
different estimates. In this briefing, we present estimates along with 
the 95 percent confidence intervals for these estimates. This means 
that there is a 95 percent probability that the actual value for the 
entire population is within the range defined by the confidence 
interval. In other words, if 100 different samples were taken, in 95 of 
those 100 samples, the actual value for the entire population would be 
within the range defined by the confidence interval, and in 5 of those 
100 samples, the value would be either higher or lower than the range 
defined by the confidence interval. 

[End of section]

Appendix II: Comments from the Department of Defense:

Note: GAO's comments supplementing those in the report's text appear at 
the end of this appendix.

UNDER SECRETARY OF DEFENSE: 
4000 DEFENSE PENTAGON: 
WASHINGTON, D.C. 20301-4000:

PERSONNEL AND READINESS:

JAN 25 2000:

Mr. Randolph C. Hite: 
Director:
Information Technology Architecture and Systems Issues: 
U.S. Government Accountability Office:
Washington, DC 20548:

Dear Mr. Hite:

This is the Department of Defense (DoD) response to the GAO draft 
report, GAO-05-189, "DOD SYSTEMS MODERNIZATION: Additional 
Improvements Needed to Management of Integrated Military Human Capital 
Program," December 14, 2004 (GAO Code 350623). A complete response to 
each point of the recommendations is enclosed.

It is my understanding that the GAO team was tasked to compare the 
process we are using for documentation of requirements with industry 
standards for best practices rather than to compare the process with 
other DoD or Federal practices. While I fully understand the interest 
in using these standards, this has led to some incongruity in the 
recommendations.

The draft report notes that best practices require that we discourage 
modification of commercial components. It also notes that any 
modifications should be justified by thorough analysis of the life-
cycle costs and benefits and by centrally controlling modifications of 
upgrades. We fully agree and have put in place a process that does just 
that, allowing modifications to the Commercial-Off-The-Shelf (COTS) 
products only if they are mission essential or cost beneficial.

The draft report also concludes that we have not done enough to ensure 
that all stakeholders have had full input to the requirements. In fact, 
as described in the enclosure, the process has allowed all of the 
requirements to be fully vetted throughout the Department. However, 
acting on all comments from all sources would have resulted in 
excessive modification of the COTS product (in violation of best 
practices as described in the draft report), so we carefully put a 
process in place that would allow us to accept comments and inputs that 
did not violate our principles (use of common business rules and data 
and no modification to COTS unless essential) while screening out those 
that did.

We also have in place a robust issue resolution process that ensures 
that all issues and views will be heard. More discussion of these and 
other points is included in the enclosure.

The DoD partially non-concurs with the recommendations in the draft 
report. For the most part, we agree with the thrust of the 
recommendations, but maintain that we are already following best 
practices to the extent that they are practicable in the Department. 
The process used for this program is innovative and ground-breaking for 
the Department. The coordination process went far beyond what is 
required by DoD regulations and the resulting set of requirements 
documentation far exceeds what has been available for any other system 
development effort. The design of the system is fully traceable to each 
element of the detailed requirements and all aspects of the system will 
be maintained using this same process to ensure that the system remains 
well-documented and future changes to policy and requirements can be 
easily incorporated.

We appreciate the opportunity to respond to the draft report. My point 
of contact is Ms. Norma St. Claire. She may be reached by email: 
norma.stclaire@osd.pentagon.mil or by telephone at (703) 696-8710.

Sincerely, 

Signed by: 

David S. C. Chu:

Enclosure: As Stated:

GAO CODE 350623/GAO-05-189:

"DOD SYSTEMS MODERNIZATION: ADDITIONAL IMPROVEMENTS NEEDED TO 
MANAGEMENT OF INTEGRATED MILITARY HUMAN CAPITAL PROGRAM":

DEPARTMENT OF DEFENSE COMMENTS TO THE RECOMMENDATIONS:

RECOMMENDATIONS: The GAO recommended that the Secretary of Defense 
direct the Assistant Secretary (NII), the Under Secretary (P&R), and 
the Under Secretary (Comptroller), in collaboration with the leadership 
of the Military Services and DFAS, to jointly ensure an integrated, 
coordinated, and risk-based approach to all DIMHRS (Personnel/Pay)* 
definition, design, development, and deployment activities. At minimum, 
this should include:

* ensuring that joint system requirements are complete and correct and 
are acceptable to user organizations;

* establishing a DoD-wide integrated governance structure for DIMHRS 
that vests an executive-level organization or entity representing the 
interests of all program stakeholders, including JR&IO, JPMO, the 
Services, and DFAS, with responsibility, accountability, and authority 
for the entire DIMHRS (Personnel/Pay) program, and ensuring that all 
stakeholder interests and positions are appropriately heard and 
considered during program reviews and before key program decisions;

* ensuring that the degree of consistency between DIMHRS and the 
evolving DoD-wide business enterprise architecture is continuously 
analyzed, and that material inconsistencies between the two, both 
potential and actual, are disclosed at all program reviews and 
decision points and in program budget submissions, along with any 
associated system risks and steps to mitigate these risks;

* developing and implementing a DoD-wide, integrated master plan/
schedule of activities that extends to all DoD program stakeholders;

* ensuring that all relevant acquisition management best practices 
associated with COTS-based systems are appropriately followed; and:

* adopting a more evidence-driven, risk-based approach to managing 
DIMHRS that adequately considers factors other than the contract 
schedule.

*DIMHRS (Personnel/Pay) is the Defense Integrated Military Human 
Resources System for Personnel and Pay. It is often referred to 
throughout the GAO report and this response as DIMHRS.

DOD RESPONSE:

A summary of the DOD responses to the recommendation points is as 
follows:

1 - To the recommendation of: ensuring that joint system requirements 
are complete and correct and are acceptable to user organizations.

DOD RESPONSE: Partially Concur with Comment. The DIMHRS (Pers/Pay) 
requirements are complete and correct to the extent that any 
documentation this massive can be correct. The design is fully 
traceable to the requirements, including the applicable financial 
system and accounting requirements. The Joint Requirements Oversight 
Council (JROC) validation of the Operational Requirements Document 
(ORD) is all that is required by DoD regulation. The rest of the 
documentation of requirements for DIMHRS was an innovative and 
unprecedented effort to ensure full traceability from documentation of 
requirements through design, development, deployment, and maintenance.

The Department has taken great pains to ensure that the system 
requirements are complete and correct and acceptable to user 
organizations.

The functional community had over a year to provide two rounds of 
comments on the detailed documentation before it was baselined.

All comments and responses were fully documented tracked, analyzed, and 
maintained. All issues were vetted.

There are forums for the discussion and resolution of any issues. The 
Functional Requirements Review Board (FRRB) meets monthly to review 
system requirements change requests. If there is disagreement on the 
disposition of any request, that becomes an issue in a robust issue 
resolution process.

All issues are fully documented and vetted through the issue resolution 
process. Final decisions are made by the Executive Steering Committee 
(with appeal to the Under Secretary of Defense (Personnel & Readiness) 
if anyone disagrees - -that has not happened yet).

The detailed design for DIMHRS (Pers/Pay) was presented in four 
incremental Critical Design Reviews: CDR 1-3 and Program CDR (PCDR). 
Representatives from the Services, the Defense Finance and Accounting 
Service (DFAS) and other stakeholder organizations were invited to 
participate and submit comments on the system design. Because the 
stakeholders had many questions on out-of-the-box PeopleSoft 
capabilities during CDR 1 - 3, PCDR was expanded to include a 
PeopleSoft "fit" demonstration that showed 37 life cycle scenarios from 
enlistment to separation.

Representatives from the Services, DFAS, and other DOD offices were 
active participants in all CDR reviews and were encouraged to submit 
comments or questions to the Joint Program Management Office (JPMO) and 
the Joint Requirements and Integration Office (JR&IO) for review. A 
number of issues were identified during CDR 3, which led to the 
establishment of the Design to Requirements Traceability (DRT) process 
with JR&IO.

During PCDR, participants were able to submit questions during the 
presentation via comment cards as well as on comment resolution 
matrices against design deliverables. 606 business process comments (a 
combination of design and requirements comments) and 17 interface 
comments were received and are currently being analyzed. None were 
critical.

The GAO report notes ambiguities that they found in their review of the 
detailed requirements. The detailed requirements were originally to 
have been documented after the award of the Developer/Implementer 
contract, to ensure that they were not worded in such a way as to 
inadvertently cause modification of the COTS product. The award of the 
contract was delayed, so the review with the Developer/Implementer took 
place after the development of the full set of detailed documentation. 
The review is now complete and all questions and ambiguities with the 
documentation have been fully resolved. The purpose of the detailed 
documentation was to ensure that the design of the program would be 
fully traceable to the requirements. They are not considered the 
vehicle for gaining user acceptance for the system. The Department has 
an extensive change management process in place for user acceptance. 
That process is already working with prototype demonstrations and 
briefings on the new system. As the system is developed, actual system 
modules will be incorporated into the demonstrations to ensure a wide 
understanding of the new system and new processes.

2 - To the recommendation of: establishing a DoD-wide integrated 
governance structure for DIMHRS that vests an executive-level 
organization or entity representing the interests of all program 
stakeholders, including JR&IO, JPMO, the Services, and DFAS, with 
responsibility, accountability, and authority for the entire DIMHRS 
(Personnel/Pay) program, and ensuring that all stakeholder interests 
and positions are appropriately heard and considered during program 
reviews and before key program decisions.

DOD RESPONSE: Partially Non Concur with Comment. The roles and 
responsibilities for the Executive Steering Committee are well-defined 
in documentation signed by the Deputy Secretary. All stakeholder 
interests are heard and there is a robust issue resolution process and 
a forum for all program reviews and key program decisions. A committee 
cannot have responsibility, accountability, or authority for a program. 
Additionally, there is already a single functional focal point for the 
program. The Joint Requirements and Integration Office (JR&IO) is the 
single source for all functional requirements. Representatives from all 
of the military components and from the Defense Finance and Accounting 
Service (DFAS) are assigned to JR&IO to support requirements definition 
and maintenance. The Under Secretary of Defense (Personnel and 
Readiness) has full responsibility and accountability for the program.

On charts 63 through 66, GAO outlines those best practices and notes 
that the Department is following some of them. Two of the best 
practices are:

Discouraging modification of commercial components and allowing it only 
if justified by thorough analysis of the life-cycle costs and benefits; 
and:

Centrally controlling modification of upgrades to deployed versions of 
system components and precluding users from making unilateral changes 
to releases.

The Services and other stakeholders had over a year to review and 
comment on the detailed documentation of the requirements. For the most 
part, the Service comments on the detailed requirements did not address 
critical issues in the management of personnel and pay. They primarily 
focused on the AS-IS processes that, in some cases, are a reflection of 
the limitations of legacy systems. The Department has proven many times 
over the years that replicating the AS-IS processes will not solve the 
critical problems in military personnel and pay management and that 
they will result in over modification of the COTS products. Providing 
stakeholders with veto power over requirements definition will result 
in extensive delays and extensive modification of the software.

3 - To the recommendation for: ensuring that the degree of consistency 
between DIMHRS and the evolving DoD-wide business enterprise 
architecture is continuously analyzed, and that material 
inconsistencies between the two, both potential and actual, are 
disclosed at all program reviews and decision points and in program 
budget submissions, along with any associated system risks and steps to 
mitigate these risks.

DOD RESPONSE: Partially Concur with Comment. The DIMHRS requirements 
comprise the Business Enterprise Architecture (BEA) for military 
personnel and pay. The Business Enterprise Architecture is fully 
incorporating all of the DIMHRS processes and interfaces. The Director 
of JR&IO, the primary functional executive for DIMHRS, is also the 
Human Resources Management Domain manager. DIMHRS is now, and will 
remain, fully consistent with the BEA as one of the first major systems 
developed using all of the principles of the BEA and the DoD 
transformation effort.

4 - To the recommendation for: developing and implementing a DoD-wide, 
integrated master plan/schedule of activities that extends to all DoD 
program stakeholders.

DOD RESPONSE: Concur.

5 - To the recommendation for: ensuring that all relevant acquisition 
management best practices associated with COTS-based systems are 
appropriately followed.

DOD RESPONSE: Partially Non Concur with Comment. All relevant 
acquisition management best practices associated with COTS-based 
systems are being appropriately followed. See detailed discussions 
below.

GAO Charts 63 - 64 list the key best practices:

* Discouraging modification of commercial components and allowing it 
only if justified by a thorough analysis of the life-cycle costs and 
benefits;

* Explicitly evaluating systems integration contractors on their 
ability to implement commercial components;

* Centrally controlling modification or upgrades to deployed versions 
of system components and precluding users from making unilateral 
changes to releases;

* Ensuring that plans explicitly provide for preparing users for the 
impact that the business processes embedded in the commercial 
components will have on their respective roles and responsibilities;

* Proactively managing the introduction and adoption of changes to how 
users will be expected to use the system to execute their jobs; and:

* Ensuring that project plans explicitly provide for the necessary time 
and resources for integrating commercial components with legacy 
systems:

GAO acknowledges that the DIMHRS program is following the first three 
of these. It also states that the program is not following the last 
three. In fact, the Department is also following all of the last three.

In defining requirements for DIMHRS (Pers/Pay), the Department 
documented every change that will be required for the Services in terms 
of current practices and policies and the future practices and 
policies. Every change was documented as an issue, with a full 
description of the AS-IS and the recommended TO-BE, including 
identification of any relevant regulations or laws. All of these 
changes were fully vetted through the functional community with all 
comments and final disposition fully documented. Although it is 
possible that additional things will come to light as design and 
development proceed, the program remains fully committed to continue to 
define, document, coordinate and resolve all of these issues.

The Department has put in place a proactive change management process 
to introduce and promote adoption of changes in process. The process 
was initiated in August of 2002 with Service Analysis Sessions. These 
were a series of two-week sessions with each Service, DFAS and others 
where the program provided one week of PeopleSoft training and one week 
of follow-on work using PeopleSoft out-of-the-box to address a wide 
range of military personnel and pay functions. The sessions were 
extremely successful and were followed up with numerous training and 
demonstrations projects since then. There have been numerous workshops 
with users on roles and responsibilities and the changing roles with 
the implementation of DIMHRS; the project team has worked with the 
Services one-on-one and in multi-Service groups on all of these issues; 
the team has developed the Joint Capabilities Demonstration which is a 
more sophisticated demonstration of PeopleSoft applied to military 
personnel and pay functions and activities; and presentations and 
demonstrations have been provided at numerous conferences and to senior 
executives throughout the Department. Further, Service members, using 
the provided tool set, have given demonstrations themselves to large 
groups of users throughout the Department. Consolidation of functions 
and duties has been a major topic for several focus groups. As the 
DIMHRS system is developed, actual DIMHRS modules will be added to the 
presentations to ensure that the new processes and the new system is 
promoted early and often.

The project plans have been confirmed by the Developer/Implementer and 
reviewed by an independent analyst as sufficient to do the work 
required to successfully design, develop and implement DIMHRS.

6 - To the recommendation for: adopting a more evidence-driven, risk-
based approach to managing DIMHRS that adequately considers factors 
other than the contract schedule.

DOD RESPONSE: Partially Non Concur. The Department is using an event-
driven, risk-based approach to DIMHRS. The contract is not the driver 
of the schedule. The Department understands the risks. If the system is 
not working, it will not be implemented. The schedule is event driven - 
-the program is not schedule driven. If events do not occur on the 
anticipated schedule, the schedule will be revised. The schedule 
reflects the best information available about when events are most 
likely to occur. 

The following are GAO's comments on the Department of Defense's letter 
dated January 25, 2005.

GAO's Comments:

1. The Department of Defense's (DOD) characterization of our objectives 
is not correct. As stated in our report, our objectives were to 
determine whether DOD had effective processes in place for managing the 
definition of requirements for the Defense Integrated Military Human 
Resources System (DIMHRS) (Personnel/Pay) and whether it established an 
integrated program management structure and followed effective 
processes for acquiring a system based on commercial software 
components. Accordingly, we assessed the processes used to manage 
DIMHRS (Personnel/Pay), and the content of the requirements, against 
relevant best practices, many of which are embodied in DOD and federal 
policies and guidance, and against federal guidance, federal accounting 
standards, and prior GAO reports.

2. We do not believe that our finding that DOD is appropriately 
limiting modification of commercial, off-the-shelf (COTS) products (a 
best practice) is incongruous with our recommendation that requirements 
be acceptable to user organizations (another best practice). 
Furthermore, our report does not recommend that DOD act on all comments 
regardless of impact. Our recommendations concerning system 
requirements are intended to provide DOD with the principles and rules 
that it should apply in executing a requirements-acceptance process 
that permits all stakeholder interests and positions to be heard, 
considered, and resolved in the context of what makes economic sense. 
Furthermore, our report makes complementary recommendations that 
discourage changes to COTS products unless fully justified on the basis 
of life-cycle costs, benefits, and risks. Finally, while we do not 
dispute whether DOD has followed a process to screen out comments that 
would have necessitated COTS modification, DIMHRS (Personnel/Pay) users 
said that this process did not allow for effective resolution of the 
comments, which is the basis for our recommendation aimed at gaining 
user acceptance of requirements.

3. We have not concluded that DOD had not done enough to ensure that 
all stakeholders have had full input to the requirements. Our 
conclusion was that DOD had not obtained user acceptance of the 
detailed requirements, and that this choice entails risks.

4. We disagree. Our report neither states nor suggests that DOD act on 
all comments that it receives on requirements from all sources. Also, 
see comment 2.

5. See comment 10.

6. We acknowledge DOD's implementation of certain best practices as 
noted in our report. However, at the time we concluded our work, DOD 
was not following all relevant and practicable best practices, as we 
discuss in our report.

7. We do not dispute DOD's comment about efforts on DIMHRS (Personnel/
Pay) relative to other system acquisitions because our review's 
objectives and approach did not extend to comparing the two. See 
comment 1 for a description of our objectives. Furthermore, while it is 
correct that DOD's regulations only require stakeholder agreement with 
the Operational Requirements Document, our evaluation was not limited 
to whether DOD was meeting its own policy; we also evaluated whether 
DOD's processes were consistent with industry and government best 
practices.

8. We do not disagree that DOD has taken important steps to meet the 
goals of requirements completeness and correctness, and we do not have 
a basis for commenting on whether the department might have completed 
important requirements-to-design traceability steps since we completed 
our work. However, as we state in our report, these tracing steps began 
in response to the inquiries we made during the course of our review. 
Furthermore, DOD's comments contain no evidence to show that it has 
addressed the limitations in the requirements' completeness and 
correctness that we cite in the report, such as those relating to the 
interface and data requirements, and they do not address the 
understandability issues we found relative to an estimated 77 percent 
of the detailed requirements. Moreover, DOD even stated in its comments 
that its latest program review revealed 606 business process comments 
and 17 interface comments that it deemed noncritical, although it noted 
that they were still being analyzed.

9. We do not dispute DOD's position that the Joint Requirements 
Oversight Council's validation of the Operational Requirements Document 
is all that is required by DOD regulation, and we do not have a basis 
for commenting on whether its documentation of requirements for DIMHRS 
(Personnel/Pay) was innovative and unprecedented. Our review objective 
relating to requirements, as stated in our report, was to address 
whether DOD has effective processes in place for managing the 
definition of the requirements. To accomplish this objective, and as 
also stated in our report, we analyzed DOD's requirements management 
efforts against recognized best practices.

10. We do not disagree that DOD has taken numerous steps to gain user 
acceptance of the system. However, as we point out in the report, user 
organizations still had questions and reservations concerning the 
requirements. Not adequately resolving these issues, and thereby 
gaining user acceptance of requirements, increases the risk that a 
system will be developed that does not meet users' needs, that users 
will not adopt the developed and deployed system, and that later system 
rework will be required to rectify this situation.

11. We do not question that DOD has reviewed the detailed requirements 
since we completed our review. However, we challenge DOD's comment that 
all questions have been resolved for two reasons. First, DOD's comments 
contain no evidence to show that it has addressed the limitations in 
the requirements' completeness and correctness that we cite in the 
report, such as those relating to the interface and data requirements, 
or the understandability issues we found relative to an estimated 77 
percent of the detailed requirements. Second, in its comments, DOD 
acknowledges that 606 questions remain regarding requirements and 
design issues.

12. We agree that the detailed requirements are not the sole vehicle 
for gaining user acceptance of the system. Rather, they are one vehicle 
to be used in a continuous process to ensure acceptability of a system 
to end users. Industry and government best practices advocate users' 
understanding and acceptance of requirements, and these practices are 
not limited to high-level requirements descriptions, but rather apply 
to more detailed requirements descriptions as well.

13. We do not dispute that the roles and responsibilities of the 
Executive Steering Committee are defined and documented. In fact, we 
cite the committee's responsibilities in our report.

14. We agree that DOD has forums and processes for communicating 
stakeholder interests. However, we do not agree that these have 
provided for effective resolution of concerns and comments, as we 
describe in our report. Also, see comment 10.

15. We disagree. In our view, any entity, whether it is an individual, 
office, or committee, can have responsibility, accountability, and 
authority for managing a program. Moreover, we intentionally worded the 
recommendation so as not to prescribe what entity should fulfill this 
role for DIMHRS (Personnel/Pay). Rather, our intent was to ensure that 
such an entity was designated and empowered.

16. We disagree. DIMHRS (Personnel/Pay) is a DOD-wide program involving 
three distinct stakeholder groups whose respective chains of command do 
not meet at any point below the Secretary and Deputy Secretary of 
Defense levels. As we state in our report, responsibility, authority, 
and accountability for DIMHRS (Personnel/Pay) is in fact diffused among 
three stakeholder groups with responsibility for requirements with the 
Joint Requirements and Integration Office, responsibility for 
acquisition with the Joint Program Management Office, and 
responsibility for transition to DIMHRS (Personnel/Pay) with the 11 end 
users' organizations. Furthermore, under the current structure, only 
one of the three stakeholder groups, the Joint Requirements and 
Integration Office (JR&IO), is accountable to the Under Secretary, and 
authority over DIMHRS (Personnel/Pay) is spread across the three 
groups. Accordingly, the intent of our recommendation is for DOD to 
create an accountability structure that can set expectations for 
stakeholders and hold them accountable.

17. See comment 2. Furthermore, we agree that the department should not 
replicate "as is" processes. However, as our report points out, the 
users' comment process did not provide for effective resolution; 
therefore, users stated that they were not willing to sign off on the 
requirements as sufficient to meet their needs. The intent of our 
recommendation is to ensure that the system's functional acceptability 
to users is reasonably ensured before the system is developed, thereby 
minimizing the risk of more expensive system rework to meet users' 
needs.

18. We do not believe and nowhere in our report do we state or suggest 
that stakeholders should be granted veto power. Also, see comments 2 
and 3.

19. We disagree. As stated in our report, DIMHRS (Personnel/Pay) had a 
preliminary architectural certification with the Business Enterprise 
Architecture (BEA) in April 2003. However, DOD could not provide us 
with documented, verifiable analysis demonstrating this consistency and 
forming the basis for the certification, in part because the BEA was 
incomplete. Rather, we were told that this certification was based on 
the DIMHRS (Personnel/Pay) program manager's stated commitment to be 
consistent at some future point, and the system is scheduled to undergo 
another certification before the system deployment decision. Moreover, 
we had previously reported that the BEA, including the military 
personnel and pay portions of the architecture, was not complete, and 
thus not in place to effectively guide and constrain system 
investments. As we state in our report, the real value in having an 
architecture is knowing, at the time when system definition, design, 
and development are occurring, what the larger blueprint for the 
enterprise is in order to guide and constrain these activities.

20. We disagree. See comment 21.

21. We disagree. As we state in our report, DOD is not following three 
relevant best practices. These practices are focused on effectively 
planning for the full complement of activities that are needed to 
prepare an organization for the institutional and individual changes 
that COTS-based system solutions introduce. Such planning is intended 
to ensure, among other things, that key change management activities, 
including the dependencies among these activities, are defined and 
agreed to by stakeholders, including ensuring that adequate resources 
and realistic time frames are established to accomplish them. In this 
regard, DOD agreed in its comments that it does not have an integrated 
master plan/schedule for the program, which is an essential tool for 
capturing the results of the proactive change management planning that 
the best practices and our recommendation advocate. Moreover, available 
plans did not include all activities that end-user organizations will 
need to make regarding organizational changes and business process 
improvements associated with the system, such as revising the duties 
that are now performed by pay specialists and personnel specialists. 
This concern was stated by representatives of the DIMHRS (Personnel/
Pay) offices in the services and DFAS, who stated that current plans do 
not adequately address the activities, time frames, and resources they 
will need to complete the transition to DIMHRS (Personnel/Pay). 
Furthermore, at the time that we completed our review, DOD had yet to 
identify all the legacy systems that would interface with DIMHRS 
(Personnel/Pay), and so DOD could not estimate the time and resources 
that will be needed to develop and implement legacy system interfaces 
with DIMHRS (Personnel/Pay). Both published research and our experience 
in evaluating the acquisition and implementation of COTS-based system 
solutions show that the absence of well-planned, proactive 
organizational and individual change management efforts can cause these 
system efforts to fail.

22. See comment 21. Furthermore, among the ambiguities in the detailed 
requirements that we cite in our report are references that do not 
clearly state the associated practice or policy.

23. See comments 8 and 22. In addition and as stated in our report, the 
process used to define detailed requirements has yet to result in user 
acceptance. Specifically, according to end-user representatives from 
each of the services, the detailed requirements were difficult to 
understand because they were shared in a piecemeal fashion and did not 
include sufficient detail. Furthermore, these representatives stated 
that they were not willing to sign off on the requirements.

24. We do not dispute that DOD has provided demonstrations of and 
training on the COTS product. However, as we point out in our report, 
users still have questions on DIMHRS (Personnel/Pay), which will be 
based on the COTS product, including how it will be used to perform 
personnel and pay functions, and how it will change the roles and 
responsibilities of end users. Moreover, proactive management of the 
organizational and individual change associated with COTS-based system 
solutions requires careful planning for the full range of activities 
needed to facilitate the introduction and adoption of the system, and 
as we state in the report and DOD agreed in its comments, the 
department does not have the kind of integrated master plan that would 
reflect such planning.

25. We do not dispute that existing plans have been reviewed and 
approved by the contractor and an independent reviewer. However, we 
disagree that these plans sufficiently incorporate all the change 
management activities that are needed to position DOD for adoption and 
use of DIMHRS (Personnel/Pay). In the absence of an integrated master 
schedule, which the department acknowledges in its comments as yet to 
be developed, DOD cannot adequately ensure that the full range of 
organizational and individual change management activities will be 
effectively performed.

26. We support DOD's stated commitment to follow a more event-driven 
risk-based approach, and have slightly modified our recommendation to 
recognize this commitment. Nevertheless, it is important to note that 
the approach that we found the department taking during the course of 
our review was schedule driven, meaning that program activities were 
truncated or performed concurrently in order to meet established 
deadlines. For example, as we describe in our report, data requirements 
(which are derived from higher-level information needs) were provided 
to the contractor before information needs were fully defined because 
the contractor needed these data requirements to complete the system 
design on schedule. Also during our review, the program had developed 
plans for accelerating system deployment in order to meet an externally 
imposed deadline. After we raised concern about the risks of 
accelerating the schedule, and the lack of adequate risk-mitigation 
strategies, DOD changed its plans.

27. At the time of our review, we observed that the contract was a 
driver for the schedule. For example and as our report states, in March 
2004, JR&IO provided a version of the detailed requirements to the 
development and integration contractor in order to meet the 
contractor's schedule, even though it had received thousands of 
comments on the requirements from users that it had yet to examine and 
resolve.

28. We support DOD's comment that it will revise the schedule if events 
do not occur as anticipated because it is consistent with our 
recommendation.

[End of section]

Appendix III: GAO Contact and Staff Acknowledgments:

GAO Contact:

Donald C. Snyder, (202) 512-7204:

Acknowledgments:

In addition to the person named above, the following persons made key 
contributions to this report: Nabajyoti Barkakati, Harold J. Brumm, 
Barbara S. Collier, Nicole L. Collier, George L. Jones, John C. Martin, 
Kenneth E. Patton, B. Scott Pettis, Mark F. Ramage, Karl W. D. Seifert, 
Robert W. Wagner, Joseph J. Watkins, and Daniel K. Wexler.

FOOTNOTES

[1] GAO, Financial Management: Defense's System for Army Military 
Payroll Is Unreliable, GAO-93-32 (Washington, D.C.: Sept. 30, 1993).

[2] GAO, Military Pay: Army Reserve Soldiers Mobilized to Active Duty 
Experienced Significant Pay Problems, GAO-04-911 (Washington, D.C.: 
Aug. 20, 2004), and Military Pay: Army National Guard Personnel 
Mobilized to Active Duty Experienced Significant Pay Problems, GAO-04-
89 (Washington, D.C.: Nov. 13, 2003).

[3] The 95 percent confidence interval for this estimate is from 60 to 
89 percent. (For more details of the results of the review and the 
scope and methodology, see slides 34 and 74 of appendix I.)

[4] GAO, Business Process Reengineering Assessment Guide Version 3, 
GAO/AIMD-10.1.15 (Washington, D.C.: May 1997).

[5] GAO, Defense IRM: Poor Implementation of Management Controls Has 
Put Migration Strategy at Risk, GAO/AIMD-98-5 (Washington, D.C.: Oct. 
20, 1997).

[6] GAO/AIMD-98-5.

[7] An enterprise architecture is a blueprint for guiding and 
constraining the definition and implementation of programs in a way 
that supports strategic plans and promotes integration, 
interoperability, and optimization of mission performance. This 
blueprint serves as the common frame of reference to inform program 
operational and technological decision making relative to, for example, 
what functions will be performed where, by whom, and using what 
information. For more information, see GAO, Information Technology: A 
Framework for Assessing and Improving Enterprise Architecture 
Management (Version 1.1), Executive Guide, GAO-03-584G (Washington, 
D.C.: April 2003).

[8] Bob Stump National Defense Authorization Act for Fiscal Year 2003, 
Pub. L. No. 107-314, ß 1004, 116 Stat. 2458, 2629-2631 (Dec. 2, 2002).

[9] GAO, DOD Business Systems Modernization: Limited Progress in 
Development of Business Enterprise Architecture and Oversight of 
Information Technology Investments, GAO-04-731R (Washington, D.C.: May 
17, 2004).

GAO's Mission:

The Government Accountability Office, the investigative arm of 
Congress, exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO's commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony:

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics.

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as "Today's Reports," on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order 
GAO Products" heading.

Order by Mail or Phone:

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to:

U.S. Government Accountability Office

441 G Street NW, Room LM

Washington, D.C. 20548:

To order by Phone:

 

Voice: (202) 512-6000:

TDD: (202) 512-2537:

Fax: (202) 512-6061:

To Report Fraud, Waste, and Abuse in Federal Programs:

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470:

Public Affairs:

Jeff Nelligan, managing director,

NelliganJ@gao.gov

(202) 512-4800

U.S. Government Accountability Office,

441 G Street NW, Room 7149

Washington, D.C. 20548: