This is the accessible text file for GAO report number GAO-03-1046 
entitled 'DOD Personnel: Documentation of the Army's Civilian 
Workforce-Planning Model Needed to Enhance Credibility' which was 
released on August 22, 2003.

This text file was formatted by the U.S. General Accounting Office 
(GAO) to be accessible to users with visual impairments, as part of a 
longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov.

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately.

On January 5, 2004, this document was revised to add various 
footnote references missing in the text of the body of the document.

Report to the Ranking Minority Member, Subcommittee on Readiness, 
Committee on Armed Services, House of Representatives:

United States General Accounting Office:

GAO:

August 2003:

DOD Personnel:

Documentation of the Army's Civilian Workforce-Planning Model Needed to 
Enhance Credibility:

GAO-03-1046:

GAO Highlights:

Highlights of GAO-03-1046, a report to the Ranking Minority Member, 
Subcommittee on Readiness, Committee on Armed Services, House of 
Representatives

Why GAO Did This Study:

Between fiscal years 1989 and 2002, the Department of Defense (DOD) 
reduced its civilian workforce by about 38 percent, with little 
attention to shaping or specifically sizing this workforce for the 
future. As a result, the civilian workforce is imbalanced in terms of 
the shape, skills, and experience needed by the department. DOD is 
taking steps to transform its civilian workforce. To assist with this 
transformation, the department is considering adopting an Army 
workforce-planning model, known as the Civilian Forecasting System 
(CIVFORS), which the Army uses to forecast its civilian workforce 
needs. Other federal agencies are also considering adopting this 
model. GAO was asked to review the adequacy of the steps the Army has 
taken to ensure the credibility of the model. 

What GAO Found:

The Army has taken adequate steps to ensure that the historical 
personnel data used in the model are sufficiently reliable and that 
the information technology structure adequately and appropriately 
supports the model. For example, the Army has established adequate 
control measures (e.g., edit checks, expert review, etc.) to ensure 
that the historical data that goes into the model are sufficiently 
reliable. Moreover, it has taken adequate steps to ensure that the 
information technology support structure (i.e., the software and 
hardware used to interface with and house the model) would enable 
continuity of operations, functionality, and system modification and 
operations. 

However, the Army has not demonstrated that it has taken adequate 
steps to ensure that the model’s forecasting capability provides the 
basis for making accurate forecasts of the Army’s civilian workforce. 
The Army’s original certification of CIVFORS in 1987 was based on a 
formal documented verification and validation of the model structure 
that has not been formally updated since that time even though the 
Army has undertaken several model improvements. According to the 
Army’s CIVFORS program manager, the Army has taken several steps, to 
include an independent review, peer reviews, and a comparison of 
forecasted data to actual data. However, documentation of these steps 
is incomplete and, therefore, does not provide adequate evidence to 
demonstrate the credibility of the forecast results. Without adequate 
documentation, the Army cannot show that it has taken sufficient steps 
to ensure the model’s credibility in terms of its forecasting 
capability; consequently, there exists a risk that the forecasts it 
produces may be inaccurate or misleading. Furthermore, without 
documentation of CIVFORS’s forecasting capability, it may be difficult 
for DOD and other federal organizations to accurately determine its 
suitability for their use. 

What GAO Recommends:

To assure the reliability of Army civilian workforce projections and 
the appropriateness of the model for use DOD-wide and by other federal 
agencies, we recommend that the Secretary of Defense direct the 
Secretary of the Army to appropriately document the forecasting 
capability of the model. 

Although DOD stated, in written comments on a draft of this report, it 
did not concur with GAO’s recommendation, the Army is taking actions 
that, in effect, implement it.

www.gao.gov/cgi-bin/getrpt?GAO-03-1046.

To view the full product, including the scope and methodology, click 
on the link above. For more information, contact Derek Stewart, 202-
512-5559, stewartd@gao.gov.
 
[End of section]

Contents:

Letter:

Results in Brief:

Background:

Civilian Workforce-Planning Model's Data Reliability and Information 
Technology Structure Are Adequate, but Forecasting Ability Not Fully 
Established:

Conclusions:

Recommendation for Executive Action:

Agency Comments and Our Evaluation:

Scope and Methodology:

Appendix I: Comments from the Department of Defense:

Appendix II: GAO Contact and Staff Acknowledgments:

Abbreviations:

CIVFORS: Civilian Forecasting System:

DOD: Department of Defense:

WASS: Workforce Analysis Support System:

United States General Accounting Office:

Washington, DC 20548:

August 22, 2003:

The Honorable Solomon P. Ortiz 
Ranking Minority Member 
Subcommittee on Readiness 
Committee on Armed Services 
House of Representatives:

Dear Mr. Ortiz:

Between fiscal years 1989 and 2002, the Department of Defense (DOD) 
reduced its civilian workforce from 1,075,437 to 670,166--about a 38 
percent reduction--with little attention to shaping or specifically 
sizing this workforce for the future. As a result, the civilian 
workforce is imbalanced in terms of the shape, skills, and experience 
needed by the department. DOD plans to downsize its civilian workforce 
by an additional 55,000 through fiscal year 2007. In addition, in April 
2003, DOD submitted a proposal to Congress that would authorize DOD to 
establish a National Security Personnel System to transform its current 
civilian personnel system.[Footnote 1] DOD is also exploring the 
feasibility of placing hundreds of thousands of civilians into 
essentially nonmilitary jobs that are currently performed by uniformed 
personnel. To assist in determining its future workforce, DOD will need 
reliable workforce planning tools, such as workforce projection models. 
According to DOD guidance, a model used to provide data for decision 
making should be accredited--that is, the model should be officially 
certified as acceptable for use for a specific purpose.

In a February 2003 testimony, the Chief of Staff of the Army stated 
that the Army has begun to transform its civilian personnel system. To 
assist with this transformation, the Army is using its workforce-
planning model, known as the Civilian Forecasting System (CIVFORS), 
which forecasts future civilian workforce needs.[Footnote 2] The Army 
is working closely with the Office of the Secretary of Defense and the 
Office of Personnel Management to demonstrate the applicability of the 
model for use DOD-wide and in other federal agencies. According to Army 
guidance (Army Regulation 5-11 and Department of the Army Pamphlet 5-
11), verification is one of the steps needed to ensure a model's 
suitability to perform its intended purpose. The verification process 
evaluates the extent to which a model has been developed using sound 
and established software engineering techniques, and it establishes 
whether the computer code correctly performs the intended functions. 
Army guidance also states that assessment of the correctness and 
forecasting capability is required.

In this report, we reviewed the adequacy of the steps the Army has 
taken to ensure the credibility of the model. In March 2003, we briefed 
your staff on our preliminary findings. To determine the Army's efforts 
to ensure the credibility of its model, we interviewed and obtained 
pertinent documentation from the Army's CIVFORS program manager. We 
also reviewed DOD and Army guidance relevant to the management of Army 
models and interviewed DOD officials to discuss their plans to adopt 
CIVFORS. We conducted our review from September 2002 to June 2003. More 
detailed information on our scope and methodology appears at the end of 
this report.

Results in Brief:

The Army's steps were adequate to ensure that the historical personnel 
data used in the model are sufficiently reliable and that the 
information technology support structure[Footnote 3] adequately and 
appropriately supports the model, but the Army has not documented its 
steps to ensure the credibility of the model's forecasting capability. 
The Army has established adequate control measures (e.g., edit checks, 
expert review, etc.) to ensure that the historical data that goes into 
the model are sufficiently reliable. Moreover, it has taken adequate 
steps to ensure that the information technology support structure would 
enable continuity of operations, functionality, and system modification 
and operations. However, the Army has not documented that it has taken 
adequate steps to ensure that the model's structure (including its 
forecasting capability and the appropriateness of its assumptions) 
provides the basis for making accurate forecasts of the Army's civilian 
workforce. The Army's original certification of CIVFORS in 1987 was 
based on a formal documented verification and validation of the model 
structure that has not been formally updated since that time, even 
though the Army has undertaken several model improvements. According to 
the Army's CIVFORS program manager, the Army has taken several steps, 
to include an independent review, peer reviews, and a comparison of 
forecasted data to actual data. However, documentation of these steps 
is incomplete and, therefore, does not provide adequate evidence to 
demonstrate the credibility of the forecast results. Without adequate 
documentation, the Army cannot show that it has taken sufficient steps 
to ensure the credibility of the model's forecasting capability; 
consequently, there exists a risk that the forecasts it produces may be 
inaccurate or misleading. Furthermore, without documentation of 
CIVFORS's forecasting capability, it may be difficult for DOD and other 
federal organizations to accurately determine its suitability for their 
use.

We are recommending that the Secretary of Defense direct the Secretary 
of the Army to appropriately document the Army's forecasting capability 
of the civilian workforce-planning model. Although DOD stated, in 
written comments on a draft of this report, it did not concur with our 
recommendation, the Army is taking actions that, in effect, implement 
it.

Background:

According to an Army Human Resource official, the Army uses the 
workforce-planning model--CIVFORS--for human resources management. 
CIVFORS is a collection of software programs that anticipate future 
impacts on the workforce so that management can plan for changes 
instead of reacting to them. The model is used to evaluate a number of 
critical areas in civilian workforce planning, including projected 
recruitment of personnel, impact of organizational realignments, and 
changes in workforce trends (such as aging, retention, and projected 
personnel shortfalls). It is a life-cycle modeling and projection tool 
that models the most significant events that describe the life-cycle 
path of personnel, which includes accessions, promotions, 
reassignments, retirements, and voluntary and involuntary separations 
over a 7-year period.

Verification and validation of models are important steps to building 
credible models because they provide the foundation for the 
accreditation process to ensure the suitability of the models for their 
intended purposes, as stated in Army guidance, Management of Army 
Models and Simulations.[Footnote 4] The verification process evaluates 
the extent to which a model has been developed using sound and 
established software engineering techniques, and it establishes whether 
the model's computer code correctly performs the intended functions. 
Model verification includes data verification, model documentation, and 
testing of the information technology structure that supports the 
model; model verification is contained in such documents as the 
programmer's manual, installation's manual, user's guide, analyst's 
manual, and trainer's manual. According to Army guidance, assessment of 
the correctness and forecasting capability of the model is also 
required, and it should be performed by a subject matter expert 
independent from the model developer; however, the developer is 
expected to conduct in-house verification and testing to assist in the 
overall model development process. Validation is the process of 
determining the extent to which the model adequately represents the 
real world.

Civilian Workforce-Planning Model's Data Reliability and Information 
Technology Structure Are Adequate, but Forecasting Ability Not Fully 
Established:

The Army has taken steps to ensure the reliability of the historical 
personnel data used by the model and the adequacy of its information 
technology structure used to support the model, but it has not provided 
documentation that it has sufficiently tested and reviewed the most 
critical aspect of the model--its forecasting capability and the 
appropriateness of its assumptions. As a result, the forecasting 
credibility of the current version of the model is not sufficiently 
validated or documented. Without proper documentation of the abilities 
of the model, there is a risk that the forecasts it produces may be 
inaccurate or misleading and the suitability for use by other 
organizations may be difficult to determine.

Historical Personnel Data Reliability Is Adequate:

The Army's review of the historical personnel data used to provide 
information for workforce planning was adequate to show that the data 
are sufficiently reliable for use in the workforce model. Data 
regarding personnel (such as date hired, education, age, grade level, 
and occupational series) are taken from the Army's Workforce Analysis 
Support System (WASS).[Footnote 5] CIVFORS uses the most recent 5 years 
of historical data to forecast the civilian workforce planning needs 
during the next 7 years.

According to Army guidance, to ensure that data are sufficiently 
reliable for use in the Army model, support documents should contain 
information about the overall characteristics of the database. 
Furthermore, the documents should show the intended range of 
appropriate uses for the model as well as constraints on its use. They 
should also include concise statements of the condition of the database 
for the purpose of indicating its stability. The Army provided most, 
but not all, of the documents referred to in Army guidance; we believe 
that the documents provided are key ones and are adequate to show that 
WASS data are sufficiently reliable for use in CIVFORS. In addition, 
the Army program manager for the CIVFORS workforce-planning model 
stated that the workforce data are checked by reviewing the arithmetic 
in the numerical algorithms to verify that there is no unexplained 
change in the size of the civilian personnel workforce contained in the 
database. Further, edit checks include matching social security numbers 
for personnel from one time period to another to account for actual 
personnel and personnel transactions processed. In addition, CIVFORS 
has automated checks for inappropriate numbers or characters. Such 
steps help to assure that the data contained in WASS accurately and 
completely reflect critical personnel aspects and transactions.

Information Technology Structure Is Adequate:

The Army's procedures for validating the information technology support 
structure (the software and hardware used to interface with and house 
the model) were also sufficient. For example, the Army (1) adequately 
documented the information technology structure to allow for continuity 
of operations, (2) tested its functionality, and (3) provided expertise 
for system modification and operation. Procedures used by the Army 
include documenting the model's system description and hardware and 
software requirements, providing system and user manuals, planning for 
configuration management,[Footnote 6] and conducting functionality 
tests to help ensure the system's usability and operability over time 
and to demonstrate the adequacy of the information technology structure 
to support use of the workforce model.

Model's Forecasting Ability Is Not Fully Established:

The Army's documentation cannot show that the forecasting ability of 
CIVFORS has been adequately evaluated and, therefore, we cannot fully 
assess the credibility of the model. According to Army guidance, 
validation is the process of determining the extent to which a model 
adequately represents the real world. According to the Army program 
manager, over a 7-year period, CIVFORS forecasts the anticipated 
impacts on the workforce based on the most significant events in the 
life-cycle path of personnel (to include accessions, promotions, 
reassignments, retirements, voluntary separations, and involuntary 
separations). Army guidance states that an independent, peer, and 
subject matter expert review of the model should be conducted. The Army 
guidance also suggests generally accepted methods, such as conducting a 
careful line-by-line examination of the model design and computer code 
and algorithms. The Army's program manager said this had been done for 
the original certification of CIVFORS in 1987. However, no formal 
document of the reviews has been prepared in the years since, even 
though the Army has undertaken several model improvements, such as (1) 
an expanded scope to include more dimensions in the modeling process; 
(2) a more integrated, streamlined process that involves fewer steps; 
and (3) greater flexibility, achieved by generalizing the formulas and 
parameters.

In addition, there is insufficient documentation regarding tests 
performed, since 1987, in which CIVFORS's forecasts for prior years are 
compared against equivalent historical data (called an "out of sample" 
test) to measure the model's forecasting capability. Such testing, 
which is one method to validate a model's forecasting capability, would 
involve using the first 5 of the last 7 years of historical data to 
forecast the 2 subsequent years. The forecasts for the last 2 years 
could then be compared to the actual historical data. The Army, 
however, performed tests comparing patterns of forecasts against 
historical data (called "in sample" tests), showing that forecasts 
reflect the same patterns as the historical data used to develop them 
for a sample of three Army major commands. However, the draft document 
that was provided to us was inadequate to fully assess the sampling 
used by the Army and the value of the tests.

Finally, the Army could not provide adequate documentation of an 
independent or peer review of the model. The Army's CIVFORS program 
manager stated that the major commands served as peer reviewers by 
conducting a comparison of their workforce data to WASS and CIVFORS 
workforce data. We believe that such assessments by users provide 
important information but do not constitute a peer review as defined in 
Army guidance. Also, the results of these assessments were not 
available for us to review. The program manager also stated that an 
independent subject matter expert reviewed the functional design and 
the code in 1999, but a formal report of the activities performed and 
the specific changes or modifications implemented during the review 
were not produced.

Documentation has often not been a priority for several reasons. 
According to the Army's CIVFORS program manager, lack of documentation 
is primarily due to limited funding, which was spent on implementing 
changes to CIVFORS and WASS rather than on the production of formal 
documents. Further, a shortage of staff (only one staff person--the 
program manager) and loss of documents during the attack on the 
Pentagon on September 11, 2001, also affected the amount of 
documentation the Army could provide us. The program manager also 
stated that some documentation was not needed because CIVFORS's design 
is predicated on proven methods in other Army active-duty, military 
manpower forecasting models. In addition, the program manager stated 
that the Army and contractors have primarily been adapting technology 
(upgrading from mainframe to personal computer to Web-based) to improve 
model functionality rather than creating new technology. However, 
without proper documentation of the abilities of the model, there 
exists a risk that the forecasts it produces may be inaccurate or 
misleading. Consequently, decisions about future workforce 
requirements may be questionable, and planning for the size, shape, and 
experience level of the future workforce may not adequately meet the 
Army's needs.

These issues may extend beyond the Army. In April 2002, DOD published a 
strategic plan for civilian personnel, which includes a goal to obtain 
management systems to support workforce planning. According to a DOD 
official responsible for civilian workforce planning tools, components 
within DOD have been requesting a modeling tool to assist them with 
civilian workforce planning. As a result, DOD has decided to test the 
Army's civilian forecasting model. In October 2002, DOD purchased 
hardware, installed modified software, and provided training to a small 
number of personnel. Recently, DOD obtained a historical database of 
civilian personnel data from the Defense Management Data Center and 
provided the database to the contractor to load into the model. Two 
agencies have volunteered to test the model: the Defense Logistics 
Agency and the Washington Headquarters Service. DOD is working to 
develop a test for these organizations using their own civilian 
personnel data to test the model. At the end of the testing period, DOD 
will assess the model to obtain a better understanding of its logic and 
determine whether or not it should be implemented departmentwide.

Conclusions:

As DOD continues to transform and downsize its civilian workforce, it 
is imperative that the department properly shape and size the 
workforce. One tool that could assist in this effort is CIVFORS--the 
Army's workforce planning model. However, proper documentation of the 
verification and validation of CIVFORS is needed before expanding its 
use. The Army has taken adequate steps to ensure that the historical 
personnel data used in the model are sufficiently reliable and the 
information technology structure appropriately supports the model; 
however, it has not fully documented that it has taken adequate steps 
to demonstrate the credibility of the model's forecasting capability. 
Further, a model should be fully scrutinized before each new 
application because a change in purpose, passage of time, or input data 
may invalidate some aspects of the existing model. Without sufficient 
documentation to demonstrate that adequate steps have been taken to 
ensure the credibility of the model's forecasting capabilities, 
decisions about the Army's future civilian workforce may be based on 
questionable data and other potential users cannot determine with 
certainty the model's suitability for their use.

Recommendation for Executive Action:

To assure the reliability of Army civilian workforce projections, as 
well as the appropriateness of the model for use DOD-wide and by other 
federal agencies, we recommend that the Secretary of Defense direct the 
Secretary of the Army to appropriately document the Army's forecasting 
capability of the model.

Agency Comments and Our Evaluation:

Although DOD stated, in written comments on a draft of this report, 
that it did not concur with our recommendation, the Army is taking 
actions that, in effect, implement it. DOD's written comments are 
contained in appendix I.

Regarding our recommendation that the Secretary of Defense direct the 
Secretary of the Army to appropriately document the Army's forecasting 
capability of the model, DOD stated that the Army recognizes the need 
to fully document its verification and validation efforts. Further, DOD 
stated the staff of the Assistant Secretary of the Army, Manpower and 
Reserve Affairs, has begun developing a verification and validation 
plan to enable outside parties to assess the suitability and 
adaptability of the model for their organizational use. This 
verification and validation process is scheduled for completion in 
September 2003. However, during our review, DOD did not provide 
information about the full scope of this verification and validation 
effort. We believe that as the Army undertakes its verification and 
validation effort, it should clearly document, as we recommended, its 
assumptions, procedures, and the results so that future users can 
replicate the tests to appropriately establish the model's validity for 
their purposes.

DOD also did not concur with our finding that the forecasting ability 
of the model has not been fully established. DOD stated that the 
ultimate test of a system is performance and that CIVFORS has been 
consistently generating Army projections with high standards of 
accuracy. We did not independently evaluate the model's accuracy. As 
our report makes clear, our basic point is that the model's forecasting 
ability has not been documented in accordance with Army guidance. We 
continue to believe that without adequate documentation, the Army 
cannot show that it has taken sufficient steps to ensure the model's 
credibility in terms of its forecasting capability. DOD also provided 
technical comments, which we incorporated where appropriate.

Scope and Methodology:

We did not independently evaluate the model or the application of the 
steps; rather, we reviewed the adequacy of the steps that the Army 
program manager stated were taken to ensure the credibility of the 
model. To determine the adequacy of the steps the Army has taken to 
ensure the credibility of its civilian workforce-forecasting model, we 
discussed CIVFORS with the Army's CIVFORS program manager in the Army 
G-1 office, Civilian Personnel Policy Directorate, who has overall 
responsibility for the workforce analysis and the forecasting system. 
In addition, Army contractor officials who are responsible for 
providing technical, analytic, and management support to operate, 
maintain, and enhance the planning tool and model participated in 
several of our discussions with the program manager. We reviewed the 
following CIVFORS's documents regarding the information technology 
support structure: the Configuration Management Manual, the System's 
Specifications, the Design/Subsystem Documentation, the Operator's 
Manual, and the User's Manual. In addition, we reviewed the 1987 and 
draft 2002 test analysis report on the Civilian Forecasting System and 
other documentation provided by the Army to obtain information on how 
the model operates according to model assumptions. We also reviewed the 
DOD Defense Modeling and Simulation Office guidance on verification and 
validation of models, the Army regulation and pamphlet pertaining to 
the management of Army models and simulations, and other literature 
regarding model credibility. We also interviewed DOD officials in the 
Civilian Personnel Management Service responsible for developing plans 
to adopt the Army's workforce forecasting model to discuss the status 
of their efforts.

We conducted our review from September 2002 to June 2003 in accordance 
with generally accepted government auditing standards.

We are sending copies of this report to the appropriate congressional 
committees, the Secretary of Defense, the Under Secretary of Defense 
for Personnel and Readiness, and the Secretary of the Army. We will 
also make copies available to others upon request. In addition, the 
report will be available at no charge on the GAO Web site at http://
www.gao.gov.

If you or your staff have any questions about this report, please 
contact me at (202) 512-5559. Key contributors to this report are 
listed in appendix II.

Sincerely yours,

Derek B. Stewart 
Director, Defense Capabilities and Management:

Signed by Derek B. Stewart: 

[End of section]

Appendix I: Comments from the Department of Defense:

OFFICE OF THE UNDER SECRETARY OF DEFENSE:

4000 DEFENSE PENTAGON WASHINGTON, D.C. 20301-4000:

AUG 8 2003:

PERSONNEL AND READINESS:

Mr. Derek B. Stewart:

Director, Defense Capabilities and Management Issues U.S. General 
Accounting Office:

441 G Street, N.W. Washington, DC 20548:

Dear Mr. Stewart:

This is the Department of Defense (DoD) response to the General 
Accounting Office (GAO) draft report, "DOD PERSONNEL: Greater Review 
and Documentation of the Army's Civilian Workforce-Planning Model 
Needed to Enhance Credibility," dated July 23, 2003, (GAO Code 350385 
Prior Code 350198/GAO-03-1046). It is important to note that the 
genesis of this report was the GAO notice of the intent to review 
"DOD's Civilian Human Capital Strategic and Workforce Planning" on May 
16, 2002; that GAO chose not to include DOD's reply in its March 2003 
report, "DoD Actions Needed to Strengthen Civilian Human Capital 
Strategic Planning and Integration with Military Personnel and Sourcing 
Decisions;" and that GAO began another "Review of DOD's Strategic 
Workforce Planning for Civilians" on June 17, 2003. Specifically listed 
as an objective in this study was for GAO to determine "What actions 
have DoD and the military services taken to ensure the credibility of 
the workforce data and the analyses used in developing the civilian 
strategic workforce plans?" While the second study is ongoing, 
recommendations concerning the Civilian Forecasting System (CIVFORS) 
must now be considered premature.

We nonconcur with the recommendation that the Secretary of Defense 
direct the Secretary of the Army to appropriately document the Army's 
forecasting capability of the model. The Department of the Army 
recognizes the need to fully document Verification and Validation (V&V) 
for its CIVFORS model. The staff of the Assistant Secretary of the 
Army, Manpower and Reserve Affairs, has begun this process, scheduled 
for completion in September 2003.

We also nonconcur with the major finding of the report that the model's 
forecasting ability is not fully established. The ultimate test of the 
system is performance. Using Army analysts to produce specific Army 
products, the CIVFORS system has been consistently generating Army 
projections with high standards of accuracy. The Army offers the 
following comments to substantiate the system's abilities:

CIVFORS has been projecting Army civilian personnel strengths, gains 
and losses since Fiscal Year (FY) 1990. Data studies provided to GAO 
documented deviation of total projected (CIVFORS forecast) Army end 
strength within one percent of actual on-board strength totals for FY90 
through FY01. In FY02 the model predicted projected end strength of 
222,600; the actual Army end strength was 223,449. The difference 
between the projected and actual was $49; this is a deviation of a 
little less than four-tenths of one percent (0.4 percent). Established 
standards of accuracy by most government and private sector 
organizations for forecasting models is 2.5 to 5 percent deviation. The 
Army model has consistently exceeded this standard during its entire 
operational life.

Documented evidence was provided to GAO to support the credibility of 
the model forecasts in the analysis/evaluation of employee transaction 
patterns. A specific example of behavior pattern forecasting was 
illustrated by the Civil Service Retirement System (CSRS) retirement 
bubble analysis. CIVFORS data predicted that the anticipated mass 
exodus of employees covered by the CSRS retirement system would not 
materialize. This conclusion, despite being in opposition to the 
established theory of the personnel community, validated CIVFORS 
capability in the area of employee gains and losses forecasting. In 
projections developed for FY02 retirements for Army Materiel Command, 
the model forecasted 166; actual retirements for FY02 were 1694, a 
deviation of 29 for a projection accuracy of 99.3 percent.

For a complex mathematical procedure to consistently perform at a 9970 
accuracy rate validates. by most conventional standards, the 
credibility of the results. Consistently accurate results produced over 
a long period of lime can only be produced by u model whose conceptual 
design is founded in mathematically sound theory and is executed by 
correctly programmed automalion code. 11 is statistically impossible 
for any system to generate 99 percent accurate forecasts for over 13 
years operating with incorrect mathematical algorithms and/or erroneous 
programming code.

We suggest that either this report be delayed until after September 30, 
2003, to allow the Army to complete its V&V process or that the 
information concerning Army CIVFORS be included in the current review 
of "DOD's Strategic Workforce Planning for Civilians.":

Sincerely,

Charles S. Abel 
Principal Deputy:

Signed by Charles S. Abel:

[End of section]

Appendix II: GAO Contact and Staff Acknowledgments:

GAO Contact:

Christine Fossett (202) 512-2956:

Acknowledgments:

In addition to the name above, David Dornisch, Barbara Johnson, Barbara 
Joyce, John Smale, Dale Wineholt, and Susan Woodward made significant 
contributions to this report.

FOOTNOTES

[1] The system is proposed in the Transformation for the 21st Century 
Act of 2003. The act also proposes other wide-ranging changes, 
affecting civilian personnel pay and performance management, collective 
bargaining, rightsizing, and other human capital areas. 

[2] The Civilian Forecasting System was adapted from an Army military 
forecasting model for civilian use in 1987.

[3] The software and hardware used to interface with and house the 
model.

[4] Headquarters, Department of the Army, Management of Army Models and 
Simulations, Army Regulation 5-11 (Washington, D.C., July 10, 1997). 
This regulation prescribes policy and guidance and assigns 
responsibilities for the management of Army models and simulations, 
including development and maintenance.

[5] WASS enables analysis of data on Army civilians from 1974 to the 
present. It has analysis capabilities that range from frequency 
distribution to trend analysis.

[6] Configuration management is the control and documentation of 
changes made to system hardware, software, and documentation throughout 
the development and operational life of the system.

GAO's Mission:

The General Accounting Office, the investigative arm of Congress, 
exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO's commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony:

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics.

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as "Today's Reports," on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order 
GAO Products" heading.

Order by Mail or Phone:

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to:

U.S. General Accounting Office

441 G Street NW,

Room LM Washington,

D.C. 20548:

To order by Phone: 	

	Voice: (202) 512-6000:

	TDD: (202) 512-2537:

	Fax: (202) 512-6061:

To Report Fraud, Waste, and Abuse in Federal Programs:

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470:

Public Affairs:

Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S.

General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C.

20548: