This is the accessible text file for GAO report number GAO-07-614 
entitled 'Veterans Benefits Administration: Progress Made in Long-Term 
Effort to Replace Benefits Payment System, but Challenges Persist' 
which was released on April 27, 2007. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to the Chairman, Committee on Veterans' Affairs, House of 
Representatives: 

United States Government Accountability Office: 

GAO: 

April 2007: 

Veterans Benefits Administration: 

Progress Made in Long-Term Effort to Replace Benefits Payment System, 
but Challenges Persist: 

GAO-07-614: 

GAO Highlights: 

Highlights of GAO-07-614, a report to the Chairman, Committee on 
Veteransí Affairs, House of Representatives 

Why GAO Did This Study: 

Since 1996, the Veterans Benefit Administration (VBA) has been working 
on an initiative to replace its aging system for paying compensation 
and pension benefits. In 2005, concerned about the slow pace of 
development, VBA contracted with the Software Engineering Institute 
(SEI) for an independent evaluation of the project, known as the 
Veterans Service Network (VETSNET). SEI advised VBA to continue working 
on the project at a reduced pace while addressing management and 
organization weaknesses that it determined had hampered the projectís 
progress. 

GAO was requested to determine to what extent the VETSNET project has 
followed the course of action recommended by SEI and describe the 
projectís current status. 

To perform its review, GAO analyzed project documentation, conducted 
site visits, and interviewed key program officials. 

What GAO Found: 

VBA is generally following the course of action recommended by SEI by 
continuing to work on the replacement initiative at a reduced pace, 
while taking action to address identified weaknesses in overall 
management and software development processes. For example, VBA 
established a new governance structure for the initiative that included 
senior management and involved all stakeholders, and it incorporated 
all critical areas of system development in an integrated master 
schedule. However, not all of SEIís management concerns have been 
addressed. For example, SEI advised VBA to ensure that stakeholders 
take ownership responsibility for the project, including the total 
system and process operating costs; however, although VBA is tracking 
costs incurred by contractors, it is not yet tracking and reporting in-
house costs incurred by the project. Further, although the project has 
improved its management processes, such as establishing a process to 
manage and stabilize system requirements, it has not yet developed 
processes for capacity planning and management. This will be important 
for ensuring that further VETSNET development does not lead to delays 
and slowdowns in processing of benefits. In addition, although the 
project has established certain performance measures, it has not yet 
established results-oriented measures for productivity and user 
satisfaction, both of which will be important for measuring progress. 
Finally, the process improvements that VBA has incorporated in the 
replacement initiative remain in draft and have not been established 
through documented policies and procedures. If VBA does not 
institutionalize these improvements, it increases the risk that they 
may not be maintained through the life of the project or be available 
for application to other development initiatives. 

After more than 10 years of effort, including the recent management, 
organizational, and process improvements, VBA has developed critical 
functionalities needed to process and pay certain original compensation 
claims using the replacement system, but it remains far from completing 
the project. According to VBA officials, all five of the major software 
applications that make up the new system are now being used to 
establish and process new compensation claims for veterans. In total, 
the replacement system is currently providing monthly compensation 
payments to almost 50,000 veterans (out of about 3 million veterans who 
receive such payments); the system was used to process about 83 percent 
of all new compensation claims completed in March 2007. Nonetheless, 
the system requires further development before it can be used to 
process claims for the full range of compensation and pension benefits 
available to veterans and their dependents. 

What GAO Recommends: 

To sustain the improved management and software development processes 
currently being used by VETSNET project management, GAO is making 
recommendations to the Secretary of Veterans Affairs in several areas, 
including cost tracking, capacity planning and management, and 
performance measures. The Secretary agreed with GAOís recommendations 
and described actions planned in response. 

[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-614]. 

To view the full product, including the scope and methodology, click on 
the link above. For more information, contact Valerie Melvin at (202) 
512-6304 or melvinv@gao.gov. 

[End of section] 

Contents: 

Letter: 

Results in Brief: 

Background: 

VBA Is Following Course Recommended by SEI, but Weaknesses Remain to Be 
Addressed: 

VETSNET Is Currently Processing a Portion of Compensation Claims, but 
Much Work Remains: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Objectives, Scope, and Methodology: 

Appendix II: Comments from the Department of Veterans Affairs: 

Appendix III: GAO Contact and Staff Acknowledgments: 

Tables: 

Table 1: Roles and Responsibilities of VETSNET Organizational Elements: 

Table 2: VETSNET Component Applications and Status of Development: 

Table 3: Software Releases Planned to Complete the VETSNET System: 

Table 4: Conversion Phases: 

Figure: 

Figure 1: VETSNET System Life Cycle and Review Concept: 

Abbreviations: 

BDN: Benefits Delivery Network: 

CIO: Chief Information Officer: 

FAS: Finance and Accounting System: 

IT: Information Technology: 

MAP-D: Modern Award Processing-Development: 

RBA 2000: Rating Board Automation 2000: 

SEI: Carnegie Mellon Software Engineering Institute: 

VA: Department of Veterans Affairs: 

VBA: Veterans Benefits Administration: 

VETSNET: Veterans Service Network: 

VRE: Vocational Rehabilitation and Employment: 

United States Government Accountability Office: 
Washington, DC 20548: 

April 27, 2007: 

The Honorable Bob Filner: 
Chairman, Committee on Veterans' Affairs: 
House of Representatives: 

Dear Mr. Chairman: 

Since 1996, the Veterans Benefits Administration (VBA), a major 
component of the Department of Veterans Affairs (VA), has been 
undertaking a project to replace its compensation and pension benefits 
payment system. VBA's existing system, the Benefits Delivery Network 
(BDN), is essential to ensuring the accurate processing of compensation 
and pension benefits payments to over 3.5 million veterans and their 
dependents each month. However, this system, which has been in 
operation for more than 40 years, is based on antiquated software 
programs, which have become increasingly difficult and costly to 
maintain. VBA is in the process of replacing the aging BDN with a 
faster, more flexible, and higher capacity system. When it began this 
project in 1996, which it generally refers to as the Veterans Service 
Network (VETSNET),[Footnote 1] VBA had planned to complete the 
replacement system in May 1998 at an estimated cost of $8 million. 

Over the years, we have reported on numerous problems that VBA has 
encountered in completing the replacement system.[Footnote 2] Our prior 
work found, for example, that the project was begun before VBA had 
fully developed its business requirements, resulting in confusion over 
the requirements to be addressed; in addition, VBA's software 
development capability was too immature to ensure that the agency could 
reliably develop and maintain high-quality software on any major 
project within cost and schedule constraints. In 2002, we offered a 
number of recommendations to improve managerial and program weaknesses, 
including that VBA appoint a project manager; thoroughly analyze its 
initiative; and develop a number of plans, including a revised 
compensation and pension replacement strategy and an integrated project 
plan. VA concurred with our recommendations, and as we last reported in 
June 2006,[Footnote 3] it took several actions to address them. For 
example, it appointed a full-time project manager, and the project team 
reported that it had completed certifications of users' requirements 
for the system's applications. Nonetheless, these actions did not 
implement all our recommendations and were not sufficient to establish 
the program on a solid footing: certain basic requirements of sound 
project management, such as an integrated project plan for the 
replacement system, continued to be lacking. 

In 2005, the VA Chief Information Officer (CIO) and Under Secretary for 
Benefits became concerned about continuing problems with the 
replacement project and contracted for an independent assessment of the 
department's options for the project, including whether the project 
should be terminated. This assessment, conducted by the Carnegie Mellon 
Software Engineering Institute (SEI), concluded that the replacement 
project faced many risks arising from management and organizational 
issues, but no technical barriers that could not be overcome.[Footnote 
4] According to SEI, a new system was still needed, and VBA would not 
be able to successfully deliver a full, workable solution unless it 
addressed its management and organizational weaknesses. SEI recommended 
that VBA continue to work on the project at a reduced pace, while 
taking an aggressive approach to addressing the identified weaknesses. 

Given the importance of ensuring the effective and efficient delivery 
of veterans' benefits, we were requested to review the department's 
continuing efforts to develop and implement the compensation and 
pension replacement system. Specifically, our objectives were to 
determine (1) to what extent VA has followed the course of action 
recommended by SEI and addressed the concerns that it raised and (2) 
the current status of the replacement project. 

In conducting this study, we visited VBA's Nashville Regional Office to 
observe processing of new benefits claims in the compensation and 
pension replacement system, and we visited the St. Petersburg Regional 
Office to observe processes and procedures used to test and validate 
key functionalities of the replacement system. We obtained and analyzed 
documents related to the replacement initiative and to SEI's review of 
the initiative. We supplemented our analysis with interviews of VBA and 
contractor personnel at headquarters and the two regional offices. 
Appendix I contains a more detailed discussion of our objectives, 
scope, and methodology. We conducted our study between April 2006 and 
April 2007 in accordance with generally accepted government auditing 
standards. 

Results in Brief: 

VBA is generally following the course of action recommended by SEI by 
continuing to work on the replacement initiative at a reduced pace, 
while taking action to address identified weaknesses in overall 
management and software development processes. For example, VBA 
established a new governance structure for the initiative that included 
senior management and involved all stakeholders, and it incorporated 
all critical areas of system development in an integrated master 
schedule. However, VBA did not address all of SEI's management 
concerns. Specifically, SEI advised VBA to ensure that stakeholders 
take ownership responsibility for the project, including the total 
system and process operating costs; however, although VBA is tracking 
costs incurred by contractors, it is not yet tracking and reporting in- 
house costs incurred by the project. Further, VBA has not yet developed 
sufficient plans for performing the substantial task of moving records 
from the BDN to the replacement system (approximately 3.5 million 
beneficiaries are currently being served by the older system). VBA did 
take steps to improve its software development processes, such as 
establishing risk and requirements management processes, but some have 
not been addressed. For example, still to be addressed are processes 
for capacity planning and management, which will be important for 
ensuring that further development does not lead to slowdowns in 
processing of benefits. Finally, VBA has not yet documented policies 
and procedures to institutionalize all the improvements that it has 
incorporated in the replacement initiative. According to the 
replacement project's management team, it made a conscious decision 
first to establish the governance and build the organization, among 
other things, and it is still prioritizing remaining tasks. However, if 
VBA does not institutionalize the improvements made, it increases the 
risk that these process improvements may not be maintained through the 
life of the project or be available for application to other 
development initiatives. 

After more than 10 years of effort, including the recent management, 
organizational, and process improvements, VBA has developed critical 
functionalities needed to process and pay certain original compensation 
claims using the replacement system, but it remains far from completing 
the project. According to VBA officials, all five of the major software 
applications that make up the new system are now being used in VA's 
regional offices to establish and process new compensation claims for 
veterans. In total, the replacement system is currently providing 
monthly compensation payments to almost 50,000 veterans (out of about 3 
million veterans who receive such payments); the system was used to 
process about 83 percent of all new compensation claims completed in 
March 2007. Nonetheless, the system requires further development before 
it can be used to process claims for the full range of compensation and 
pension benefits available to veterans and their dependents. In 
addition, VBA still faces the substantial task of converting records 
for the approximately 3.5 million beneficiaries currently being served 
by the BDN to the replacement system. 

To sustain the improved management and software development processes 
currently being used by VETSNET project management, we are making 
recommendations to the Secretary of Veterans Affairs in several areas, 
including cost tracking, capacity planning and management, and 
performance measures. 

In providing written comments on a draft of this report, the Secretary 
of Veterans Affairs agreed with our conclusions and concurred with the 
report's recommendations. (The department's comments are reproduced in 
app. II.) The comments described actions planned that respond to our 
recommendations, as well as providing further information on relevant 
actions already taken. If the planned actions are properly implemented, 
they could help strengthen the department's management of the 
replacement system project and improve the chances that the system will 
be successfully completed. 

Background: 

VBA provides benefits for veterans and their families through five 
programs: (1) compensation and pension, (2) education, (3) vocational 
rehabilitation and employment (VRE) services, (4) loan guaranty, and 
(5) life insurance. It relies on the BDN to administer benefit programs 
for three of VBA's five programs: compensation and pension, education, 
and VRE services. 

Replacing the aging BDN has been a focus of systems development efforts 
at VBA since 1986.[Footnote 5] Originally, the administration planned 
to modernize the entire system, but after experiencing numerous false 
starts and spending approximately $300 million on the overall 
modernization of the BDN, VBA revised its strategy in 1996. It narrowed 
its focus to replacing only those functionalities that support the 
compensation and pension program, and began developing a replacement 
system, which it called VETSNET. 

As reported by the department in its fiscal year 2008 budget 
submission, the compensation and pension program is the largest of the 
three programs that the BDN supports: 

* The compensation and pension program paid about $35 billion in 
benefits in fiscal year 2006 to about 3.6 million veterans or veterans' 
family members. 

- Of this amount, compensation programs paid benefits of about $31 
billion to about 3.1 million recipients. 

- Pension programs paid benefits of about $3.5 billion to about 535,000 
recipients. 

* The education program paid about $2.8 billion to about 498,000 
veterans or their dependents in fiscal year 2006.[Footnote 6] 

* The VRE services program paid about $574 million for VRE services in 
2006 and provides rehabilitation services to approximately 65,700 
disabled veteran participants per year.[Footnote 7] 

The Processes Supporting the Variety of Compensation and Pensions 
Benefits Are Complex: 

One of the challenges of developing the replacement system is that it 
must include processes to support the administration of a complex set 
of benefits. Different categories of veterans and their families are 
eligible for a number of different types of benefits and payments, some 
of which are based on financial need. Compensation programs, which are 
based on service-connected disability or death, provide direct payments 
to veterans and/or veterans' dependents and survivors. These programs 
are not based on income. Pension benefits programs, on the other hand, 
are income based; these are designed to provide income support to 
eligible veterans and their families who experience financial hardship. 
Eligible veterans are those who served in wartime and are permanently 
and totally disabled for reasons that are not service-connected (or who 
are age 65 or older). Veterans are also eligible for burial benefits. 

Survivor benefits may be paid to eligible survivors of veterans, 
depending on the circumstances. Some of these benefits are based on 
financial need, such as death pensions for some surviving spouses and 
children of deceased wartime veterans, and Dependency and Indemnity 
Compensation to some surviving parents. 

Finally, certain benefits may be paid to third parties, such as 
individuals to whom a veteran has given power of attorney or medical 
service providers designated to receive payments on the veteran's 
behalf. 

Generally, VBA administers benefit programs through 57 veterans 
benefits regional offices[Footnote 8] in a process that requires a 
number of steps, depending on the type of claim. When a veteran 
submits, for example, a compensation claim to any of the regional 
offices, a veterans service representative must obtain the relevant 
evidence to evaluate the claim (such as the veteran's military service 
records, medical examinations, and treatment records from VA medical 
facilities or private medical service providers). In the case of 
pension claims, income information would also be collected. 

Once all the necessary evidence has been compiled, a rating specialist 
evaluates the claim and determines whether the claimant is eligible for 
benefits. If the veteran is determined to be eligible for disability 
compensation, the Rating Veterans Service Representative assigns a 
percentage rating based on the veteran's degree of disability. This 
percentage is used in calculating the amount of payment. 

Benefits received by veterans are subject to change depending on 
changing circumstances. More than half of VBA's workload consists of 
dealing with such changes. If a veteran believes that a service- 
connected condition has worsened, for example, the veteran may ask for 
additional benefits by submitting another claim.[Footnote 9] The first 
claim submitted by a veteran is referred to as the original claim, and 
a subsequent change is referred to as a reopened claim. 

Our Prior Products Identified Weaknesses in the Development and 
Implementation of the Replacement System: 

Since its inception, VETSNET has been plagued by problems. Over the 
years, we have reported on the project, highlighting concerns about 
VBA's software development capabilities. 

In 1996, our assessment of the department's software development 
capability determined that it was immature.[Footnote 10] In our 
assessment, we specifically examined VETSNET and concluded that VBA 
could not reliably develop and maintain high-quality software on any 
major project within existing cost and schedule constraints. The 
department showed significant weaknesses in requirements management, 
software project planning, and software subcontract management, with no 
identifiable strengths. We also testified that VBA did not follow sound 
systems development practices on VETSNET, and we concluded that its 
modernization efforts had inherent risks. 

Between 1996 and 2002, we continued to identify the department's weak 
software development capability as a significant factor contributing to 
persistent problems in developing and implementing the system. We also 
reported that VBA continued to work on VETSNET without an integrated 
project plan. As a result, the development of the system continued to 
suffer from problems in several areas, including project management, 
requirements development, and testing. 

Over the years, we made several recommendations aimed at improving VA's 
software development capabilities. Among our recommendations was that 
the department take actions to achieve greater maturity in its software 
development processes[Footnote 11] and that it delay any major 
investment in software development (beyond that needed to sustain 
critical day-to-day operations) until it had done so. In addition, we 
made specific recommendations aimed at improving VETSNET development. 
For example, we recommended that VA appoint a project manager, 
thoroughly analyze its current initiative, and develop a number of 
plans, including a revised compensation and pension replacement 
strategy and an integrated project plan. 

VA concurred with our recommendations and took several actions to 
address them. For example, it appointed a full-time project manager and 
ensured that business needs were met by certification of user 
requirements for the system applications. The actions taken addressed 
some of our specific concerns; however, they were not sufficient to 
fully implement our recommendations or to establish the program on a 
sound footing. 

SEI Assessed System Replacement Initiative and Recommended a Changed 
Approach: 

As a result of continuing concerns about the replacement project, in 
2005 VA's CIO and its Under Secretary for Benefits contracted for an 
independent assessment of the department's options for the 
initiative.[Footnote 12] The chosen contractor, SEI, is a federally 
funded research and development center operated by Carnegie Mellon 
University. Its mission is to advance software engineering and related 
disciplines to ensure the development and operation of systems with 
predictable and improved cost, schedule, and quality. 

SEI recommended that the department reduce the pace of development 
while at the same time taking an aggressive approach to dealing with 
management and organizational weaknesses hampering VBA's ability to 
complete the replacement system. According to SEI, these management and 
organizational concerns needed to be addressed before the replacement 
initiative or any similar project could deliver a full, workable 
solution. 

For example, the contractor stressed the importance of setting 
realistic deadlines and commented that there was no credible evidence 
that VETSNET would be complete by the target date, which at the time of 
the review had slipped to December 2006. According to the assessment, 
because this deadline was unrealistic, VBA needed to plan and budget 
for supporting the BDN so that its ability to pay veterans' benefits 
would not be disrupted. SEI also noted that different organizational 
components had independent schedules and priorities, which caused 
confusion and deprived the department of a program perspective. 
Further, the contractor concluded that VBA needed to give priority to 
establishing sound program management to ensure that the project could 
meet targeted dates. These and other observations were consistent with 
our long-standing concerns regarding fundamental deficiencies in VBA's 
management of the project. 

To help VBA implement the overall recommendation, the contractor's 
assessment included numerous discussions of activities needed to 
address these areas of concern, which can be generally categorized as 
falling into two major types: 

* Overall management concerns with regard to the initiative included: 

- governance structure, including assigning ownership for the project 
and its costs; 

- project planning, including the development schedule and capacity 
planning; and: 

- conversion of records currently on the BDN to the replacement system. 

* Software development process improvements were needed in the 
following areas: 

- risk management, 

- requirements management, 

- defect management, and: 

- program measures. 

VBA Is Following Course Recommended by SEI, but Weaknesses Remain to Be 
Addressed: 

As recommended by SEI, VBA is continuing to work on the replacement 
initiative at a reduced pace and taking action to address identified 
weaknesses in the project's overall management and software development 
processes. For example, VBA has established a new governance structure 
and has developed an integrated master schedule that provides 
additional time and includes the full range of project activities. 
However, additional effort is needed to complete a number of the 
corrective actions, such as improving project accountability through 
monitoring and reporting all project costs. Further, VBA has not yet 
institutionalized many of the improvements that it has undertaken for 
the initiative. In particular, process improvements remain in draft and 
have not been established through documented policies and procedures. 
According to the VETSNET management team, it gave priority to other 
activities, such as establishing appropriate governance and 
organizational structures, and it is still gathering information to 
assist in prioritizing the activities that remain. Nonetheless, if VBA 
does not institutionalize these improvements, it increases the risk 
that these process improvements may not be maintained through the life 
of the project or be available for application to other development 
initiatives. 

VBA Is Taking Action on Certain Overall Management Concerns: 

SEI concluded that VBA's management issues would need to be addressed 
as part of the implementation of its overall recommendation. SEI's 
overall management concerns focused on the project governance, project 
planning, and conversion of records currently on the BDN to the 
replacement system. 

Top Management Included in Revised Governance Structure: 

SEI guidance for software development stresses the need for 
organizational commitment and the involvement of senior management in 
overall project governance.[Footnote 13] In its assessment, SEI noted 
that because management of the VETSNET project had been assigned to 
VBA's information technology (IT) group, certain activities critical to 
the veterans' benefits program, but not traditionally managed by the IT 
group, had not been visible to the project's management. The contractor 
pointed out that the IT group, business lines, and regional offices 
needed to share ownership and management of the replacement project 
through an established governance process and that the project 
management office should include business representatives. According to 
SEI, the project needed to establish ownership responsibility, 
including addressing total system and process operating costs. 

In response to the assessment, VBA developed a new governance structure 
for the initiative, which the Under Secretary for Benefits approved in 
March 2006. In the new structure, the VETSNET Executive Board that had 
been in place was expanded and reorganized to serve as a focal point 
and major governance mechanism for the replacement initiative. A 
Special Assistant (reporting directly to the Under Secretary) was 
appointed to coordinate and oversee the initiative as the head of the 
VETSNET Executive Team, which was established to provide day-to-day 
operational control and oversight of the replacement initiative. 
Implementation Teams were also established to conduct the day-to-day 
activities associated with implementing the initiative. This governance 
structure established a process for IT, business lines, and regional 
offices to share ownership and management, as SEI advised. 

The roles and membership of each of the organizational elements in the 
new governance structure are described in table 1. 

Table 1: Roles and Responsibilities of VETSNET Organizational Elements: 

Organization: VETSNET Executive Board; 
Role: Provides executive direction to the VETSNET Executive Team; 
Membership: Deputy Under Secretary for Benefits; Special Assistant to 
the Under Secretary for Benefits; VETSNET Program Manager; VA Office of 
Information and Technology Principal Associate Deputy Assistant 
Secretary; Chief Financial Officer; Director, Compensation and Pension 
Service; Director, Office of Performance Analysis and Integrity; 
Associate Deputy Under Secretary for Policy and Programs; Associate 
Deputy Under Secretary for Field Operations; VBA Chief Information 
Officer. 

Organization: VETSNET Executive Team; 
Role: Responsible for the delivery of the project and acts as the 
review and approval authority for the Implementation Teams; 
Membership: VETSNET Program Executive (Special Assistant to Under 
Secretary); VETSNET Program Manager; Contract support; Business 
Architect; Technical Architect; MITRE support. 

Organization: VETSNET Implementation Teams; 
Role: Responsible for the day-to-day activities necessary to develop 
and implement the replacement system; 
Membership: Compensation and Pension; Office of Resource Management; 
Office of Performance Analysis and Integrity; Office of Field 
Operation; Office of Information Management, VA Central Office; St. 
Petersburg Systems Development Center; Hines Information Technology 
Center; Austin Automation Center. 

Source: GAO analysis of VBA information. 

[End of table] 

When the new governance structure was approved in March 2006, the Under 
Secretary ensured that those involved in the project gave it high 
priority, directing certain key personnel (such as members of the 
executive and implementation teams) to make the initiative their 
primary responsibility, and other personnel (technical staff that 
provide support to other systems) with collateral (non-VETSNET) duties 
to make the project their first priority. He also placed limitations on 
the transfer of personnel away from the project, recognizing the 
importance of staff continuity in successfully completing the 
initiative. Staff members assigned project responsibilities could be 
reassigned (i.e., given nonpromotion, lateral reassignments) only with 
approval from the Under Secretary or his deputy. 

By implementing the new governance and organizational structure and 
ensuring that the project has priority, VBA partially responded to 
SEI's concerns in this area; however, VBA has not yet taken action with 
regard to ownership responsibility for total system and process 
operating costs, as SEI advised. According to administration officials, 
the replacement initiative is an in-house, contractor-assisted 
development effort, in which three different contractors provide 
support for program management, system development, and testing and 
validation of requirements. VA reported VETSNET system costs to the 
Congress totaling about $89 million for fiscal years 1996 through 2006, 
with additional estimated costs for completion of the initiative in 
2009 of about $62.4 million. However, according to project management 
officials, these costs do not include expenditures for in-house 
development work. This in-house work involves many VA personnel, as 
well as travel to various locations for testing and other project 
related activities. Thus, considerable costs other than contract cost 
have been incurred, which have not been tracked and reported as costs 
for the replacement initiative. Without comprehensive tracking and 
reporting of costs incurred by the replacement project, the ability of 
VBA and the Congress to effectively monitor progress could be impaired. 

VBA Took Action to Address Project Planning and Management: 

A second major area of overall management concern was project planning. 
In particular, the lack of an integrated master schedule for the 
VETSNET project was a major concern articulated by SEI, as well as in 
our prior work. An integrated project plan and schedule should 
incorporate all the critical areas of system development and be used as 
a means of determining what needs to be done and when, as well as 
measuring progress. Such an integrated schedule should consider all 
dependencies and include subtasks so that deadlines are realistic, and 
it should incorporate review activities to allow oversight and approval 
by high-level managers. Among other things, the program plan should 
also include capacity requirements for resources and technical 
facilities to support development, testing, user validation, and 
production. 

SEI was specifically concerned that releases with overlapping 
functionality were being developed at the same time, with insufficient 
time to document or test requirements; this approach constrained 
resources and added complexity because of the need to integrate 
completed applications and newly developed functionality. In addition, 
SEI observed that the VETSNET program suffered from lack of sufficient 
test facilities because it did not have enough information to plan for 
adequate capacity. 

In response to these project planning concerns, VETSNET management, 
with contractor support, developed an integrated master schedule to 
guide development and implementation of the remaining functionalities 
for the replacement system. The VETSNET Integrated Master Schedule, 
finalized in September 2006, includes an end-to-end plan and a master 
schedule. According to VBA, the end-to-end plan documents the end state 
of the project from a business perspective, which had not previously 
been done. The master schedule identifies the necessary activities to 
manage and control the replacement project through completion. The 
schedule also describes a new software release process that provides 
more time to work on requirements definition and testing, and allows 
for more cross-organizational communications to lessen the possibility 
of not meeting requirements. 

In addition, the new release process includes a series of management 
reviews to help control the software development process and ensure 
that top management has continuous visibility of project related 
activities. These reviews occur at major steps in the system 
development life cycle (as described in fig. 1: initiation, preliminary 
design, and so on). Such reviews are intended to ensure that the 
VETSNET Executive Team and the VETSNET Executive Board agree and accept 
that the major tasks of each step have been properly performed. 

Figure 1: VETSNET System Life Cycle and Review Concept: 

[See PDF for image] 

Source: VBA. 

[End of figure] 

Nonetheless, while the Integrated Master Schedule is an important 
accomplishment, it may not ensure that the project sufficiently 
addresses capacity planning, one of SEI's areas of concern. According 
to its assessment, capacity requirements for the fully functional 
production system were unclear. Capacity planning is important because 
program progress depends on the availability of necessary system 
capacity to perform development and testing; adjustments to such 
capacity take time and must be planned. If systems do not have adequate 
capacity to accommodate workload, interruptions or slowdowns could 
occur. According to SEI, capacity adjustments cannot be made instantly, 
and program progress will suffer without sufficient attention to 
resource requirements. 

However, the VETSNET Integrated Master Schedule does not identify 
activities or resources devoted to capacity planning. According to 
officials, the capacity of the corporate environment (that is, 
corporate information systems, applications, and networks) is being 
monitored by operational teams with responsibility for maintaining this 
environment. According to project officials, VETSNET representatives 
participate in daily conference calls in which the performance of 
corporate applications is discussed, and changes in application 
performance are reported to the VETSNET developers for investigation 
and corrective action. Project officials reported that when a 
performance degradation occurred in some transactions during 
performance testing, it was determined that additional computing 
capacity was needed and would be acquired. One reason why the 
occurrence of degradation had not been anticipated by the VETSNET 
project was that capacity planning had not taken place. Unless it 
ensures that capacity planning and activities are included in the 
Integrated Master Schedule, the replacement project may face other 
unanticipated degradations that it must react to after the fact, thus 
jeopardizing the project's cost, schedule, and performance. 

Conversion Efforts Were Suspended as Advised, but Issues Remain: 

In its assessment, SEI questioned VBA's approach to developing 
functionality while concurrently converting records from the BDN to the 
replacement system. It noted that VBA had chosen to complete software 
development according to location rather than according to the type of 
functionality. Specifically, in 2004, VBA began an effort to remove all 
claims activity (both new and existing claims) at one regional office 
(Lincoln, Nebraska) from the BDN to the replacement system, developing 
the software as necessary to accommodate processing the types of claims 
encountered at that site. The intention was to address each regional 
office in turn until all sites were converted. According to SEI, this 
approach had resulted in the development being stalled by obstacles 
arising from the variety of existing claims.[Footnote 14] The 
contractor advised VBA to focus first on developing functionality to 
process original claims and discontinue efforts to convert existing 
claims until all the necessary functionality had been developed, and 
the replacement system's ability to handle new cases of any complexity 
had been proven by actual experience. 

In accordance with this advice, VBA stopped converting existing records 
from the BDN and changed its focus to developing the necessary 
functionality to process all new compensation claims. According to the 
integrated master schedule, conversion activities are now timed to 
follow the release of the needed functionality. That is, according to 
the schedule, VBA plans to begin converting each type of record from 
the BDN only after the necessary functionality for the replacement 
system has been developed and deployed to process that type of record. 

In addition, the project is mitigating risk by resuming conversions 
beginning with a test phase. Its strategy is first to convert records 
for terminated claims--claims that are no longer being paid. Conversion 
of the terminated records will be followed by additional conversions of 
records for claims receiving payment at Lincoln and Nashville (these 
two sites are being used to test system functionality during 
development). The VETSNET leadership will consider testing complete 
with the successful conversion at these two sites. 

However, SEI raised three additional issues with regard to the 
conversion of records that VBA has not fully addressed: 

* First, SEI expressed concerns that conversion failures could lead to 
substantial numbers of records being returned to the BDN. Because of 
differences in the database technologies used for the old system and 
the replacement system, certain types of errors in BDN records cause 
conversion to fail (according to SEI, approximately 15 percent of all 
these records are estimated to have such errors). If records fail to 
convert correctly, they may need to be returned to the BDN so that 
benefits can continue to be paid. However, this process is not simple 
and may involve manually reentering the records.[Footnote 15] 

* Second, SEI observed that VBA was also depending on manual processes 
for determining that records were converted successfully, including the 
use of statistically random samples, and that it was aiming to ensure 
correctness to a confidence level of 95 percent. However, in the 
absence of a straightforward method for automatically returning records 
to the BDN, SEI considered the 5 percent risk of error unacceptable for 
conversions of large numbers of records. 

* Finally, SEI observed that the lack of automated methods and the 
complexity of the processes meant that conversions required careful 
planning and assurance that adequate staff would be available to 
validate records when the conversions took place. 

However, the VETSNET leadership has not developed any strategy to 
address the possibility that a large number of cases might need to be 
returned to the BDN during the testing phase. For example, it has not 
included this possibility as a risk in its risk management plan. The 
absence of a strategy to address this possibility could lead to delays 
in program execution. 

Further, VBA has not yet decided whether a possible 5 percent error 
rate is acceptable or developed a plan for addressing the resulting 
erroneous records. If VBA does not address these issues in its 
planning, it increases the risk that veterans may not receive accurate 
or timely payments. 

Finally, the VETSNET leadership has not yet developed detailed plans 
that include the scheduled conversions for each regional office and 
identified staff to perform the necessary validation. Having such plans 
would reduce the risk that the conversion process could be delayed or 
fail. 

VBA Is Improving Software Development Processes, but Improvements Are 
Not Yet Institutionalized: 

In addition to actions addressing the overall management concerns 
identified by SEI, VBA has steps action to improve its software 
development processes in risk management, requirements management, 
defect/change management, and performance measures. SEI described 
weaknesses in all of these areas. The steps taken have generally been 
effective in addressing the identified weaknesses, but VBA has not yet 
institutionalized many of these improvements. According to the VETSNET 
management team, it made a conscious decision first to establish the 
governance, build the organization, implement processes to gain 
control, and gather additional information about the project to assist 
in prioritizing the remaining activities. The team also stated that 
some of the processes are no longer VBA's responsibility but are now 
that of the newly realigned Office of Information and 
Technology.[Footnote 16] Nonetheless, if VA does not develop and 
establish documented policies and procedures to institutionalize these 
improvements, they may not be maintained through the life of the 
project or available to be applied to other development initiatives. 

Risk Management Plan Has Been Revised: 

Risk management is a process for identifying and assessing risks, their 
impact and status, the probability of their occurrence, and mitigation 
strategies. Effective risk management includes the development of a 
risk management plan and tracking and reporting progress against the 
plan. According to SEI, to the extent that risk management existed at 
all in the replacement program, it was conducted on a pro forma basis 
without real effect on program decisions. SEI said that risks and risk 
mitigation activities needed to be incorporated into all aspects of 
program planning, budgeting, scheduling, execution, and review. 

In response to these concerns, VBA has instituted risk management 
activities that, if properly implemented, should mitigate the risks 
associated with the project. Specifically, the VETSNET team, with 
contractor support, developed a risk management plan that was adopted 
in January 2007. The plan includes procedures for identifying, 
validating, analyzing, assessing, developing mitigation strategies for, 
controlling and tracking, reporting, and closing risks. It also 
establishes criteria for assessing the severity of the risks and their 
impact. 

The VETSNET leadership also developed a Risk Registry database, and its 
contractor reviewed and prioritized the open risks. Each open risk was 
evaluated, and a proposed disposition of the risk was submitted to 
VETSNET management. Of the 39 open risks, all but 3 had been addressed 
as of January 2007. 

The development documentation for each planned software release also 
includes sections on risk. In accordance with these plans, the VETSNET 
leadership is currently capturing potential risks and tracking action 
items and issues. At weekly status meetings, VETSNET leadership reviews 
Risk Registry reports of open risks. According to the contractor, the 
reports identify each risk and provide information on its age, 
ownership, and severity. 

However, these risk management activities have not yet been 
institutionalized through the definition and establishment of 
associated policies and procedures. If it does not institutionalize 
these improvements, VBA increases the possibility that the VETSNET 
project's improvements in risk management may not be maintained through 
the life of the project. 

Requirements Management Has Improved: 

Requirements management is a process for establishing and maintaining a 
common understanding between the business owners and the developers of 
the requirements to be addressed, as well as verifying that the system 
meets the agreed requirements. SEI's report commented that the VETSNET 
project requirements were not stable, and that the business owners 
(including subject-matter experts) and developers were separated by 
many organizational layers, resulting in confusion and delays in 
development of the system. SEI suggested that VA restructure project 
activities to focus on defining an effective requirements process. 
According to SEI, the project needed to ensure that subject-matter 
experts were included in developing requirements and that evaluation 
criteria were established for prioritizing requests for changes to 
requirements. Finally, business owners should confirm that the system 
is meeting organizational needs. 

VBA has instituted requirements management activities that, if properly 
implemented, should help avoid the instability and other requirements 
problems identified by SEI. Specifically, VBA took steps to establish a 
requirements management process and to stabilize the requirements. For 
example, the development release process in the Integrated Master 
Schedule includes a phase for requirements identification. In addition, 
the project has established and begun applying evaluation criteria to 
prioritize change requests for its development releases. Further, until 
all claims are completely migrated from the BDN to the replacement 
system, in July 2006, the Under Secretary directed that any additional 
requirements would have to have his approval. 

Responding to SEI's advice regarding the involvement of subject-matter 
experts and business owners, VBA designed the new release process to 
directly involve subject-matter experts in requirements workshops. 
Further, the business teams participate in user-acceptance testing. 

However, these requirements management activities have not yet been 
institutionalized through the definition and establishment of policies 
and procedures. Until they are established, VBA runs the risk that the 
improved processes will not be maintained through the life of the 
VETSNET project or used in other software development projects. 

Management Attention Is Being Focused on Major Software Defects: 

SEI raised numerous concerns regarding the defect process for the 
replacement system. These concerns for defect management included (1) 
identification of defects, (2) determination of cause, and (3) 
disposition of defects--either by correction or workaround. According 
to SEI guidance, defect management prevents known defects from 
hampering the progress of the program. The management process should 
include clearly identifying and tracking defects, analyzing defects to 
establish their cause, tracking their disposition, clearly identifying 
the rationale for not addressing any defects (as well as proposing 
workarounds), and making information on defects and their resolution 
broadly available. SEI's report stated that VBA needed to distinguish 
defects from changes to requirements and develop a process for defect 
management. 

To respond to these concerns and focus program management attention on 
major defects, the VETSNET Executive Team, with contractor support, 
conducted an audit of existing defects and revised the defect 
management process. The audit of the defect database determined that 
the VETSNET database used to capture software defects also included 
change requests; as a result, work required to address processes that 
did not work properly was not distinguished from requests for added or 
changed functionality, which would require review and approval before 
being addressed. To address this issue, the team separated defects from 
change requests, and a new severity rating scale was developed. All 
open defects were recategorized to ensure the major defects would 
receive appropriate program management attention. Also, all defect 
categorizations must meet the approval of the VETSNET Business 
Architect and are scheduled for action as dictated by the severity 
level. 

Although these steps address many of SEI's concerns regarding VBA's 
defect management process, more remains to be done before the process 
is institutionalized. The Program Management Office has reported that 
actions to revise the defect management process are complete, but the 
process description is still in draft, and policies and procedures have 
not been fully established. Without institutionalized policies and 
procedures for the defect management process, it may not be maintained 
consistently through the life of the project. 

The Replacement Initiative Is Tracking Certain Performance Measures: 

According to SEI, performance measures are the only effective mechanism 
that can provide credible evidence of a program's progress. The chosen 
measures must link directly to the expected accomplishments and goals 
of the system, and they must be applied across all activities of the 
program. In its report, SEI stated that although VBA was reporting 
certain types of performance measures, it was not relating these to 
progress in system development. For example, VBA reported the total 
number of veterans paid, but did not provide estimates of how many 
additional veterans would be paid when the system incorporated specific 
functionalities that were under development. SEI suggested several 
measures that would provide more evidence of progress, such as 
increases in the percentage of original claims being paid by the 
replacement system, as well as user satisfaction and productivity gains 
resulting from use of the replacement system applications at regional 
offices. 

In response to these concerns, the replacement project has begun 
tracking a number of the measures suggested by SEI, including: 

* increases in the percentage of original claims being paid by the 
replacement system, 

* increases in the percentage of veterans' service representatives 
using the new system, 

* decreases in the percentage of original claims being entered in the 
BDN rather than the replacement system. 

Although these measures provide indications of VA's progress, other 
measures that could demonstrate the effectiveness of the replacement 
system have not been developed. For example, VBA has not developed 
results-oriented measures to capture user satisfaction or productivity 
gains from the system. Without measuring user satisfaction, VBA has 
reduced assurance that the replacement system will be accepted by the 
users. In addition, measures of productivity would provide VBA with 
another indication of progress toward meeting business needs. 

VETSNET Is Currently Processing a Portion of Compensation Claims, but 
Much Work Remains: 

After more than 10 years of effort, including the recent management, 
organizational, and process improvements, VBA has achieved critical 
functionalities needed to process and pay certain original compensation 
claims using the replacement system, but it remains far from completing 
the project. For example, the replacement system is currently being 
used to process a portion of the original claims that veterans file for 
compensation. Nonetheless, the system requires further development 
before it can be used to process claims for the full range of 
compensation and pension benefits available to veterans and their 
dependents. In addition, VBA still faces the substantial task of moving 
approximately 3.5 million beneficiaries who are currently being served 
by the BDN to the replacement system. 

As designed, VETSNET consists of five major system applications that 
are used in processing benefits: 

* Share--used to establish claims;[Footnote 17] it records and updates 
basic information about veterans and dependents both in the BDN and the 
replacement system. 

* Modern Award Processing-Development (MAP-D)--used to manage the 
claims development process, including the collection of data to support 
the claims and the tracking of claims. 

* Rating Board Automation 2000 (RBA 2000)--provides laws and 
regulations pertaining to disabilities, which are used by rating 
specialists in evaluating and rating disability claims. 

* Award Processing (Awards)--used to prepare and calculate the benefit 
award based on the rating specialist's determination of the claimant's 
percentage of disability. It is also used to authorize the claim for 
payment. 

* Finance and Accounting System (FAS)--used to develop the actual 
payment record. FAS generates various accounting reports and supports 
generation and audit of benefit payments. 

According to VBA officials, all five of the software applications that 
make up the new system are now being used in VA's 57 regional offices 
to establish and process new compensation claims for veterans. As of 
March 2007, VBA leadership reported that the replacement system was 
providing monthly compensation payments to almost 50,000 veterans (out 
of about 3 million veterans who receive such payments). In addition, 
the replacement system has been processing a steadily increasing 
percentage of all new compensation claims completed: this measure was 
47 percent in January 2007, increasing to 60 percent in February and 83 
percent in March. 

Nonetheless, considerable work must be accomplished before VBA will be 
able to rely on the replacement system to make payments to all 
compensation and pension beneficiaries. Specifically, while all five 
software applications can now be used to process original compensation 
claims for veterans, two of the applications--Awards and FAS--require 
further development before the system will be able to process claims 
for the full range of benefits available to veterans and their 
dependents. Table 2 shows the status of development of all five 
applications. 

Table 2: VETSNET Component Applications and Status of Development: 

Application: Share; 
Development status: Completed; 
Deployment status: Fully deployed at all regional offices. 

Application: MAP-D; 
Development status: Completed; 
Deployment status: Fully deployed at all regional offices. 

Application: RBA 2000; 
Development status: Completed; 
Deployment status: Fully deployed at all regional offices. 

Application: Awards; 
Development status: Functionality to process third-party/nonveteran 
payee claims is in application testing; Requirements for processing 
survivor benefits are being developed; Functionality to process 
pensions is not yet under development; 
Deployment status: In partial use in all regional offices. 

Application: FAS; 
Development status: Functionality to process third- party/nonveteran 
payee claims is in application testing; Functionality to generate 
management reports is partially in requirements development and 
partially in application testing; Requirements for processing survivor 
benefits are being developed; Functionality to process pensions is not 
yet under development; 
Deployment status: In partial use in all regional offices. 

Source: GAO analysis of VBA data. 

[End of table] 

According to VBA officials, Awards and FAS do not yet have the 
capability to process original claims for payment to recipients other 
than veterans: that is, the applications do not have the functionality 
to process claims for survivor benefits[Footnote 18] and third-party/ 
nonveteran payee claims.[Footnote 19] In addition, further development 
of these applications is needed to process pension benefits for 
qualified veterans and their survivors. Until enhancements are made to 
Awards and FAS, these claims must continue to be processed and paid 
through the BDN. Also, according to VBA, FAS does not yet have the 
capability to generate all the necessary accounting reports that 
support the development of benefits payments to claimants. 

As described earlier, VBA now has an Integrated Master Schedule that 
incorporates the activities that VBA needs to manage in order to 
complete the replacement project. According to the schedule, the 
remaining capabilities necessary to process compensation and pension 
claims are to be developed and deployed in three software releases, as 
shown in table 3. 

Table 3: Software Releases Planned to Complete the VETSNET System: 

Release number and title: 1. Complete Compensation; 
Functionality to be attained: Process all compensation claims and add 
significant FAS functionality (generate management reports); 
Estimated completion date: August 2007. 

Release number and title: 2. Nonincome-Based Survivor Benefits; 
Functionality to be attained: Process burial, accrued, and survivor 
claims; 
Estimated completion date: February 2008. 

Release number and title: 3. Income-Based Pension; 
Functionality to be attained: Process nonservice-connected pension 
(veteran and survivor) and parents' Dependent and Indemnity 
Compensation Benefit; 
Estimated completion date: August 2008. 

Source: VBA. 

[End of table] 

As table 3 shows, VBA does not expect to complete the development of 
the functionalities needed to process all new compensation and pension 
claims until August 2008. However, according to VBA, the estimated 
completion date is the planned date for completing all development and 
testing, but it is not necessarily the date when users will be able to 
begin using the new system. Before such use can begin, other activities 
need to occur. For example, users must receive training, and VETSNET 
program management must authorize the use of the system at each 
regional office. 

In addition to its remaining software development activities, VBA also 
faces the challenge of converting records for claims currently paid by 
the BDN to the replacement system. Existing compensation and pension 
cases on the BDN number about 3 million and about 535,000, 
respectively. Table 4 shows the phases in which VA is planning to 
perform conversions, according to its Integrated Master Schedule. 

Table 4: Conversion Phases: 

Phases: Phase 1; 
Conversions: Conversion testing on terminated records and selected live 
records; 
Estimated start date: March 2007; 
Estimated completion date: August 2007. 

Phases: Phase 2; 
Conversions: Conversion of live compensation records; 
Estimated start date: January 2008; 
Estimated completion date: October 2008. 

Phases: Phase 3; 
Conversions: Conversion of pension records; 
Estimated start date: November 2008; 
Estimated completion date: June 2009. 

Source: GAO analysis of VBA data. 

[End of table] 

As the table shows, VBA conversion efforts began in March 2007. VBA 
first performed conversion testing on 310,000 terminated (that is, 
inactive) compensation cases so that it could develop and apply lessons 
learned to the conversion of live records. According to VETSNET 
officials, VBA planned to continue testing by converting live cases at 
two regional offices (Lincoln and Nashville) that were used as testing 
sites during development. It then plans to perform the conversion of 
all live compensation cases. After the compensation conversion is 
complete, VBA plans to begin efforts to convert pension benefits cases. 

Based on VETSNET project documentation, activities supporting the 
releases have so far been performed on time, consistent with the 
milestones in the recently finalized Integrated Master Schedule. For 
example, VA completed the Project Initiation and Review Authorization 
for Release 1 on September 7, 2006, as scheduled (see fig. 1, shown 
earlier in the report, for the phases of system development and the 
required milestone reviews). It also completed the Preliminary Design 
Review and the Critical Design Review as scheduled (on November 20 and 
December 22, respectively). Planning for Release 2 is also on schedule: 
a kickoff meeting was held on January 24, 2007, which established the 
scope of the release, and the Project Initiation and Review 
Authorization was conducted on February 8. 

Conclusions: 

VA has responded to SEI's assessment by making significant changes in 
its approach to the project and its overall management, including 
slowing the pace of development, establishing a new governance 
structure, and ensuring staff resources. However, VBA has not yet 
addressed all the issues raised by the SEI assessment. That is, it has 
not ensured ownership responsibility for total system and process 
operating costs, because it is not currently monitoring and reporting 
in-house expenditures for the project. It has not defined processes and 
resources for capacity planning for the project. In addition, VBA has 
not yet addressed issues related to the conversion of records now on 
the BDN to the replacement system. Specifically, it has not addressed 
the risk that large numbers of records may need to be returned to the 
BDN, decided on the degree of confidence it will require that records 
are converted accurately, or developed complete plans for converting 
and validating records. In addition, although VBA has improved key 
processes for managing the software development, these processes have 
not yet been institutionalized in defined policies and procedures, and 
performance measures of productivity and user satisfaction have not 
been developed. VETSNET management has stated that it gave priority to 
other activities, such as establishing appropriate governance and 
organizational structures, and that it is still gathering information 
to assist in prioritizing the activities that remain. 

Much work remains to be done to complete the VETSNET initiative. 
Although VBA has substantially increased the number of claims being 
paid by the replacement system, it must not only finish the development 
and deployment of the software, it must also convert the over 3.5 
million records now on the BDN to the replacement system. Addressing 
the remaining issues identified by SEI would improve VBA's chances of 
successfully completing the replacement system and ending reliance on 
the aged BDN to pay compensation and pension benefits. 

Recommendations for Executive Action: 

To enhance the likelihood that the replacement system will be 
successfully completed and implemented, we are recommending that the 
Secretary of Veterans Affairs take the following five actions: 

* Direct the CIO to institute measures to track in-house expenditures 
for the project. 

* Direct the VETSNET project to include activities for capacity 
planning in the VETSNET Integrated Master Schedule and ensure that 
resources are available for these activities. 

* Direct VBA to (1) develop a strategy to address the risk that large 
numbers of records may need to be returned to the BDN; (2) determine 
whether a greater confidence level for accuracy should be required in 
the conversion process; and (3) develop a detailed validation plan that 
includes the scheduled conversions for each regional office and the 
validation team members needed for that specific conversion. 

* Direct the CIO to document and incorporate the improved processes for 
managing risks, requirements, and defects into specific policy and 
guidance for the replacement initiative and for future use throughout 
VBA. 

* Direct the replacement project to develop effective results-oriented 
performance measures that show changes in efficiency, economy, or 
improvements in mission performance, as well as measures of user 
satisfaction, and to monitor and report on the progress of the 
initiative according to these measures. 

Agency Comments and Our Evaluation: 

In providing written comments on a draft of this report, the Secretary 
of Veterans Affairs agreed with our conclusions and concurred with the 
report's recommendations. (The department's comments are reproduced in 
app. II.) The comments described actions planned that respond to our 
recommendations, such as incorporating processes developed for the 
VETSNET project in standard project management policies, processes, and 
procedures that would be used for all IT projects in the department. In 
addition, the comments provided further information on actions already 
taken, such as details of the records conversion process. If the 
planned actions are properly implemented, they could help strengthen 
the department's management of the replacement system project and 
improve the chances that the system will be successfully completed. 

We are sending copies of this report to the Chairman and Ranking 
Minority Member of Committee on Veterans' Affairs. We are also sending 
copies to the Secretary of Veterans Affairs and appropriate 
congressional committees. We will make copies available to other 
interested parties upon request. Copies of this report will also be 
made available at no charge on GAO's Web site at http://www.gao.gov. 

Should you or your staff have any questions about this report, please 
contact me at (202) 512-6304 or by e-mail at melvinv@gao.gov. Contact 
points for our Offices of Congressional Relations and Public Affairs 
may be found on the last page of this report. Key contributors to this 
report are listed in appendix III. 

Sincerely yours, 

Signed by: 

Valerie C. Melvin: 
Acting Director: 
Human Capital and Management Information Systems: 

[End of section] 

Appendix I: Objectives, Scope, and Methodology: 

Our objectives were to determine (1) to what extent the Department of 
Veterans Affairs (VA) has followed the course of action recommended by 
the Carnegie Mellon Software Engineering Institute (SEI) and addressed 
the concerns that it raised and (2) the current status of the 
replacement project, the Veterans Service Network (VETSNET). 

To determine the actions taken to implement SEI's recommended approach 
and address the concerns it raised, we: 

* determined the recommended actions by analyzing the report; 

* compared the concerns identified in the assessment to actions 
planned, actions undertaken but not completed, and actions implemented 
by VA officials or contractors; 

* interviewed contractor, VA, and VETSNET program office officials to 
gain an understanding about processes developed and procedures 
implemented; and: 

* obtained and reviewed relevant VA and contractor documents that 
disclosed or validated VA responses to SEI's concerns. 

To determine the status of system development efforts and the extent 
that tasks planned for the initiative were completed, we analyzed VA 
and contractor documentation regarding system operations and 
development, time frames, and activities planned. We analyzed VA 
documents that disclosed costs to date and costs planned for completion 
of the initiative. We did not assess the accuracy of the cost data 
provided to us. We supplemented our analyses with interviews of VA and 
contractor personnel involved in the replacement initiative. 

We visited the Nashville and St. Petersburg regional offices to observe 
the replacement system in operation and the processes and procedures 
used to test and validate the replacement system as it was being 
developed and implemented. We analyzed VA documentation and relevant 
evidence from contractors involved in the replacement effort to 
establish the work remaining to complete the project. Finally, we 
interviewed cognizant VA and contractor officials, responsible for 
developing, testing, and implementing the replacement system. 

We performed our work at VA offices in Washington, D.C., and at VA 
regional offices in Nashville, Tennessee, and St. Petersburg, Florida, 
from April 2006 to April 2007 in accordance with generally accepted 
government auditing standards. 

[End of section] 

Appendix II: Comments from the Department of Veterans Affairs: 

The Secretary Of Veterans Affairs: 
Washington: 

April 18, 2007: 

Ms. Valerie C. Melvin: 
Acting Director: 
Human Capital and Management Information Systems: 
Information Technology Team: 
U. S. Government Accountability Office: 
441 G Street, NW: 
Washington, DC 20548: 

Dear Ms. Melvin: 

The Department of Veterans Affairs (VA) has reviewed your draft report, 
Veterans Benefits Administration: Progress Made in Long-Term Effort to 
Replace Benefits Payment System, but Challenges Persist, GAO 07-614) 
and agrees with your conclusions and concurs in your recommendations. 

VA is eager to move to a new automated system that will streamline its 
benefit claims processing functions in a manner that is accurate and 
timely to serve our Nation's veterans. The Government Accountability 
Office's recommendations should provide valuable assistance in 
achieving that goal. The enclosure addresses your recommendations in 
detail. 

I appreciate the opportunity to comment on your draft report. 

Sincerely yours, 

Signed by: 

R. James Nicholson: 

Enclosure: 

Department of Veterans Affairs (VA) comments to Government 
Accountability Office (GAO) draft report Veterans Benefits 
Administration: Progress Made in Long-Term Effort to Replace Benefits 
Payment System, but Challenges Persist (GAO-07-614): 

To enhance the likelihood that the replacement system will be 
successfully completed and implemented, GAO recommends that the 
Secretary of Veterans Affairs take the following actions: 

* direct the CIO to institute measures to track in-house expenditures 
for the project; 

Concur - VA's Chief Information Officer (CIO) is committed to tracking 
the in-house expenditures required to complete the Veterans Service 
Network (VETSNET) project. In addition to tracking Government full time 
employee (FTE) and contractor costs, the VETSNET Project Management 
Office will take responsibility for tracking travel costs associated 
with the project to include those related to application testing and 
training. 

* direct the VETSNET project to include activities for capacity 
planning in the VETSNET Integrated Master Schedule and ensure that 
resources are available for these activities; 

Concur - The Veterans Benefits Administration (VBA) has already 
implemented capacity planning activities in conjunction with the 
Integrated Master Schedule (IMS). As noted during the review, capacity 
planning was difficult in advance of the IMS development, because VBA 
had not identified when specific functionality would be delivered and 
available for claims processing. The IMS identifies when specific types 
of functionality will be delivered into the production environment and 
when records will be converted from the Benefits Delivery Network 
(BDN). Increased levels of claims processing are tied to each of these 
releases and conversions, which in turn, allows for more effective 
capacity planning. As reported, VBA has a staff dedicated to 
performance monitoring and capacity planning (note, this organization 
now falls under the Department's Office of Information and Technology). 
Performance testing, which is a component of capacity planning, is a 
defined element of the overall VETSNET development and testing 
timeline. Results of performance testing for each release are factored 
in and used to validate capacity assumptions. To ensure functionality 
already delivered to production continues to meet performance 
expectations, VETSNET leadership attends daily conference calls 
regarding corporate database performance. This ensures any issues that 
may unexpectedly arise are rapidly addressed. In addition, VETSNET 
leadership has fully briefed its overall schedule and claims processing 
estimates to the capacity planning staff to assist that staff in better 
understanding VETSNET's plans and needs. Finally, to ensure sufficient 
attention is given this vital element of the project, VBA has provided 
for assistance in capacity planning in MITRE Corporation's 2007 
contract for VETSNET support. 

* direct VBA to (1) develop a strategy to address the risk that large 
numbers of records may need to be returned to the BDN; (2) determine 
whether a greater confidence level for accuracy should be required in 
the conversion process; and (3) develop a detailed validation plan that 
includes the scheduled conversions for each regional office and the 
validation team members needed for that specific conversion; 

Concur-(1) As part of addressing the Software Engineering Institute's 
(SEI) recommendations, and in developing the Integrated Master 
Schedule, VBA has taken significant steps in addressing the conversion 
of existing records from the BDN to the corporate database. As 
reported, VBA stopped converting records from the BDN in order to 
develop a more coordinated approach. Conversions are now planned for 
execution in accordance with delivery of application functionality into 
production, followed by a period of stabilization. Since the initial 
conversions of VA Regional Office (VARO) Lincoln, Nebraska, records, 
data show that the only reason records required return to BDN was 
because application functionality no longer supported the record. For 
example, the record of a veteran in receipt of disability compensation 
was successfully converted from the BDN. The veteran then elected to 
receive non-service connected pension as the greater monetary benefit. 
Since pension is not fully supported in VETSNET, the veteran's record 
required return to the BDN to continue payment. At no time has a record 
required return to the BDN due to the conversion being "wrong" or 
paying the veteran incorrectly. 

VBA's current strategy, as noted in the report, is to convert a large 
volume of terminated records, followed by conversions of records at 
Lincoln and then VARO Nashville, Tennessee. These conversions were 
designed to test both conversion processes and validation methods to 
ensure the correct conversion of the selected records. A "correct" 
conversion is defined as uninterrupted payment at the correct rate. In 
the event modifications to our conversion methods are required, the 
schedule allows sufficient time to address those changes. 

In addition, a core conversion team has been assembled and is testing 
and validating. This team includes technical resources, individuals 
from headquarters staff of the Compensation and Pension Service (C&P), 
Office of Resource Management, and field resources for both the claims 
processing and finance activities. These individuals will serve as the 
core team throughout the entirety of our conversion process. 

(2) During these initial test conversions, physical record validation 
is performed at the 95 percent confidence level. In addition, a number 
of automated reports have been and are being developed, to provide 
additional oversight for the process. These reports will compare the 
state of records pre-and post-conversion, for example, the number of 
records submitted for conversion and the number converted; the amount 
of payment associated with the records pre-and post-conversion, etc. 
Similarly, an automated testing process was developed and is in use to 
validate payment processing for application development purposes. This 
automated process is also being used to ensure the converted records 
conform to the required format for Treasury processing. 

(3) As noted previously, a core conversion team has been assembled and 
is actively engaged in testing and validation efforts. These 
individuals will remain on the core team throughout the conversion 
process, continuing to test and validate the conversion logic and 
outcome for each scheduled conversion. As incorporated in the IMS, 
conversions will follow the release and stabilization of application 
functionality in production. Live compensation records are planned for 
three or four conversions, depending upon the number of records we 
decide to convert in a given "lot". The number of records will be 
determined once we complete the current testing phase underway. The 
order of regional offices will also be determined at that time, based 
on a variety of factors, to include business process changes currently 
underway in VBA, as well as workload considerations. Service-connected 
death records (Dependency and Indemnity Compensation) will be converted 
in one lot, following release of that functionality. Similarly, 
existing income-based pension records will be converted in two lots, 
once functionality is in place. The decision to convert in two lots is 
based on workload considerations for the Pension Maintenance Centers. 

VBA and MITRE Corporation believe these efforts will significantly 
reduce the risks associated with conversions. During its weekly 
meetings, the VETSNET Executive Team formally tracks these risks. 

* direct the CIO to document and incorporate the improved processes for 
managing risks, requirements, and defects into specific policy and 
guidance for the replacement initiative and for future use throughout 
VBA; and: 

Concur - As part of the centralization of information technology (IT) 
within VA, the CIO has undertaken the effort of implementing best 
business practices to assist VA in its consolidation. The goal is to 
establish application development processes that employ industry best 
practices and that have the potential to accelerate the successful 
completion of existing projects and ensure continued success in future 
projects. 

As part of this effort, VETSNET's project management processes, 
including risk management, requirements management, and testing 
methodology to include the management of defects are under review. 
VETSNET project management processes will be incorporated into a set of 
standard project management policies, processes and procedures for all 
IT projects in VA. 

The VETSNET governance model includes the VETSNET Executive Team 
consisting of representatives from the technical, project management, 
and business lines. The CIO has identified this governance model as the 
model for all VA enterprise-wide IT projects. The governance model is 
being implemented in other VA priority IT development programs, such as 
Financial and Logistics Integrated Technology Enterprise (FLITE). 

* direct the replacement project to develop effective results-oriented 
performance measures that show changes in efficiency, economy, or 
improvements in mission performance, as well as measures of user 
satisfaction, and to monitor and report on the progress of the 
initiative according to these measures. 

Concur - As noted in the report, VBA has implemented a number of 
performance metrics to track progress toward transition of all C&P 
claims processing and payments from BDN to VETSNET. Some examples were 
provided in the body of the report. In addition, VBA recently completed 
its periodic Work Measurement Study, which assesses the level of effort 
required for various C&P work tasks. One element of this year's study 
will assess task time for work processed using VETSNET Award, as well 
as through BDN. The other components of the VETSNET suite (i.e., MAP-D 
and RBA2000) have already been in full deployment for a number of 
years, and exceed functionality currently in BDN. In that regard, 
comparisons to BDN processing are not applicable. The efficiencies 
gained by these applications are directly related to the reusability of 
data and the ability to determine the status of a veteran's claim 
without the need to review the physical claims file. This reduces staff 
time required to physically locate and review claims files, provides 
enhanced customer service, and facilitates the sharing of work across 
regional offices to balance workload and provide more timely service to 
veterans. 

With respect to user satisfaction, while VBA does not formally survey 
users as to their level of satisfaction, we have a variety of forums in 
place to solicit and receive feedback from the user community. This 
includes, for example, an electronic mailbox to receive questions or 
comments, and regular "super user" conference calls to discuss issues 
and suggestions. In addition, the Special Assistant to the Under 
Secretary regularly addresses regional office managers to ensure 
information reaches the executive level, and to address any concerns or 
comments they may have. As SEI noted, there is a strong "pull" from the 
user community for expanded VETSNET functionality. That has been borne 
out by the increase in claims processing volume once field stations 
were authorized to increase the scope of claims available for 
processing. A more formal measure of user satisfaction can be 
implemented, but we believe our core measure of project success is 
reflected by the metrics currently in place. These metrics track the 
current volume of claims processing and beneficiaries being paid 
through VETSNET, as well as tracking the activities necessary to 
transition all C&P processing off of the BDN. 

[End of section] 

Appendix III: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Valerie Melvin, (202) 512-6304 or melvinv@gao.gov: 

Staff Acknowledgments: 

In addition to the individual named above, key contributions were made 
to this report by Barbara Oliver, Assistant Director; Nabajyoti 
Barkakati; Barbara Collier; Neil Doherty; Matt Grote; Robert Williams; 
and Charles Youman. 

FOOTNOTES 

[1] It also refers to the initiative as the compensation and pension or 
C&P replacement system. 

[2] GAO, Software Capability Evaluation: VA's Software Development 
Process Is Immature, GAO/AIMD-96-90 (Washington, D.C.: June 19, 1996); 
Veterans Benefits Modernization: VBA Has Begun to Address Software 
Development Weaknesses but Work Remains, GAO/AIMD-97-154 (Washington, 
D.C.: Sept. 15, 1997); VA Information Technology: Progress Continues 
Although Vulnerabilities Remain, GAO/T-AIMD-00-321 (Washington, D.C.: 
Sept. 21, 2000); VA Information Technology: Important Initiatives 
Begun, Yet Serious Vulnerabilities Persist, GAO-01-550T (Washington, 
D.C.: Apr. 4, 2001); VA Information Technology: Management Making 
Important Progress in Addressing Key Challenges, GAO-02-1054T 
(Washington, D.C.: Sept. 26, 2002); and Information Technology: VA and 
DOD Face Challenges in Completing Key Efforts, GAO-06-905T (Washington, 
D.C.: June 22, 2006). 

[3] GAO-06-905T. 

[4] Kathryn Ambrose, William Novak, Steve Palmquist, Ray Williams, and 
Carol Woody, Report of the Independent Technical Assessment on the 
Department of Veterans Affairs VETSNET Program (Carnegie Mellon 
Software Engineering Institute, September 2005). 

[5] The BDN currently runs on aging software: COBOL programs and a 
nonrelational database. Analysts have indicated that moving from a 
nonrelational database of the BDN type to a more modern relational 
database is a challenging task. 

[6] This program provides veterans, service members, reservists, and 
certain veterans' dependents with educational resources. 

[7] To help veterans with service-connected disabilities become 
employable and obtain and maintain suitable employment, the program 
provides a range of direct and supportive services. These services 
include comprehensive evaluation of rehabilitation needs (vocational or 
independent living); training and employment services to obtain or 
maintain suitable employment; and independent living services, such as 
training and specialized equipment to enable independence in the 
activities of daily living. 

[8] Not all regional offices process all the different types of 
benefits. For example, adjustments to pension claims are processed at 
three pension maintenance centers in St. Paul, Milwaukee, and 
Philadelphia, and education claims are processed only at the regional 
offices in Atlanta, Buffalo, Muskogee, and St. Louis. The Philadelphia 
Regional Office and Insurance Center has sole responsibility for 
insurance benefit processing, and nine Regional Loan Centers administer 
the loan guaranty program. 

[9] Claims may also be reopened if a veteran provides additional 
information on a claim that was denied. 

[10] GAO/AIMD-96-90. 

[11] Specifically, at the repeatable level of process maturity, basic 
project management processes are established to track cost, schedule, 
and functionality, and the necessary process discipline is in place to 
repeat earlier successes on projects with similar applications. 

[12] Kathryn Ambrose, William Novak, Steve Palmquist, Ray Williams, and 
Carol Woody, Report of the Independent Technical Assessment on the 
Department of Veterans Affairs VETSNET Program (Carnegie Mellon 
Software Engineering Institute, September 2005). 

[13] CMMI Product Team, CMMI for Development, Version 1.2 (Carnegie 
Mellon Software Engineering Institute, August 2006). 

[14] As previously noted, there are variations in types of compensation 
and pension benefits, including death pensions for survivors of 
deceased wartime veterans, burial benefits, Dependency and Indemnity 
Compensation to some surviving parents, and benefits paid to third 
parties, such as people to whom a veteran has given power of attorney, 
medical service providers, and so on. In addition, different forms of 
payment must be accommodated (check or electronic funds transfer). 

[15] SEI quoted program staff as calling the process "extraordinarily 
time consuming," saying that "It takes 5 minutes to get a veteran into 
VETSNET and then 5 days to get him back out." 

[16] This realignment was approved on February 27, 2007. 

[17] The functionalities for the Search and Participant Profile, 
formerly a separate application, are now in Share. 

[18] Survivor benefits may be paid to survivors of veterans who were 
eligible for either compensation or pension benefits, depending on the 
circumstances. 

[19] Examples of third-party/nonveteran payees include people to whom a 
veteran has given power of attorney, medical service providers 
designated to receive payments, and so on. 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site (www.gao.gov). Each weekday, GAO posts 
newly released reports, testimony, and correspondence on its Web site. 
To have GAO e-mail you a list of newly posted products every afternoon, 
go to www.gao.gov and select "Subscribe to Updates." 

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office 441 G Street NW, Room LM 
Washington, D.C. 20548: 

To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202) 
512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400 U.S. 
Government Accountability Office, 441 G Street NW, Room 7125 
Washington, D.C. 20548: 

Public Affairs: 

Paul Anderson, Managing Director, AndersonP1@gao.gov (202) 512-4800 
U.S. Government Accountability Office, 441 G Street NW, Room 7149 
Washington, D.C. 20548: