This is the accessible text file for GAO report number GAO-07-614 entitled 'Veterans Benefits Administration: Progress Made in Long-Term Effort to Replace Benefits Payment System, but Challenges Persist' which was released on April 27, 2007. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Report to the Chairman, Committee on Veterans' Affairs, House of Representatives: United States Government Accountability Office: GAO: April 2007: Veterans Benefits Administration: Progress Made in Long-Term Effort to Replace Benefits Payment System, but Challenges Persist: GAO-07-614: GAO Highlights: Highlights of GAO-07-614, a report to the Chairman, Committee on Veterans’ Affairs, House of Representatives Why GAO Did This Study: Since 1996, the Veterans Benefit Administration (VBA) has been working on an initiative to replace its aging system for paying compensation and pension benefits. In 2005, concerned about the slow pace of development, VBA contracted with the Software Engineering Institute (SEI) for an independent evaluation of the project, known as the Veterans Service Network (VETSNET). SEI advised VBA to continue working on the project at a reduced pace while addressing management and organization weaknesses that it determined had hampered the project’s progress. GAO was requested to determine to what extent the VETSNET project has followed the course of action recommended by SEI and describe the project’s current status. To perform its review, GAO analyzed project documentation, conducted site visits, and interviewed key program officials. What GAO Found: VBA is generally following the course of action recommended by SEI by continuing to work on the replacement initiative at a reduced pace, while taking action to address identified weaknesses in overall management and software development processes. For example, VBA established a new governance structure for the initiative that included senior management and involved all stakeholders, and it incorporated all critical areas of system development in an integrated master schedule. However, not all of SEI’s management concerns have been addressed. For example, SEI advised VBA to ensure that stakeholders take ownership responsibility for the project, including the total system and process operating costs; however, although VBA is tracking costs incurred by contractors, it is not yet tracking and reporting in- house costs incurred by the project. Further, although the project has improved its management processes, such as establishing a process to manage and stabilize system requirements, it has not yet developed processes for capacity planning and management. This will be important for ensuring that further VETSNET development does not lead to delays and slowdowns in processing of benefits. In addition, although the project has established certain performance measures, it has not yet established results-oriented measures for productivity and user satisfaction, both of which will be important for measuring progress. Finally, the process improvements that VBA has incorporated in the replacement initiative remain in draft and have not been established through documented policies and procedures. If VBA does not institutionalize these improvements, it increases the risk that they may not be maintained through the life of the project or be available for application to other development initiatives. After more than 10 years of effort, including the recent management, organizational, and process improvements, VBA has developed critical functionalities needed to process and pay certain original compensation claims using the replacement system, but it remains far from completing the project. According to VBA officials, all five of the major software applications that make up the new system are now being used to establish and process new compensation claims for veterans. In total, the replacement system is currently providing monthly compensation payments to almost 50,000 veterans (out of about 3 million veterans who receive such payments); the system was used to process about 83 percent of all new compensation claims completed in March 2007. Nonetheless, the system requires further development before it can be used to process claims for the full range of compensation and pension benefits available to veterans and their dependents. What GAO Recommends: To sustain the improved management and software development processes currently being used by VETSNET project management, GAO is making recommendations to the Secretary of Veterans Affairs in several areas, including cost tracking, capacity planning and management, and performance measures. The Secretary agreed with GAO’s recommendations and described actions planned in response. [Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-614]. To view the full product, including the scope and methodology, click on the link above. For more information, contact Valerie Melvin at (202) 512-6304 or melvinv@gao.gov. [End of section] Contents: Letter: Results in Brief: Background: VBA Is Following Course Recommended by SEI, but Weaknesses Remain to Be Addressed: VETSNET Is Currently Processing a Portion of Compensation Claims, but Much Work Remains: Conclusions: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendix I: Objectives, Scope, and Methodology: Appendix II: Comments from the Department of Veterans Affairs: Appendix III: GAO Contact and Staff Acknowledgments: Tables: Table 1: Roles and Responsibilities of VETSNET Organizational Elements: Table 2: VETSNET Component Applications and Status of Development: Table 3: Software Releases Planned to Complete the VETSNET System: Table 4: Conversion Phases: Figure: Figure 1: VETSNET System Life Cycle and Review Concept: Abbreviations: BDN: Benefits Delivery Network: CIO: Chief Information Officer: FAS: Finance and Accounting System: IT: Information Technology: MAP-D: Modern Award Processing-Development: RBA 2000: Rating Board Automation 2000: SEI: Carnegie Mellon Software Engineering Institute: VA: Department of Veterans Affairs: VBA: Veterans Benefits Administration: VETSNET: Veterans Service Network: VRE: Vocational Rehabilitation and Employment: United States Government Accountability Office: Washington, DC 20548: April 27, 2007: The Honorable Bob Filner: Chairman, Committee on Veterans' Affairs: House of Representatives: Dear Mr. Chairman: Since 1996, the Veterans Benefits Administration (VBA), a major component of the Department of Veterans Affairs (VA), has been undertaking a project to replace its compensation and pension benefits payment system. VBA's existing system, the Benefits Delivery Network (BDN), is essential to ensuring the accurate processing of compensation and pension benefits payments to over 3.5 million veterans and their dependents each month. However, this system, which has been in operation for more than 40 years, is based on antiquated software programs, which have become increasingly difficult and costly to maintain. VBA is in the process of replacing the aging BDN with a faster, more flexible, and higher capacity system. When it began this project in 1996, which it generally refers to as the Veterans Service Network (VETSNET),[Footnote 1] VBA had planned to complete the replacement system in May 1998 at an estimated cost of $8 million. Over the years, we have reported on numerous problems that VBA has encountered in completing the replacement system.[Footnote 2] Our prior work found, for example, that the project was begun before VBA had fully developed its business requirements, resulting in confusion over the requirements to be addressed; in addition, VBA's software development capability was too immature to ensure that the agency could reliably develop and maintain high-quality software on any major project within cost and schedule constraints. In 2002, we offered a number of recommendations to improve managerial and program weaknesses, including that VBA appoint a project manager; thoroughly analyze its initiative; and develop a number of plans, including a revised compensation and pension replacement strategy and an integrated project plan. VA concurred with our recommendations, and as we last reported in June 2006,[Footnote 3] it took several actions to address them. For example, it appointed a full-time project manager, and the project team reported that it had completed certifications of users' requirements for the system's applications. Nonetheless, these actions did not implement all our recommendations and were not sufficient to establish the program on a solid footing: certain basic requirements of sound project management, such as an integrated project plan for the replacement system, continued to be lacking. In 2005, the VA Chief Information Officer (CIO) and Under Secretary for Benefits became concerned about continuing problems with the replacement project and contracted for an independent assessment of the department's options for the project, including whether the project should be terminated. This assessment, conducted by the Carnegie Mellon Software Engineering Institute (SEI), concluded that the replacement project faced many risks arising from management and organizational issues, but no technical barriers that could not be overcome.[Footnote 4] According to SEI, a new system was still needed, and VBA would not be able to successfully deliver a full, workable solution unless it addressed its management and organizational weaknesses. SEI recommended that VBA continue to work on the project at a reduced pace, while taking an aggressive approach to addressing the identified weaknesses. Given the importance of ensuring the effective and efficient delivery of veterans' benefits, we were requested to review the department's continuing efforts to develop and implement the compensation and pension replacement system. Specifically, our objectives were to determine (1) to what extent VA has followed the course of action recommended by SEI and addressed the concerns that it raised and (2) the current status of the replacement project. In conducting this study, we visited VBA's Nashville Regional Office to observe processing of new benefits claims in the compensation and pension replacement system, and we visited the St. Petersburg Regional Office to observe processes and procedures used to test and validate key functionalities of the replacement system. We obtained and analyzed documents related to the replacement initiative and to SEI's review of the initiative. We supplemented our analysis with interviews of VBA and contractor personnel at headquarters and the two regional offices. Appendix I contains a more detailed discussion of our objectives, scope, and methodology. We conducted our study between April 2006 and April 2007 in accordance with generally accepted government auditing standards. Results in Brief: VBA is generally following the course of action recommended by SEI by continuing to work on the replacement initiative at a reduced pace, while taking action to address identified weaknesses in overall management and software development processes. For example, VBA established a new governance structure for the initiative that included senior management and involved all stakeholders, and it incorporated all critical areas of system development in an integrated master schedule. However, VBA did not address all of SEI's management concerns. Specifically, SEI advised VBA to ensure that stakeholders take ownership responsibility for the project, including the total system and process operating costs; however, although VBA is tracking costs incurred by contractors, it is not yet tracking and reporting in- house costs incurred by the project. Further, VBA has not yet developed sufficient plans for performing the substantial task of moving records from the BDN to the replacement system (approximately 3.5 million beneficiaries are currently being served by the older system). VBA did take steps to improve its software development processes, such as establishing risk and requirements management processes, but some have not been addressed. For example, still to be addressed are processes for capacity planning and management, which will be important for ensuring that further development does not lead to slowdowns in processing of benefits. Finally, VBA has not yet documented policies and procedures to institutionalize all the improvements that it has incorporated in the replacement initiative. According to the replacement project's management team, it made a conscious decision first to establish the governance and build the organization, among other things, and it is still prioritizing remaining tasks. However, if VBA does not institutionalize the improvements made, it increases the risk that these process improvements may not be maintained through the life of the project or be available for application to other development initiatives. After more than 10 years of effort, including the recent management, organizational, and process improvements, VBA has developed critical functionalities needed to process and pay certain original compensation claims using the replacement system, but it remains far from completing the project. According to VBA officials, all five of the major software applications that make up the new system are now being used in VA's regional offices to establish and process new compensation claims for veterans. In total, the replacement system is currently providing monthly compensation payments to almost 50,000 veterans (out of about 3 million veterans who receive such payments); the system was used to process about 83 percent of all new compensation claims completed in March 2007. Nonetheless, the system requires further development before it can be used to process claims for the full range of compensation and pension benefits available to veterans and their dependents. In addition, VBA still faces the substantial task of converting records for the approximately 3.5 million beneficiaries currently being served by the BDN to the replacement system. To sustain the improved management and software development processes currently being used by VETSNET project management, we are making recommendations to the Secretary of Veterans Affairs in several areas, including cost tracking, capacity planning and management, and performance measures. In providing written comments on a draft of this report, the Secretary of Veterans Affairs agreed with our conclusions and concurred with the report's recommendations. (The department's comments are reproduced in app. II.) The comments described actions planned that respond to our recommendations, as well as providing further information on relevant actions already taken. If the planned actions are properly implemented, they could help strengthen the department's management of the replacement system project and improve the chances that the system will be successfully completed. Background: VBA provides benefits for veterans and their families through five programs: (1) compensation and pension, (2) education, (3) vocational rehabilitation and employment (VRE) services, (4) loan guaranty, and (5) life insurance. It relies on the BDN to administer benefit programs for three of VBA's five programs: compensation and pension, education, and VRE services. Replacing the aging BDN has been a focus of systems development efforts at VBA since 1986.[Footnote 5] Originally, the administration planned to modernize the entire system, but after experiencing numerous false starts and spending approximately $300 million on the overall modernization of the BDN, VBA revised its strategy in 1996. It narrowed its focus to replacing only those functionalities that support the compensation and pension program, and began developing a replacement system, which it called VETSNET. As reported by the department in its fiscal year 2008 budget submission, the compensation and pension program is the largest of the three programs that the BDN supports: * The compensation and pension program paid about $35 billion in benefits in fiscal year 2006 to about 3.6 million veterans or veterans' family members. - Of this amount, compensation programs paid benefits of about $31 billion to about 3.1 million recipients. - Pension programs paid benefits of about $3.5 billion to about 535,000 recipients. * The education program paid about $2.8 billion to about 498,000 veterans or their dependents in fiscal year 2006.[Footnote 6] * The VRE services program paid about $574 million for VRE services in 2006 and provides rehabilitation services to approximately 65,700 disabled veteran participants per year.[Footnote 7] The Processes Supporting the Variety of Compensation and Pensions Benefits Are Complex: One of the challenges of developing the replacement system is that it must include processes to support the administration of a complex set of benefits. Different categories of veterans and their families are eligible for a number of different types of benefits and payments, some of which are based on financial need. Compensation programs, which are based on service-connected disability or death, provide direct payments to veterans and/or veterans' dependents and survivors. These programs are not based on income. Pension benefits programs, on the other hand, are income based; these are designed to provide income support to eligible veterans and their families who experience financial hardship. Eligible veterans are those who served in wartime and are permanently and totally disabled for reasons that are not service-connected (or who are age 65 or older). Veterans are also eligible for burial benefits. Survivor benefits may be paid to eligible survivors of veterans, depending on the circumstances. Some of these benefits are based on financial need, such as death pensions for some surviving spouses and children of deceased wartime veterans, and Dependency and Indemnity Compensation to some surviving parents. Finally, certain benefits may be paid to third parties, such as individuals to whom a veteran has given power of attorney or medical service providers designated to receive payments on the veteran's behalf. Generally, VBA administers benefit programs through 57 veterans benefits regional offices[Footnote 8] in a process that requires a number of steps, depending on the type of claim. When a veteran submits, for example, a compensation claim to any of the regional offices, a veterans service representative must obtain the relevant evidence to evaluate the claim (such as the veteran's military service records, medical examinations, and treatment records from VA medical facilities or private medical service providers). In the case of pension claims, income information would also be collected. Once all the necessary evidence has been compiled, a rating specialist evaluates the claim and determines whether the claimant is eligible for benefits. If the veteran is determined to be eligible for disability compensation, the Rating Veterans Service Representative assigns a percentage rating based on the veteran's degree of disability. This percentage is used in calculating the amount of payment. Benefits received by veterans are subject to change depending on changing circumstances. More than half of VBA's workload consists of dealing with such changes. If a veteran believes that a service- connected condition has worsened, for example, the veteran may ask for additional benefits by submitting another claim.[Footnote 9] The first claim submitted by a veteran is referred to as the original claim, and a subsequent change is referred to as a reopened claim. Our Prior Products Identified Weaknesses in the Development and Implementation of the Replacement System: Since its inception, VETSNET has been plagued by problems. Over the years, we have reported on the project, highlighting concerns about VBA's software development capabilities. In 1996, our assessment of the department's software development capability determined that it was immature.[Footnote 10] In our assessment, we specifically examined VETSNET and concluded that VBA could not reliably develop and maintain high-quality software on any major project within existing cost and schedule constraints. The department showed significant weaknesses in requirements management, software project planning, and software subcontract management, with no identifiable strengths. We also testified that VBA did not follow sound systems development practices on VETSNET, and we concluded that its modernization efforts had inherent risks. Between 1996 and 2002, we continued to identify the department's weak software development capability as a significant factor contributing to persistent problems in developing and implementing the system. We also reported that VBA continued to work on VETSNET without an integrated project plan. As a result, the development of the system continued to suffer from problems in several areas, including project management, requirements development, and testing. Over the years, we made several recommendations aimed at improving VA's software development capabilities. Among our recommendations was that the department take actions to achieve greater maturity in its software development processes[Footnote 11] and that it delay any major investment in software development (beyond that needed to sustain critical day-to-day operations) until it had done so. In addition, we made specific recommendations aimed at improving VETSNET development. For example, we recommended that VA appoint a project manager, thoroughly analyze its current initiative, and develop a number of plans, including a revised compensation and pension replacement strategy and an integrated project plan. VA concurred with our recommendations and took several actions to address them. For example, it appointed a full-time project manager and ensured that business needs were met by certification of user requirements for the system applications. The actions taken addressed some of our specific concerns; however, they were not sufficient to fully implement our recommendations or to establish the program on a sound footing. SEI Assessed System Replacement Initiative and Recommended a Changed Approach: As a result of continuing concerns about the replacement project, in 2005 VA's CIO and its Under Secretary for Benefits contracted for an independent assessment of the department's options for the initiative.[Footnote 12] The chosen contractor, SEI, is a federally funded research and development center operated by Carnegie Mellon University. Its mission is to advance software engineering and related disciplines to ensure the development and operation of systems with predictable and improved cost, schedule, and quality. SEI recommended that the department reduce the pace of development while at the same time taking an aggressive approach to dealing with management and organizational weaknesses hampering VBA's ability to complete the replacement system. According to SEI, these management and organizational concerns needed to be addressed before the replacement initiative or any similar project could deliver a full, workable solution. For example, the contractor stressed the importance of setting realistic deadlines and commented that there was no credible evidence that VETSNET would be complete by the target date, which at the time of the review had slipped to December 2006. According to the assessment, because this deadline was unrealistic, VBA needed to plan and budget for supporting the BDN so that its ability to pay veterans' benefits would not be disrupted. SEI also noted that different organizational components had independent schedules and priorities, which caused confusion and deprived the department of a program perspective. Further, the contractor concluded that VBA needed to give priority to establishing sound program management to ensure that the project could meet targeted dates. These and other observations were consistent with our long-standing concerns regarding fundamental deficiencies in VBA's management of the project. To help VBA implement the overall recommendation, the contractor's assessment included numerous discussions of activities needed to address these areas of concern, which can be generally categorized as falling into two major types: * Overall management concerns with regard to the initiative included: - governance structure, including assigning ownership for the project and its costs; - project planning, including the development schedule and capacity planning; and: - conversion of records currently on the BDN to the replacement system. * Software development process improvements were needed in the following areas: - risk management, - requirements management, - defect management, and: - program measures. VBA Is Following Course Recommended by SEI, but Weaknesses Remain to Be Addressed: As recommended by SEI, VBA is continuing to work on the replacement initiative at a reduced pace and taking action to address identified weaknesses in the project's overall management and software development processes. For example, VBA has established a new governance structure and has developed an integrated master schedule that provides additional time and includes the full range of project activities. However, additional effort is needed to complete a number of the corrective actions, such as improving project accountability through monitoring and reporting all project costs. Further, VBA has not yet institutionalized many of the improvements that it has undertaken for the initiative. In particular, process improvements remain in draft and have not been established through documented policies and procedures. According to the VETSNET management team, it gave priority to other activities, such as establishing appropriate governance and organizational structures, and it is still gathering information to assist in prioritizing the activities that remain. Nonetheless, if VBA does not institutionalize these improvements, it increases the risk that these process improvements may not be maintained through the life of the project or be available for application to other development initiatives. VBA Is Taking Action on Certain Overall Management Concerns: SEI concluded that VBA's management issues would need to be addressed as part of the implementation of its overall recommendation. SEI's overall management concerns focused on the project governance, project planning, and conversion of records currently on the BDN to the replacement system. Top Management Included in Revised Governance Structure: SEI guidance for software development stresses the need for organizational commitment and the involvement of senior management in overall project governance.[Footnote 13] In its assessment, SEI noted that because management of the VETSNET project had been assigned to VBA's information technology (IT) group, certain activities critical to the veterans' benefits program, but not traditionally managed by the IT group, had not been visible to the project's management. The contractor pointed out that the IT group, business lines, and regional offices needed to share ownership and management of the replacement project through an established governance process and that the project management office should include business representatives. According to SEI, the project needed to establish ownership responsibility, including addressing total system and process operating costs. In response to the assessment, VBA developed a new governance structure for the initiative, which the Under Secretary for Benefits approved in March 2006. In the new structure, the VETSNET Executive Board that had been in place was expanded and reorganized to serve as a focal point and major governance mechanism for the replacement initiative. A Special Assistant (reporting directly to the Under Secretary) was appointed to coordinate and oversee the initiative as the head of the VETSNET Executive Team, which was established to provide day-to-day operational control and oversight of the replacement initiative. Implementation Teams were also established to conduct the day-to-day activities associated with implementing the initiative. This governance structure established a process for IT, business lines, and regional offices to share ownership and management, as SEI advised. The roles and membership of each of the organizational elements in the new governance structure are described in table 1. Table 1: Roles and Responsibilities of VETSNET Organizational Elements: Organization: VETSNET Executive Board; Role: Provides executive direction to the VETSNET Executive Team; Membership: Deputy Under Secretary for Benefits; Special Assistant to the Under Secretary for Benefits; VETSNET Program Manager; VA Office of Information and Technology Principal Associate Deputy Assistant Secretary; Chief Financial Officer; Director, Compensation and Pension Service; Director, Office of Performance Analysis and Integrity; Associate Deputy Under Secretary for Policy and Programs; Associate Deputy Under Secretary for Field Operations; VBA Chief Information Officer. Organization: VETSNET Executive Team; Role: Responsible for the delivery of the project and acts as the review and approval authority for the Implementation Teams; Membership: VETSNET Program Executive (Special Assistant to Under Secretary); VETSNET Program Manager; Contract support; Business Architect; Technical Architect; MITRE support. Organization: VETSNET Implementation Teams; Role: Responsible for the day-to-day activities necessary to develop and implement the replacement system; Membership: Compensation and Pension; Office of Resource Management; Office of Performance Analysis and Integrity; Office of Field Operation; Office of Information Management, VA Central Office; St. Petersburg Systems Development Center; Hines Information Technology Center; Austin Automation Center. Source: GAO analysis of VBA information. [End of table] When the new governance structure was approved in March 2006, the Under Secretary ensured that those involved in the project gave it high priority, directing certain key personnel (such as members of the executive and implementation teams) to make the initiative their primary responsibility, and other personnel (technical staff that provide support to other systems) with collateral (non-VETSNET) duties to make the project their first priority. He also placed limitations on the transfer of personnel away from the project, recognizing the importance of staff continuity in successfully completing the initiative. Staff members assigned project responsibilities could be reassigned (i.e., given nonpromotion, lateral reassignments) only with approval from the Under Secretary or his deputy. By implementing the new governance and organizational structure and ensuring that the project has priority, VBA partially responded to SEI's concerns in this area; however, VBA has not yet taken action with regard to ownership responsibility for total system and process operating costs, as SEI advised. According to administration officials, the replacement initiative is an in-house, contractor-assisted development effort, in which three different contractors provide support for program management, system development, and testing and validation of requirements. VA reported VETSNET system costs to the Congress totaling about $89 million for fiscal years 1996 through 2006, with additional estimated costs for completion of the initiative in 2009 of about $62.4 million. However, according to project management officials, these costs do not include expenditures for in-house development work. This in-house work involves many VA personnel, as well as travel to various locations for testing and other project related activities. Thus, considerable costs other than contract cost have been incurred, which have not been tracked and reported as costs for the replacement initiative. Without comprehensive tracking and reporting of costs incurred by the replacement project, the ability of VBA and the Congress to effectively monitor progress could be impaired. VBA Took Action to Address Project Planning and Management: A second major area of overall management concern was project planning. In particular, the lack of an integrated master schedule for the VETSNET project was a major concern articulated by SEI, as well as in our prior work. An integrated project plan and schedule should incorporate all the critical areas of system development and be used as a means of determining what needs to be done and when, as well as measuring progress. Such an integrated schedule should consider all dependencies and include subtasks so that deadlines are realistic, and it should incorporate review activities to allow oversight and approval by high-level managers. Among other things, the program plan should also include capacity requirements for resources and technical facilities to support development, testing, user validation, and production. SEI was specifically concerned that releases with overlapping functionality were being developed at the same time, with insufficient time to document or test requirements; this approach constrained resources and added complexity because of the need to integrate completed applications and newly developed functionality. In addition, SEI observed that the VETSNET program suffered from lack of sufficient test facilities because it did not have enough information to plan for adequate capacity. In response to these project planning concerns, VETSNET management, with contractor support, developed an integrated master schedule to guide development and implementation of the remaining functionalities for the replacement system. The VETSNET Integrated Master Schedule, finalized in September 2006, includes an end-to-end plan and a master schedule. According to VBA, the end-to-end plan documents the end state of the project from a business perspective, which had not previously been done. The master schedule identifies the necessary activities to manage and control the replacement project through completion. The schedule also describes a new software release process that provides more time to work on requirements definition and testing, and allows for more cross-organizational communications to lessen the possibility of not meeting requirements. In addition, the new release process includes a series of management reviews to help control the software development process and ensure that top management has continuous visibility of project related activities. These reviews occur at major steps in the system development life cycle (as described in fig. 1: initiation, preliminary design, and so on). Such reviews are intended to ensure that the VETSNET Executive Team and the VETSNET Executive Board agree and accept that the major tasks of each step have been properly performed. Figure 1: VETSNET System Life Cycle and Review Concept: [See PDF for image] Source: VBA. [End of figure] Nonetheless, while the Integrated Master Schedule is an important accomplishment, it may not ensure that the project sufficiently addresses capacity planning, one of SEI's areas of concern. According to its assessment, capacity requirements for the fully functional production system were unclear. Capacity planning is important because program progress depends on the availability of necessary system capacity to perform development and testing; adjustments to such capacity take time and must be planned. If systems do not have adequate capacity to accommodate workload, interruptions or slowdowns could occur. According to SEI, capacity adjustments cannot be made instantly, and program progress will suffer without sufficient attention to resource requirements. However, the VETSNET Integrated Master Schedule does not identify activities or resources devoted to capacity planning. According to officials, the capacity of the corporate environment (that is, corporate information systems, applications, and networks) is being monitored by operational teams with responsibility for maintaining this environment. According to project officials, VETSNET representatives participate in daily conference calls in which the performance of corporate applications is discussed, and changes in application performance are reported to the VETSNET developers for investigation and corrective action. Project officials reported that when a performance degradation occurred in some transactions during performance testing, it was determined that additional computing capacity was needed and would be acquired. One reason why the occurrence of degradation had not been anticipated by the VETSNET project was that capacity planning had not taken place. Unless it ensures that capacity planning and activities are included in the Integrated Master Schedule, the replacement project may face other unanticipated degradations that it must react to after the fact, thus jeopardizing the project's cost, schedule, and performance. Conversion Efforts Were Suspended as Advised, but Issues Remain: In its assessment, SEI questioned VBA's approach to developing functionality while concurrently converting records from the BDN to the replacement system. It noted that VBA had chosen to complete software development according to location rather than according to the type of functionality. Specifically, in 2004, VBA began an effort to remove all claims activity (both new and existing claims) at one regional office (Lincoln, Nebraska) from the BDN to the replacement system, developing the software as necessary to accommodate processing the types of claims encountered at that site. The intention was to address each regional office in turn until all sites were converted. According to SEI, this approach had resulted in the development being stalled by obstacles arising from the variety of existing claims.[Footnote 14] The contractor advised VBA to focus first on developing functionality to process original claims and discontinue efforts to convert existing claims until all the necessary functionality had been developed, and the replacement system's ability to handle new cases of any complexity had been proven by actual experience. In accordance with this advice, VBA stopped converting existing records from the BDN and changed its focus to developing the necessary functionality to process all new compensation claims. According to the integrated master schedule, conversion activities are now timed to follow the release of the needed functionality. That is, according to the schedule, VBA plans to begin converting each type of record from the BDN only after the necessary functionality for the replacement system has been developed and deployed to process that type of record. In addition, the project is mitigating risk by resuming conversions beginning with a test phase. Its strategy is first to convert records for terminated claims--claims that are no longer being paid. Conversion of the terminated records will be followed by additional conversions of records for claims receiving payment at Lincoln and Nashville (these two sites are being used to test system functionality during development). The VETSNET leadership will consider testing complete with the successful conversion at these two sites. However, SEI raised three additional issues with regard to the conversion of records that VBA has not fully addressed: * First, SEI expressed concerns that conversion failures could lead to substantial numbers of records being returned to the BDN. Because of differences in the database technologies used for the old system and the replacement system, certain types of errors in BDN records cause conversion to fail (according to SEI, approximately 15 percent of all these records are estimated to have such errors). If records fail to convert correctly, they may need to be returned to the BDN so that benefits can continue to be paid. However, this process is not simple and may involve manually reentering the records.[Footnote 15] * Second, SEI observed that VBA was also depending on manual processes for determining that records were converted successfully, including the use of statistically random samples, and that it was aiming to ensure correctness to a confidence level of 95 percent. However, in the absence of a straightforward method for automatically returning records to the BDN, SEI considered the 5 percent risk of error unacceptable for conversions of large numbers of records. * Finally, SEI observed that the lack of automated methods and the complexity of the processes meant that conversions required careful planning and assurance that adequate staff would be available to validate records when the conversions took place. However, the VETSNET leadership has not developed any strategy to address the possibility that a large number of cases might need to be returned to the BDN during the testing phase. For example, it has not included this possibility as a risk in its risk management plan. The absence of a strategy to address this possibility could lead to delays in program execution. Further, VBA has not yet decided whether a possible 5 percent error rate is acceptable or developed a plan for addressing the resulting erroneous records. If VBA does not address these issues in its planning, it increases the risk that veterans may not receive accurate or timely payments. Finally, the VETSNET leadership has not yet developed detailed plans that include the scheduled conversions for each regional office and identified staff to perform the necessary validation. Having such plans would reduce the risk that the conversion process could be delayed or fail. VBA Is Improving Software Development Processes, but Improvements Are Not Yet Institutionalized: In addition to actions addressing the overall management concerns identified by SEI, VBA has steps action to improve its software development processes in risk management, requirements management, defect/change management, and performance measures. SEI described weaknesses in all of these areas. The steps taken have generally been effective in addressing the identified weaknesses, but VBA has not yet institutionalized many of these improvements. According to the VETSNET management team, it made a conscious decision first to establish the governance, build the organization, implement processes to gain control, and gather additional information about the project to assist in prioritizing the remaining activities. The team also stated that some of the processes are no longer VBA's responsibility but are now that of the newly realigned Office of Information and Technology.[Footnote 16] Nonetheless, if VA does not develop and establish documented policies and procedures to institutionalize these improvements, they may not be maintained through the life of the project or available to be applied to other development initiatives. Risk Management Plan Has Been Revised: Risk management is a process for identifying and assessing risks, their impact and status, the probability of their occurrence, and mitigation strategies. Effective risk management includes the development of a risk management plan and tracking and reporting progress against the plan. According to SEI, to the extent that risk management existed at all in the replacement program, it was conducted on a pro forma basis without real effect on program decisions. SEI said that risks and risk mitigation activities needed to be incorporated into all aspects of program planning, budgeting, scheduling, execution, and review. In response to these concerns, VBA has instituted risk management activities that, if properly implemented, should mitigate the risks associated with the project. Specifically, the VETSNET team, with contractor support, developed a risk management plan that was adopted in January 2007. The plan includes procedures for identifying, validating, analyzing, assessing, developing mitigation strategies for, controlling and tracking, reporting, and closing risks. It also establishes criteria for assessing the severity of the risks and their impact. The VETSNET leadership also developed a Risk Registry database, and its contractor reviewed and prioritized the open risks. Each open risk was evaluated, and a proposed disposition of the risk was submitted to VETSNET management. Of the 39 open risks, all but 3 had been addressed as of January 2007. The development documentation for each planned software release also includes sections on risk. In accordance with these plans, the VETSNET leadership is currently capturing potential risks and tracking action items and issues. At weekly status meetings, VETSNET leadership reviews Risk Registry reports of open risks. According to the contractor, the reports identify each risk and provide information on its age, ownership, and severity. However, these risk management activities have not yet been institutionalized through the definition and establishment of associated policies and procedures. If it does not institutionalize these improvements, VBA increases the possibility that the VETSNET project's improvements in risk management may not be maintained through the life of the project. Requirements Management Has Improved: Requirements management is a process for establishing and maintaining a common understanding between the business owners and the developers of the requirements to be addressed, as well as verifying that the system meets the agreed requirements. SEI's report commented that the VETSNET project requirements were not stable, and that the business owners (including subject-matter experts) and developers were separated by many organizational layers, resulting in confusion and delays in development of the system. SEI suggested that VA restructure project activities to focus on defining an effective requirements process. According to SEI, the project needed to ensure that subject-matter experts were included in developing requirements and that evaluation criteria were established for prioritizing requests for changes to requirements. Finally, business owners should confirm that the system is meeting organizational needs. VBA has instituted requirements management activities that, if properly implemented, should help avoid the instability and other requirements problems identified by SEI. Specifically, VBA took steps to establish a requirements management process and to stabilize the requirements. For example, the development release process in the Integrated Master Schedule includes a phase for requirements identification. In addition, the project has established and begun applying evaluation criteria to prioritize change requests for its development releases. Further, until all claims are completely migrated from the BDN to the replacement system, in July 2006, the Under Secretary directed that any additional requirements would have to have his approval. Responding to SEI's advice regarding the involvement of subject-matter experts and business owners, VBA designed the new release process to directly involve subject-matter experts in requirements workshops. Further, the business teams participate in user-acceptance testing. However, these requirements management activities have not yet been institutionalized through the definition and establishment of policies and procedures. Until they are established, VBA runs the risk that the improved processes will not be maintained through the life of the VETSNET project or used in other software development projects. Management Attention Is Being Focused on Major Software Defects: SEI raised numerous concerns regarding the defect process for the replacement system. These concerns for defect management included (1) identification of defects, (2) determination of cause, and (3) disposition of defects--either by correction or workaround. According to SEI guidance, defect management prevents known defects from hampering the progress of the program. The management process should include clearly identifying and tracking defects, analyzing defects to establish their cause, tracking their disposition, clearly identifying the rationale for not addressing any defects (as well as proposing workarounds), and making information on defects and their resolution broadly available. SEI's report stated that VBA needed to distinguish defects from changes to requirements and develop a process for defect management. To respond to these concerns and focus program management attention on major defects, the VETSNET Executive Team, with contractor support, conducted an audit of existing defects and revised the defect management process. The audit of the defect database determined that the VETSNET database used to capture software defects also included change requests; as a result, work required to address processes that did not work properly was not distinguished from requests for added or changed functionality, which would require review and approval before being addressed. To address this issue, the team separated defects from change requests, and a new severity rating scale was developed. All open defects were recategorized to ensure the major defects would receive appropriate program management attention. Also, all defect categorizations must meet the approval of the VETSNET Business Architect and are scheduled for action as dictated by the severity level. Although these steps address many of SEI's concerns regarding VBA's defect management process, more remains to be done before the process is institutionalized. The Program Management Office has reported that actions to revise the defect management process are complete, but the process description is still in draft, and policies and procedures have not been fully established. Without institutionalized policies and procedures for the defect management process, it may not be maintained consistently through the life of the project. The Replacement Initiative Is Tracking Certain Performance Measures: According to SEI, performance measures are the only effective mechanism that can provide credible evidence of a program's progress. The chosen measures must link directly to the expected accomplishments and goals of the system, and they must be applied across all activities of the program. In its report, SEI stated that although VBA was reporting certain types of performance measures, it was not relating these to progress in system development. For example, VBA reported the total number of veterans paid, but did not provide estimates of how many additional veterans would be paid when the system incorporated specific functionalities that were under development. SEI suggested several measures that would provide more evidence of progress, such as increases in the percentage of original claims being paid by the replacement system, as well as user satisfaction and productivity gains resulting from use of the replacement system applications at regional offices. In response to these concerns, the replacement project has begun tracking a number of the measures suggested by SEI, including: * increases in the percentage of original claims being paid by the replacement system, * increases in the percentage of veterans' service representatives using the new system, * decreases in the percentage of original claims being entered in the BDN rather than the replacement system. Although these measures provide indications of VA's progress, other measures that could demonstrate the effectiveness of the replacement system have not been developed. For example, VBA has not developed results-oriented measures to capture user satisfaction or productivity gains from the system. Without measuring user satisfaction, VBA has reduced assurance that the replacement system will be accepted by the users. In addition, measures of productivity would provide VBA with another indication of progress toward meeting business needs. VETSNET Is Currently Processing a Portion of Compensation Claims, but Much Work Remains: After more than 10 years of effort, including the recent management, organizational, and process improvements, VBA has achieved critical functionalities needed to process and pay certain original compensation claims using the replacement system, but it remains far from completing the project. For example, the replacement system is currently being used to process a portion of the original claims that veterans file for compensation. Nonetheless, the system requires further development before it can be used to process claims for the full range of compensation and pension benefits available to veterans and their dependents. In addition, VBA still faces the substantial task of moving approximately 3.5 million beneficiaries who are currently being served by the BDN to the replacement system. As designed, VETSNET consists of five major system applications that are used in processing benefits: * Share--used to establish claims;[Footnote 17] it records and updates basic information about veterans and dependents both in the BDN and the replacement system. * Modern Award Processing-Development (MAP-D)--used to manage the claims development process, including the collection of data to support the claims and the tracking of claims. * Rating Board Automation 2000 (RBA 2000)--provides laws and regulations pertaining to disabilities, which are used by rating specialists in evaluating and rating disability claims. * Award Processing (Awards)--used to prepare and calculate the benefit award based on the rating specialist's determination of the claimant's percentage of disability. It is also used to authorize the claim for payment. * Finance and Accounting System (FAS)--used to develop the actual payment record. FAS generates various accounting reports and supports generation and audit of benefit payments. According to VBA officials, all five of the software applications that make up the new system are now being used in VA's 57 regional offices to establish and process new compensation claims for veterans. As of March 2007, VBA leadership reported that the replacement system was providing monthly compensation payments to almost 50,000 veterans (out of about 3 million veterans who receive such payments). In addition, the replacement system has been processing a steadily increasing percentage of all new compensation claims completed: this measure was 47 percent in January 2007, increasing to 60 percent in February and 83 percent in March. Nonetheless, considerable work must be accomplished before VBA will be able to rely on the replacement system to make payments to all compensation and pension beneficiaries. Specifically, while all five software applications can now be used to process original compensation claims for veterans, two of the applications--Awards and FAS--require further development before the system will be able to process claims for the full range of benefits available to veterans and their dependents. Table 2 shows the status of development of all five applications. Table 2: VETSNET Component Applications and Status of Development: Application: Share; Development status: Completed; Deployment status: Fully deployed at all regional offices. Application: MAP-D; Development status: Completed; Deployment status: Fully deployed at all regional offices. Application: RBA 2000; Development status: Completed; Deployment status: Fully deployed at all regional offices. Application: Awards; Development status: Functionality to process third-party/nonveteran payee claims is in application testing; Requirements for processing survivor benefits are being developed; Functionality to process pensions is not yet under development; Deployment status: In partial use in all regional offices. Application: FAS; Development status: Functionality to process third- party/nonveteran payee claims is in application testing; Functionality to generate management reports is partially in requirements development and partially in application testing; Requirements for processing survivor benefits are being developed; Functionality to process pensions is not yet under development; Deployment status: In partial use in all regional offices. Source: GAO analysis of VBA data. [End of table] According to VBA officials, Awards and FAS do not yet have the capability to process original claims for payment to recipients other than veterans: that is, the applications do not have the functionality to process claims for survivor benefits[Footnote 18] and third-party/ nonveteran payee claims.[Footnote 19] In addition, further development of these applications is needed to process pension benefits for qualified veterans and their survivors. Until enhancements are made to Awards and FAS, these claims must continue to be processed and paid through the BDN. Also, according to VBA, FAS does not yet have the capability to generate all the necessary accounting reports that support the development of benefits payments to claimants. As described earlier, VBA now has an Integrated Master Schedule that incorporates the activities that VBA needs to manage in order to complete the replacement project. According to the schedule, the remaining capabilities necessary to process compensation and pension claims are to be developed and deployed in three software releases, as shown in table 3. Table 3: Software Releases Planned to Complete the VETSNET System: Release number and title: 1. Complete Compensation; Functionality to be attained: Process all compensation claims and add significant FAS functionality (generate management reports); Estimated completion date: August 2007. Release number and title: 2. Nonincome-Based Survivor Benefits; Functionality to be attained: Process burial, accrued, and survivor claims; Estimated completion date: February 2008. Release number and title: 3. Income-Based Pension; Functionality to be attained: Process nonservice-connected pension (veteran and survivor) and parents' Dependent and Indemnity Compensation Benefit; Estimated completion date: August 2008. Source: VBA. [End of table] As table 3 shows, VBA does not expect to complete the development of the functionalities needed to process all new compensation and pension claims until August 2008. However, according to VBA, the estimated completion date is the planned date for completing all development and testing, but it is not necessarily the date when users will be able to begin using the new system. Before such use can begin, other activities need to occur. For example, users must receive training, and VETSNET program management must authorize the use of the system at each regional office. In addition to its remaining software development activities, VBA also faces the challenge of converting records for claims currently paid by the BDN to the replacement system. Existing compensation and pension cases on the BDN number about 3 million and about 535,000, respectively. Table 4 shows the phases in which VA is planning to perform conversions, according to its Integrated Master Schedule. Table 4: Conversion Phases: Phases: Phase 1; Conversions: Conversion testing on terminated records and selected live records; Estimated start date: March 2007; Estimated completion date: August 2007. Phases: Phase 2; Conversions: Conversion of live compensation records; Estimated start date: January 2008; Estimated completion date: October 2008. Phases: Phase 3; Conversions: Conversion of pension records; Estimated start date: November 2008; Estimated completion date: June 2009. Source: GAO analysis of VBA data. [End of table] As the table shows, VBA conversion efforts began in March 2007. VBA first performed conversion testing on 310,000 terminated (that is, inactive) compensation cases so that it could develop and apply lessons learned to the conversion of live records. According to VETSNET officials, VBA planned to continue testing by converting live cases at two regional offices (Lincoln and Nashville) that were used as testing sites during development. It then plans to perform the conversion of all live compensation cases. After the compensation conversion is complete, VBA plans to begin efforts to convert pension benefits cases. Based on VETSNET project documentation, activities supporting the releases have so far been performed on time, consistent with the milestones in the recently finalized Integrated Master Schedule. For example, VA completed the Project Initiation and Review Authorization for Release 1 on September 7, 2006, as scheduled (see fig. 1, shown earlier in the report, for the phases of system development and the required milestone reviews). It also completed the Preliminary Design Review and the Critical Design Review as scheduled (on November 20 and December 22, respectively). Planning for Release 2 is also on schedule: a kickoff meeting was held on January 24, 2007, which established the scope of the release, and the Project Initiation and Review Authorization was conducted on February 8. Conclusions: VA has responded to SEI's assessment by making significant changes in its approach to the project and its overall management, including slowing the pace of development, establishing a new governance structure, and ensuring staff resources. However, VBA has not yet addressed all the issues raised by the SEI assessment. That is, it has not ensured ownership responsibility for total system and process operating costs, because it is not currently monitoring and reporting in-house expenditures for the project. It has not defined processes and resources for capacity planning for the project. In addition, VBA has not yet addressed issues related to the conversion of records now on the BDN to the replacement system. Specifically, it has not addressed the risk that large numbers of records may need to be returned to the BDN, decided on the degree of confidence it will require that records are converted accurately, or developed complete plans for converting and validating records. In addition, although VBA has improved key processes for managing the software development, these processes have not yet been institutionalized in defined policies and procedures, and performance measures of productivity and user satisfaction have not been developed. VETSNET management has stated that it gave priority to other activities, such as establishing appropriate governance and organizational structures, and that it is still gathering information to assist in prioritizing the activities that remain. Much work remains to be done to complete the VETSNET initiative. Although VBA has substantially increased the number of claims being paid by the replacement system, it must not only finish the development and deployment of the software, it must also convert the over 3.5 million records now on the BDN to the replacement system. Addressing the remaining issues identified by SEI would improve VBA's chances of successfully completing the replacement system and ending reliance on the aged BDN to pay compensation and pension benefits. Recommendations for Executive Action: To enhance the likelihood that the replacement system will be successfully completed and implemented, we are recommending that the Secretary of Veterans Affairs take the following five actions: * Direct the CIO to institute measures to track in-house expenditures for the project. * Direct the VETSNET project to include activities for capacity planning in the VETSNET Integrated Master Schedule and ensure that resources are available for these activities. * Direct VBA to (1) develop a strategy to address the risk that large numbers of records may need to be returned to the BDN; (2) determine whether a greater confidence level for accuracy should be required in the conversion process; and (3) develop a detailed validation plan that includes the scheduled conversions for each regional office and the validation team members needed for that specific conversion. * Direct the CIO to document and incorporate the improved processes for managing risks, requirements, and defects into specific policy and guidance for the replacement initiative and for future use throughout VBA. * Direct the replacement project to develop effective results-oriented performance measures that show changes in efficiency, economy, or improvements in mission performance, as well as measures of user satisfaction, and to monitor and report on the progress of the initiative according to these measures. Agency Comments and Our Evaluation: In providing written comments on a draft of this report, the Secretary of Veterans Affairs agreed with our conclusions and concurred with the report's recommendations. (The department's comments are reproduced in app. II.) The comments described actions planned that respond to our recommendations, such as incorporating processes developed for the VETSNET project in standard project management policies, processes, and procedures that would be used for all IT projects in the department. In addition, the comments provided further information on actions already taken, such as details of the records conversion process. If the planned actions are properly implemented, they could help strengthen the department's management of the replacement system project and improve the chances that the system will be successfully completed. We are sending copies of this report to the Chairman and Ranking Minority Member of Committee on Veterans' Affairs. We are also sending copies to the Secretary of Veterans Affairs and appropriate congressional committees. We will make copies available to other interested parties upon request. Copies of this report will also be made available at no charge on GAO's Web site at http://www.gao.gov. Should you or your staff have any questions about this report, please contact me at (202) 512-6304 or by e-mail at melvinv@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix III. Sincerely yours, Signed by: Valerie C. Melvin: Acting Director: Human Capital and Management Information Systems: [End of section] Appendix I: Objectives, Scope, and Methodology: Our objectives were to determine (1) to what extent the Department of Veterans Affairs (VA) has followed the course of action recommended by the Carnegie Mellon Software Engineering Institute (SEI) and addressed the concerns that it raised and (2) the current status of the replacement project, the Veterans Service Network (VETSNET). To determine the actions taken to implement SEI's recommended approach and address the concerns it raised, we: * determined the recommended actions by analyzing the report; * compared the concerns identified in the assessment to actions planned, actions undertaken but not completed, and actions implemented by VA officials or contractors; * interviewed contractor, VA, and VETSNET program office officials to gain an understanding about processes developed and procedures implemented; and: * obtained and reviewed relevant VA and contractor documents that disclosed or validated VA responses to SEI's concerns. To determine the status of system development efforts and the extent that tasks planned for the initiative were completed, we analyzed VA and contractor documentation regarding system operations and development, time frames, and activities planned. We analyzed VA documents that disclosed costs to date and costs planned for completion of the initiative. We did not assess the accuracy of the cost data provided to us. We supplemented our analyses with interviews of VA and contractor personnel involved in the replacement initiative. We visited the Nashville and St. Petersburg regional offices to observe the replacement system in operation and the processes and procedures used to test and validate the replacement system as it was being developed and implemented. We analyzed VA documentation and relevant evidence from contractors involved in the replacement effort to establish the work remaining to complete the project. Finally, we interviewed cognizant VA and contractor officials, responsible for developing, testing, and implementing the replacement system. We performed our work at VA offices in Washington, D.C., and at VA regional offices in Nashville, Tennessee, and St. Petersburg, Florida, from April 2006 to April 2007 in accordance with generally accepted government auditing standards. [End of section] Appendix II: Comments from the Department of Veterans Affairs: The Secretary Of Veterans Affairs: Washington: April 18, 2007: Ms. Valerie C. Melvin: Acting Director: Human Capital and Management Information Systems: Information Technology Team: U. S. Government Accountability Office: 441 G Street, NW: Washington, DC 20548: Dear Ms. Melvin: The Department of Veterans Affairs (VA) has reviewed your draft report, Veterans Benefits Administration: Progress Made in Long-Term Effort to Replace Benefits Payment System, but Challenges Persist, GAO 07-614) and agrees with your conclusions and concurs in your recommendations. VA is eager to move to a new automated system that will streamline its benefit claims processing functions in a manner that is accurate and timely to serve our Nation's veterans. The Government Accountability Office's recommendations should provide valuable assistance in achieving that goal. The enclosure addresses your recommendations in detail. I appreciate the opportunity to comment on your draft report. Sincerely yours, Signed by: R. James Nicholson: Enclosure: Department of Veterans Affairs (VA) comments to Government Accountability Office (GAO) draft report Veterans Benefits Administration: Progress Made in Long-Term Effort to Replace Benefits Payment System, but Challenges Persist (GAO-07-614): To enhance the likelihood that the replacement system will be successfully completed and implemented, GAO recommends that the Secretary of Veterans Affairs take the following actions: * direct the CIO to institute measures to track in-house expenditures for the project; Concur - VA's Chief Information Officer (CIO) is committed to tracking the in-house expenditures required to complete the Veterans Service Network (VETSNET) project. In addition to tracking Government full time employee (FTE) and contractor costs, the VETSNET Project Management Office will take responsibility for tracking travel costs associated with the project to include those related to application testing and training. * direct the VETSNET project to include activities for capacity planning in the VETSNET Integrated Master Schedule and ensure that resources are available for these activities; Concur - The Veterans Benefits Administration (VBA) has already implemented capacity planning activities in conjunction with the Integrated Master Schedule (IMS). As noted during the review, capacity planning was difficult in advance of the IMS development, because VBA had not identified when specific functionality would be delivered and available for claims processing. The IMS identifies when specific types of functionality will be delivered into the production environment and when records will be converted from the Benefits Delivery Network (BDN). Increased levels of claims processing are tied to each of these releases and conversions, which in turn, allows for more effective capacity planning. As reported, VBA has a staff dedicated to performance monitoring and capacity planning (note, this organization now falls under the Department's Office of Information and Technology). Performance testing, which is a component of capacity planning, is a defined element of the overall VETSNET development and testing timeline. Results of performance testing for each release are factored in and used to validate capacity assumptions. To ensure functionality already delivered to production continues to meet performance expectations, VETSNET leadership attends daily conference calls regarding corporate database performance. This ensures any issues that may unexpectedly arise are rapidly addressed. In addition, VETSNET leadership has fully briefed its overall schedule and claims processing estimates to the capacity planning staff to assist that staff in better understanding VETSNET's plans and needs. Finally, to ensure sufficient attention is given this vital element of the project, VBA has provided for assistance in capacity planning in MITRE Corporation's 2007 contract for VETSNET support. * direct VBA to (1) develop a strategy to address the risk that large numbers of records may need to be returned to the BDN; (2) determine whether a greater confidence level for accuracy should be required in the conversion process; and (3) develop a detailed validation plan that includes the scheduled conversions for each regional office and the validation team members needed for that specific conversion; Concur-(1) As part of addressing the Software Engineering Institute's (SEI) recommendations, and in developing the Integrated Master Schedule, VBA has taken significant steps in addressing the conversion of existing records from the BDN to the corporate database. As reported, VBA stopped converting records from the BDN in order to develop a more coordinated approach. Conversions are now planned for execution in accordance with delivery of application functionality into production, followed by a period of stabilization. Since the initial conversions of VA Regional Office (VARO) Lincoln, Nebraska, records, data show that the only reason records required return to BDN was because application functionality no longer supported the record. For example, the record of a veteran in receipt of disability compensation was successfully converted from the BDN. The veteran then elected to receive non-service connected pension as the greater monetary benefit. Since pension is not fully supported in VETSNET, the veteran's record required return to the BDN to continue payment. At no time has a record required return to the BDN due to the conversion being "wrong" or paying the veteran incorrectly. VBA's current strategy, as noted in the report, is to convert a large volume of terminated records, followed by conversions of records at Lincoln and then VARO Nashville, Tennessee. These conversions were designed to test both conversion processes and validation methods to ensure the correct conversion of the selected records. A "correct" conversion is defined as uninterrupted payment at the correct rate. In the event modifications to our conversion methods are required, the schedule allows sufficient time to address those changes. In addition, a core conversion team has been assembled and is testing and validating. This team includes technical resources, individuals from headquarters staff of the Compensation and Pension Service (C&P), Office of Resource Management, and field resources for both the claims processing and finance activities. These individuals will serve as the core team throughout the entirety of our conversion process. (2) During these initial test conversions, physical record validation is performed at the 95 percent confidence level. In addition, a number of automated reports have been and are being developed, to provide additional oversight for the process. These reports will compare the state of records pre-and post-conversion, for example, the number of records submitted for conversion and the number converted; the amount of payment associated with the records pre-and post-conversion, etc. Similarly, an automated testing process was developed and is in use to validate payment processing for application development purposes. This automated process is also being used to ensure the converted records conform to the required format for Treasury processing. (3) As noted previously, a core conversion team has been assembled and is actively engaged in testing and validation efforts. These individuals will remain on the core team throughout the conversion process, continuing to test and validate the conversion logic and outcome for each scheduled conversion. As incorporated in the IMS, conversions will follow the release and stabilization of application functionality in production. Live compensation records are planned for three or four conversions, depending upon the number of records we decide to convert in a given "lot". The number of records will be determined once we complete the current testing phase underway. The order of regional offices will also be determined at that time, based on a variety of factors, to include business process changes currently underway in VBA, as well as workload considerations. Service-connected death records (Dependency and Indemnity Compensation) will be converted in one lot, following release of that functionality. Similarly, existing income-based pension records will be converted in two lots, once functionality is in place. The decision to convert in two lots is based on workload considerations for the Pension Maintenance Centers. VBA and MITRE Corporation believe these efforts will significantly reduce the risks associated with conversions. During its weekly meetings, the VETSNET Executive Team formally tracks these risks. * direct the CIO to document and incorporate the improved processes for managing risks, requirements, and defects into specific policy and guidance for the replacement initiative and for future use throughout VBA; and: Concur - As part of the centralization of information technology (IT) within VA, the CIO has undertaken the effort of implementing best business practices to assist VA in its consolidation. The goal is to establish application development processes that employ industry best practices and that have the potential to accelerate the successful completion of existing projects and ensure continued success in future projects. As part of this effort, VETSNET's project management processes, including risk management, requirements management, and testing methodology to include the management of defects are under review. VETSNET project management processes will be incorporated into a set of standard project management policies, processes and procedures for all IT projects in VA. The VETSNET governance model includes the VETSNET Executive Team consisting of representatives from the technical, project management, and business lines. The CIO has identified this governance model as the model for all VA enterprise-wide IT projects. The governance model is being implemented in other VA priority IT development programs, such as Financial and Logistics Integrated Technology Enterprise (FLITE). * direct the replacement project to develop effective results-oriented performance measures that show changes in efficiency, economy, or improvements in mission performance, as well as measures of user satisfaction, and to monitor and report on the progress of the initiative according to these measures. Concur - As noted in the report, VBA has implemented a number of performance metrics to track progress toward transition of all C&P claims processing and payments from BDN to VETSNET. Some examples were provided in the body of the report. In addition, VBA recently completed its periodic Work Measurement Study, which assesses the level of effort required for various C&P work tasks. One element of this year's study will assess task time for work processed using VETSNET Award, as well as through BDN. The other components of the VETSNET suite (i.e., MAP-D and RBA2000) have already been in full deployment for a number of years, and exceed functionality currently in BDN. In that regard, comparisons to BDN processing are not applicable. The efficiencies gained by these applications are directly related to the reusability of data and the ability to determine the status of a veteran's claim without the need to review the physical claims file. This reduces staff time required to physically locate and review claims files, provides enhanced customer service, and facilitates the sharing of work across regional offices to balance workload and provide more timely service to veterans. With respect to user satisfaction, while VBA does not formally survey users as to their level of satisfaction, we have a variety of forums in place to solicit and receive feedback from the user community. This includes, for example, an electronic mailbox to receive questions or comments, and regular "super user" conference calls to discuss issues and suggestions. In addition, the Special Assistant to the Under Secretary regularly addresses regional office managers to ensure information reaches the executive level, and to address any concerns or comments they may have. As SEI noted, there is a strong "pull" from the user community for expanded VETSNET functionality. That has been borne out by the increase in claims processing volume once field stations were authorized to increase the scope of claims available for processing. A more formal measure of user satisfaction can be implemented, but we believe our core measure of project success is reflected by the metrics currently in place. These metrics track the current volume of claims processing and beneficiaries being paid through VETSNET, as well as tracking the activities necessary to transition all C&P processing off of the BDN. [End of section] Appendix III: GAO Contact and Staff Acknowledgments: GAO Contact: Valerie Melvin, (202) 512-6304 or melvinv@gao.gov: Staff Acknowledgments: In addition to the individual named above, key contributions were made to this report by Barbara Oliver, Assistant Director; Nabajyoti Barkakati; Barbara Collier; Neil Doherty; Matt Grote; Robert Williams; and Charles Youman. FOOTNOTES [1] It also refers to the initiative as the compensation and pension or C&P replacement system. [2] GAO, Software Capability Evaluation: VA's Software Development Process Is Immature, GAO/AIMD-96-90 (Washington, D.C.: June 19, 1996); Veterans Benefits Modernization: VBA Has Begun to Address Software Development Weaknesses but Work Remains, GAO/AIMD-97-154 (Washington, D.C.: Sept. 15, 1997); VA Information Technology: Progress Continues Although Vulnerabilities Remain, GAO/T-AIMD-00-321 (Washington, D.C.: Sept. 21, 2000); VA Information Technology: Important Initiatives Begun, Yet Serious Vulnerabilities Persist, GAO-01-550T (Washington, D.C.: Apr. 4, 2001); VA Information Technology: Management Making Important Progress in Addressing Key Challenges, GAO-02-1054T (Washington, D.C.: Sept. 26, 2002); and Information Technology: VA and DOD Face Challenges in Completing Key Efforts, GAO-06-905T (Washington, D.C.: June 22, 2006). [3] GAO-06-905T. [4] Kathryn Ambrose, William Novak, Steve Palmquist, Ray Williams, and Carol Woody, Report of the Independent Technical Assessment on the Department of Veterans Affairs VETSNET Program (Carnegie Mellon Software Engineering Institute, September 2005). [5] The BDN currently runs on aging software: COBOL programs and a nonrelational database. Analysts have indicated that moving from a nonrelational database of the BDN type to a more modern relational database is a challenging task. [6] This program provides veterans, service members, reservists, and certain veterans' dependents with educational resources. [7] To help veterans with service-connected disabilities become employable and obtain and maintain suitable employment, the program provides a range of direct and supportive services. These services include comprehensive evaluation of rehabilitation needs (vocational or independent living); training and employment services to obtain or maintain suitable employment; and independent living services, such as training and specialized equipment to enable independence in the activities of daily living. [8] Not all regional offices process all the different types of benefits. For example, adjustments to pension claims are processed at three pension maintenance centers in St. Paul, Milwaukee, and Philadelphia, and education claims are processed only at the regional offices in Atlanta, Buffalo, Muskogee, and St. Louis. The Philadelphia Regional Office and Insurance Center has sole responsibility for insurance benefit processing, and nine Regional Loan Centers administer the loan guaranty program. [9] Claims may also be reopened if a veteran provides additional information on a claim that was denied. [10] GAO/AIMD-96-90. [11] Specifically, at the repeatable level of process maturity, basic project management processes are established to track cost, schedule, and functionality, and the necessary process discipline is in place to repeat earlier successes on projects with similar applications. [12] Kathryn Ambrose, William Novak, Steve Palmquist, Ray Williams, and Carol Woody, Report of the Independent Technical Assessment on the Department of Veterans Affairs VETSNET Program (Carnegie Mellon Software Engineering Institute, September 2005). [13] CMMI Product Team, CMMI for Development, Version 1.2 (Carnegie Mellon Software Engineering Institute, August 2006). [14] As previously noted, there are variations in types of compensation and pension benefits, including death pensions for survivors of deceased wartime veterans, burial benefits, Dependency and Indemnity Compensation to some surviving parents, and benefits paid to third parties, such as people to whom a veteran has given power of attorney, medical service providers, and so on. In addition, different forms of payment must be accommodated (check or electronic funds transfer). [15] SEI quoted program staff as calling the process "extraordinarily time consuming," saying that "It takes 5 minutes to get a veteran into VETSNET and then 5 days to get him back out." [16] This realignment was approved on February 27, 2007. [17] The functionalities for the Search and Participant Profile, formerly a separate application, are now in Share. [18] Survivor benefits may be paid to survivors of veterans who were eligible for either compensation or pension benefits, depending on the circumstances. [19] Examples of third-party/nonveteran payees include people to whom a veteran has given power of attorney, medical service providers designated to receive payments, and so on. GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site (www.gao.gov). Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to www.gao.gov and select "Subscribe to Updates." Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. Government Accountability Office 441 G Street NW, Room LM Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400 U.S. Government Accountability Office, 441 G Street NW, Room 7125 Washington, D.C. 20548: Public Affairs: Paul Anderson, Managing Director, AndersonP1@gao.gov (202) 512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, D.C. 20548: