This is the accessible text file for GAO report number GAO-03-1045 entitled 'Veterans' Benefits: Improvements Needed in the Reporting and Use of Data on the Accuracy of Disability Claims Decisions' which was released on September 30, 2002. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States General Accounting Office: GAO: Report to Congressional Requesters: September 2003: Veterans' Benefits: Improvements Needed in the Reporting and Use of Data on the Accuracy of Disability Claims Decisions: GAO-03-1045: GAO Highlights GAO Highlights of GAO-03-1045, a report to congressional requesters. Why GAO Did This Study: The Veterans Benefits Administration (VBA) has a large inventory of claims for benefits under its compensation and pension programs. The Secretary of the Department of Veterans Affairs has pledged to substantially reduce this inventory in order to improve timeliness. In response, VBA emphasized producing more claims decisions per year. GAO was asked to ascertain how accuracy has changed since VBA increased its emphasis on production and to report on the agency's efforts to ensure the accuracy of its decisions. What GAO Found: From fiscal years 2001 to 2002, VBA's accuracy of decision-making in the disability compensation and pension benefit programs declined from 89 percent to 81 percent. The agency had reported a slight improvement in accuracy between fiscal years 2001 and 2002—from 78 percent to 80 percent. However, we found that these two annual figures were not comparable because the agency had substantially changed the way it measured accuracy for fiscal year 2002. Although VBA acknowledged a change in its accuracy measure in its annual report to the Congress, the agency did not revise its 2001 figure to allow for an appropriate comparison with 2002. VBA officials GAO spoke with suggested several factors that may have contributed to the decline in accuracy. We were not able to quantify the relative contribution of these factors. These factors included VBA's emphasis on increasing claims decisions, the specific processing requirements of the Veterans Claims Assistance Act (VCAA) of 2000, and the relative inexperience of VBA's claims processing staff. To help ensure accountability for accuracy, VBA set accuracy standards for its regional offices. Although VBA has regional office-level accuracy data, it has not made full use of this information to encourage better performance from regional offices with low accuracy scores. For example, in fiscal year 2002, VBA did not require offices with poor accuracy to prepare improvement plans and gave performance awards to two offices that clearly failed to meet VBA's accuracy goal. Figure: VBA Compensation and Pension Rating Products Decided and Accuracy Rates, Fiscal Years 2001-2002: Production: FY 2001, Number completed: 481,000; FY 2002, Number completed: 797,000. Accuracy: FY 2001, Percentage: 89%; FY 2002, Percentage: 81%. Source: GAO analysis of Veterans Benefits Administration data. Note: Accuracy rates in the above chart are estimates based on samples of rating products completed in fiscal years 2001 and 2002 and have margins of error of about plus or minus 1 percentage point at the 95- percent level of confidence. We determined these accuracy rates by comparing substantially similar questions for the 2 fiscal years that focused on whether the decision to grant or deny benefits was correct. The fiscal year 2002 accuracy score includes a specific question on proper VCAA pre-decision notification that VBA did not include in its fiscal year 2001 quality reviews. [End of figure] What GAO Recommends: GAO recommends that VBA: * report the accuracy of VBA disability compensation and pension claims decisions to the Congress and other stakeholders in a manner that allows for valid comparisons of accuracy across fiscal years and; * better hold regional offices accountable for the accuracy of their claims decisions, by increasing the use of its regional office accuracy data. VBA concurred with GAO's recommendations and provided additional information about its efforts to improve accuracy. [hyperlink, http://www.gao.gov/products/www.gao.gov/cgi-bin/getrpt?GA0- 03-1045]. To view the full report, including the scope and methodology, click on the link above. For more information, contact Cynthia A. Bascetta at (202) 512-7215 or bascettac@gao.gov. [End of section] Contents: Letter: Results in Brief: Background: Accuracy Declined in Fiscal Year 2002: VBA Could Have Done More to Hold Regional Offices Accountable for Accuracy: Conclusions: Recommendations: Agency Comments and Our Response: Appendix I: Comments from the Veterans Benefits Administration: Figure: Figure 1: Fiscal Year 2001 STAR Rating Checklist—-Criteria for a Correct VBA Decision: Abbreviations: BVA: Board of Veterans' Appeals: SIPA: Systematic Individual Performance Assessment: STAR: Systematic Technical Accuracy Review: VA: Department of Veterans Affairs: VBA: Veterans Benefits Administration: VCAA: Veterans Claims Assistance Act of 2000: VSR: veterans service representative: [End of section] United States General Accounting Office: Washington, DC 20548: September 30, 2003: The Honorable Arlen Specter: Chairman: The Honorable Bob Graham: Ranking Minority Member: Committee on Veterans' Affairs: United States Senate: The Honorable John D. Rockefeller IV: United States Senate: At the beginning of fiscal year 2002, over 400,000 claims for disability compensation and pension benefits were waiting for Veterans Benefits Administration (VBA) decisions. On average, veterans were waiting about 6 months and many were waiting more than a year. VBA has been making a concerted effort to reduce the inventory of claims and complete decisions in an average of 100 days by the end of fiscal year 2003. In order to improve the timeliness of its disability claims decisions, VBA established a goal of completing about 839,000 decisions in fiscal year 2002 and reducing its inventory of pending claims to about 316,000 by the end of that year. During fiscal year 2002, VBA increased its claims decisions by two-thirds from about 481,000 to about 797,000 and reduced its inventory of pending claims by about one-fifth from about 421,000 to about 346,000. For fiscal year 2003, VBA's goals are to produce about 806,000 decisions and further reduce inventory to 250,000 claims. In light of VBA's emphasis on completing more claims decisions more quickly, you asked us to assist the Committee in its oversight of VBA by examining the agency's efforts to ensure the accuracy of decision- making in its disability compensation and pension benefits programs. In response, we assessed (1) how accuracy may have changed since VBA increased its emphasis on production and (2) how well VBA ensures the accuracy of its decisions. We reviewed VBA's efforts to measure and report accuracy for fiscal years 2001 and 2002.[Footnote 1] To determine how accuracy had changed over that time, we independently calculated and compared VBA's accuracy rates for fiscal years 2001 and 2002. As part of our analysis of how well VBA ensures national and regional office accuracy, we reviewed VBA's accuracy measurement system, known as the Systematic Technical Accuracy Review (STAR) program, including how VBA incorporates accuracy into its performance standards for both regional office directors and claims processing staff. In addition, we examined VBA's criteria for regional office performance awards and the results of reviews by VBA survey teams of regional offices. We discussed factors affecting accuracy with staff at VBA regional offices in Phoenix, Arizona; St. Petersburg, Florida; Boston, Massachusetts; Pittsburgh, Pennsylvania; and Houston, Texas.[Footnote 2] We also discussed accuracy issues with representatives of the American Legion, Disabled American Veterans, Paralyzed Veterans of America, and Veterans of Foreign Wars to gain their perspectives on VBA efforts to improve accuracy. We conducted our review from January through August of 2003 in accordance with generally accepted government auditing standards. Results in Brief: From fiscal years 2001 to 2002, VBA's accuracy of decision making in the compensation and pension programs declined from 89 percent to 81 percent.[Footnote 3] We determined these accuracy rates by comparing substantially similar questions for the 2 fiscal years that focused on whether the decision to grant or deny benefits was correct. The agency had reported a slight improvement in accuracy between fiscal years 2001 and 2002—from 78 percent to 80 percent. However, we found that these two annual figures were not comparable because the agency had substantially changed the way it measured accuracy for fiscal 2002. Although VBA acknowledged a change in its accuracy measure in its annual report to the Congress, the agency did not revise its 2001 figure to allow for an appropriate comparison with 2002. VBA officials attributed the decline in accuracy during this period to several factors. We were not able to quantify the relative contribution of these factors. These factors included headquarters emphasis on production, the specific processing requirements of the Veterans Claims Assistance Act of 2000, and the relative inexperience of VBA's claims processing staff. To help ensure accountability for accuracy, VBA set accuracy standards for its regional offices. However, VBA has not fully used data gathered on the accuracy of its regional office decisions to encourage better regional office performance. For example, while some regional offices were required to prepare plans to improve claims processing production, similar plans were not required for offices to improve accuracy. Also, two regional offices received overall performance awards despite not meeting VBA's accuracy standard. Unless VBA strikes a balance between production goals and accuracy standards, it cannot ensure that as it improves decision timeliness, it will also improve decision accuracy. This report contains recommendations to the Secretary of Veterans Affairs to direct VBA to improve its reporting on and use of accuracy data. VBA concurred with our recommendations and provided additional information about its efforts to improve accuracy. Background: Through its disability compensation program, VBA pays monthly benefits to veterans with service-connected disabilities (injuries or diseases incurred or aggravated while on active military duty) according to the severity of the disability. VBA's pension benefit program pays monthly benefits to wartime veterans who have low incomes and are permanently and totally disabled for reasons not service-connected.' In addition, VBA pays dependency and indemnity compensation to some deceased veterans' spouses, children, and parents and to survivors of service members who died on active duty. In fiscal year 2002, VBA paid over $22 billion in disability compensation to an average of about 2.4 million veterans and over 300,000 survivors. VBA also paid over $3 billion in pensions to an average of about 580,000 veterans and survivors. The amount of disability compensation largely depends on the degree to which a veteran is disabled. VBA determines the degree to which veterans are disabled in 10 percent increments on a scale of 0 to 100 percent. Basic monthly payments range from $104 for 10 percent disability to $2,193 for 100 percent disability. About 65 percent of veterans receiving disability compensation have disabilities rated at 30 percent and lower; about 8 percent have disabilities rated at 100 percent. The most common impairments for veterans who began receiving compensation in fiscal year 2001 were tinnitus (ex., a ringing in the ear), auditory acuity impairment rated at 0 percent (i.e., mild hearing difficulties), skeletal conditions, arthritis due to trauma, post-traumatic stress disorder, and scars. Eligibility and priority for other Department of Veterans Affairs (VA) benefits and services such as health care and vocational rehabilitation are affected by these VA disability ratings. When a veteran submits a claim to any of VBA's 57 regional offices, a veterans service representative (VSR) is responsible for obtaining the relevant evidence to evaluate the claim. Such evidence includes veterans' military service records, medical examinations and treatment records from VA medical facilities, and treatment records from private medical service providers. Once a claim is developed (i.e., has all the necessary evidence), a rating VSR, also called a rating specialist, evaluates the claim, determines whether the claimant is eligible for benefits, and assigns a rating based on degree of disability. Veterans with multiple disabilities receive a single composite rating. For veterans claiming pension eligibility, the regional office determines if the veteran served in a period of war, is permanently and totally disabled for reasons not service-connected, and meets the income thresholds for eligibility. A veteran who disagrees with the regional office's decision for either program can appeal sequentially to VA's Board of Veterans' Appeals (BVA), the U.S. Court of Appeals for Veterans Claims, and the U.S. Court of Appeals for the Federal Circuit. In fiscal year 1999, VBA implemented a quality review system to improve the measurement of the accuracy of its claims decisions. Under STAR, VBA selects a random sample of completed claims decisions each month from each of its 57 regional offices. STAR quality review staff review these claims using a standard checklist. If a claim has any error, VBA counts the entire claim as incorrect for accuracy rate computation purposes. STAR then returns the case file and the results of the review to the regional office that made the decision. If an error was found, the regional office is required to either correct it or request reconsideration of the error by STAR. VBA made several significant changes to its STAR system in fiscal year 2002. First, in response to our March 1999 recommendation, VBA made the STAR review staff organizationally independent of the regional offices to ensure the independence of the reviews.[Footnote 5] Second, VBA more than doubled the number of rating decisions it reviewed (from 3,209 in fiscal year 2001 to 6,646 in fiscal year 2002) in order to obtain more precise regional office level accuracy scores. Third, VBA improved its key measure to focus more on whether decisions to grant or deny benefits were correct. Previously, VBA's key measure also included decision documentation and notification issues. Figure 1 shows the areas covered by the checklist used to measure accuracy in fiscal year 2001. The revised key accuracy measure includes only the first four areas of the checklist. Figure 1: Fiscal Year 2001 STAR Rating Checklist—-Criteria for a Correct VBA Decision: [Refer to PDF for image: list] Address all issues – The decision addressed all claimed disabilities and benefits, and any disabilities or benefits that could be inferred from the available evidence. Proper development – VBA made an adequate attempt to obtain all evidence needed to decide the claim. Grant or deny – The decision to grant or deny was correct; if granted, the percentage evaluation was correct. Award actions – Where benefits were granted, the effective dates of benefits and payment amounts were correct. Reasons and bases – The decision discussed all applicable evidence and the basis of the decision was explained. Notification – The claimant and power of attorney, if applicable, were correctly notified of the decision and were notified of appeal rights. Source: Veterans Benefits Administration. Accuracy Declined in Fiscal Year 2002: From fiscal years 2001 to 2002, VBA's accuracy of decision making in the compensation and pension programs declined from 89 percent to 81 percent. We determined these accuracy rates by comparing substantially similar questions for the 2 fiscal years that focused on whether the decision to grant or deny benefits was correct.[Footnote 6] However, the agency reported that it had improved its accuracy for that period. This discrepancy occurred because VBA did not report comparable accuracy data for the 2 fiscal years. VBA reported its fiscal year 2002 accuracy based on a revised measure that excluded two areas of the checklist—reasons and bases and notification. VBA then compared this to its fiscal year 2001 accuracy, which included these two areas of the checklist. VBA officials attributed the decline in accuracy during this period to several factors. We were not able to quantify the relative contribution of these factors. These factors included headquarters emphasis on production, the specific processing requirements of the Veterans Claims Assistance Act of 2000, and the relative inexperience of VBA's claims processing staff. In fiscal year 2002, VBA made progress in increasing production of claims decisions and reducing inventory. The number of claims decisions rose substantially from about 481,000 in fiscal year 2001 to about 797,000 in fiscal year 2002. During the same time period, the number of claims VBA received increased from about 674,000 to 722,000. The higher level of production permitted VBA to reduce its inventory of pending claims by 18 percent from about 421,000 to about 346,000. Although accuracy actually declined, VBA reported in its performance report to the Congress for fiscal year 2002 that the accuracy of its rating decisions for compensation and pension benefits had improved slightly. VBA reported that accuracy improved from 78 percent in fiscal 2001 to 80 percent in fiscal 2002.[Footnote 7] However, these rates were not comparable because the new "benefit entitlement accuracy" rate for fiscal year 2002 was based on fewer areas of the checklist that focused on whether and to what extent the claimant received a benefit. In its fiscal year 2002 performance report, VA made note of this change but did not recalculate the fiscal 2001 rate to provide a correct comparison. When we recomputed the agency's accuracy rate for fiscal 2001 using its revised criteria, the result was 89 percent, not 78 percent as reported. As a result, the correct comparison showed a decline in accuracy from 89 percent to 81 percent. The agency corroborated our finding after performing its own computation at our request. VBA officials we spoke with cited a number of factors that affected accuracy over the fiscal years 2001 to 2002 period. A key factor cited was VBA's emphasis on production, such that accuracy became a lesser priority. In 2001, the Secretary of Veterans Affairs set an ambitious goal to reduce the agency's average time to complete a rating decision from 173 days (in fiscal year 2000) to 100 days by the end of fiscal year 2003. Also, the Secretary tasked VBA with reducing the inventory of rating claims to 250,000 by the end of September 2003. VBA managers responded by emphasizing the completion of more rating cases each month. The result has been a significant increase in production and a significant reduction in the rating inventory since the end of fiscal year 2001. In fiscal year 2002, VBA incorporated its new production targets into its measures for regional office performance.[Footnote 8] Regional office directors became accountable for specific targets for production, inventory reduction, and timeliness improvement. The agency also established regional office performance award criteria that gave more weight to efficiency than accuracy. Three of the four criteria for cash awards were based on production and timeliness and one on accuracy; bonuses could be received without meeting the accuracy criteria. VBA officials also attributed accuracy problems to the addition of more specific processing requirements under the Veterans Claims Assistance Act (VCAA) of 2000. The act broadened VBA's responsibility to assist veterans — such as notifying them of needed evidence and helping them develop that evidence for their claims. VBA incorporated VCAA requirements into its STAR rating accuracy checklist in fiscal year 2002. In the first 4 months of fiscal year 2002, VCAA errors accounted for almost half of all errors identified by STAR rating reviewers. In April 2002, VBA required each regional office to re- train its claims processing staff on the new requirements.[Footnote 9] Over the last 8 months of fiscal year 2002, the proportion of VCAA- related errors declined to about one-third of all errors. VBA officials also cited the relative inexperience of VBA's claims processing staff. Regional office officials noted that they had many VSRs and rating specialists with insufficient training and experience to be fully proficient at developing and making decisions on claims. For example, officials at two offices we visited noted that their accuracy was affected by large numbers of inexperienced staff, due to a large increase in staff size or a relatively high rate of turnover. VBA has found that attrition rates for new claims processing staff hired over a 3-year period ranged from 0 percent to 49 percent at some regional offices.[Footnote 10] VBA Could Have Done More to Hold Regional Offices Accountable for Accuracy: While VBA has established accuracy standards for its regional offices, it has not made the best use of its accuracy data on regional offices. VBA collected regional office accuracy data in fiscal year 2002 but has not fully used the information to evaluate regional office performance, correct errors, and identify needed training that could reduce errors. Unless VBA better uses these data to hold regional offices accountable for accuracy, it cannot ensure that as decisions become timelier, they also become more accurate. VBA is developing an agencywide system for evaluating individual claims processing employees' performance against VBA's individual accuracy standards. In fiscal year 2002, VBA improved its STAR review system to provide independent review of decisions and more precise accuracy data at the regional office level. Regional accuracy scores are estimates based on samples of cases and revealed high and low performers relative to the agency's fiscal year 2002 accuracy goal of 85 percent. Eleven regional offices clearly failed to meet the goal while 3 clearly met or exceeded the goal.[Footnote 11] However, VBA managers did not fully use the regional office-specific information generated by the STAR unit as a basis for improving accuracy at the regional office level. For example, VBA did not require regional offices that failed to meet the national accuracy goal of 85 percent to prepare strategies for improvement. In contrast, in fiscal year 2002, 13 regional offices that did not meet established targets for production, inventory, and timeliness were required to develop corrective action plans. Also, as noted earlier, VBA did not make rating accuracy a prerequisite for receiving a performance award. Three of the four criteria for cash awards were based on production and timeliness and one on accuracy; a bonus could be received without meeting the accuracy criteria. In fiscal year 2002, 2 offices received performance awards despite clearly not meeting VBA's accuracy goal. These offices had accuracy rates of 71 percent and 75 percent, respectively. VBA's regional offices did not always properly address STAR errors that were returned to them, a violation of VBA policy that regional offices either correct errors or formally request that the STAR unit reconsider its error call. VBA survey teams, which review judgmental samples of STAR cases with errors, found that the 18 regional offices they visited from October 2001 through December 2002 had failed to respond properly to over 40 percent of the errors reviewed. At 14 of the 18 offices, the survey teams found problems significant enough to recommend improvements in STAR error handling. For example, teams recommended that several offices establish management controls to ensure that they properly address all errors identified by the STAR unit. Effective April 1, 2003, VBA established a new tracking system that required regional offices to report to headquarters on actions taken to address all errors identified by the STAR unit. In addition to not correcting all errors, several of the regional offices visited by the survey teams were not making adequate use of STAR errors to identify trends that could be used to provide feedback and training to claims processing staff. Officials at regional offices we visited commented that this information is useful for identifying trends for feedback and training purposes. For example, in fiscal year 2002, the results of STAR reviews helped to alert management of the Boston Regional Office that many of its claims processors were having difficulties with the new VCAA legislation, permitting management to provide special training for the Boston staff. The agency survey teams recommended that 7 of the regional offices reviewed between October 2001 and December 2002 take steps to use STAR reviews for feedback and training. In addition to regional office-level accuracy standards, VBA has also established quantifiable accuracy standards for its experienced claims processing employees, as part of these employees' annual performance evaluations.[Footnote 12] The accuracy standards for experienced rating specialists require that they correctly decide 85 percent of 60 randomly selected rating decisions per year (5 decisions per month). [Footnote 13] VBA is developing an agencywide review process, the Systematic Individual Performance Assessment (SIPA), to assess compliance with the individual accuracy standards. VBA developed a uniform checklist, similar to the STAR checklist for reviewing rating decisions, which offices could use to assess rating specialists' accuracy. Unlike the STAR system, reviews are conducted before decisions are finalized. In April 2003, VBA surveyed regional offices to determine the status of their efforts to measure individual accuracy and assess the staff resources needed to implement SIPA VBA-wide. Individual reviews were being conducted by supervisors, such as team coaches, and nonsupervisory staff, such as Decision Review Officers, who are responsible for handling appeals. At the time of our review, VBA was in the process of deciding who would perform the reviews. In its efforts to develop SIPA as a consistent review process, VBA is considering issues such as ensuring that consistent feedback is provided to the employees and all regional offices are using the same review checklists. Conclusions: While VBA's emphasis on production succeeded in increasing the number of claims decisions made during fiscal year 2002, the accuracy of those decisions declined. Because VBA did not report comparable accuracy rates for fiscal years 2001 and 2002 in its performance and accountability report, Members of Congress, Department of Veterans Affairs management, and claimants did not know of this decline in accuracy. The lack of accurate performance data hampers stakeholders' ability to monitor VBA's performance in serving the nation's veterans. In addition, VBA could better use the accuracy data it has to hold its regional offices more accountable for accuracy. While VBA has made progress in measuring regional office accuracy, it has not made full use of the results to reward high performing offices or to improve poorly performing offices. VBA has made significant progress in getting its compensation and pension claims workload under control to improve timeliness for veterans waiting for decisions on their claims. However, VBA's accuracy has declined. Unless VBA balances its emphasis on producing more and faster decisions with an increased emphasis on accuracy, it will be difficult to ensure that veterans are getting the benefits to which they are entitled in a timely manner. Recommendations: We recommend that the Secretary of Veterans Affairs direct the Under Secretary for Benefits to: * report the accuracy of VBA disability compensation and pension claims decisions to the Congress and other stakeholders in a manner that allows for valid comparisons of accuracy across fiscal years, and; * better hold regional offices accountable for the accuracy of their claims decisions, by increasing the use of its regional office accuracy data, while at the same time maintaining an appropriate emphasis on production. Agency Comments and Our Response: In its written comments on a draft of this report (see appendix 1), VBA concurred with our recommendations. However, VBA suggested that more effective use of STAR had been made to improve performance than our report concludes. We did note that VBA used STAR data to improve accuracy. Specifically, we noted in our report that VCAA-related errors declined to about one-third of all errors after VBA identified the errors through STAR and required retraining. We support VBA's continued efforts to emphasize accuracy and timeliness as critical components of quality. We will send copies of this report to the Secretary of Veterans Affairs, appropriate congressional committees, and other interested parties. We will also make copies of this report available to others on request. The report will also be available at no charge on GAO's Web site at [hyperlink, http://www.gao.gov]. If you or your staffs have any questions regarding this report, please call me at (202) 512-7215 or Irene Chu, Assistant Director, at (202) 512-7102. In addition to those named, Susan Bernstein, Kristine Braaten, Kevin Jackson, Joseph Natalicchio, Martin Scire, and Greg Whitney made key contributions to this report. Signed by: Cynthia A. Bascetta: Director, Education, Workforce, and Income Security Issues: [End of section] Appendix I: Comments from the Veterans Benefits Administration: The Under Secretary Of Veterans Affairs For Benefits: Washington, D.C. 20420: September 24, 2003: Ms. Cynthia A. Bascetta: Director, Education, Workforce, and Income Security Issues: U.S. General Accounting Office: 441 G Street, NW: Washington, DC 20548: Dear Ms. Bascetta: The Department of Veterans Affairs (VA) has reviewed your draft report, Veterans' Benefits: Improvements Needed in the Reporting and Use of Data on the Accuracy of Disability Claims Decisions (GAO-03- 1045). Your report includes two basic themes: 1) the Veterans Benefits Administration (VBA) reported accuracy in a manner that obscured true accuracy trends and 2) VBA failed to effectively use review results to improve regional office performance. The two recommendations included in the report reflect these two themes. While we believe the recommendations are reasonable and we will certainly implement those recommendations, we do not believe the report accurately reflects VBA's major efforts to improve service to veterans. We firmly believe that quality is not limited to technical accuracy, but must also include providing veterans with a timely response to their claims. We approached the challenge of reducing the backlogs and minimizing the time claimants must wait for a decision on their claims with a sense of urgency and determination, and we have made extraordinary progress. At the same time, we have maintained the same high and uncompromising standards for compliance with statute, regulation, and policy guidelines that have existed since the creation of the Systematic Technical Accuracy Review (STAR) quality assurance program in 1999. In compliance with prior GAO recommendations, we strengthened the STAR program effective the beginning of FY 2002, which was one of the two years under review. Measurement of regional office accuracy was reassigned from a local to a national review to ensure independence of the review process and to enhance our ability to hold regional offices accountable for quality service delivery. We also changed the STAR report format in response to a recommendation by the VA Claims Processing Task Force that accuracy be redefined to include errors that affect entitlement, amount of benefit allowed, and the effective date of the award of benefits. This change was intended to describe claims processing accuracy more consistently with stakeholder perceptions of what our accuracy statistics indicate — i.e., was the decision correct and is the claimant receiving the correct benefit amount from the correct date. We continue to set aggressive goals for quality improvement; the benefit entitlement accuracy goal for FY 2003 is 88%, increasing to 90% in FY 2004 and 92% in FY 2005. It is important to note that while STAR reports were revised, all elements previously subject to review remain part of the review process. Decision documentation and notification requirements, also critical elements of fairness and good customer service, are separately measured and reported. Your report concludes that VBA has not made effective use of the STAR data to improve performance and infers that we failed to take steps to improve accuracy when deficiencies were apparent. We believe that our STAR results from FY 2002 actually provide strong evidence to refute that conclusion. Following enactment of the Veterans Claims Assistance Act (VCAA) in FY 2001, VBA established implementing regulations and guidance for the new law, and then incorporated review of the VCAA requirements in the STAR program effective the beginning of FY 2002. Through effective monitoring and assessment of the 1st quarter results, we identified deficiencies in regional offices' compliance with the new VCAA notification and development requirements. These deficiencies were the primary cause of a dramatic 15-point decline in the accuracy level — from 89% in the 4th quarter of FY 2001 to 74% in the 1st quarter of FY 2002. We immediately and successfully intervened to correct the problem, requiring mandatory VCAA retraining of all regional office employees involved in claims processing. Certification of completion of this VCAA training by every regional office director was required. Our Area Directors provided aggressive oversight of the quality levels at the individual regional offices and focused their support on the offices having the most difficulty in bringing up their quality levels. As a result of our management actions, benefit entitlement accuracy improved 7 points in the 2nd quarter, and our 4th quarter accuracy was 10 points above the 1st quarter figure. Even though there has been steady improvement in benefit entitlement accuracy over the past year and a half, we have not yet reached our goal. Further, we note there has been a small decline in decision documentation and notification accuracy. Accordingly, quarterly regional office reports of completed corrective actions for every error recorded on STAR review are now required. Site visit teams monitor the information provided in these reports against claim folder reviews to validate report accuracy. Area Directors will continue their oversight of all of our quality improvement efforts, using all of the information available to them to increase accountability for performance. We believe that our STAR program is an effective tool for measuring quality, identifying areas for improvement, and acting as an agent for change. However, we also believe that periodically it may be appropriate to modify the program to better meet program and stakeholder requirements. While we did identify in VA's Annual Performance and Accountability Report that our STAR reports had changed, describe the nature of the change, and report both the benefit entitlement and decision documentation/notification results, we did not recalculate prior reports using the new report criteria to facilitate trend analysis. GAO's recommendation in this area is appropriate and we will gladly comply. VBA will also continue to increase accountability at the regional office level by requiring formal quality improvement plans from all offices failing to meet annual target levels for quality each fiscal year. Thank you for the opportunity to comment on your draft report. Sincerely, Signed by: Daniel L. Cooper: [End of section] Footnotes: [1] This report focuses on the accuracy of rating decisions. Rating decisions are primarily decisions on original claims for compensation and pension benefits and reopened claims. For example, veterans may file reopened claims if they believe that their service-connected conditions have worsened. [2] Except for Pittsburgh, we selected the regional offices we visited based on a combination of their fiscal year 2002 rating accuracy and the number of rating decisions they produced. We visited the Pittsburgh Regional Office to observe how VBA conducts an inspection of a regional office's compensation and pension operations as well as to discuss factors affecting accuracy. [3] The margin of error for rating accuracy VBA-wide for fiscal years 2001 and 2002 was about 1 percentage point at a 95-percent level of confidence. [4] Veterans who are 65 years or older do not have to be permanently and totally disabled to become eligible for pension benefits, as long as they meet the other requirements for income and military service. VBA also pays pensions to surviving spouses and unmarried children of deceased wartime veterans. [5] For more information, see U.S. General Accounting Office, Veterans' Benefits Claims: Further Improvements Needed in Claims- Processing Accuracy, [hyperlink, http://www.gao.gov/products/GAO/HEHS-99-35] (Washington, D.C.: March 1, 1999) and Veterans' Benefits: Quality Assurance for Disability Claims Processing, [hyperlink, http://www.gao.gov/products/GAO-01-930R] (Washington, D.C.: August 23, 2001). [6] The fiscal year 2002 accuracy score includes a specific question on proper VCAA pre-decision notification that VBA did not include in its fiscal year 2001 quality reviews. [7] Due to the time lag involved between receiving claims files and conducting the accuracy reviews of disability compensation and pension decisions as well as deadlines for incorporating data into its annual accountability report, reported accuracy rates can differ from final rates for the reported year. VBA's final rating accuracy rate for fiscal year 2002 was 81 percent. [8] VBA established specific monthly production targets for its regional offices in December 2001. For more information on these targets see U.S. General Accounting Office, Veterans' Benefits: Despite Recent Improvements, Meeting Claims Processing Goals Will Be Challenging, [hyperlink, http://www.gao.gov/products/GAO-02-645T] (Washington, D.C.: April 26, 2002). [9] For more information on VCAA implementation, see U.S. General Accounting Office, Veterans' Benefits: VBA's Efforts to Implement the Veterans Claims Assistance Act Need Further Monitoring, [hyperlink, http://www.gao.gov/products/GAO-02-412] (Washington, D.C.: July 1, 2002). [10] For more information, see U.S. General Accounting Office, Veterans Benefits Administration: Better Collection and Analysis of Attrition Data Needed to Enhance Workforce Planning, GAO-03-491 (Washington, D.C.: April 28, 2003). [11] It could not be determined if the remaining offices met the accuracy goal because of the margins of error associated with their accuracy scores. In fiscal year 2002, the margins of error for rating accuracy for each regional office ranged from 4 to 9 percentage points at the 95-percent level of confidence. [12] In general, inexperienced claims processing employees have all of their work reviewed. [13] These standards, according to VBA officials, are intended to be floors; individual regional offices can set higher standards. [End of section] GAO’s Mission: The General Accounting Office, the investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through the Internet. GAO’s Web site [hyperlink, http://www.gao.gov] contains abstracts and full text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents using key words and phrases. You can print these documents in their entirety, including charts and other graphics. Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site daily. The list contains links to the full-text document files. To have GAO e-mail this list to you every afternoon, go to [hyperlink, http://www.gao.gov] and select “Subscribe to daily E-mail alert for newly released products” under the GAO Reports Order GAO Products heading. Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. General Accounting Office: 441 G Street NW, Room LM: Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000: TDD: (202) 512-2537: Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Public Affairs: Jeff Nelligan, managing director, NelliganJ@gao.gov: (202) 512-4800: U.S. General Accounting Office: 441 G Street NW, Room 7149: Washington, D.C. 20548: