Veterans' Disability Benefits:

Improvements Could Further Enhance Quality Assurance Efforts

GAO-15-50: Published: Nov 19, 2014. Publicly Released: Nov 19, 2014.

Additional Materials:

Contact:

Daniel Bertoni
(202) 512-7215
bertonid@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

What GAO Found

The Veterans Benefits Administration (VBA)—within the Department of Veterans Affairs—measures and reports the accuracy of its disability compensation claim decisions in two ways: (1) by claim and (2) by disabling condition, though its approach has limitations. When calculating accuracy rates for either measure through its Systematic Technical Accuracy Review (STAR), VBA does not always follow generally accepted statistical practices, resulting in imprecise performance information. For example, VBA does not adjust its accuracy estimates to reflect that it samples the same number of claims for review from each regional office—despite their varying workloads—and thus produces imprecise estimates of national and regional accuracy. Further, VBA reviews about 39 percent (over 5,000) more claims nationwide than is necessary to achieve its desired precision in reported accuracy rates, thereby diverting limited resources from other important quality assurance activities, such as targeted reviews of error-prone cases. In addition to issues with its statistical practices, VBA's process for selecting claims for STAR review creates an underrepresentation of claims that are moved between regional offices, which may inflate accuracy estimates because these claims have had historically lower accuracy rates. Finally, VBA has not clearly explained in public reports the differences in how its two accuracy measures are calculated or their associated limitations, as suggested by best practices for federal performance reporting.

VBA has taken steps to enhance and coordinate its other quality assurance activities, but GAO found shortcomings in how VBA is implementing and evaluating these activities. To improve local accuracy, VBA created regional office quality review teams (QRTs) with staff dedicated primarily to performing local accuracy reviews. QRTs assess individual claims processor performance and conduct special reviews to forestall certain types of errors. In addition, VBA began using questionnaires for assessing decision-making consistency, which are more efficient to administer than VBA's prior approach to conducting consistency studies. VBA also coordinates quality assurance efforts by disseminating national accuracy and consistency results, trends, and related guidance to regional offices for use in training claims processors. Further, VBA uses STAR results to inform other quality assurance activities, such as focusing certain QRT reviews on commonly made errors. However, GAO identified implementation shortcomings that may detract from the effectiveness of VBA's quality assurance activities. For example, contrary to accepted practices for ensuring the clarity and validity of questionnaires, VBA did not pre-test its consistency questionnaires to ensure the clarity of questions or validity of the expected results, although VBA officials indicated that they plan to do so for future questionnaires. In contrast with federal internal control standards that call for capturing and distributing information in a form that allows people to efficiently perform their duties, staff in the four regional offices that we visited had trouble finding the guidance they needed to do their work, which could affect the accuracy as well as the speed with which staff decide claims. Federal standards also call for knowing the value of efforts such as quality assurance activities and monitoring their performance over time; however, VBA has not evaluated the effect of its special QRT reviews or certain consistency studies on improving targeted accuracy rates, and lacks clear plans to do so.

Why GAO Did This Study

With a backlog of disability compensation claims, VBA faces difficulties in improving the accuracy and consistency of the claim decisions made by staff in its 57 regional offices. To help achieve its goal of 98 percent accuracy by fiscal year 2015, VBA recently implemented a new way of measuring accuracy and changed several quality assurance activities to assess the accuracy and consistency of decisions and to provide feedback and training to claims processors. GAO was asked to examine VBA's quality assurance activities.

This report evaluates (1) the extent to which VBA effectively measures and reports the accuracy of its disability compensation claim decisions and (2) whether VBA's other quality assurance activities are coordinated and effective. GAO analyzed VBA claims and STAR accuracy data from fiscal year 2013 (the most recent fiscal year for which complete data are available); reviewed relevant federal laws, VBA guidance, and other documents relevant to quality assurance activities; and interviewed VBA staff from headquarters and four VBA regional offices (selected to achieve variety in geography, workload, and accuracy rates), as well as veteran service organization officials.

What GAO Recommends

GAO is making eight recommendations to VA to improve its measurement and reporting of accuracy, review the multiple sources of policy guidance available to claims processors, enhance local data systems, and evaluate the effectiveness of quality assurance activities. VA concurred with all of GAO's recommendations.

For more information, contact Daniel Bertoni at (202) 512-7215 or bertonid@gao.gov.

Recommendations for Executive Action

  1. Status: Closed - Implemented

    Comments: In February 2015, VBA changed its sample selection methodology to ensure that redistributed claims are not underrepresented by selecting claims on the basis of regional office completion regardless of whether the claim was redistributed.

    Recommendation: To help improve the quality of VBA's disability compensation claim decisions, the Secretary of Veterans Affairs should direct the Under Secretary for Benefits to take steps to ensure that redistributed claims and those moved between regional offices are not underrepresented in the STAR sample.

    Agency Affected: Department of Veterans Affairs

  2. Status: Closed - Implemented

    Comments: According to VA's December 2015 update, VBA has consolidated its policy and procedural guidance into a single source available to regional office staff via the agency's intranet. Officials stated that the M21-1 Adjudication Procedures Manual has improved search capabilities and is the authoritative source for claims processing guidance. The manual's content is also available to external stakeholders through a separate link, KnowVA (www.knowva.ebenefits.va.gov), and will be updated simultaneously.

    Recommendation: To help improve the quality of VBA's disability compensation claim decisions, the Secretary of Veterans Affairs should direct the Under Secretary for Benefits to review the multiple sources of policy guidance VBA provides to determine ways to consolidate them or otherwise improve their availability and accessibility for use by staff in regional offices.

    Agency Affected: Department of Veterans Affairs

  3. Status: Closed - Implemented

    Comments: VBA has created an abstract that describes its sample design, how it calculates accuracy estimates, and assessment criteria for claim-based and issue-based errors. This abstract is publicly available on VBA's performance website. VBA reported in December 2015 that this abstract now includes an explanation of why the issue-based accuracy estimate is higher than the claim-based accuracy estimate, to help users make meaningful comparisons between the two estimates.

    Recommendation: To help improve the quality of VBA's disability compensation claim decisions, the Secretary of Veterans Affairs should direct the Under Secretary for Benefits to increase transparency in explaining how the claim-based and issue-based accuracy rates are calculated as well as their key limitations when publicly reporting these metrics.

    Agency Affected: Department of Veterans Affairs

  4. Status: Open

    Comments: In August 2017, VA reported that VBA had submitted business requirements to add new functionality in VA's electronic claims processing system, the Veterans Benefit Management System (VBMS), that would allow the agency to identify deficiencies during the claims process. If an error is detected, the claim would be paused to allow an in-process review (IPR), conducted by quality review teams, to occur. However, VA reported that these new business requirements will not be implemented due to limited VA Office of Information Technology (OIT) funding available for OIT updates in fiscal year 2018 and beyond. This recommendation will remain open until November 2018, in case circumstances or funding change.

    Recommendation: To help improve the quality of VBA's disability compensation claim decisions, the Secretary of Veterans Affairs should direct the Under Secretary for Benefits to take steps to ensure that any future upgrades to local data systems allow QRTs to pause the claims process when errors are detected and enable QRTs to better track error trends.

    Agency Affected: Department of Veterans Affairs

  5. Status: Open

    Comments: As of August 2017, VA reported that VBA has developed a new quality assurance database that will capture data from all types of quality reviews at various stages of the claims process, to include local regional office individual quality reviews, in-process reviews, national STAR reviews and consistency studies. VBA stated that this new database will support increased data analysis capabilities and allow VBA to evaluate the effectiveness of quality assurance activities through improved and vigorous error trend analysis. To close this recommendation, the agency will need to provide documentation of the comprehensive quality assurance data being captured, and how it is being used to either evaluate QA activities or make specific improvements with potential for improving accuracy.

    Recommendation: To help improve the quality of VBA's disability compensation claim decisions, the Secretary of Veterans Affairs should direct the Under Secretary for Benefits to take additional steps to evaluate the effectiveness of quality assurance activities to identify opportunities to improve or better target these activities.

    Agency Affected: Department of Veterans Affairs

  6. Status: Closed - Implemented

    Comments: VA reported in July 2015 that VBA finalized a statistical program to reflect its approach to selecting claims by regional office, which vary in size. VBA now weights its accuracy estimates to account for this.

    Recommendation: To help improve the quality of VBA's disability compensation claim decisions, the Secretary of Veterans Affairs should direct the Under Secretary for Benefits to leverage appropriate expertise to help VBA weigh its accuracy estimates to reflect the sample design for reviewed claims.

    Agency Affected: Department of Veterans Affairs

  7. Status: Closed - Implemented

    Comments: VA reported that VBA finalized a statistical program to calculate weighted claim-based accuracy estimates, and has modified its Monday Morning Workload Report (MMWR) to report margin of error associated with regional office and national claim-based accuracy estimates. In February 2016, VBA reported claim-based and issue-based accuracy estimates in VA?s 2017 President?s Budget Submission and in VA?s Annual Performance Plan and Report, and included margins of error for these estimates. In addition, in April 2016, VBA updated its Performance website with additional information about its accuracy estimates. The website now states that VBA?s quality assurance sample design is designed to yield estimates within a 5 percent margin of error and a confidence level of 95 percent.

    Recommendation: To help improve the quality of VBA's disability compensation claim decisions, the Secretary of Veterans Affairs should direct the Under Secretary for Benefits to leverage appropriate expertise to help VBA determine and report the confidence intervals associated with its reported accuracy estimates.

    Agency Affected: Department of Veterans Affairs

  8. Status: Closed - Implemented

    Comments: Effective February 2015, VBA changed its approach for calculating the regional office sample size for its Systematic Technical Accuracy Review (STAR).

    Recommendation: To help improve the quality of VBA's disability compensation claim decisions, the Secretary of Veterans Affairs should direct the Under Secretary for Benefits to leverage appropriate expertise to help VBA re-examine its approach to calculating the regional office sample size for STAR.

    Agency Affected: Department of Veterans Affairs

 

Explore the full database of GAO's Open Recommendations »

Nov 8, 2017

Oct 19, 2017

Oct 12, 2017

Sep 29, 2017

Sep 22, 2017

Sep 19, 2017

Sep 13, 2017

Sep 12, 2017

Sep 6, 2017

Looking for more? Browse all our products here