Social Security Administration:

More Effort Needed to Assess Consistency of Disability Decisions

GAO-04-656: Published: Jul 2, 2004. Publicly Released: Jul 2, 2004.

Additional Materials:

Contact:

Daniel Bertoni
(202) 512-9889
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

Each year, about 2.5 million people file claims with the Social Security Administration (SSA) for disability benefits. If the claim is denied at the initial level, the claimant may appeal to the hearings level. The hearings level has allowed more than half of all appealed claims, an allowance rate that has raised concerns about the consistency of decisions made at the two levels. To help ensure consistency, SSA began a "process unification" initiative in 1994 and recently announced a new proposal to strengthen its disability programs. This report examines (1) the status of SSA's process unification initiative, (2) SSA's assessments of possible inconsistencies in decisions between adjudication levels, and (3) whether SSA's new proposal incorporates changes to improve consistency in decisions between adjudication levels.

SSA has only partially implemented its process unification initiative. Although the agency initially made improvements in its policies and training intended to address inconsistency in decisions made at the two adjudication levels, it has not continued to actively pursue these efforts. Further, as part of this initiative, the agency implemented a review of hearings level decisions to identify ways to improve training and policies, but no new improvements were made as a result of the review. Finally, the agency began tests of two process changes intended to improve the consistency of decision making between the two adjudication levels. One test, which is ongoing, was not well designed and therefore will not provide conclusive results. The other test was abandoned because of implementation difficulties. SSA's assessments have not provided a clear understanding of the extent and causes of possible inconsistencies in decisions between adjudication levels. The two measures SSA uses to monitor inconsistency of decisions have weaknesses, such as not accounting for the many factors that can affect decision outcomes, and therefore do not provide a true picture of the changes in consistency. Furthermore, SSA has not sufficiently assessed the causes of possible inconsistency. For example, SSA conducted an analysis in 1994 that identified potential areas of inconsistency, but it did not employ more sophisticated techniques--such as multivariate analyses, followed by in-depth case studies--that would allow the agency to identify and address the key areas and leading causes of possible inconsistency. SSA has yet to repeat or expand upon this 10-year-old study. SSA's new proposal incorporates changes intended to improve consistency in decisions between levels. However, challenges may hinder its implementation. Most stakeholder groups for adjudicators and claimant representatives told us that a number of aspects of the proposal hold promise for improving consistency. These included one change, being tested as part of the process unification initiative, that requires state adjudicators to more fully develop and document their decisions, as well as several new changes, such as providing both adjudication levels with equal access to medical expertise. However, stakeholder groups also told us that insufficient resources and other obstacles might hinder the implementation of some changes. Adding to uncertainties about the proposal's overall success is its dependence on a new electronic folder system that would allow cases to be easily accessed by various adjudicators across the country. However, this technically complex project has not been fully tested.

Status Legend:

More Info
  • Review Pending-GAO has not yet assessed implementation status.
  • Open-Actions to satisfy the intent of the recommendation have not been taken or are being planned, or actions that partially satisfy the intent of the recommendation have been taken.
  • Closed-implemented-Actions that satisfy the intent of the recommendation have been taken.
  • Closed-not implemented-While the intent of the recommendation has not been satisfied, time or circumstances have rendered the recommendation invalid.
    • Review Pending
    • Open
    • Closed - implemented
    • Closed - not implemented

    Recommendations for Executive Action

    Recommendation: To move successfully forward with agency efforts to make more consistent decisions, including efforts incorporated in the Commissioner's proposal for an improved disability claims process and quality assurance system, SSA should quickly expand its assessment of the areas and causes of inconsistency in decisions between adjudication levels. In doing so, SSA should consider making near-term and cost-effective enhancements to its current approach for assessing the consistency of decisions, including publishing the methods and findings of all analyses, to keep internal and external stakeholders aware of the agency's efforts to assess consistency and demonstrate improvement over time.

    Agency Affected: Social Security Administration

    Status: Closed - Not Implemented

    Comments: As part of its new decision making approach, the Disability Service Improvement (DSI) initiative, the agency planned to collect, assess, and distribute information on the consistency of decisions between adjudication levels. SSA planned to have feedback loops, which would allow its decision makers to understand why their decisions were reversed. It also planned to implement a quality assurance review of the decision rationales, which would feed into monthly internal reports identifying problem areas to help management improve the quality and consistency of decision making. SSA also planned to have additional in-line and end-of-line reviews, but to date had not clarified how it would report the findings of these reviews. Also unclear was the extent to which the agency would keep external stakeholders informed about its progress in addressing consistency in decisions. However, the agency used a proactive and collaborative approach to share information and included internal and external stakeholders in its development and implementation of its new decision making process, which began implementation in one region in August 2006. In 2007, the agency's Office of Quality Performance developed an intranet site to share its quality review analysis with internal stakeholders and planned to provide information to state decision makers via various management meetings. The agency also had plans underway to create a new in-depth, multi-level case analysis to identify inconsistencies between adjudication levels, but had not made it clear how the results of such analysis would be shared with internal and external stakeholders. The DSI initiative was discontinued under the new Commission and SSA has not indicated to us that it has distributed to internal or external stakeholders new information on reasons for inconsistent disability decisions that it may have gleaned from the DSI effort.

    Recommendation: To move successfully forward with agency efforts to make more consistent decisions, including efforts incorporated in the Commissioner's proposal for an improved disability claims process and quality assurance system, SSA should quickly expand its assessment of the areas and causes of inconsistency in decisions between adjudication levels. In doing so, SSA should consider making near-term and cost-effective enhancements to its current approach for assessing the consistency of decisions, including having adjudicators and reviewers from each level study cases in depth to help pinpoint the causes of inconsistency, once potential areas of inconsistency between levels are identified.

    Agency Affected: Social Security Administration

    Status: Closed - Not Implemented

    Comments: SSA tested a new approach to decision making that included several steps that were intended to allow adjudicators and reviewers an opportunity to more fully evaluate disability cases and understand the causes of inconsistent decisions. Under the new approach, referred to as the Disability Service Improvement (DSI) initiative, state and federal decision makers used a standard decision form to document the rationale for a decision and the reversal of a prior decision. SSA also planned to provide feedback loops to allow prior decision makers to learn why their decisions were reversed. In addition, its Office of Quality Performance planned to collect and analyze the content of the forms to identify problem areas and provide monthly reports to help managers improve the quality and consistency of decision making. Furthermore, the Decision Review Board envisioned under DSI planned to review ALJ decisions to identify trends and causes of errors and inconsistencies. SSA tested its new approach in 2006 in its Boston Region and planned to evaluate the results prior to rolling out changes to its other regions. SSA's Office of Quality Performance also planned to conduct some quality review of federal decisions and evaluate the data for feedback to both federal and state levels. In addition, the agency had new plans to conduct in-depth multi-level cases analysis to identify decision inconsistencies. However, as of September 2008, the agency has not begun conducting any new multi-level analysis or provided any information on decisional inconsistency identified from the DSI initiative, which was never fully implemented and has been discontinued.

    Recommendation: To move successfully forward with agency efforts to make more consistent decisions, including efforts incorporated in the Commissioner's proposal for an improved disability claims process and quality assurance system, SSA should quickly expand its assessment of the areas and causes of inconsistency in decisions between adjudication levels. In doing so, SSA should consider making near-term and cost-effective enhancements to its current approach for assessing the consistency of decisions, including expanding the biennial case review by requiring disability examiners and medical consultants to review the hearing tapes to ensure that reviewers have the complete case before them (including the types and sources of testimonial evidence provided during the hearings) when evaluating the ALJs' decisions.

    Agency Affected: Social Security Administration

    Status: Closed - Not Implemented

    Comments: Under the prior SSA Commissioner, a new approach to decision making, the Disability Service Improvement (DSI) initiative, was tested. Under DSI, a new Decision Review Board (DRB) would review a sample of Adjudicative Law Judges' (ALJ) decisions--a sample selected using a predictive model for error-prone cases. In reviewing each case, the DRB should have had the ALJ's rationales for his case decision. It was not clear if the DRB would also have the complete case file, including the ALJ's hearing tape, for review purposes. The DRB would have also used the available information to look for trends and causes of inconsistent and error-prone cases. In addition to the DRB, SSA told us that it had plans under way to conduct new in-depth multi-level case analyses to identify inconsistencies across adjudication levels. However, as of September 2008, the agency has not conducted any new multi-level analysis or provided any information on decisional inconsistency identified from the DSI initiative, which was never fully implemented and has been discontinued.

    Recommendation: To move successfully forward with agency efforts to make more consistent decisions, including efforts incorporated in the Commissioner's proposal for an improved disability claims process and quality assurance system, SSA should quickly expand its assessment of the areas and causes of inconsistency in decisions between adjudication levels. In doing so, SSA should consider making near-term and cost-effective enhancements to its current approach for assessing the consistency of decisions, including performing more sophisticated multivariate analysis on the biennial case review data in order to pinpoint the most significant case characteristics influencing allowance decisions and to distinguish factors that might be contributing either appropriately or inappropriately to allowance decisions.

    Agency Affected: Social Security Administration

    Status: Closed - Not Implemented

    Comments: In moving forward with the SSA Commissioner's new approach to disability decision making, the agency planned to develop, but has not implemented, a more sophisticated approach to analyzing the significant case characteristics that contribute to inconsistency in decision making between adjudication levels. For example, SSA planned to develop a sophisticated predictive model (using data mining techniques) to identify error-prone cases for review by the Decision Review Board envisioned under its Disability Service Improvement (DSI) initiative that would replace SSA's Appeals Council. During the early implementation in the Boston region, the DRB planned to review all Adjudicative Law Judge (ALJ) decisions to develop and validate the model as it looks for the trends and causes of error-prone decisions. To further support its new DSI initiative, SSA planned to change other aspects of its quality assurance system to promote the quality and consistency of decisions, such as the creation of a central quality assurance review to eliminate the potential for differences among regional reviews and the use of standardized electronic tools for decision writing with feedback loops and quality assurance analysis to identify and address inaccurate and inconsistent decisions. SSA has also told us that it had plans underway to conduct new in-depth multi-level case analyses to identify inconsistencies across adjudication levels. However, as of September 2008, SSA has not begun conducting any new multi-level analyses or provided any information on decisional inconsistency identified during DSI initiative, which was discontinued before being fully implemented.

    Recommendation: To move successfully forward with agency efforts to make more consistent decisions, including efforts incorporated in the Commissioner's proposal for an improved disability claims process and quality assurance system, SSA should quickly expand its assessment of the areas and causes of inconsistency in decisions between adjudication levels. In doing so, SSA should consider making near-term and cost-effective enhancements to its current approach for assessing the consistency of decisions, including reestablishing ongoing analyses of case characteristics as part of its biennial case review, in line with efforts undertaken for the review report published in 1994.

    Agency Affected: Social Security Administration

    Status: Closed - Not Implemented

    Comments: SSA no longer conducts the biennial case review, but planned to take several steps to expand its assessment of the areas and causes of inconsistency in decisions between the state and federal adjudication levels as it implemented major changes to its disability decision making process. SSA began implementing its new Disability Service Improvement (DSI) process in August 2006 in its Boston Region. The agency has evaluated the Boston results in preparation for a national rollout decision. Several of the DSI process changes, if adopted, could provide the agency with information on decision inconsistency in line with efforts undertaken as part of its biennial review. For example, state and federal decision makers plan to use standardized forms to consistently document the rationale for their decisions, addressing areas that have contributed to errors and inconsistency. The forms and feedback loops would have allowed decision makers to learn why their decisions were reversed or remanded. Once in full use, SSA's Office of Quality Performance (OQP) planned to collect and analyze the content of the forms to identify problem areas and provide monthly reports to help managers improve the quality and consistency of decision making. OQP has also performed analyses of decisions made under DSI at the new federal level to identify factors associated with the allowance of cases which were denied at the prior state level. SSA also indicated that the Disability Review Board envisioned under DSI would be in a good position to provide feedback regarding decisions made by adjudicators at the hearings and prior levels. Although the agency has discontinued DSI and the associated pilot, it told us that it had other plans underway to conduct in-depth multilevel case analyses to identify inconsistencies between adjudication levels. As of September 2008, the agency has not conducted any new multi-level analyses or provided any information or results on decisional inconsistency that may have resulted from the DSI pilot.

    Recommendation: To move successfully forward with agency efforts to make more consistent decisions, including efforts incorporated in the Commissioner's proposal for an improved disability claims process and quality assurance system, SSA should quickly expand its assessment of the areas and causes of inconsistency in decisions between adjudication levels. In doing so, SSA should consider making near-term and cost-effective enhancements to its current approach for assessing the consistency of decisions, including using the information from these improved analyses to develop a more focused and effective strategy for ensuring uniform application of SSA's guidance and to improve the consistency of decisions. To accomplish this, SSA should clarify guidance for making disability decisions and develop mandatory training for adjudicators on issues identified as contributing to inconsistency.

    Agency Affected: Social Security Administration

    Status: Closed - Not Implemented

    Comments: SSA planned to improve its collection of management information (MI) to monitor and improve its new decision-making approach, referred to as the Disability Service Improvement (DSI) initiative. As part of this effort, SSA consulted with internal stakeholders and incorporated their needs into its new MI strategy. SSA planned to use information from quality reviews and its electronic disability system, to capture information about decision making and enhance its guidance and training. SSA, however, has not provided any details on how it would accomplish this task. Furthermore, in preparing to implement its new approach, SSA provided training on how to use the new writing tools (along with other aspects of the new process) and planned to use on-line guidance for decision makers to help improve the quality and consistency of decisions. Furthermore, SSA's Office of Quality Performance planned to analyze information from the new on-line writing tools and provide feedback to both state and federal decision makers. However, as of September 2008, SSA has not provided any indication that efforts undertaken as part of DSI have provided any information on decisional inconsistency and has discontinued the DSI initiative.

    Jul 25, 2013

    May 16, 2013

    Aug 2, 2012

    Sep 30, 2011

    Apr 28, 2011

    Jul 29, 2010

    May 24, 2010

    Dec 3, 2009

    Oct 27, 2009

    Oct 14, 2009

    Looking for more? Browse all our products here