This is the accessible text file for GAO report number GAO-04-466 entitled 'Electronic Disability Claims Processing: SSA Needs to Address Risks Associated with Its Accelerated Systems Development Strategy' which was released on March 26, 2004. This text file was formatted by the U.S. General Accounting Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Report to the Chairman, Subcommittee on Social Security, Committee on Ways and Means, House of Representatives: March 2004: ELECTRONIC DISABILITY CLAIMS PROCESSING: SSA Needs to Address Risks Associated with Its Accelerated Systems Development Strategy: [Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-04-466]: GAO Highlights: Highlights of GAO-04-466, a report to the Chairman, Subcommittee on Social Security, Committee on Ways and Means, House of Representatives Why GAO Did This Study: The Social Security Administration’s (SSA) AeDib initiative is designed to provide SSA with a more efficient, paperless system that will enable its disability components to electronically view and share claims data and process claims electronically. Yet previous GAO reviews found that SSA’s accelerated strategy to develop AeDib involved risks that could threaten a complete and successful transition to this capability. At the Subcommittee’s request, GAO reviewed AeDib to assess (1) SSA’s progress and strategy, (2) the adequacy of measures taken to avoid software development problems similar to those encountered in SSA’s previous efforts, (3) the adequacy of cost-benefit analyses, and (4) SSA’s consultation with stakeholders. What GAO Found: SSA is continuing its work on the AeDib initiative and is in various stages of completing its electronic disability system; however, its accelerated strategy continues to involve risks. Specifically, GAO found that the agency is relying on limited pilot testing to help guide business and technical decisions and ensure that technology supporting the electronic disability system will work as intended (see table). Further, it is beginning its national rollout without ensuring that all critical problems identified in the pilot testing have been resolved and without conducting testing adequate to evaluate the performance of all system components collectively. Without resolution of critical problems and full testing, SSA cannot be assured that interrelated components will work together successfully. Pilot Tests are Limited in Scope, Involving Few Examiners: State/pilot start date: North Carolina/July 2003; Number/percentage of examiners participating: 21 of 254/8 percent. State/pilot start date: Illinois/September 2003; Number/percentage of examiners participating: 15 of 250/6 percent. State/pilot start date: California/October 2003; Number/percentage of examiners participating: 2 of 620/0.3 percent. Source: GAO analysis of SSA data. [End of table] While SSA has procedures to guide its software development, it could provide no evidence that it was consistently applying them in this case. In addition, while SSA has identified AeDib system and security risks, it has not finalized mitigation strategies. Without these measures being in place, SSA stands at greater vulnerability to circumstances that could impede project success. The agency also has not validated its analysis to ensure the reasonableness of estimated AeDib costs and benefits. While indicating that it would use pilot test results to validate cost-benefit estimates, officials have not indicated when this will be accomplished. This leaves SSA without a validated cost-benefit analysis, and the assurance that its AeDib cost estimates are reliable and that anticipated benefits will therefore be realized. Finally, SSA reports that it has increased its communications with AeDib stakeholders and users; however, state officials dealing with disability determinations have varying perspectives. A national organization representing these state officials continues to voice concerns about SSA’s approach. And while the Commissioner states that SSA is consulting with stakeholders and the medical community, the agency has not articulated a comprehensive plan for ensuring that the concerns of this population are addressed. What GAO Recommends: GAO recommends that before proceeding with national rollout of AeDib, the Commissioner of Social Security, among other steps, ensure that critical problems have been resolved and full testing completed, expedite completion of risk-mitigation strategies, and validate cost- benefit estimates. In commenting on a draft of this report, SSA stated that it would conduct studies to help validate AeDib cost assumptions but disagreed with GAO’s other recommendations. In GAO’s view, it is essential that SSA fully address all recommendations. www.gao.gov/cgi-bin/getrpt?GAO-04-466. To view the full product, including the scope and methodology, click on the link above. For more information, contact Linda D. Koontz at (202) 512-6240 or koontzl@gao.gov. [End of section] Contents: Letter: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendixes: Appendix I: GAO's January 30, 2004, Briefing: Appendix II: Comments from the Social Security Administration: Letter March 26, 2004: The Honorable E. Clay Shaw, Jr. Chairman, Subcommittee on Social Security: Committee on Ways and Means: House of Representatives: Dear Mr. Chairman: As you know, the Social Security Administration (SSA) is currently pursuing a major initiative to establish an electronic disability claims processing capability. With this initiative--known as AeDib--SSA aims to develop and implement a more efficient, paperless system that will enable its disability processing components to electronically view and share claims information, including large volumes of medical images, files, and other documents currently maintained in paper folders. SSA's pursuit of an electronic disability process represents a positive and necessary step toward more efficient delivery of benefits payments and services to an increasing beneficiary population. However, as we have previously reported,[Footnote 1] the agency's accelerated strategy to develop the electronic disability system involves risks that could pose a threat to the complete and successful transition to this capability. At your request, we have been reviewing SSA's continued actions on the AeDib initiative. Specifically, our objectives were to: * assess SSA's progress toward and strategy for achieving AeDib, * determine the adequacy of measures SSA is taking with AeDib to avoid the kinds of software development problems that it encountered with its prior disability system initiative and to identify and mitigate risks to a successful development, * assess the adequacy of SSA's analysis of AeDib costs and benefits, and: * assess SSA's consultation with and support from key stakeholders. To address these objectives, we analyzed project management and technical documentation describing SSA's plans, strategies, and progress related to developing and implementing the electronic disability system. We identified and evaluated measures--including institutional software capability mechanisms--that the agency had taken to guide its development of the AeDib software and mitigate program and project risks. We also assessed the adequacy of SSA's analyses of AeDib costs and benefits, including evaluating them against federal guidance and industry best practices and analyzing the impact of critical pilot evaluation data on costs and benefits. In addition, we analyzed SSA's plans and actions for consulting with and identifying and resolving issues of key disability stakeholders (i.e., state Disability Determination Services [DDS] offices, medical providers, and SSA field offices). We performed our work in accordance with generally accepted government auditing standards, from August 2003 through January 2004. On January 30, 2004, we provided your staff with a briefing on the results of our study. The slides from that briefing are included as appendix I to this report. The purpose of this report is to provide the published briefing slides to you and to officially transmit our recommendations to the Commissioner of Social Security. In summary, our briefing made four main points: * While the AeDib initiative is important to achieving more efficient delivery of disability payments, SSA's accelerated strategy continues to involve risks that threaten its ultimate success. SSA is relying on limited pilot testing to help guide business and technical decisions and ensure that technology supporting the electronic disability folder will work as intended. Further, the agency is beginning its national rollout without ensuring that all critical problems identified during the pilot tests have been resolved and without conducting end-to-end tests to evaluate the functionality and performance of all electronic disability system components collectively. SSA has maintained that its pilot tests of the components would be sufficient to evaluate the system; however, without end-to-end testing, SSA lacks assurance that the interrelated components will work together successfully. * While SSA has established processes and procedures to guide its software development activities, such as project management plans, it could not provide evidence that it was consistently applying these procedures to AeDib. In addition, while SSA has identified AeDib system and security risks, it has not finalized mitigation strategies. Without concurrence on software validation and system certifications, or risk mitigation strategies, SSA lacks assurance that its system will be acceptable to end users and it will not be positioned to effectively prevent circumstances that could impede project success. * SSA has not validated its analysis to ensure the reasonableness of estimated AeDib costs and benefits. The agency stated that it would use pilot test results to validate its cost-benefit estimates; however, its officials have not indicated when this will be accomplished. Without a validated cost-benefit analysis, SSA lacks assurance that its AeDib cost estimates are reliable and that anticipated benefits will be realized. * While SSA has stated that it has increased its communications with AeDib stakeholders and users and instituted mechanisms to facilitate its communications, state DDSs have varying views regarding their involvement. Specifically, while state disability offices participating in the AeDib pilots agree that communications have improved, a national organization representing these offices continues to have concerns about SSA's approach. In addition, although the Commissioner has stated that SSA is consulting with stakeholders and the medical community, SSA has not articulated a comprehensive plan for ensuring that all of these concerns are addressed. In light of the risks associated with the AeDib initiative, SSA's successful transition to an electronic disability claims process could be jeopardized unless it addresses the weaknesses described above. Recommendations for Executive Action: To reduce the risks associated with SSA's strategy to develop an electronic disability claims processing system, we recommend that the Commissioner of Social Security, before continuing with the AeDib national rollout, * ensure that all critical problems identified in pilot testing of the electronic disability system components are resolved and that end-to- end testing of the interrelated systems is performed; * ensure that users have approved the software developed and that systems have been certified for production; * establish a revised time frame for and expedite actions toward finalizing AeDib risk mitigation strategies; * validate all AeDib cost and benefit estimates; and: * implement a communications plan that clearly and comprehensively conveys SSA's approach for effectively addressing disability stakeholders' and users' concerns and ensuring their full involvement in the AeDib initiative. Agency Comments and Our Evaluation: In providing written comments on a draft of this report (reprinted in app. II), the Commissioner of Social Security reiterated the department's general disagreement with the findings discussed in our briefing slides. The Commissioner added that, in considering our recommendations, the agency had not been compelled to change its direction on the AeDib initiative. In particular, SSA continued to disagree with the need for end-to-end testing, stating that to perform such testing would delay the agency's ability to realize benefits from this initiative. SSA estimated that following our recommendation would have delayed the project, and the resulting benefits, by at least 3 years. Given the technological complexity of the AeDib project, we remain convinced that end-to-end testing is essential to ensuring that the multiple interrelated components of the electronic disability system will operate as intended. As our report noted, at the time that SSA began national implementation of the electronic disability system, it had conducted only limited pilot testing of the Document Management Architecture that is fundamental to the electronic disability folder. It also had not fully evaluated the results of its pilot tests or ensured that all critical problems were resolved. Further, we disagree with SSA's contention that end-to-end testing would delay the AeDib project by 3 years. In referring to this time frame, SSA has emphasized the processing of its most comprehensive cases--those in which individuals pursue their disability claims through all levels of agency appeal-- which reportedly take an average of 1,153 days to reach a final decision. However, not every case that SSA processes requires this level of appeal; thus, some cases can be processed in less time. In addition, SSA has reported that, due to backlogs, cases that go through all levels of appeal spend nearly 50 percent of that time in queue, waiting for agency action. Given these factors, we believe SSA has an opportunity to apply a more expedited approach to end-to-end testing, focused on evaluating certain claims that require some, but not all levels of agency appeal, and processing them outside the normal backlog of cases. Without full awareness and resolution of critical problems and end-to-end testing, SSA cannot ensure that the electronic disability system components will work together effectively. Thus, SSA could incur future unanticipated costs to correct the system, and stands to increase its risk of not realizing the very benefits that it seeks, by proceeding with AeDib without this critical testing. In addition, SSA said it considered moot our recommendation that the agency ensure that users have approved new software and that it certify its systems for production, stating that the agency has always performed these tasks. In addition, SSA said that our concern in this area had pertained to the lack of wet signatures[Footnote 2] on its project documents. In speaking to this issue, our primary concern was not that SSA lacked wet signatures on its software development documentation. Rather, during the course of our review, agency officials were unable to provide readily verifiable evidence that users had approved software developed for the electronic disability system, or that systems had been certified for production. At the end of our study, SSA provided certain documentation supporting its software validation and certification efforts; however, the documents lacked sufficient details for us to fully evaluate and reach conclusions about the extent to which all software had been validated by the users and certified for production. For any project, and especially one of AeDib's size and complexity, having readily verifiable evidence that systems were developed in accordance with specified requirements is essential to ensuring that users' needs will be met by the software developed and that systems can be certified for production. Regarding AeDib program and project risks, SSA stated that it believed the success of AeDib over the last 26 months had attested to the agency's diligent and ongoing efforts to mitigate risks. Nonetheless, as our report stresses, strategies for addressing program-level and security-level risks are needed to help prevent circumstances that could impede successful project outcomes. In the absence of explicit mitigation strategies, SSA increases the probability of failing to meet project commitments before they can manifest themselves in cost, schedule, and performance shortfalls. Accordingly, it is essential that SSA establish a time frame for and expedite actions toward finalizing risk-mitigation strategies for the AeDib project. Regarding our recommendation that SSA validate all AeDib cost and benefit estimates, the agency reiterated its intent to conduct studies to help validate planning assumptions, and noted that work supporting this action was under way. Finally, SSA stated that it continually reassesses its communications, and that while it can always improve, it is satisfied that information is being shared, and that stakeholder voices are being heard. Further, SSA stated that the stakeholders' differing opinions about how to design the AeDib system do not reflect an issue of communication, but rather, the reality of bringing change into a very complex and diverse environment. Our report noted that SSA had taken additional actions to work collaboratively with stakeholders. Further, we recognize that changing conditions can elicit varying opinions regarding the actions being taken on a project. Nonetheless, stakeholders continue to voice serious concerns regarding the manner in which SSA is communicating with them about AeDib, necessitating that the agency remain diligent in pursuing a mutually agreed-upon understanding with stakeholders on systems development and implementation issues. SSA's success in fully implementing AeDib depends heavily on resolving all outstanding issues and concerns that could affect the ultimate use and outcome of the intended electronic capability. By implementing a communications plan that clearly and comprehensively conveys its approach for effectively addressing disability stakeholders' and users' concerns and ensuring their full involvement in the AeDib initiative, SSA could strengthen the likelihood that it will achieve a successful project outcome that effectively meets the needs of all critical players in the disability claims process. We are sending copes of this report to the Commissioner of Social Security, and to the Director, Office of Management and Budget. Copies will also be available at no charge on our Web site at [Hyperlink, http://www.gao.gov.]. Should you have any question on matters contained in this report, please contact me at (202) 512-6240, or Valerie Melvin, Assistant Director, at (202) 512-6304. We can also be reached by e-mail at [Hyperlink, koontzl@gao.gov] and [Hyperlink, melvinv@gao.gov], respectively. Other key contributors to this report were Michael A. Alexander, Tonia B. Brown, Harold J. Brumm, Mary J. Dorsey, and Michael P. Fruitman. Sincerely yours, Signed by: Linda D. Koontz: Director, Information Management Issues: [End of section] Appendixes: Appendix I: GAO's January 30, 2004, Briefing: Update on the Social Security Administration's Accelerated Electronic Disability Strategy: Briefing for the Staff of the Subcommittee on Social Security: House Ways and Means Committee: January 30, 2004: Purpose and Outline: Purpose: To provide an updated assessment of SSA's accelerated electronic disability (AeDib) claims processing initiative: Outline: * Introduction: * Objectives: * Scope and Methodology: * Results in Brief: * Background: * Objective 1: Strategy: * Objective 2: Software Development and Risk Management: * Objective 3: Cost-Benefit Analysis: * Objective 4: Stakeholder/User Involvement: * Conclusions: * Recommendations: * Agency Comments and Our Evaluation: Introduction: Over the years, SSA has faced increasing challenges in ensuring acceptable levels of service to the millions of disabled individuals and beneficiaries. A major barrier to improving efficiency in the disability determination process has been SSA's continuing reliance on paper folders to process and adjudicate claims. As a result, SSA is currently pursuing a major systems-related initiative to enhance its disability claims processing capability. A key goal of this initiative, known as the Accelerated Electronic Disability (AeDib) System, is to develop by late January 2004, a disability claims process with the capability to electronically view and share claims information among all processing components. GAO previously reported [NOTE 1] on SSA's progress in developing the AeDib initiative, noting that its strategy involved risks that could jeopardize the agency's successful transition to an electronic disability process. Objectives: As requested by the Chairman of the subcommittee, our objectives were to: * assess SSA's progress toward and strategy for achieving AeDib, * determine the adequacy of measures SSA is taking with AeDib to avoid the kinds of software development problems that it encountered with its prior disability system initiative and to identify and mitigate risks to a successful development effort, * assess the adequacy of SSA's analysis of AeDib costs and benefits, and: * assess SSA's consultation with and support from key stakeholders. Scope and Methodology: Analyzed project management and technical documentation describing SSA's plans, strategies, and progress related to developing and implementing AeDib: Evaluated the extent to which SSA has implemented measures-including institutional software capability mechanisms-to avoid the types of software development problems identified in prior reviews and established strategies for mitigating program and project risks: Assessed the adequacy of SSA's analyses of AeDib costs and benefits, including evaluating them against federal guidance and industry best practices and analyzing the impact of critical pilot evaluation data on costs and benefits: Assessed SSA's plans and actions to effectively consult with and identify and resolve issues raised by key disability stakeholders (i.e., state Disability Determination Services [DDS] offices, medical providers, and SSA field offices): Conducted site visits at the North Carolina DDS office and SSA field offices in Raleigh and Durham to observe pilot tests supporting the electronic folder capability, and teleconferences with officials in the California and Illinois DDSs and the SSA field office in Hillside, Illinois to discuss pilot test progress and results: Interviewed officials in selected DDS offices that had been included in our prior review[NOTE 2] (Delaware, Iowa, Mississippi, Nebraska, New York, Virginia, and Wisconsin) to update our assessments of SSA's involvement of stakeholders in the AeDib initiative: Interviewed headquarters officials in SSA's Offices of Disability Programs, Operations, Systems, Hearings and Appeals, and the Inspector General to obtain their perspectives on the AeDib project: Conducted our review in accordance with generally accepted government auditing standards, from August 2003 through January 2004: Results in Brief: Objective 1: SSA's Progress and Strategy for Achieving AeDib: AeDib represents an important step toward more efficient delivery of disability payments and SSA is in various stages of completing key components of the initiative. SSA's strategy, however, involves risks that could threaten project success. Specifically, SSA is relying on limited pilot tests to help guide business and technical decisions and ensure that technology supporting the electronic disability folder will work as intended. Further, SSA is beginning its national rollout without ensuring that all critical problems identified in its pilot testing have been resolved and without conducting end-to-end testing to evaluate the functionality and performance of all electronic disability system components collectively. SSA maintained that its pilot tests of the electronic disability system components would be sufficient to evaluate the system. However, in the absence of end-to-end testing, SSA lacks assurance that the interrelated electronic disability system components will work together successfully. Results in Brief: Objective 2; Previous Problems and Mitigating Risks: SSA has established processes and procedures to guide its software development activities, including using plans to manage projects and tracking and overseeing initiatives to measure progress. However, SSA could not provide evidence that it is consistently applying these procedures to the AeDib initiative. For example, our review found no evidence of approved software validation plans and system certifications to show that users' requirements were satisfied. In addition, while SSA had identified AeDib system and security risks, it had not finalized mitigation strategies needed to help ensure a successful project outcome. Without concurrence on software validation and system certifications, or risk mitigation strategies, SSA lacks assurance that its system will be acceptable to end users and it will not be positioned to effectively prevent circumstances that could impede project success. Results in Brief: Objective 3 Adequacy of AeDib Cost-Benefit Analysis: SSA has not validated its cost-benefit analysis to ensure the reasonableness of estimated AeDib costs and benefits. SSA stated that it would use pilot test results to validate its cost-benefit estimates; however, its officials have not indicated when this will be done. Without a validated cost-benefit analysis, SSA lacks assurance that its AeDib cost estimates are reliable and that anticipated benefits will be realized. Results in Brief: Objective 4 Consultation With/Support from Stakeholders: SSA stated that it has increased its communications with AeDib stakeholders/users and has instituted some mechanisms to facilitate its communications. However, state DDSs have varying views regarding their involvement. Specifically, while state disability offices involved in AeDib pilots agreed that communications have improved, a national organization representing these offices continues to have concern about SSA's approach. In addition, although the Commissioner has stated that SSA is consulting with stakeholders and the medical community, SSA has not articulated a comprehensive plan for ensuring that all of their concerns are addressed: Results in Brief: Objective 4; Recommendations for Improvement: Given the importance of the AeDib initiative to SSA's future service delivery capability, it is essential that the agency satisfy itself that the electronic disability system will perform as intended with minimal risk before it is deployed nationwide. Therefore, to help improve the success of this initiative, we are making recommendations to the Commissioner of Social Security on (1) resolving critical problems identified and conducting end-to-end testing, (2) ensuring user concurrence on software validation and systems certifications, (3) finalizing AeDib risk mitigation strategies, (4) validating AeDib costs and benefits, and (5) improving agency communications with stakeholders and users. SSA officials reviewed a draft of our briefing slides, and their comments have been incorporated where appropriate. SSA took issue with many of our findings. For example, it disagreed with the need for end- to-end testing, and also disagreed with our finding that it had not followed all agency processes and procedures for software development. We discuss these comments and our response to them in the Agency Comments and Our Evaluation section. Background: SSA's Disability Program: The Disability Insurance and Supplemental Security Income programs are the nation's largest providers of federal income assistance to disabled individuals, with SSA making payments of approximately $86 billion to about 10 million beneficiaries in 2002. SSA's process of approving or denying disability benefits is complex and involves multiple partners at both the state and federal levels in determining a claimant's eligibility. * SSA's 1,300 field offices and its Office of Hearings and Appeals, along with 54 state DDSs, are the primary players in processing disability claims. * Physicians and other members of the medical community support disability determinations by providing the medical evidence needed to evaluate disability claims. To address concerns regarding the disability program's efficiency, in 1992 SSA initiated a redesign of the disability claims process, emphasizing the use of automation to achieve an electronic (paperless) processing capability. However, SSA encountered problems with this prior initiative, the Reengineered Disability System (RDS), and in 1999 suspended the project after approximately 7 years and about $71 million reportedly spent on the initiative [NOTE 3]. SSA's Disability Program: In August 2000 SSA renewed its plan to achieve paperless disability decision making through the use of an electronic disability folder and automated case processing systems for disability claims adjudication/ review by the end of 2005. In the spring of 2002, the Commissioner of Social Security accelerated the strategy to more quickly automate the disability claims process. Under the accelerated strategy-called AeDib- SSA planned to begin implementing its electronic disability system by January 2004. Projects Supporting AeDib: SSA is undertaking five key projects to support the AeDib strategy: 1. an Electronic Disability Collect System, giving SSA field offices the capability to electronically capture and store in an electronic folder information about a claimant's disability; 2. a Document Management Architecture (data repository and scanning and imaging capabilities) to allow claimant information and medical evidence to be captured, stored, indexed, and shared electronically among disability processing components; 3. Internet applications to enable SSA to obtain disability claims and medical information from claimants via the Internet; 4. a DDS systems migration and electronic folder interface to enhance existing case processing systems and enable state DDS offices to operate on a common platform and share information in the electronic folder; and: 5. a Case Processing and Management System for the Office of Hearings and Appeals that will interface with the electronic folder and enable staff to track, manage, and complete case-related tasks electronically. Projects Supporting AeDib: The AeDib strategy focuses on developing the capability for claimant information and large volumes of medical images, files, and other documents that are currently maintained in paper folders to be stored in electronic folders, and then accessed, viewed, and shared by the disability processing offices. According to SSA, the Electronic Disability Collect System and the Document Management Architecture are the two fundamental elements needed to achieve the electronic disability folder. SSA is using an incremental, phased approach to implementing AeDib. The agency planned to rely on pilot tests and evaluations to help guide business and technical decisions about the electronic disability folder, including critical decisions regarding the Document Management Architecture. SSA had planned to complete its pilot tests of the Document Management Architecture in December 2003, and begin its national implementation of the electronic disability folder in late January 2004, with scanning and imaging capabilities and interface software being rolled out over 18 months. SSA's Approach: AeDib is a major capital investment project, and its February 2003 cost-benefit analysis estimates 10-year life-cycle costs of about $900 million and benefits of approximately $1.3 billion. SSA reported actual AeDib costs of approximately $91 million for fiscal year 2000 through fiscal year 2003. SSA identified two AeDib funding sources for IT hardware, software, and services: * information technology systems/automation investment fund, and: * limitation on administrative expenses budget. July 2003 Findings: In July 2003 testimony [NOTE 4] we noted that SSA: * had performed important tasks for the initial electronic capability, but had substantial work to accomplish in order to begin implementing the electronic folder by late January 2004; * was not expected to complete pilot tests that were to provide critical information about the electronic folder's performance until late December 2003 and had not developed a strategy for conducting end- to-end testing to demonstrate that the individual components would work together reliably; * had not comprehensively assessed AeDib project risks or developed risk mitigation strategies; and: * had taken some steps to involve key stakeholders in the system's development, but had not resolved all outstanding concerns that could affect the use and ultimate success of the intended electronic capability. September 2003 Findings: In September 2003 [NOTE 5] we reported that SSA: may have underestimated the accelerated electronic disability system's costs and overstated corresponding benefits in its AeDib cost-benefit analysis, and: had applied software development practices that could help the agency avoid software development problems experienced with its prior electronic disability initiative-the Reengineered Disability System. Objective 1: Strategy; Progress Update: Since July, SSA has continued its work toward completing the AeDib initiative, and is in various stages of development. * SSA has implemented all planned releases of the Electronic Disability Collect System (as of January 20, 2004). * SSA has completed and placed into production three Internet applications to aid claimants in filing for disability benefits and services online. * SSA has enhanced the DDS legacy systems by installing hardware upgrades and moving systems to a common platform; it has also migrated legacy systems claims processing software in all but 3 of the 54 state DDS offices. However, it has yet to complete the legacy system electronic folder interface software that will enable existing SSA and DDS systems to interface with the electronic folder. SSA reported that it plans to complete its implementation of the legacy system electronic folder interface software by October 2004. * The Case Processing Management System is being pilot tested in a standalone environment at five Office of Hearings and Appeals sites. Objective 1: Strategy; Testing: Consistent with generally accepted best practices, SSA should ensure that the system it delivers successfully meets key business and technical requirements for reliably exchanging data among disability processing components and is protected from errors and vulnerabilities that can disrupt service. Accomplishing this requires SSA to conduct complete and thorough testing to provide reasonable assurances that systems perform as intended. This includes tests and evaluations of pilot projects to obtain data on the system's functional performance, and end-to-end tests to ensure that the interrelated systems will operate together effectively. SSA planned to rely on the pilot tests and evaluations to guide critical decisions about its document management technology and verify that this system component works in the actual user environment. For example, SSA stated that Document Management Architecture pilots would be used to test electronic folder interface and DDS site configurations for the electronic disability system's national implementation. SSA and DDS offices stated that they intended to expand the size and scope of the pilot tests to analyze the system's performance. SSA had planned to complete these pilot tests in December 2003. Since last July, SSA has been pilot testing the Document Management Architecture in three locations-North Carolina, Illinois, and California. SSA has been incrementally providing increased electronic folder functionality. These pilot tests have demonstrated that the DDSs can view medical evidence captured in the Document Management Architecture; however, the pilots also identified numerous technical/operational problems, which SSA is in the process of resolving: * DDS pilot systems experienced software glitches (i.e., screen lock- ups, inability to save highlighted text, and inability to enhance images) after new releases. * Scanning quality for some documents, such as graphical images, was poor. Response times for retrieving medical evidence stored in the data repository was slow (California/Midas). * 2-D barcode resolution diminished with multiple faxing and copying, preventing effective identification of the case file and the assigned DDS. * Existing 19-inch monitors proved too small for DDS claims examiners to simultaneously view two different documents (e.g., forms and medical evidence) on split screens. Although the pilot tests are proceeding, they are limited in scope, involving a relatively small number of examiners and disability cases: North Carolina (pilot started July 29, 2003): * 21 of 254 examiners participating (8 percent): * Has closed 561 of 1,246 cases received for the pilot: * Closes about 147,000 cases annually: Illinois (pilot started Sept. 8, 2003): * 15 of 250 examiners participating (6 percent): * Has closed 282 of 1,165 cases received for the pilot: * Closes about 164,000 disability cases annually: California (pilot started Oct. 15, 2003): * 2 of 620 examiners participating (0.3 percent): * Has closed 47 of 160 cases received for the pilot: * Closes about 400,000 disability cases annually: Although SSA had planned to expand the pilot tests to additional users, the technical/operational problems encountered during the pilots (e.g., software glitches experienced after new releases) have limited the expansion, and consequently, the opportunity to analyze system performance under increasingly heavy workloads (e.g., higher transaction rates.): The pilot tests, originally scheduled to be completed in December 2003, are also behind schedule. Further, it is not clear when SSA plans to complete the pilot tests. On January 13, SSA's Deputy Commissioner for Systems stated that, beginning on January 20, SSA planned to release to the three pilot states, the interface software and enhancements that would provide disability users the capability to electronically process an entire disability case. The Deputy Commissioner added that, barring significant problems, SSA planned to begin the national rollout of this functionality in Mississippi on January 26, 2004. Thus, SSA would begin its national rollout before fully evaluating pilot test results and ensuring that all critical problems are resolved. Without fully evaluating pilot test results and ensuring the resolution of all critical problems before national rollout of the electronic disability system, SSA not only risks being unable to ensure that the electronic folder will perform as intended, but also risks being unable to rely on the new system to sustain current disability workloads. Given AeDib's technological complexity and multiple interrelated components, coupled with limited pilot testing of the Document Management Architecture and the numerous problems encountered during that testing, end-to-end testing remains essential to helping SSA reduce system risks prior to national roll-out. Our prior work has noted the need for such testing to ensure that interrelated systems that collectively support a core business area or function will work as intended in a true operational environment. [NOTE 6] End-to-end testing evaluates both the functionality and performance of all systems components, enhancing an organization's ability to trust the system's reliability. In addition to its limited pilot testing of the Document Management Architecture, performance and other problems that SSA has encountered while developing and implementing other critical AeDib components further underscore the need for end-to-end testing. For example: * since implementing the Electronic Disability Collect System, SSA has encountered software performance issues, such as slow response times when moving from screen to screen, which have contributed to longer waits in reception areas, scheduling fewer claimant appointments, and backlogs in post-entitlement work; and: * edits for claimant information provided on Internet applications generate additional edits after the data are input to the Electronic Disability Collect System. Having to respond to the additional edits has contributed to increased SSA field office claims representatives' workloads. Resolving all critical problems identified in its testing of the AeDib components and then assessing how these components function in an integrated environment before proceeding to national roll-out are essential to gauging the system's overall successful operation. Nonetheless, SSA does not plan to conduct end-to-end testing of the electronic disability system prior to national implementation. SSA has maintained that its pilot tests of each component of the electronic disability system would be sufficient to evaluate the system's functionality and performance. Without resolution of all critical problems and end-to-end testing, SSA cannot make fully informed decisions about the system it delivers, risks having a system that will not work effectively, and could incur future unanticipated costs for systems adjustments or corrections. Objective 2: Software Development/Risk Management Previous Problems: Our prior reports [NOTE 7] have noted that effective software development processes and procedures are essential to producing high- quality software products and ensuring that the software works as intended. In response to problems encountered during its prior effort with the Reengineered Disability System, SSA established key software process improvement procedures to guide its software development activities. These procedures include: * using plans to manage projects, * tracking and overseeing initiatives to measure progress, * performing quality assurance reviews to ensure compliance with policies and procedures, and: * conducting software validation and systems certifications to ensure the effectiveness of and users' satisfaction with the software. Implementation of these processes should help the agency ensure that the system being developed is ready for production and will meet users' requirements. SSA is adhering to some of its software development processes and procedures for the AeDib initiative. For example, it is using plans to monitor software development activities and is performing quality assurance reviews. However, the agency has not consistently implemented certain key procedures. For example, we found no evidence that: * users approved software validation plans to show that they agreed with what was being produced, and: * all systems currently in production were certified for release, to indicate that the systems were developed in accordance with specified requirements. Without software validation and systems certification, SSA lacks assurance that the system is ready for production and will be acceptable to its end users. Objective 2: Software Development/Risk Management Mitigating Risks: Best practices and federal guidance advocate risk management to identify facts and circumstances that decrease the probability of failing to meet project commitments before they manifest themselves as cost, schedule, and performance shortfalls. SSA has developed a risk management plan to guide the identification and mitigation of AeDib program and project risks. The plan requires SSA to ensure that the necessary risk assessments and mitigation strategies are developed and implemented to reduce risks and achieve schedule and performance goals. SSA has identified system and security risks for the five AeDib projects. For example, it noted: * the need for Document Management Architecture interfaces with other AeDib components to be properly defined, tested, integrated, and installed to avoid jeopardizing AeDib success, and: * the need for completion and approval of systems security plans for all five projects. However, SSA has not yet finalized the mitigation strategies needed to help prevent circumstances that could impede successful project outcomes. SSA had previously stated that its mitigation strategies would be completed by December 15, 2003. However, according to agency officials, the draft strategies are currently under review by the Office of the Chief Information Officer and a date for finalizing them has not been established. Until the strategies are finalized, SSA will not be in a position to cost-effectively plan for and prevent circumstances that could impede project success. Objective 3: Cost-Benefit Analysis Adequacy: Office of Management and Budget guidance [NOTE 8] states that the purpose of a cost-benefit analysis is to promote efficient resource allocation through well-informed decision making when initiating, renewing, or expanding programs or projects that would result in a series of measurable benefits or costs extending for 3 or more years. SSA's guidance states that a cost-benefit analysis is to be conducted when there is a major capital investment, potential major human resources impact, and the investment is in competition with other projects for funding and resources. We previously reported that SSA's estimates of costs and benefits were based on critical assumptions that were not substantiated: * Critical infrastructure costs, such as outsourced scanning and imaging (keying and indexing), telecommunications, and disaster recovery, were not included in the analysis. * The estimated electronic receipt of 30 percent of medical evidence by 2004 was not based on sound data or tested in pilots. Objective 3: Cost-Benefit Analysis Adequacy: Validating the estimates is necessary to review assumptions that were made, and to continuously update and ensure that costs and benefits reflect the current situation and are accurate and realistic. Our review determined that SSA's analysis had not been validated to ensure the reasonableness of estimated AeDib costs and benefits. SSA agreed with our assessment and stated that it planned to use the results of pilot tests to validate cost and performance data included in the analysis. However, agency officials could not state when the validation of costs and benefits would be performed. Without a validated cost-benefit analysis, SSA lacks assurance that its AeDib cost estimates are reliable and that anticipated benefits will be realized. Objective 4: Stakeholder/User Involvement Consultation and Support: As partners in the disability determination process, stakeholders can offer valuable insight regarding existing work processes and information technology needs, and their involvement in the systems development initiative is essential for ensuring AeDib's acceptance and use. Our work on cross-organizational collaborative projects identified effective communication and outreach as a key practice that contributes to success. [NOTE 9] The research indicated that to achieve this, a specific outreach plan specifying tasks and mechanisms may be needed. In July we reported on the need for SSA to resolve stakeholder concerns to ensure AeDib's acceptance and use, and to take additional steps to consult with the medical community. [NOTE 10] SSA acknowledged the importance of ensuring sound relations with stakeholders and the need to take additional actions toward addressing stakeholders' concerns. Objective 4: Stakeholder/User Involvement Consultation and Support: SSA stated that it has increased its communications about AeDib with state DDSs and medical providers, and the agency has established some specific communication mechanisms (e.g., ad hoc meetings, Internet television broadcasts, and an AeDib intranet site). State DDSs, however, have varying views and concerns about their involvement with the AeDib initiative. DDSs associated with the Document Management Architecture pilot (North Carolina, Illinois, and California) agreed that SSA has increased communications and is working more closely with them. Four states that are not participating in the pilot (Iowa, Nebraska, New York, and Virginia) cited no improvement in SSA's overall communications with them about the development and implementation of the electronic disability system. The National Council of Disability Determination Directors, which represents the DDSs, continues to have concerns about SSA's communications approach, stating that the AeDib steering committee is only being used as a vehicle for reporting decisions already made, and that improved communication with DDSs is still needed. Objective 4: Stakeholder/User Involvement Consultation and Support: While the Commissioner previously stated that SSA was consulting with stakeholders and the medical community, agency officials could not articulate a comprehensive plan for effectively involving all stakeholders and consulting with the medical community. Without a clear and comprehensive plan for communicating with its stakeholders and the medical community, SSA risks not obtaining vital end-user buy-in and acceptance of the electronic disability system and the medical community's support in providing electronic medical evidence that is crucial to achieving anticipated benefits. Conclusions: SSA is beginning its implementation of an electronic disability system without fully evaluating pilot test results or ensuring that all critical problems have been addressed, and with no plans for end-to-end testing to guide its system implementation. Thus, SSA lacks assurance that the system will operate as intended in a national environment- making the agency highly susceptible to failure in its current effort to achieve an electronic disability claims processing capability. While SSA has established software process improvement procedures, critical user concurrence for all software validations and systems certifications are not evident. Without such concurrence, SSA lacks assurance that it is building a system that will meet users' requirements. Further, although SSA had planned to complete strategies for mitigating program and project risks by last December, it has yet to finalize them. SSA is beginning its national rollout of the electronic disability system without a validated cost-benefit analysis. Lacking a reliable analysis based on validated data, the agency is assuming the risk that its anticipated costs may be inconsistent with planned spending for the initiative and that anticipated measurable benefits may not be realized. Conclusions: Finally, SSA has taken some steps to improve its communications with key AeDib stakeholders and users; however, the agency has not yet articulated a comprehensive plan for ensuring that all stakeholders and users are fully involved in the initiative and that their concerns are adequately addressed. State DDSs currently have mixed views regarding their involvement, with DDS representatives continuing to see a need for improved communications with SSA about critical AeDib decisions. Recommendations: Given the technological complexity, scope, and size of the AeDib initiative, it is essential that SSA take the necessary steps to avoid the substantial risks of moving into production with a system that does not perform as intended. Accordingly, we recommend that the Commissioner of Social Security, before proceeding with the AeDib national rollout, * ensure that all critical problems identified in pilot testing of the electronic disability system components are resolved and that end-to- end testing of the interrelated systems is performed; * ensure that users have approved of software developed and that systems have been certified for production; * establish a revised time frame for and expedite actions toward finalizing AeDib risk mitigation strategies, * validate all AeDib cost and benefit estimates, and: * implement a communications plan that clearly and comprehensively conveys SSA's approach for effectively addressing disability stakeholders' and users' concerns and ensuring their full involvement in the AeDib initiative. Agency Comments and Our Evaluation: SSA's Deputy Commissioner for Finance, Assessment and Management provided an e-mail message containing the agency's comments on a draft of the briefing slides. In its comments, SSA took issue with many of our findings. SSA disagreed with the need for end-to-end testing, stating that such testing would have been too slow for its accelerated strategy. In addition, it stated that examples discussed in our slides did not support our position, noting, for example, that the Electronic Disability Collect System performance issues highlighted in our slides had not appeared until the software was in full production across the country. We maintain that end-to-end testing is essential for AeDib. The acceleration of such a technologically complex initiative, with its multiple interrelated components, elevates the need for end-to-end testing to help SSA reduce system risks, many of which were identified during pilot tests of each component. The fact that performance issues did not appear until the Electronic Disability Collect System software was in production, and that high edit rates were discovered when data from Internet applications were input into that system, is further evidence of the need for end-to-end testing to provide SSA with the opportunity to evaluate both the functionality and performance of all systems components prior to national rollout. Regarding examples of technical problems that we noted for the Document Management Architecture pilots, SSA stated that (1) it had completed a technical evaluation of the Document Management Architecture in three pilot states and (2) that four of five examples identified in our slides were resolved before SSA made a final decision to roll out to Mississippi. In addition, SSA stated that our finding regarding the 19- inch monitor size prejudged a conclusion that the screens were too small. While SSA stated that it had completed technical evaluations of the Document Management Architecture pilot efforts in the three pilot offices, it did not provide evidence of its complete evaluations during our review. Specifically, SSA provided evaluation results covering tests performed in the North Carolina and Illinois DDSs only during the July through September 2003 time frame. Further, on January 20, SSA provided a software release supporting the electronic folder capability to the pilot offices, leaving little time to ensure that the software was fully tested and evaluated and that all critical problems were resolved prior to beginning its national rollout. In addition, while we acknowledge that SSA may have resolved four of the five problems noted in our slides, the examples that we cited constituted only a subset of the numerous problems that were noted in the Document Management Architecture evaluations provided for our review. Our analysis of the North Carolina and Illinois pilot evaluations and SSA's problem tracking system report, and our discussions with officials of all three state DDSs participating in the pilots, revealed that, as certain software performance issues were resolved, other problems appeared, and that DDS staff participating in the pilots were overwhelmed as workloads and backlogs increased. Finally, it was not our judgment or conclusion that SSA's monitor size was too small. Rather, DDS claims examiners informed us that the 19- inch monitors were too small for viewing two documents on split screen. For example, during our visit to the North Carolina pilot site, claims examiners participating in tests of the Document Management Architecture and Electronic Disability Collect System stated that the small size of the monitors had affected their ability to effectively adjudicate a claim. In discussing this matter with SSA's Deputy Associate Commissioner for Systems and Associate Commissioner for Disability Programs, both stated that the agency was aware of DDS staffs' concerns about the size of the monitors and that SSA planned to send an ergonomics team to those sites to investigate this and other issues affecting staff using the system. At the time of review, the study had not been completed. In discussing the October 2004 completion date of its legacy systems migration, SSA stated that, while technically correct, our discussion did not acknowledge that DDS legacy software migration had been completed in all but 3 of the 54 state DDS offices and that those three offices would be completed in February 2004. We have modified our slides to reflect that SSA has completed legacy software migration in all but three DDS offices. SSA also stated that the DDSs have or will shortly have the Electronic Disability Collect System electronic folder interface (EFI). It noted that its more expansive EFI software is currently running in 11 states, and expansion of that will occur in concert with the Document Management Architecture rollout. SSA's recent documentation stated that it has made the Electronic Disability Collect System electronic folder interface software available to 15 DDS states, including the three Document Management Architecture pilot offices. However, SSA has yet to complete the necessary DDS legacy systems software-including its critical Document Management Architecture interface software-to enable these components to process disability claims. Until this is accomplished, existing disability processing components will not have the capability to interface with or share information captured in the full electronic folder. SSA stated that it has followed processes and procedures for software development and disagreed with our findings that it did not have approved software validation plans and had not properly certified all software releases. While we agree that SSA has established key software process improvement procedures to guide its software development activities, we found that SSA was not consistently adhering to its established procedures. At the time of our review, SSA could not provide the necessary documentation to demonstrate that users had approved of the AeDib software's being produced. SSA's Inspector General had reported the same findings in July 2003, and in discussions with us last December, reiterated this continuing concern. SSA stated that its security risk assessment documentation is no longer in draft and can be made available to us for review. SSA's actions toward finalizing its security level risk assessments for AeDib are commendable, and we look forward to reviewing this documentation. However, our recent reviews of AeDib found that the agency had not finalized mitigation strategies for its program-level or security-level risk assessments. SSA officials informed us that the draft mitigation strategies were under review by the Office of the Chief Information Officer and that no date for finalizing them had been established. We continue to believe that these strategies are needed to help prevent circumstances that could impede successful project outcomes. SSA believes that its 30-percent electronic medical evidence is a viable goal, and continued to disagree with our view that its cost- benefit analysis did not consider proper infrastructure costs. However, SSA did agree that validation of its cost and benefit assumptions is important and stated that it is actively planning that work. We are encouraged that SSA is planning to validate its cost and benefit assumptions. In noting SSA's 30-percent electronic medical evidence goal, our intent was to highlight the importance of SSA's validating its cost-benefit analysis to ensure the reasonableness of its estimated costs and benefits. The assumption that we highlighted was one of many included in SSA's cost-benefit analysis that had not been validated. Further, it continues to be our position that all relevant costs need to be reflected in SSA's cost-benefit analysis. Office of Management and Budget guidance states that cost-benefit analyses should include comprehensive estimates of all direct and indirect costs associated with a project. Without a validated cost-benefit analysis, and the inclusion of missing critical cost elements, SSA lacks assurance that its AeDib cost estimates are reliable and that anticipated benefits will be realized. Finally, regarding our findings on communication and consultation with disability stakeholders, SSA stated that AeDib information is pervasive and available to all stakeholders/users, and that the agency continues to work collaboratively with them. It added that outreach to external groups will ramp up in relation to national rollout. We acknowledge that SSA has made additional efforts to work collaboratively with its stakeholders regarding AeDib. However, we found during our analyses of documentation and in discussions with disability stakeholders that there are still varying views and concerns about SSA's approach for implementing AeDib. Due to the complexity of the AeDib initiative, and the many concerns that continue to be voiced about SSA's approach, we continue to believe that having a clear and comprehensive plan before national rollout is vital to ensuring end- user buy-in and acceptance of the electronic disability system, and to realizing anticipated benefits. NOTES: [1] U.S. General Accounting Office, Electronic Disability Claims Processing: Social Security Administration's Accelerated Strategy Faces Significant Risks, GAO-03-984T (Washington, D.C.: July 24, 2003) and U.S. General Accounting Office, Social Security Administration: Subcommittee Questions Concerning Efforts to Automate the Disability Claims Process, GAO-03-1113R (Washington, D.C.: Sept. 5, 2003). [2] GAO-03-984T. [3] U.S. General Accounting Office, Social Security Administration: Technical and Performance Challenges Threaten the Progress of Modernization, GAO/AIMD-98-136 (Washington, D.C.: June 19, 1998). [4] GAO-03-984T. [5] GAO-03-1113R. [6] U.S. General Accounting Office, Year2000 Computing Crisis: A Testing Guide, GAO/AIMD-10.1.21 (Washington, D.C.: November 1998). [7] U.S. General Accounting Office, Customs Service Modernization: Serious Management and Technical Weaknesses Must Be Corrected, GAO/ AIMD-99-41 (Washington, D.C., Feb. 26,1999); Customs Service Modernization: Ineffective Software Development Processes Increase Customs System Development Risks, GAO/AIMD-99-35 (Washington, D.C., February 11, 1999); and Air Traffic Control: Immature Software Acquisition Processes Increase FAA System Acquisition Risks, GAO-AIMD- 97-47 (Washington, D.C., March 21, 1997). [8] Office of Management and Budget Circular A-94 (Revised Transmittal Memo No. 64), October 29, 1992. [9] U.S. General Accounting Office, Electronic Government. Potential Exists for Enhancing Collaboration on Four Initiatives, GAO-04-6 (Washington, D.C.: Oct. 10, 2003). [10] GAO-03-1113R. [End of section] Appendix II: Comments from the Social Security Administration: SOCIAL SECURITY: The Commissioner March 15, 2004: Ms. Linda D. Koontz: Director, Information Management Issues U.S. General Accounting Office: 441 G Street, NW Washington, D.C. 20548: Dear Ms. Koontz: Thank you for the opportunity to review and comment on the draft report "Electronic Disability Claims Processing: SSA Needs to Address Risks Associated with Its Accelerated Systems Development Strategy" (GAO-04- 466). Our comments on the report are enclosed. If you have any questions, please have your staff contact Candace Skurnik, Director, Audit Management and Liaison Staff at (410) 965- 4636. Sincerely, Signed by: Jo Anne B. Barnhart: Enclosures (2): SOCIAL SECURITY ADMINISTRATION BALTIMORE MD 21235-0001: COMMENTS ON THE GENERAL ACCOUNTING OFFICE (GAO) DRAFT REPORT "ELECTRONIC DISABILITY CLAIMS PROCESSING: SSA NEEDS TO ADDRESS RISKS ASSOCIATED WITH ITS ACCELERATED SYSTEMS DEVELOPMENT STRATEGY" (GAO-04- 466): Thank you for sharing the draft report with us. We note that this draft contains essentially the same slides you provided us to comment on earlier this year. Given that, and since you already presented this material to the sub-committee staff in January, we are not going to re-state our disagreements with your findings. In the interest of efficiency, we request that you include this note, plus our prior comments from Mr. Sopper (copy attached) in your published report. Considering GAO's long audit presence on the eDib project, I know your interest in our progress is keen and I am sure that you will be pleased to hear that things continue to move forward. Our first full functionality Disability Determination Service (DDS) site in Mississippi recently expanded to 24 examiners using the new system. South Carolina is scheduled to go live on March 22, and then Tennessee on April 5. All told, we have used the system to receipt in over 5,700 cases in DDS sites, and have processed over 1,900 cases. Although it is too early to draw conclusions about processing time, comparing 2003 to 2002 overall disability processing time, we saw a 7- day improvement. We believe 5 days of this are attributable to the rollout of our Electronic Disability Collect System. In addition, early indicators for North Carolina show that, over the last six months, as they have piloted the electronic folder, they processed their claims 6 days faster than the national average. Further, we have over 30,000 documents in our electronic evidence repository, and the percent of evidence that bypasses scanning and comes in electronically currently stands at 38 percent in Mississippi. Turn-key pilots of the Case Processing Management System (CPMS) in five hearing offices are running smoothly, and we will begin a five month national rollout of CPMS in April. We will also begin installing full electronic folder capability in hearing offices this summer. You have made five recommendations in this draft and, as with all GAO views, we have given them careful consideration. However, as noted below, none of your recommendations compel us to change direction. 1. Systems Testing: We continue to disagree with you on end-to-end testing. If we had followed your recommendation, we estimate that it would have delayed the project by at least three years-and of course, that is three years of benefits that would have: been missed. We do continue to aggressively troubleshoot the technical bugs that are inevitable with any complex new IT system, and so far our track-record has been strong. 2. System Development Process: Your recommendations that we ensure users have approved new software, and that we certify systems for production are moot. We have always done both of these things, and will continue to do so. We believe your concern tracks back to the lack of "vet" signatures on some of the hundreds of project documents we have given you. If that is the case, and especially since this project is all about movement to an electronic environment, we are puzzled that you would be uncomfortable reviewing documents from our artifact files that have been electronically signed. 3. Risk: Every large IT project presents a wide array of risks. We believe the success of this project over the last 26 months attests to our diligent and ongoing efforts to mitigate them. 4. Validating Planning Assumptions: As stated in our prior comments, we will conduct special studies to help validate key eDib planning assumptions. This work is underway. In the mean time, technical data on systems performance, operational management information, and progress reports from sites all support continued momentum on rollout. 5. Communication: Finally, we continually reassess how we are doing with communication. While we can always improve, I am satisfied that information is being shared, and that stakeholder voices are being heard. There have been differing opinions about how we should design this system, and we have given thorough consideration to a wide variety of views, but we have had to make the difficult choices and move on. This is not an issue of communication, but rather the reality of bringing change into a very complex and diverse environment. Because of the intelligence, creativity, hard work, and tremendous dedication of literally thousands of SSA and DDS employees across the country, our massive eDib project is going very well. After two years, we are essentially on schedule, staying within our project budget, and delivering the core functionality as originally conceived. In the near term, eDib will help us to process claims and appeals faster and more efficiently. Beyond that, eDib will become the enabling platform we will use to implement more strategic improvements to the disability determination process. We have never lost sight that the real benefactors of this new system will be the American people who seek government services at extremely vulnerable periods in their lives. I am proud of what we have accomplished so far, and I have confidence in our ability to fully implement the system in the next year and half. From: Sopper, Dale: Sent: Wednesday, January 28, 2004 2:31 PM To: 'Koontzl@gao.gov': Cc: 'MelvinV@gao.gov': Subject: Comments on GAO draft briefing slides systems.doc: Thank you for the opportunity to review the latest GAO draft talking points on AeDib. We take issue with many of your findings. We continue to disagree with you on the need for end-to-end testing. This project is all about accelerating strategically important IT. Simply put, your notion of testing would have been too slow. In addition, the examples you cite don't support your broader position. Our performance issues with EDCS didn't appear until the software was in full production across the country. And the high edit rate for Internet self-help claims was not a surprise because of deliberate user-centered design choices to minimize online edits for first-time filers. You cite five examples of technical problems SSA has experienced with the DDS pilots. It's important to note that 1) we did complete a technical evaluation of DMA in the three pilot States; and 2) the first four of your examples were resolved before we made a final decision to begin roll-out of DMA in Mississippi. Your fifth example on monitor size pre judges a conclusion that 19" screens are too small. We do not agree that this conclusion is warranted at this time. * You state that SSA will not complete our DDS legacy software migration till October, 2004. While technically true because of unique challenges posed by two States which have independent systems, you should acknowledge that all but 3 of the DDS's that use a standard legacy system have already been migrated, and that those three will be completed in February. In addition, all of the DDS's that use a standard legacy system have (or will shortly have) EDCS EFI. Our more expansive EFI software is currently running in 11 States, and expansion of that will occur in concert with DMA roll-out. We have followed agency processes and procedures for software development and disagree with your qualifier that we are only following some steps. The two examples you cite are not correct. The staff groups that represent users in our process have approved the software validation plans. In addition, all of our software releases have been properly certified. * Our security risk assessment documentation is no longer in draft and can be made available to GAO for review. We continue to believe that our 30 percent EME target is a viable goal. And we continue to disagree with your view that our CBA did not consider proper infrastructure costs. However, we do agree that an evaluation of AeDib which extends beyond technical performance and can help validate CBA assumptions is important and we are actively planning that work. * Finally, in regard to your findings on communication and consultation, AeDib information is pervasive and available to all SSA and DDS employees, including employees who are not involved in current pilots. We continue to work collaboratively with SSA and DDS stakeholders, and all opinions and ideas are considered in decision-making. Outreach to external groups will ramp up in relation to national roll-out. In conclusion, we think the acceleration of this very complex and strategically vital project has gone remarkably well. We fully expect many more technical and management challenges to come up during roll- out, but our significant accomplishments over the last 22 months have positioned us to overcome them, and to successfully implement e-Dib. [End of section] FOOTNOTES [1] U.S. General Accounting Office, Electronic Disability Claims Processing: Social Security Administration's Accelerated Strategy Faces Significant Risks, GAO-03-984T (Washington, D.C.: July 24, 2003) and Social Security Administration: Subcommittee Questions Concerning Efforts to Automate the Disability Claims Process, GAO-03-1113R (Washington, D.C.: Sept. 5, 2003). [2] A wet signature refers to an original signature written on a piece of paper, as opposed to a fax copy or an agreement offered verbally or electronically. GAO's Mission: The General Accounting Office, the investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through the Internet. GAO's Web site ( www.gao.gov ) contains abstracts and full-text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents using key words and phrases. You can print these documents in their entirety, including charts and other graphics. Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as "Today's Reports," on its Web site daily. The list contains links to the full-text document files. To have GAO e-mail this list to you every afternoon, go to www.gao.gov and select "Subscribe to e-mail alerts" under the "Order GAO Products" heading. Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. General Accounting Office 441 G Street NW, Room LM Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000: TDD: (202) 512-2537: Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov Automated answering system: (800) 424-5454 or (202) 512-7470: Public Affairs: Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S. General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C. 20548: