This is the accessible text file for GAO report number GAO-14-308 entitled 'Information Technology: SSA Needs to Address Limitations in Management Controls and Human Capital Planning to Support Modernization Efforts' which was released on May 8, 2014. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States Government Accountability Office: GAO: Report to the Chairman, Subcommittee on Social Security, Committee on Ways and Means, House of Representatives: May 2014: Information Technology: SSA Needs to Address Limitations in Management Controls and Human Capital Planning to Support Modernization Efforts: GAO-14-308: GAO Highlights: Highlights of GAO-14-308, a report to the Chairman, Subcommittee on Social Security, Committee on Ways and Means, House of Representatives. Why GAO Did This Study: SSA relies on IT for delivering Social Security services to virtually every American. The agency reportedly spent about $1.5 billion for IT in fiscal year 2013, and it plans to continue modernizing its aging systems. Management controls and human capital are critical in helping ensure effective and efficient IT project implementation. GAO was asked to examine SSA's IT modernization efforts. The study (1) assessed selected IT investments to determine the extent to which they adhere to SSA's investment management controls and are improving services and (2) determined how SSA's IT human capital program, including the identification and implementation of critical skills and competencies, is supporting its current and future modernization efforts. To do so, GAO reviewed key management controls for one project from each of five SSA-defined project types, including one project with the highest resources for its type and four randomly selected projects; compared human capital planning documents with relevant guidance; and interviewed relevant SSA officials. What GAO Found: The Social Security Administration's (SSA) selected information technology (IT) projects did not fully adhere to management controls called for by its IT project management guidance, which are essential to effectively oversee and monitor IT investments. Such controls include, among others, a cost-benefit analysis, risk mitigation plan, and project schedule. For the five projects selected, SSA developed the majority of the documents required to demonstrate adherence to management controls; however, most had limitations. One project that was required to complete 11 control documents had developed 5 without limitations, but the remaining 6 had limitations. For example, while certain risks to the project were identified, the documentation did not include risk mitigation plans, which are essential for avoiding, reducing, and controlling the probability of the occurrence of identified risks. Across the five projects, the most common limitations included a lack of traceability (which is needed to track project history and demonstrate that requirements are met) and inaccurate or incomplete information, such as project schedules that had inaccuracies in key milestone dates. The limitations could be attributed to, among other things, IT oversight systems that did not include all needed data or fully support traceability, and a quality assurance process that was not effectively implemented. The agency recently took steps that should help improve its quality assurance process. Further, while SSA stated that its projects have resulted in improved services, it was not able to demonstrate this. In particular, while three of the five projects identified performance measures, these measures generally were not specific enough to determine projects' contributions to improved services, and baselines against which to measure improvement were not established. Ensuring that management controls are consistently and effectively implemented would help ensure the efficient use of agency resources. SSA's IT human capital program has identified skills and competencies to support certain workforce needs, but lacks adequate planning for the future. The agency has developed IT human capital planning documents, such as its recent Information Resources Management plan and skills inventory gap reports, which identified near-term needs, such as skill sets for the following 2 years. Nevertheless, SSA has not adequately planned for longer-term needs because its human capital planning and analysis are not aligned with long-term goals and objectives and the agency does not have a current succession plan for its IT efforts. The agency has recognized challenges with regard to employee retirements and a recent hiring freeze, which have put constraints on resources for certain investments. While SSA officials stated that an updated human capital operating plan will be completed in June 2014, they could not specify how it would address future IT human capital needs. Until these needs are identified, SSA may lack critical plans for addressing IT resources and skills to support agency-wide IT investment goals. What GAO Recommends: GAO is recommending that SSA (1) perform effective oversight to ensure control documents are developed, complete, and accurate and that oversight systems include needed data and support traceability; (2) ensure project control documents identify specific performance measures and baselines; and (3) identify long-term IT needs in its updated human capital operating plan. SSA agreed with GAO's recommendations. View [hyperlink, http://www.gao.gov/products/GAO-14-308]. For more information, contact Valerie C. Melvin at (202) 512-6304 or melvinv@gao.gov. [End of section] Contents: Letter: Background: Selected IT Projects Did Not Fully Adhere to SSA's Investment Management Controls or Demonstrate Improved Services: SSA's IT Human Capital Program Has Identified Skills and Competencies to Support Certain Needs, but Lacks Adequate Planning for the Future: Conclusions: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendix I: Objectives, Scope, and Methodology: Appendix II: Description of Selected Projects and Evaluation of Implementation of Management Controls: Appendix III: Comments from the Social Security Administration: Appendix IV: GAO Contact and Staff Acknowledgments: Tables: Table 1: Selected SSA Management Controls by Project Type: Required and Conditionally Required Documents: Table 2: Results of Selected Projects' Development of Control Documents: Table 3: Status of Control Documents for Ready Retirement Release 1: Table 4: Status of Control Documents for Internet 3441: Table 5: Status of Control Documents for Medicare Annual: Table 6: Status of Control Documents for ECMS Release 3: Table 7: Status of Control Documents for Help Desk Activities Project: Figure: Figure 1: SSA Office of Systems Organizational Chart: Abbreviations: CIO: Chief Information Officer: CMMI: Capability Maturity Model Integration: COBOL: Common Business Oriented Language: ECMS: Earnings Case Management System: EGADS: Electronic General Auditable Documents Store: EVM: earned value management: IRM: Information Resources Management: IT: information technology: OMB: Office of Management and Budget: SEI: Software Engineering Institute: SITAR: Strategic Information Technology Assessment and Review: SSA: Social Security Administration: VISOR: Vital Signs & Observations Report: [End of section] United States Government Accountability Office: GAO: 441 G St. N.W. Washington, DC 20548: May 8, 2014: The Honorable Sam Johnson: Chairman: Subcommittee on Social Security: Committee on Ways and Means: House of Representatives: Dear Mr. Chairman: The Social Security Administration (SSA) is responsible for delivering Social Security services that touch the lives of virtually every American. To provide these services, the agency relies on a variety of information technology (IT) systems, ranging from those that support the processing and payment of Disability Insurance and Supplemental Security Income benefits to those that facilitate the calculation and withholding of Medicare premiums. For fiscal year 2013, the agency reported that it spent approximately $1.5 billion for IT. Many of SSA's systems have aged and present various challenges to the efficiency of the agency's existing IT environment. Consequently, the agency has committed to, and has initiated numerous efforts toward, investing in modern technologies needed to update its aging IT infrastructure and improve services.[Footnote 1] Its efforts in this regard are to be guided by management processes and controls that, according to the agency, exist to help ensure that its investments meet the strategic and business objectives of the agency. Given the importance of technology in supporting SSA's ability to meet its mission, you requested that we examine the agency's modernization efforts. Specifically, our objectives were to (1) assess selected IT investments to determine the extent to which they adhere to SSA's investment management controls and are improving services; and (2) determine how the agency's IT human capital program, including the identification and implementation of critical skills and competencies, is supporting its current and future modernization efforts. To address the first objective, we selected a non-generalizable sample of IT projects, using data files provided by SSA that included investment data from fiscal years 2010 to 2012. Our criteria considered each of the project categories that SSA has defined and the different sizes of projects based on actual work years of effort. Specifically, we selected one project because it had the most resources (i.e., work years) dedicated to it for that project category; we then selected four others randomly from the remaining project categories. In addition, we analyzed industry best practices and guidance used to effectively manage IT projects. We also analyzed policies and guidance reflected in the agency's IT project management directive and quality assurance processes to identify the types of control documents, such as project proposals, cost-benefit analyses, and project schedules, that SSA relies on for managing its projects. Among those identified, we judgmentally chose 11 types of management control documents for our evaluation. These 11 controls were called for in the agency's IT project management directive and quality assurance processes, as well as other external best practices; covered key management processes-- select, control, and evaluate; and spanned each phase of the projects' life cycle. In addition, they were represented in each randomly selected project. Subsequently, for each of the five selected projects, we evaluated the effectiveness of the controls implemented. To do so, we obtained the applicable documentation from SSA, where available, and assessed it against the agency's guidance and other best practices. Specifically, we compared the documents to the agency's IT project management guidance, as well as best practices outlined in the Software Engineering Institute's (SEI) Capability Maturity Model for Integration (CMMI) for Development (version 1.3), our IT investment management framework, and our standards for internal control.[Footnote 2] In assessing the documents, we identified any limitations in the controls, such as data inaccuracies or inconsistencies. Further, to determine whether the agency's IT investments had improved services, we analyzed information that was identified in project documentation for measuring the performance of each project. We also interviewed project managers, project team members, and other agency officials, including the Deputy Commissioner for Systems/Chief Information Officer (CIO), to understand the agency's investment management processes and its IT project management guidance and controls. For the second objective, we obtained and analyzed available human capital plans and data, including the agency's Information Resources Management (IRM) Strategic Plan for fiscal years 2014-2017, 2-year IT skills inventory gap reports, IT staffing data for fiscal years 2008- 2014, agency-wide succession plan, and agency-wide human capital plan. We also noted any human capital issues that were identified during our interviews of agency officials or reviews of documentation supporting the five selected projects. We assessed the reliability of the data that we used to support the findings in this report by reviewing relevant IT investment files and program documentation to substantiate evidence obtained from agency databases. We analyzed agency database instructions and replicated the system processes with SSA data files and compared the results with agency-provided data. We also corroborated documentation on program processes and projects obtained from SSA through interviews with SSA officials. We determined that the data used in this report are sufficiently reliable. We conducted this performance audit from November 2012 to May 2014, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Additional details on our objectives, scope, and methodology can be found in appendix I. Background: SSA's mission is to deliver Social Security services that meet the changing needs of the public. The Social Security Act and amendments [Footnote 3] established the programs that SSA administers. Among these, the Old Age, Survivors, and Disability Insurance program-- commonly referred to as Social Security--is one of the nation's largest federal retirement and disability programs.[Footnote 4] Financed by two trust funds,[Footnote 5] the program provides monthly benefits to retired and disabled workers, their spouses, children, and the survivors of insured workers. In addition, Supplemental Security Income is a needs-based program that is financed from general tax revenues. It is designed to provide benefits to aged adults and to blind or disabled adults and children who have limited income and resources.[Footnote 6] According to SSA, in fiscal year 2013, about 63 million people received Social Security or Supplemental Security Income benefits for a total of about $855 billion in benefits paid. Collectively, about 165 million people work and pay Social Security taxes, and 88 percent of individuals age 65 and over receive Social Security benefits. Organizationally, SSA is very large. It is headed by the Commissioner, who is assisted by the Deputy Commissioner and various other executive officials, including the Chief and Deputy Chief of Staff, Executive Secretary, and eight deputy commissioners who are responsible for the agency's various business components. The Commissioner is supported by about 75,000 federal and state employees who are located at SSA's headquarters in Baltimore, Maryland, and throughout a decentralized network of about 1,500 offices that includes 10 regional offices, 6 processing centers, and approximately 1,260 field offices, as well as teleservice centers, hearing offices, and 54 state and territorial Disability Determination Services offices. In support of their operations, the offices perform an assortment of interrelated and interdependent business functions, including operational, policy, financial management, and legislative relations. SSA's IT Environment: SSA relies extensively on computer hardware and software to carry out its core mission functions. In fiscal year 2013, the agency reported expenses totaling $11.6 billion to support its mission and programs-- approximately $1.5 billion of which was spent for IT. Specifically, computer systems are used to administer programs and support related administrative needs that include, among others, * handling millions of transactions via SSA's toll-free telephone number; * maintaining records for the millions of beneficiaries and other recipients of the agency's programs, including Supplemental Security Income and Old Age, Survivors, and Disability Insurance; * evaluating evidence and making determinations of eligibility for benefits on new claims; * issuing new and replacement Social Security cards; * processing and posting employer-reported earnings to workers' records; * processing continuing disability reviews;[Footnote 7] and: * processing non-disability Supplemental Security Income redeterminations.[Footnote 8] Many of SSA's existing computer systems[Footnote 9] were developed in the 1960s and 1970s, and while the agency has performed technical and functional upgrades throughout the years to accommodate legislative and policy changes, these legacy systems have aged. Accordingly, as they have aged, the systems have presented various challenges to the efficiency of SSA's existing IT environment. * Many of its programs are written in Common Business Oriented Language (COBOL),[Footnote 10] which is one of the oldest computer programming languages.[Footnote 11] The agency reported that it runs hundreds of COBOL applications and that this language has served the agency for over 40 years. It currently has roughly 60 million lines of COBOL in production that support the agency's high-transaction volume and enable the agency to meet its regulatory, benefit, and reporting requirements. While SSA officials referenced a study indicating that the benefits of a wholesale replacement of COBOL would not outweigh the risks because of the large size and cost of rewriting their COBOL applications,[Footnote 12] the Assistant Deputy Commissioner for Systems/Deputy CIO said the agency has a plan to migrate away from this language and that new applications are being developed in more modern programming languages, such as Java. * The National Computer Center is SSA's facility that currently houses its nationwide computer operations. Mainframe systems within this data center maintain critical demographic, wage, and benefit information essential for providing service to millions of individuals, as well as other federal, state, and local agencies, daily. However, the center is over 30 years old and many of its infrastructure systems are well past their designed life cycle. SSA has expressed concerns that the center could deteriorate to the point that a major failure to its systems could jeopardize its ability to handle increasing workloads without interruption. The agency is in the process of developing a new data center to replace its existing center in Baltimore, Maryland. * While SSA has efforts under way to convert to a more efficient database system for its critical mission support files and uses more modern databases for its new applications, the agency continues to use its Master Data Access Method database system,[Footnote 13] which does not support industry standards for automatic data access. Although it has converted several files, such as the Master Earnings Record and Supplemental Security Record, to a more modern database, the agency has not yet completed the conversion for one of its largest files--the Master Beneficiary Record file.[Footnote 14] * SSA stores, processes, and shares increasing amounts of data with public-and private-sector partners, and the agency faces an increasing need to transition to web-based online access for its data and services. SSA's Organization Responsible for IT: The Office of Systems, headed by the Deputy Commissioner for Systems (who also serves as the agency's CIO), is SSA's primary organization responsible for overseeing the design, development, and maintenance of the technology resources that support the agency. The Deputy Commissioner for Systems/CIO reports to the Commissioner of Social Security. According to the agency, 3,403 IT positions are assigned to the Office of Systems.[Footnote 15] In addition, the office relies on contractors to fill certain skill gaps, where necessary. The office reported that in fiscal year 2013 approximately 1,402 contractor work years were used to support IT projects. Staff in the Office of Systems are assigned to eight component offices: * Office of Applications and Supplemental Security Income Systems-- responsible for most phases in the systems development life cycle for the Supplemental Security Income, Quality Assurance, Customer Help Information, and Representative Payee programs. * Office of Disability Systems--develops, implements, and maintains electronic systems to support disability programs. * Office of Earnings, Enumeration, and Administrative Systems-- designs, develops, and maintains SSA's earnings, enumeration, and administrative systems. * Office of Enterprise Support, Architecture, and Engineering-- identifies the strategic IT resources needed to support SSA business processes and operations. * Office of Information Security--manages and directs the agency's overall information systems security program. * Office of Retirement and Survivors Insurance Systems--responsible for programmatic and management information systems supporting these programs, as well as for the post-entitlement activities associated with the Disability Insurance program. * Office of Systems Electronic Services--directs the development of software that supports electronic service-delivery initiatives. * Office of Telecommunications and System Operations--responsible for supporting the computer systems and networks infrastructure. In addition, the Office of Systems is responsible for the IT portion of the human capital planning that supports agency-wide human capital goals. For example, the Office of Systems submits an annual Human Capital Management Report to SSA's Office of Human Resources.[Footnote 16] The report provides an assessment, analysis, and the results of human capital strategies, activities, and operations related to IT. Figure 1 provides a simplified organizational chart depicting the agency's IT organization and component offices. Figure 1: SSA Office of Systems Organizational Chart: [Refer to PDF for image: Organizational Chart] Top level: Office of Systems: Office of the Deputy Commissioner/Chief Information Officer. Second level, reporting to Office of Systems: Office of Applications and Supplemental Security Income Systems; Office of Disability Systems; Office of Earnings, Enumeration, and Administrative Systems; Office of Enterprise Support, Architecture, and Engineering; Office of Information Security; Office of Retirement and Survivors Insurance Systems; Office of Systems Electronic Services; Office of Telecommunications and Systems Operations. Source: SSA. [End of figure] SSA's Management of IT Investments: SSA's governance structure for the review and management of IT investments is documented in its Capital Planning and Investment Control guidance. This guidance assigns responsibility for the investment management process to the agency's deputy commissioners and other top-level executives. Further, it describes the policies and processes guiding management decisions on the selection, control, and evaluation of all investments. In this regard, the Deputy Commissioner for Systems/CIO is to ensure that IT is acquired in accordance with the Capital Planning and Investment Control procedures. Additionally, this official is responsible for reviewing and obtaining the Commissioner's approval of the agency's annual IT budget and investment plan. SSA also has a Strategic Information Technology Assessment and Review (SITAR) board, which is chaired by the Deputy Commissioner for Systems/CIO and includes the Principal Deputy Commissioner, other deputy commissioners, and other senior executives responsible for the business units. These key stakeholders represent the business units that sponsor the investments and work with the Office of Systems to establish project requirements and strategic direction; they also help oversee the investments' development. SSA uses its Capital Planning and Investment Control process to manage its software development projects. According to the agency, the investment management process is intended to meet the objectives of the Clinger-Cohen Act by providing a framework for selecting, controlling, and evaluating investments to help ensure they meet the strategic and business objectives of the agency.[Footnote 17] During the investment selection phase, new projects are to be proposed by a sponsor--either from a business unit for mission-related projects or from the Office of Systems--and assigned to a portfolio.[Footnote 18] Proposals that identify business needs are to be developed based on the Commissioner's priorities or on gap analyses performed by portfolio teams.[Footnote 19] The portfolio executives and their teams are to review business-sponsored IT investment proposals and recommend and submit proposals to the Office of Systems to develop resource estimates on the proposals. Portfolio executives develop prioritized lists of proposed projects based on several factors, such as the availability of agency resources, and prepare recommendations on specific proposals for the agency IT plan. Next, the prioritized lists are to be combined into the plan for approval by the SITAR board. The plan is to be comprised of proposed investments for the next 2 fiscal years, and provide information on work year requirements.[Footnote 20] In addition, expected benefits and returns on investment are to be included for new projects. The SITAR board is responsible for approving the agency IT plan on an annual basis and is to modify the plan as needed based on changing priorities. The plan is then sent to the Commissioner, who provides final approval of the specific project proposals presented in the plan. During the control phase, the Office of Systems is responsible for holding monthly meetings with project managers who are assigned to monitor investment projects. During these meetings, projects that are not meeting cost and schedule expectations are to be identified and corrective actions are to be initiated. One of the objectives of these meetings is to resolve problems related to underperforming projects without elevating them to the level of the SITAR board. During the months in which the board meetings are scheduled, the Deputy Commissioner for Systems/CIO meets with staff to address any concerns about investments that may be raised during the meetings. If concerns are raised at a meeting, the Deputy Commissioner for Systems/CIO is to provide information to the SITAR board about these investments. In addition, the board is to receive profiles on the status of each of the agency's major IT investments. These profiles should include reports on actual and expended work years, costs, schedules, and any variances. During the evaluation phase, the Capital Planning and Investment Control guide calls for the Deputy Commissioner for Systems/CIO to conduct post-implementation reviews on projects that have been completed and deployed for at least 3 months. The purpose of these reviews is to compare actual project results against planned results in order to assess performance and identify areas where future decision making can be improved. SSA uses two primary oversight systems to monitor the IT projects that are selected and developed, as well as to store information on unique projects that are completed. Specifically, these systems are Electronic General Auditable Documents Store (EGADS) and Prism. * EGADS is the agency's internally developed system for storing and accessing final approved versions of project life-cycle documents, such as IT proposals, among others.[Footnote 21] Project managers and other officials, such as subject matter experts, are responsible for providing updated documents for storage in the system. * Prism is the agency's web-based project and portfolio management tool that is intended to track the progress of projects.[Footnote 22] The tool is designed to provide senior management and stakeholders with an overview of a project's key performance indicators, such as project status, schedule, resources, and risks. Project managers are responsible for managing and updating their project information in Prism, which is used for reporting during monthly management meetings. The agency classifies its IT projects according to five major categories: (1) development, (2) maintenance, (3) cyclical, (4) planning and analysis only, and (5) National Computer Center.[Footnote 23] * Development projects are those that involve the creation of a new software application, enhancements to an existing application, or installation of or enhancements to new architecture or hardware. * Maintenance projects pertain to activities required to keep a system application running. Once a project is completed, SSA places the system into maintenance mode.[Footnote 24] * Cyclical projects, such as cost-of-living adjustments, are performed on a routine basis each fiscal year, and can be done annually, quarterly, or within some other defined time frame. According to SSA, these types of projects typically do not include a total redesign or large enhancement, although some cyclical projects involve smaller enhancement efforts. * Planning and analysis only projects are undertaken for the specific purpose of determining if a potential IT solution would be feasible from a business and technical perspective. * National Computer Center projects are undertaken to keep the center running, and involve such matters as help desk activities, monitoring activities, and storage management. * Of these five categories, SSA designates certain ones as Executive Oversight projects. These projects are the agency's highest priority and receive additional senior management oversight. SSA reported that, from fiscal year 2010 to fiscal year 2012, it devoted approximately 10,739 work years to the five categories, representing 1,950 actual projects.[Footnote 25] The resources consisted of SSA personnel, as well as contractor support that, according to the agency, cost approximately $600 million for fiscal year 2012. SSA's projects varied in size from large major modernization efforts, representing hundreds of work years annually to small investments of less than 2 work years in effort.[Footnote 26] In addition, the time frames for completing the projects varied. For example, while certain projects were active and spanned over a decade, others were initiated and completed within a year. For 2010 to 2012, SSA reported that approximately 70 percent of its projects were active; 25 percent were completed; and 5 percent were withdrawn, on hold, or approved, but not yet started.[Footnote 27] To oversee these projects, SSA managers are to adhere to the Office of Systems' Project Management Directive,[Footnote 28] which identifies numerous project management controls that support key areas of the agency's Capital Planning and Investment Control process and are applicable to a project throughout its life cycle. Specifically, certain control documents call for critical IT investment information to support the select, control, or evaluate phase(s). For example, IT proposals support the select phase, project schedules support the control phase, and lessons learned support the evaluation phase. Yet other controls, such as risk management, support all three phases. The project managers store the control documents in the key management information oversight systems, EGADS and Prism. According to the project management directive, the control document can be required, not required, or conditionally required,[Footnote 29] depending on the type of project. For example, a project schedule is required for planning and analysis only, development, and cyclical projects, but is conditionally required for a maintenance project and is not required for a National Computer Center project. Additionally, a cost-benefit analysis is required for planning and analysis only, development, and National Computer Center projects, but is not required for cyclical and maintenance projects. In total, the project management directive identified about 46 types[Footnote 30] of control documents for managing those IT projects selected for our study. [Footnote 31] Table 1 describes the importance of the 11 types of control documents that we examined and the kinds of projects for which they were either required or conditionally required.[Footnote 32] Table 1: Selected SSA Management Controls by Project Type: Required and Conditionally Required Documents: SSA control document[A]: IT proposal[C]; Description of control document: An IT proposal is a critical document supporting SSA's Capital Planning and Investment Control process. The IT proposal provides information supporting management decisions on approving and selecting a project. The proposal should include, among other things, an explanation of how a project will support the agency's strategic objectives, the business case, and an analysis of costs and benefits to assist management in its consideration and approval. It helps to justify the business needs of an IT project and identifies the key executive sponsor and business customers (or end users); Project type(s) for which the control document is required: * Development; * Planning and analysis only; * National Computer Center; Project type(s) for which the control document is conditionally required[B]: * None. SSA control document[A]: Risk management[D]; Description of control document: To ultimately succeed, most IT investments need a continuous focus on interim results and successful risk management strategies, among other things. Risk management supports SSA's Capital Planning and Investment Control process and is a continuous, forward-looking process that should identify and address issues that could endanger achievement of critical project objectives. It involves defining a risk management strategy, identifying and analyzing risks, and handling identified risks, including the implementation of risk mitigation plans as needed. Risks are one of the key performance measures tracked within SSA's project oversight system; Project type(s) for which the control document is required: * Development; * Planning and analysis only; * Cyclical; Project type(s) for which the control document is conditionally required[B]: * Maintenance[E]; * National Computer Center[E]. SSA control document[A]: Cost-benefit analysis; Description of control document: A cost-benefit analysis is a technique used to compare the various costs associated with an investment with the benefits that it proposes to return. OMB requires agencies to conduct analyses on federal programs that include comprehensive estimates of the expected benefits and costs to society. SSA, in its Capital Planning and Investment Control process and project management guidance, also requires that clear and descriptive analyses of the costs and benefits, including the expected return on its IT investments, be developed and updated. For SSA, the cost- benefit analysis is a determining factor for ascertaining which projects best serve and benefit the agency when there are limited resources for which various projects proposals are competing; Project type(s) for which the control document is required: * Development; * Planning and analysis only; * National Computer Center; Project type(s) for which the control document is conditionally required[B]: * None. SSA control document[A]: Project schedule; Description of control document: The project schedule is an essential project management document that will help the project manager set milestones, track tasks, and monitor the project's progression throughout the life cycle. The schedule is called for in SSA's project management guidance and supports the agency's Capital Planning and Investment Control process. The project schedule is one of the key performance measures tracked within SSA's project oversight system; Project type(s) for which the control document is required: * Development; * Planning and analysis only; * Cyclical; Project type(s) for which the control document is conditionally required[B]: * Maintenance[F]. SSA control document[A]: Business process description; Description of control document: The business process description is a user document that describes the business process as it currently exists or will exist without regard to technology, languages, or platform on which the software is developed. The changes to an agency's business processes that are required by the new IT investment need to be documented and understood by users, sponsors, and system programmers to effectively implement the project; Project type(s) for which the control document is required: * Development; * Planning and analysis only; Project type(s) for which the control document is conditionally required[B]: * Cyclical[G]; * Maintenance[G]. SSA control document[A]: Project scope agreement; Description of control document: This control document, described in SSA's procedures and project management directive, supports the agency's Capital Planning and Investment Control process and is intended to clearly define the scope of the project to include project- specific functionality, business goals, and requirements. Without this critical information on the project's scope, it is difficult to determine if the agency has met its goals. Further, the scope agreement is important to determine if the project has missing functionality and therefore may not satisfy business goals and customer needs; Project type(s) for which the control document is required: * Development; * Cyclical[H]; Project type(s) for which the control document is conditionally required[B]: * Planning and analysis only[I]. SSA control document[A]: Service request; Description of control document: A service request serves as SSA's internal control mechanism to ensure that new or revised code has been appropriately authorized and tested before being moved into production. It includes the requirements, estimated and actual dates of development, testing and implementation, validation-related information, and related management approvals to develop and validate the software; Project type(s) for which the control document is required: * Development; * Cyclical; * Maintenance; Project type(s) for which the control document is conditionally required[B]: * None. SSA control document[A]: System release certification; Description of control document: For each service request, the agency is required to develop a system release certification to control and manage software placed in production. The agency's instructions for completing the system release certification note that if the software is released with known problems, the risks associated with those problems should also be explained. It follows software through its life cycle and is used as a mechanism for controlling and managing movement of software into production and includes management approval certifying that code is completed, fully tested, validated, and acceptable for release to production; Project type(s) for which the control document is required: * Development; * Cyclical; * Maintenance; Project type(s) for which the control document is conditionally required[B]: * None. SSA control document[A]: Function point estimate; Description of control document: Function point estimates support SSA's Capital Planning and Investment Control process and other key control guidance, such as GAO's Cost Estimating and Assessment Guide, and SSA's Project Management Directive, and are a structured technique of estimating size. It is a method to break systems into smaller components, so they can be better understood and analyzed. Function points are a unit measure for software much like an hour is to measuring time. The initial function point analysis provides an estimate of the proposed project's size, complexity, and effort. The final function point estimate is compared to the functionality of the delivered project and provides valuable information with regard to the project's actual size and effort; Project type(s) for which the control document is required: * None; Project type(s) for which the control document is conditionally required[B]: * Development[J]; * Cyclical[J,K]. SSA control document[A]: Lessons learned[L]; Description of control document: Lessons learned support the agency's Capital Planning and Investment Control process and are a set of best practices and areas for improvement gathered throughout the life of a project and are used as input for future projects to know what does and what does not promote project success. For some IT investments, "lessons learned" are conducted with similar objectives, but less extensive assessments in order to understand areas to improve in future projects; Project type(s) for which the control document is required: * Development; Project type(s) for which the control document is conditionally required[B]: * None. SSA control document[A]: Quality assurance; Description of control document: The purpose of quality assurance reviews is to provide staff and management with objective insight into the processes and associated work products. In doing so, the review should (1) objectively evaluate performed processes and work products against applicable requirements, (2) identify noncompliance, (3) provide feedback on the results, and (4) ensure that noncompliance issues are addressed; Project type(s) for which the control document is required: * None; Project type(s) for which the control document is conditionally required[B]: * Development[M]; * Cyclical[M]. Source: GAO analysis of SSA's Office of Systems project management directive and guidance and industry IT best practices. Notes: [A] SSA first defined the control documentation requirements for each of its project types in its January 2009 Office of Systems' project management directive. The requirements from that version are represented in the table. [B] Conditionally required control documents are to be completed if those projects meet certain conditions defined in the agency's project management directive. These conditions are further described in subsequent notes. [C] SSA changed its directive in January 2010 to require this document for maintenance and cyclical projects. [D] Risk management is a continuous activity throughout SSA's System Development Life cycle. [E] SSA uses management judgment to determine whether this document should be developed. [F] Development of this document depends on the functionality involved. [G] This control document is completed if a business process description does not exist or if a new business process requires an update of the existing description. [H] SSA changed its directive in February 2009 to conditionally require this document for cyclical projects with 2 or more work years. [I] A project may do a project feasibility analysis instead of a project scope agreement. [J] This control document is to be completed for all Executive Oversight projects. [K] SSA changed its directive in February 2009 to conditionally require this document for Executive Oversight cyclical projects with 2 or more work years. [L] Lessons learned were included in SSA's post-release review reports which were required of development projects in January 2009. SSA removed the post-release review report from its directive in January 2011, and a separate lessons learned document was added in July 2012. At that time, lessons learned became a conditionally required document as they were optional for all development, planning, and analysis only, and cyclical projects. [M] Control document required for all Executive Oversight projects. [End of table] The five projects selected for our study included one project from each project type, with the applicable controls depending in part on the type of project. Specifically, * Ready Retirement Release 1 was a development project and was required to develop all 11 types of controls included in our review. [Footnote 33] The project was an approximately 52 work-year Executive Oversight project that was part of a larger, broader initiative referred to as Ready Retirement. The larger initiative is intended to make the retirement application process more efficient by providing a streamlined web interface for the public and SSA employees to process claims, thereby increasing automation and improving service delivery. The Ready Retirement project is ongoing and has had several releases since its initial inception. * Internet 3441 was a maintenance project and was required to develop 2 of the 11 control types. The project was an approximately 4.74 work- year effort focused on maintaining a web application used to gather SSA Form 3441 (i.e., disability report appeal) information from the public. The project continued until September 2010 when, according to agency officials, it was combined with another maintenance project. * Medicare Annual was a cyclical project, which required 6 of the 11 control types.[Footnote 34] It was an approximately 0.83 work-year project, and was part of a larger effort to update the Medicare Part D systems.[Footnote 35] The goal of the Medicare Annual project was to select a portion of all subsidy-eligible individuals for an eligibility redetermination. This redetermination was based on characteristics which were more likely to have a change in the subsidy amount. For example, a change in income, resources, and/or household composition can affect an individual's eligibility for the subsidy and the amount of the subsidy. * Earnings Case Management System (ECMS) Release 3 was a planning and analysis only project and required 6 of the 11 control types.[Footnote 36] The project was approximately a 0.44 work-year effort that was initiated and withdrawn in fiscal year 2011. ECMS Release 3 was part of a larger project intended to support SSA's Earnings Redesign initiative and had a goal of transitioning the earnings correction process from using outdated terminal interface "green screens" to a more modern intranet application by fiscal year 2015.[Footnote 37] This particular project focused on assessing the existing business processes related to updating the master earnings file. * Help Desk Activities is a National Computer Center project that required 2 of the 11 control types. The project was initiated in 2002 and from the initiation through fiscal year 2012, it accumulated 800 work years.[Footnote 38] This project supports SSA's infrastructure and is within the Office of Telecommunications and Systems Operations. Examples of the project's major functions include providing a call center to answer users' questions, providing technology support, ensuring continued operation of the local area network, and providing hardware and software support. The Help Desk Activities also include providing a support team to coordinate and resolve programmatic system software issues, such as slow response times associated with disability claims processing systems. GAO and Others Have Highlighted the Importance of SSA's IT Investment Controls: SSA's Inspector General and we have reported on challenges the agency has faced in the overseeing its IT investments and the significance of implementing controls to effectively manage the agency's projects. * SSA's Office of the Inspector General reported in September 2006 that, while the agency had generally implemented an earned value management (EVM)[Footnote 39] system to manage its major IT projects in accordance with Office of Management and Budget guidance, some of the requirements were not based on the agency's cost-benefit analyses, which limited the effectiveness of the EVM process and the agency's IT development project management.[Footnote 40] Accordingly, the Inspector General made recommendations aimed at addressing these limitations. SSA agreed with the Inspectors General's recommendations and has taken steps to address the weaknesses. * We reported on SSA's IT investment management approach in 2008 and found that it was largely consistent with leading investment management practices, but it was not applying its investment management process to all of its investments.[Footnote 41] Specifically, we noted the agency was executing a majority of the key practices needed to build the foundation for managing its IT projects as investments, as discussed in our IT investment management framework, and it had made progress in establishing the key practices for managing investments as a portfolio. However, SSA was not applying its investment management process to a major portion of its IT budget. Among other things, we recommended that the CIO develop and implement policies and procedures to manage IT acquisitions as investments and manage them using the investment management framework. SSA disagreed with this recommendation and stated that its process treated these acquisitions as investments and maintained them by using an investment management framework, though not the one described in our framework. However, we noted that these acquisitions were not subject to the agency's investment management select, control, and evaluate processes and were not managed by its investment board. We further stated that, given that IT products and services made up the majority of SSA's IT budget, the investment board's involvement was essential to helping ensure effective management of acquisitions. * In April 2012 we also reported that SSA had undertaken numerous modernization efforts, but it lacked effective measurement tools to determine progress.[Footnote 42] We noted that, since 2001, the agency had reported spending about $5 billion on the modernization of its systems and it had undertaken hundreds of modernization projects each year from 2001 to 2011. Nonetheless, SSA still had major efforts under way to transition from its aging systems to a more modernized IT environment. Further, SSA had not fully established quantifiable performance measures for all its modernization projects or performed post-implementation reviews, which we had previously recommended and which would enable the agency to effectively measure its progress. We also reported that SSA lacked up-to-date and comprehensive plans to guide its modernization efforts. Accordingly, among other things, we recommended that SSA develop comprehensive metrics to effectively gauge modernization progress. SSA neither agreed nor disagreed with the recommendations, but described steps it was taking that would address elements of the recommendations related to IT oversight and updated its comprehensive Information Resources Management (IRM) plan in 2012. * Further, in October 2013 SSA's Office of Inspector General reported that it could not determine whether SSA had realized the planned cost savings for its IT initiatives because the agency had not calculated actual savings after project implementation.[Footnote 43] The report also noted that SSA did not have a process to assess the overall effectiveness of its IT capital planning and investment control practices. As a result, the agency did not know whether its IT investments achieved the planned savings or any productivity improvements. The Inspector General recommended that SSA continue implementing a cost-effective post-implementation review process to verify whether its IT investments are meeting planned savings. SSA agreed with the Office of Inspector General's recommendation and said the agency, in July 2013, had developed the framework for post- implementation reviews to assess IT project performance, and will continue to improve the process to verify that its investments are meeting planned savings. * Most recently, in a December 2013 report that discussed agencies' reporting on the IT Dashboard,[Footnote 44] we noted that, according to SSA officials, the agency creates new milestones and corresponding baselines for IT investments at the start of every fiscal year that are based on annual funding amounts received from SSA's SITAR process.[Footnote 45] This process, known as rebaselining, erases past cost and schedule variances, prevents the agency from monitoring year- to-year investment performance, and makes it difficult to estimate and understand life-cycle costs. We stressed that while this rebaselining only affected one investment in that review, the process had the potential to impact SSA's entire IT investment portfolio. We recommended that SSA revise its investment management processes to ensure consistency with the baselining practices identified in our guidance on the management of capital investments. SSA agreed with the recommendation and noted that it would take steps to revise its process. Selected IT Projects Did Not Fully Adhere to SSA's Investment Management Controls or Demonstrate Improved Services: The five IT projects that we reviewed did not fully implement project management controls consistent with SSA guidance and industry practices. While SSA had developed the majority of required control documents, only six fully reflected the content identified in SSA guidance as essential to effective project management. Most of the control documents developed had limitations, such as a lack of traceability (which is needed to trace and track the history of projects and demonstrate that the projects met requirements) and inaccurate or incomplete information. The limitations could be attributed, in part, to the use of oversight systems that did not include all needed data or have the capability to fully support traceability and a quality assurance function that was limited in scope and, thus, inadequate to assess control documents effectively. Further, while SSA indicated that its projects have resulted in improvements to services, it was unable to demonstrate how the selected projects had contributed such improvements. Selected IT Projects Had Mixed Results in Applying Required Management Controls: As discussed earlier, to manage and oversee its IT projects, SSA managers and project teams are to adhere to the Office of Systems' Project Management Directive to ensure that quality products that satisfy customer requirements are delivered on time and within budget. In addition, disciplined project management practices call for the development of project details such as objectives, scope of work, schedules, costs, and requirements against which projects can be managed and executed. This can be facilitated by developing work products that document these details to ensure the projects are being managed according to specified practices. Toward this end, the SSA management directive requires project teams to develop essential documentation (depending on project type) to support the execution and management of projects. However, for the five projects in our review, SSA did not consistently apply its project management guidance. Specifically, the five projects developed most of the required control documents. However, while six of these were developed without limitations, the majority had limitations, reducing assurance that the projects were being managed in accordance with disciplined practices. Table 2 shows, for each project, whether required control documents were developed for each project and had limitations. Table 2: Results of Selected Projects' Development of Control Documents: Project: Ready Retirement Release 1; Total required control documents: 11; Control documents developed with no limitations: 5; Control documents developed with limitations: 6; Control documents not developed: 0. Project: Internet 3441; Total required control documents: 2; Control documents developed with no limitations: 0; Control documents developed with limitations: 2; Control documents not developed: 0. Project: Medicare Annual; Total required control documents: 6; Control documents developed with no limitations: 1; Control documents developed with limitations: 3; Control documents not developed: 2. Project: ECMS Release 3; Total required control documents: 6; Control documents developed with no limitations: 0; Control documents developed with limitations: 3; Control documents not developed: 3. Project: Help Desk Activities; Total required control documents: 2; Control documents developed with no limitations: 0; Control documents developed with limitations: 2; Control documents not developed: 0. Source: GAO analysis of SSA data. Note: The control documents in the table include four that SSA's project management directive designated as required based on certain conditions. Specifically, a function point estimate and quality assurance reviews were required for Executive Oversight development projects, and Ready Retirement Release 1 met this criteria requiring these control documents. Further, ECMS Release 3 was to complete a project scope agreement or its alterative, a project feasibility analysis, and SSA did not complete either document. Lastly, a business process description for the Medicare Annual project was required to be created or updated if the project created a new business process and that control document was provided. In other cases, project control documents that were conditionally required were not completed based on a project's judgment that the condition did not apply; in those cases, we did not consider the documents as "not developed." [End of table] The most common limitations that we noted in the project control documentation were the lack of full traceability or inaccurate or incomplete information. A more detailed discussion of each of the five projects and the extent to which they applied required management controls follows. Ready Retirement Release 1: As directed by SSA's guidance, 11 specific control documents were required for this project, and the project developed all 11. Of these, 5 were developed without identified limitations, including the IT proposal, cost-benefit analysis, business process description, project scope agreement, and lessons learned. These controls helped guide the project by providing critical information on the project's strategic objectives, expected benefits and risks supporting the agency's SITAR process, and helped ensure that the requirements met user needs. Further, lessons learned for Ready Retirement Release 1 identified best practices and areas for improvement which could be used as input for the success of future projects. However, the other six control documents had limitations. Specifically, while project team members produced documentation that identified certain risks to the project, the documentation did not include mitigation plans, which are essential for avoiding, reducing, and controlling the probability of the occurrence of identified risks. Office of Systems officials stated that the mitigation plans were not available because these documents were stored in the agency's previous oversight system, VISOR, which has been replaced with a more current system, Prism. The officials explained that documents in the previous system are no longer accessible. In addition, the project schedule was not available in the project oversight system, limiting access to critical information, such as milestones, tasks, and progress through the life cycle--information that is needed for project oversight. Further, service requests and system release certifications, which document the software requirements to be developed and identify that they have been fully tested and approved, were not traceable to the project release. According to CMMI, project requirements and related documentation should be traceable to each other; this is essential for tracking project history and demonstrating that projects meet requirements. [Footnote 46] Thus the project lacked sufficient documentation to allow oversight officials to independently determine that developed functionality met established requirements. Further, the function point estimate, which is used to estimate project size, complexity, and effort, was not used to manage the project. Finally, the project's quality assurance was not effectively implemented in that it did not evaluate the effectiveness of required management controls, to ensure that all findings and compliance issues were properly recorded for the project. Internet 3441: For this project, two control documents were required--service requests and system release certifications--and both were developed. However, both had limitations. Specifically, the service requests were not traceable to project releases and did not account for all project releases, which, as called for by CMMI and internal control standards,[Footnote 47] is necessary for ensuring that system changes are accurately documented. In addition, while the system release certification noted that defects had been revealed during software testing, the certification did not identify these problems and their resolution, if any, to help demonstrate that the software was acceptable for release into production. Medicare Annual: Of the six control documents required for this project, four were developed. However, the project could not produce evidence that it had developed a risk management document. According to Office of Systems officials, this documentation was not readily accessible because it was stored in the previous oversight system, VISOR. In addition, the agency could not provide evidence that a project schedule had been developed. While the project manager provided a schedule related to the larger effort this project is associated with, it was at too high a level and not detailed enough to show how it related to the project. Of the documentation that was developed, the project scope agreement did not have limitations. The agreement identified the business goals, user goals, and requirements applicable to the project. By doing so, SSA established defined project boundaries that reflected a mutual agreement between the systems project manager and the business project manager, which is critical to establishing an understanding between all involved parties and providing a shared vision for the completed project. However, three other documents that were developed had deficiencies that limited traceability between requirements and functionality developed, as called for by CMMI and internal control standards.[Footnote 48] Specifically, the description of the relevant business processes was not traceable to the project or the specific changes to functionality it was to implement, and changes to the business processes were not clearly linked to the scope and objectives of the project. In addition, service requests associated with the project were not accurately linked to work occurring in the relevant year, or were not accurately linked to the project. Finally, the system release certifications lacked traceability to the service requests and were not properly linked to the appropriate year's work. These inaccuracies and inconsistencies made it unclear what work was actually performed for which project. ECMS Release 3: This project developed only three of the six required control documents. In this regard, it could not demonstrate that it had developed a risk management document, which project officials stated was stored in the previous project management system, VISOR. In addition, project officials said they decided not to pursue the development of the cost-benefit analysis and project scope agreement for ECMS Release 3 because they viewed the project as completed after the delivery of the business process description. However, according to SSA's project management guidance, the development of these control documents is required, and other documentation for this particular project showed that the project was withdrawn and control documentation that was developed did not substantiate the completion of the project as claimed by the project manager. Without control documents defining the project's scope or a cost-benefit analyses, it is unclear if the project was cost effective and the status of the project--if the project was withdrawn or completed as originally proposed--is uncertain. In addition, limitations were apparent in the three control documents that did exist. For example, the scope described in the IT proposal, which according to SSA's guidance is to support management decisions to approve and select a project, was not specific to the outcome of this planning and analysis project. Rather, the scope described in the proposal was for the larger development effort of which this project was a part and which is to be completed by fiscal year 2015. Not defining the specific project scope can hinder the ability to set clear expectations and measure benefits. Further, the project schedule was inaccurate and incomplete in that it was not updated, showed tasks as complete which had not been completed, and was not clearly aligned with the scope of the project described in other documentation. Finally, the business process description, which is to describe changes to a business process required by the new IT investment, was not traceable to other key documentation, such as the project scope description (which had not been developed). In the absence of these critical pieces of information, it was unclear how needed business process changes were related to specific project requirements. Help Desk Activities: This project was required to develop two control documents and included evidence that it had done so. Nonetheless, both had limitations. In particular, the IT proposal did not present the expected performance of the project and, according to the project managers, contained a $2 million error in estimated costs. Thus, the project lacked needed information for making investment decisions. In addition, costs and benefits for this project were included in a higher-level cost-benefit analysis for SSA's data center investment, instead of being identified separately. This meant that SSA had less assurance that actual costs and benefits for this project were reflected in investment decisions. Appendix II provides a more detailed discussion of the implementation of management controls for the five selected projects. Limitations in Oversight Systems and Quality Assurance Contributed to Inconsistent Implementation of Management Controls: The inconsistent development of required control documents for the five selected projects can be attributed in part to limitations in the systems SSA uses to oversee and manage its IT projects and to a quality assurance process that was limited in scope. In several cases, certain management control documents were not available and Office of Systems officials said this was because these documents were stored in the agency's previous oversight system, VISOR, which contains historical project data and is no longer operational. The agency's replacement system, Prism, contains information only on projects started in fiscal year 2012 or after, and officials stated they did not see the value of transferring historical data to the new system. However, in cases where projects are part of larger, ongoing initiatives, having continuing access to control documents that reflect the history of the investment or related projects can help better inform management and facilitate oversight of these related initiatives. For example, Ready Retirement 1 is part of a broader effort consisting of multiple, interrelated components. Without access to historical status reports and project work products containing information on resources, schedules, and risks, SSA limits its ability to use actual project data to improve project management and increases its need to rely on project managers' institutional knowledge. In addition, the lack of full traceability among project control documents existed in part because certain oversight systems, such as EGADS and its related system for processing service requests, were not originally designed to distinguish and fully trace project control documents at detail levels. Traceability is part of requirements management that is fundamental to having a controlled and disciplined systems engineering process.[Footnote 49] In discussing this matter, the Director of Planning and Management Analysis Staff within the Office of the Deputy Commissioner for Systems told us that, in August 2013, the agency had begun planning for a redesign of its older applications, including EGADS and its service request processing system, to provide the ability to associate control documents within and among related projects, among other things. Office of Systems officials said that the projected completion date for this redesign is June 2014. Further, according to the project scope agreement for this effort, the redesign is expected to provide a more structured workflow, eliminate the redundancy of data by leveraging data from other systems such as Prism, and automate several of the manual processes. Redesigning these systems to support full traceability should facilitate a more disciplined system development process. Given that the agency manages hundreds of projects a year and that SSA officials have stated that project managers have the ability to combine, split up, and/or rename projects on a yearly basis, full traceability, supported by readily available and complete project information, is essential. Other weaknesses in the implementation of management controls, such as incomplete and inaccurate information in required documents and work products, can be attributed in part to weaknesses in SSA's quality assurance process. According to best practices such as SEI's CMMI, process and product quality assurance should evaluate work products against applicable process descriptions, standards, and procedures. Further, federal internal control standards call for evaluations of management controls, such as quality assurance reviews, to focus on the effectiveness of controls' implementation, and this monitoring activity should ensure that findings are promptly resolved and reported to management.[Footnote 50] While the Office of Systems has a quality assurance process to ensure projects are following the procedures defined by the office, the quality assurance reviews that were conducted were limited in their scope and did not evaluate how effectively the controls were implemented. Specifically, reviews did not evaluate all project types and were limited to projects over a certain size. For example, the Office of Systems' guidance limits reviews to Executive Oversight development projects and Executive Oversight cyclical projects that are 2 work years and over.[Footnote 51] According to the quality assurance representatives, in fiscal year 2012 these criteria limited the reviews performed to approximately 10 percent of the IT projects that year (approximately 60 out of 600). However, SSA may have been better positioned to identify and address the types of control limitations that we identified if its quality assurance process had been more rigorous and representative of the agency's investment portfolio, including major product types of varying sizes. SSA quality assurance representatives acknowledged that the reviews need improvement and, in this regard, said that the agency was in the process of improving its quality assurance review process to expand the scope and depth of coverage. Specifically, the officials stated that in September 2012 the agency conducted an internal appraisal of its quality assurance program which found that stakeholders had an inconsistent understanding of quality assurance goals, formal quality assurance training was not available, reviews did not focus enough on issues with process implementation, findings were not recorded using a method that allowed for analysis and identification of needed improvements, and process trends were not communicated to senior management on a recurring basis. The officials said the agency developed and, in October 2013, began implementing a strategy to address these issues, which was to include audits of control documents in the agency's project oversight systems. The audits also were to include all Executive Oversight projects and up to four random non- Executive Oversight projects representing each Office of Systems component and project type, except National Computer Center projects. [Footnote 52] Further, the implementation plan called for the findings of the audit to be documented and reported to project managers in a quality assurance tool. In this regard, the Associate Commissioners were to begin receiving quarterly reports on quality assurance beginning in January 2014. However, an Office of Systems official subsequently informed us that this deadline was not met because of the need to address requested changes to the report. According to an Office of Systems official, the first of these reports was presented to the Associate Commissioners in late March 2014, and the agency provided us a copy of this report in early April. According to our review, the report found that each type of review that SSA conducted--process review, product review, and audit of control documents--had identified findings such as limitations in project schedules, the lack of traceability in requirements, risks not being discussed with stakeholders, and control documents not being stored appropriately, which were similar to the limitations that we identified. While the report noted that some of the findings had been closed, others remained open and the report offered recommendations for addressing them. According to the Office of Systems official, findings that were not closed during the first review will be reported in the second quarterly report. These steps, if implemented on an ongoing basis, should lead to improvements in the quality assurance process, and reduce the risk that the agency's controls will not be effectively implemented. SSA Has Not Demonstrated How the Selected Projects Have Improved Services: Best practices in IT investment management, such as those described in our investment management framework, emphasize the importance of monitoring the performance of investments by, among other things, collecting information such as costs, benefits, schedule, risk assessments, performance metrics, and system functionality to support executive decision making.[Footnote 53] In this regard, establishing a baseline and specific performance measures that include how IT contributes to achieving improved program outcomes is essential for periodically evaluating a project as it progresses and further assessing a project after its completion. This assessment allows IT projects to be compared with one another based on selected criteria, such as expected benefits or improvements, among others, to effectively manage and prioritize the portfolio of projects. This critical investment information supports executive decision making that should ensure IT projects are aligned with the agency's strategic plan and meet business needs in an effective and efficient manner. SSA has stated that its projects have contributed to improved services.[Footnote 54] However, for the three projects in our study that had identified measures of improved services,[Footnote 55] the agency was not able to demonstrate the extent to which the projects actually contributed to improved services. This is because the performance measures identified were not specific to the goals of the individual projects, and the projects did not identify measurable baselines against which to gauge progress. * The IT proposal for the Ready Retirement Release 1 project stated that it would be considered successful if it simplified the claims application process and implemented consistent business practices for taking claims. Yet, the proposal did not provide any specific performance measures that described how these success factors could be achieved. In addition, the project did not establish a performance baseline against which to measure improvements. The agency attributed some benefits to the project, such as an increase of 250,000 Medicare- only claims submitted from 2011 to 2012 and in the results of its customer satisfaction survey. However, these benefits do not clearly reflect the stated goals of simplifying the claims application process and implementing consistent business practices for taking claims. In particular, the agency did not provide the measures that were used to determine the scores on the customer satisfaction survey, nor did it provide evidence that it had specifically measured results during the time frame when Ready Release 1 was deployed. As a result, the extent to which this project effectively contributed to improved services is unclear. * The Earnings Case Management System Release 3 project established a goal to transition a specific process--the earnings correction process--to a more modern intranet application. However, this goal may have been more appropriate for measuring the overall Earnings Case Management System project development rather than for the planning and analyses only segment of the project. According to the project manager, the intent of the planning and analyses only portion of the project was to review existing earnings business processes to identify information needs that have changed in this area over the years. While the project manager stated that the planning and analysis only project was complete and that this segment of the overall project was successful because a business process description was created, other agency documentation noted that the planning and analysis only project was withdrawn. Thus it is uncertain to what extent this project may have contributed to improved services. * For the Help Desk Activities project, specific performance measures were not defined in the project proposal and then mapped to other identified performance goals. SSA did identify other metrics to assess help desk performance, such as stating that its call response time should be less than 90 seconds and its abandoned call rate should be less than 12 percent, and project officials stated that these metrics allow the agency to assess its overall key goals such as systems availability. However, these metrics related to overall customer service-level agreements and systems availability. Further, project officials noted that other documentation related to Help Desk Activities included performance measures, such as systems' availability being expected at 99 percent for online applications, but these measures applied to its overall data center investment, and the documentation did not describe how previously identified Help Desk Activities metrics contributed or were related to these higher-level benefits. While the agency's most recent Information Resources Management plan does include performance measures for various high-level IT-related "domains," without project-specific performance measures linked to expected benefits, SSA is not positioned to demonstrate the extent to which projects are contributing to the agency's business goals, such as a specific amount a project is expected to increase service performance. Consequently, SSA is not sufficiently positioned to justify IT investments that are intended to improve its services to the public. Moreover, without documented and approved baselines of proposed benefits to effectively measure against, it is unclear which projects are more efficient in meeting business needs. SSA's IT Human Capital Program Has Identified Skills and Competencies to Support Certain Needs, but Lacks Adequate Planning for the Future: Key to an agency's success in modernizing its IT systems is sustaining a workforce with the necessary knowledge, skills, and abilities to execute a range of management functions that support the agency's mission and goals.[Footnote 56] Achieving such a workforce is dependent on having effective human capital management, [Footnote 57] which includes assessing current and future agency skill needs by, for example, analyzing the gaps between current skills and future needs and developing strategies for filling the gaps, as well as planning for succession. Taking such steps would be consistent with activities outlined in human capital management models that we and the Office of Personnel Management have developed.[Footnote 58] In addition, GAO's Standards for Internal Control in the federal government stresses that management should consider how best to retain valuable employees, plan for their eventual succession, and ensure continuity of needed skills and abilities.[Footnote 59] In addressing its IT human capital, the Office of Systems identified certain critical skills and competencies supporting its workforce needs in its Information Resources Management (IRM) Strategic Plan for fiscal years 2014 through 2017, and in a series of gap reports that it developed in 2008, 2010, and 2012. For example, the IRM plan provided the approximate number of IT specialists in the agency's workforce, and described the agency's use of contractors. Further, the agency's IT human capital needs were reflected in the 2- year skills inventory gap reports that the Office of Systems had developed. The most recent such report was developed in 2012 and covered fiscal years 2013 and 2014. An executive officer in the Office of Systems stated that the agency prepares gap reports every 2 years and that, during the fiscal year in which the report is being prepared, supervisors identify the current skills of staff, including their levels of ability--beginner, intermediate, and advanced--and project what will be needed in the following 2 years. For example, each of the reports identified a number of general and specific skill sets needed, such as COBOL and Java.[Footnote 60] In addition, to help address any identified skill gaps, the Office of Systems prepares a training plan explaining how it intends to fill the gaps. The plan includes information identifying courses that are to be offered to staff to develop the needed skills. Beyond providing training, the office also pointed to its reliance on contractors to fill skill gaps. Nonetheless, while SSA had addressed its IT human resources in this manner, it had not taken steps that identified the agency's needs beyond fiscal year 2014. While identifying skills and competencies that are clearly linked to the agency's mission and longer-term IT goals is essential, neither of SSA's key human capital planning documents--the IRM plan and the agency-wide Strategic Human Capital Plan--provided information about future IT human resources needs. * The IRM plan for fiscal years 2014 to 2017 identified strategic goals such as "deliver innovative quality services" and "build a model workforce to deliver high quality service." Further, the plan stated that the agency intends to use knowledge management initiatives, technology training programs, and recruitment and retention strategies to mitigate any potential loss of institutional knowledge and to maintain its IT workforce. It also described several IT "domains," their current states, and high-level 2-year and "out-year" plans for each domain.[Footnote 61] However, the plan did not identify how human resource levels and staff development strategies link to and support longer-term plans such as how many staff will be needed to execute the future plans identified in its IT domains. * SSA's most recent agency-wide strategic human capital plan, for fiscal years 2009 through 2012, also did not identify future needs for IT resources. Although this plan identified general agency goals, such as to develop leaders at all levels through comprehensive succession management and improve employee performance by fostering better management, it did not specifically address what IT human capital resources would be needed to support these goals. In addition, SSA does not have a current succession plan for the replacement of experienced staff supporting its IT efforts. The agency's most recent succession plan was issued in 2006, and we recently recommended that it take steps to update the plan.[Footnote 62] This is particularly important given previous experiences that have impacted the agency's workforce since 2006, including a hiring freeze and greater movement toward online services. For example, the Office of Systems has reported experiencing significant decreases over recent years in its entry-level staff. Specifically, from fiscal year 2010 to fiscal year 2013, the number of staff in lower pay grades (i.e., with less seniority) dropped from 318 to 62. In contrast, the number of staff in the higher grades steadily increased from fiscal year 2008 to fiscal year 2013. In addition, since 2008, about 20 percent of the Office of Systems' IT staff have been eligible to retire annually, which is a significant portion of the workforce. According to officials in the Office of Earnings, Enumeration, and Administrative Systems, retirements and the loss of experienced staff have made it difficult to allocate needed skills to projects. Without planning for the replacement of experienced staff with critical skills, the Office of Systems may lack the resources needed to respond to requests for IT support in an effective manner. According to an official in the Office of Human Resources, multiple human capital plans, including an IT human capital plan, are to be folded into a single, updated plan--referred to as the Human Capital Operating Plan. The agency had expected to have this plan by February 2014. However, an Office of Systems official subsequently stated that the agency's deadline for developing the revised, consolidated Human Capital Operating Plan had changed to June 2014. According to this official, the new estimated completion date for the Human Capital Operating Plan should allow more time for appropriate stakeholder input. Nonetheless, the officials could not provide specific information on how future IT human capital needs are expected to be identified or addressed in these revised plans, beyond stating that they would include the agency's updated succession plan. Until SSA ensures that its human capital planning and analysis addresses the specific competencies and skills critical to meeting its future IT needs, the agency jeopardizes its ability to deliver IT support that effectively contributes to its systems modernization efforts and improved services. Conclusions: Although SSA had applied management controls to the selected IT projects in our review, limitations raise concerns about their effectiveness. Specifically, while the five projects developed most of the project documentation needed to show adherence to management controls, the majority had limitations. In particular, missing and inaccurate control documents call into question whether projects are efficiently and effectively achieving their objectives and whether investment decisions are well supported. These limitations can be attributed to key oversight systems lacking accurate and complete project information, as well as their systems' inability to support full traceability, and to limited and inadequate quality assurance reviews of project documents, although the agency has recently taken steps to improve its quality assurance process. In addition, the absence of specific performance measures and baselines makes it difficult to fully substantiate how effectively and efficiently IT investments have improved the agency's services. Given the hundreds of IT projects SSA manages each year, ensuring that management controls are consistently and effectively implemented is critical to the efficient use of agency resources. Regarding human capital, SSA has identified certain critical skills and competencies; however, an absence of planning beyond 2014 may jeopardize the agency's ability to carry out its modernization efforts. In particular, the agency has not identified IT-specific human capital needs to support its future goals and it does not have a current IT succession plan for replacing experienced staff. While agency officials stated that they expect to develop a revised plan for addressing IT human capital needs, milestones for this have recently been pushed back, calling into question the agency's ability to effectively plan for the resources needed to support its IT modernization efforts. The importance of addressing these issues is highlighted by the agency recognizing the significant decreases in entry-level staff and its hiring freeze as challenges to effectively staffing certain investments. Without having human capital plans and analyses that identify and address the agency's needs for critical IT skills and competencies, SSA risks not having the resources and skills sets necessary to carry out its future IT plans in support of the agency's mission and goals. Recommendations for Executive Action: To address SSA's project management and human capital deficiencies for its IT modernization efforts, we recommend that the Commissioner of Social Security direct the Deputy Commissioner for Systems/Chief Information Officer to take the following three actions: * Perform effective oversight to ensure that key management control documents for ongoing and future projects are developed, complete, accurate, and readily accessible in oversight systems to better support management, traceability, and project analysis of IT investments. * Assess control documents supporting the selection of investments, such as IT proposals, to ensure that they fully identify specific performance measures and baselines to gauge project success. * Identify future IT needs, including skills needed to support long- term goals and priorities, in the agency's updated human capital operating plan and associated analysis. Agency Comments and Our Evaluation: We received written comments on a draft of this report from SSA, which are reprinted in appendix III. In its comments, SSA stated that it agreed with our three recommendations and described steps it plans to address them. For example, the agency stated that it will continue to improve its governance process by requiring both process and product reviews of selected IT projects, improve its work to identify performance measures, and include a comprehensive assessment of IT human capital needs in its updated IT Human Capital Plan. In addition, regarding our first recommendation, SSA agreed that our findings were accurate; however, it noted that our review covered a small, non-generalizable sample of IT projects. While we acknowledge that the results of our review cannot be generalized to the entire set of SSA's IT projects, the projects we reviewed did represent the major categories of the agency's investments. Further, several of the issues we identified were attributable to weaknesses in oversight systems, which could affect multiple projects. Therefore, it is important for SSA to continue to strengthen the oversight of its IT management and governance process. SSA also provided technical comments, which we have incorporated as appropriate. We are sending copies of this report to the Chairman of the Committee on Ways and Means, House of Representatives, the Commissioner of the Social Security Administration, appropriate congressional committees, and other interested parties. In addition, the report is available at no charge on the GAO website at [hyperlink, http://www.gao.gov]. If you or your staff have any questions on matters discussed in this report, please contact me at (202) 512-6304 or melvinv@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. Sincerely yours, Signed by: Valerie C. Melvin: Director: Information Management and Technology Resources Issues: [End of section] Appendix I: Objectives, Scope, and Methodology: Our objectives were to (1) assess selected information technology (IT) investments to determine the extent to which they adhere to the Social Security Administration's (SSA) investment management controls and are improving services; and (2) determine how SSA's IT human capital program, including the identification and implementation of critical skills and competencies, is supporting its current and future modernization efforts. To address the first objective, we selected a non-generalizable sample of five IT projects, which were based on analyses of data files provided by SSA and selection criteria designed to include a variety of the agency's IT investment categories and sizes. The review was designed to include one of each of the five project types-- development, maintenance, cyclical, planning and analysis only, and National Computer Center--defined by the agency. Specifically, after the data files were determined to be sufficiently reliable for selecting projects, we judgmentally selected Ready Retirement Release 1 because it was the largest development project in the core services portfolio from fiscal years 2010 to 2012 and was related to SSA's major investment, Ready Retirement. The remaining four projects, Earnings Case Management System (ECMS) Release 3, Medicare Annual, Internet 3441, and Help Desk Activities were chosen randomly from the planning and analysis, cyclical, maintenance, and National Computer Center categories, respectively. We analyzed industry best practices and guidance used to effectively control, oversee, and manage IT projects. We also analyzed SSA policies and guidance to determine key documents associated with the agency's IT project management directive and quality assurance processes to identify the types of management control documents, such as project proposals, cost-benefit analyses, and project schedules, that SSA relies on for managing its investments. Among those identified, we chose 11 types of management controls for our evaluation.[Footnote 63] The 11 types of documents we selected support key management activities at different points throughout a project's life cycles. Further, they were called for in the agency's IT project management directive and quality assurance processes, as well as other external best practices; spanned each phase of the agency's project life-cycle process; and were represented in each randomly selected project. The 11 control types also were intended to cover key management processes that are undertaken during the select, control, and evaluative phase(s) and provide critical information regarding the resources and costs committed as well as the potential benefits or improvements to service. For each of the five projects, we obtained the associated control documents from SSA, where available. We then assessed each of the five projects to determine the controls that were applicable, based on the project type, and whether they were developed for the selected projects. We analyzed the documents against the agency's guidance and other best practices. Specifically, we compared the control documents to the agency's IT project management guidance, as well as best practices in the Software Engineering Institute's (SEI) Capability Maturity Model for Integration (CMMI) for Development (version 1.3), our IT investment management framework, and our standards for internal control.[Footnote 64] The best practices and standards call for control documentation to, among other things, be traceable, accurate, consistent, and effectively implemented. We assessed the control documentation and determined if any limitations, such as inaccuracies or inconsistencies, in these areas existed. To determine whether SSA had improved services, we analyzed information that was identified in project documentation for measuring the performance of each investment, where available. Specifically, we analyzed the agency's project proposals and supporting documentation to determine if quantifiable performance measures and baselines, called for in best practices such as our IT investment management framework, were identified and measurable for each project. We also interviewed project managers, project team members, as well as other agency officials, including the Deputy Commissioner for Systems/Chief Information Officer, to understand the agency's processes and control documents. To address the second objective, we obtained and analyzed SSA human capital plans and data, including its Information Resources Management (IRM) Strategic Plan for fiscal years 2012 through 2016 and the most recent updated plan for fiscal years 2014 through 2017, 2-year IT skills inventory gap reports, IT staffing data for fiscal years 2008- 2014, agency-wide succession plan, and its agency-wide human capital plan. We compared the agency's plans and documents to best practices we have identified in strategic human capital management.[Footnote 65] We also noted any human capital issues that were identified in interviews with agency officials or documentation for the selected investments above. We assessed the reliability of the data that we used to support the findings in this report by reviewing relevant IT investment files and program documentation to substantiate evidence obtained from agency databases. We analyzed agency database instructions and replicated the system processes with SSA data files and compared the results with agency-provided data. We also corroborated documentation on program processes and projects obtained from SSA through interviews with agency officials to clarify program processes and documentation. We determined that the data used in this report are sufficiently reliable. We have also made appropriate attribution indicating the sources of the data. We conducted this performance audit from November 2012 to May 2014 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. [End of section] Appendix II: Description of Selected Projects and Evaluation of Implementation of Management Controls: The following provides a description of each project and our evaluation of control documents called for by SSA's Office of Systems Project Management Directive and other guidance: * Ready Retirement Release 1. Ready Retirement Release 1, which was to provide online services for Medicare Only and Month of Election,[Footnote 66] was an approximately a 52 work-year Executive Oversight development project completed in 2010. This project was part of a larger, broader initiative referred to as Ready Retirement. The larger initiative was intended to make SSA's retirement application process more efficient by bypassing the manual claims processes and providing a streamlined web interface for the public and SSA employees to process retirement claims, thereby increasing automation and improving service delivery.[Footnote 67] * For this project, all 11 of the control documents we selected from SSA's Office of Systems' Project Management Directive were relevant, with 9 being required and 2 being conditionally required.[Footnote 68] SSA developed all 11 documents; however, 6 had limitations. Table 3 describes the 11 control documents, whether or not they were developed, and any limitations with the documents. Table 3: Status of Control Documents for Ready Retirement Release 1: Required control document: IT proposal; GAO evaluation: No limitation identified; Description: SSA developed an IT proposal for this project and no limitation was identified. Required control document: Risk management; GAO evaluation: Incomplete; Description: This project was required to manage risks, which includes, among other things, identifying and analyzing risks as well as developing risk mitigation plans.[A] Timely mitigation of risks helps the agency avoid threats that may endanger a project's completion[B, C]. Despite the need to identify and mitigate risks, risks were not accessible in the project oversight system and the agency did not provide risk mitigation plans for this IT project. Specifically, while the project manager stated that risks were in the project oversight system, they could not provide them because the system was no longer accessible. Agency officials stated the historical project status information, that included risks, was not transferred to the newer project management oversight system if the project was no longer active. Instead, SSA provided other project documentation that identified risks related to policy changes, business practices, and skilled developers. However, there were no mitigation plans identified for these risks. Required control document: Cost-benefit analysis; GAO evaluation: No limitation identified; Description: SSA developed a cost-benefit analysis for this project and no limitation was identified. Required control document: Project schedule; GAO evaluation: Inaccuracies; Description: This project was required to develop a project schedule, which is an essential project management document for setting milestones, tracking tasks, and monitoring the project's progression throughout its life cycle.[A, B, C] The official project schedule was provided by the project manager, but it was not within the project management oversight system called for by the project management directive, potentially limiting access to critical information for project oversight. Additionally, inaccuracies within key milestone dates were identified in project schedules assessed by quality assurance reviews. For example, key project milestone dates, such as moving the system into production, did not match the implementation date within an oversight system. While the quality assurance reviews did not provide a reason for the difference in the dates, it did indicate that dates in the oversight system are used to determine goals and accurate dates are needed to determine if the project achieved those goals. Required control document: Business process description; GAO evaluation: No limitation identified; Description: SSA developed a business process description for this project and no limitation was identified. Required control document: Project scope agreement; GAO evaluation: No limitation identified; Description: SSA developed a project scope agreement for this project and no limitation was identified. Required control document: Service request; GAO evaluation: Lack of traceability; Description: This project was required to develop service requests, which is SSA's specific name for a control document that provides system development documentation for the IT software requirements, supporting the project's scope and objectives.[A, B] This would include what requirements have been requested to be changed, how the software will be tested, and related management approvals. This control document is critical for managers to effectively monitor the project and, as such, should show the requirements associated with new or revised code have been appropriately authorized and tested before it is moved into production. To help ensure that requirements are developed according to the project scope and plans, traceability among individual requests and other related projects is essential for effectively managing requirements. According to CMMI, requirements and related documentation should be traceable to each other, and internal control standards state how documentation should facilitate tracing transactions through processing.[C, D] The service requests were not traceable to the project release that was reviewed because SSA's management system--the service request system--does not have the capability to capture the release number associated with the Ready Retirement Release 1 project. As a result, the agency does not have capability to fully trace requirements. Required control document: System release certification; GAO evaluation: Lack of traceability; Description: System release certifications were required to be developed for this project to ensure that software developed for a service request has been completed, fully tested, and is acceptable for release into production.[A, B] The control document should note if the software is released with known problems and the risks associated with those problems.[E] The traceability between requirements and related testing should be documented to help ensure all requirements are developed as planned.c For this project, the system release certifications stated that there were known problems and defects in testing.f Project officials identified defects in a separate system that were stated to be related to the project; however, these defects did not identify the related system release certification number. Traceability between the defect and its related system release certification would allow the project manager and systems developers to know if a defect had been resolved. Traceability is important particularly because SSA, in its lessons learned for this project, noted that the thoroughness of defect documentation was identified as an area that needed improvement. Required control document: Function point estimate*; GAO evaluation: Ineffective implementation; Description: This project conditionally required a function point estimate, which can be used to estimate the project's size, complexity, and effort and can provide important measurement information when compared to the actual functionality of a delivered project.[A, B] While initial and final function point estimates were developed for this project, the project manager stated that they did not use the estimates in managing the project. Further, while there was a difference between the initial and final function point estimate, the officials could not provide a specific reason for the difference. However, the project manager acknowledged that there are several possible reasons for the difference, such as added requirements. Additionally, the lessons learned noted that appropriate stakeholders were not involved in the initial function point estimate. Required control document: Lessons learned; GAO evaluation: No limitation identified; Description: SSA developed lessons learned for this project and no limitation was identified. Required control document: Quality assurance*; GAO evaluation: Ineffective implementation; Description: This project conditionally required quality assurance reviews, which can focus on the effectiveness of controls' implementation and should ensure that findings are promptly resolved and reported to management.[A, B, C, D, G] The quality assurance documents were questionnaires that included a checklist but did not fully evaluate the effectiveness of the controls that were performed for this project. The quality assurance documents confirmed that certain controls were completed, such as the function point analysis, however it did not note that the project manager did not use the function point estimates to assist in project management activities. Further, while the quality assurance documents noted that key milestone dates, such as moving the system into production and the implementation date, did not match, this was not reported as an issue or finding. SSA recognized that the quality assurance reviews need improvements. In this regard, SSA is in the process of improving its quality assurance review to expand the scope and depth of coverage. * Conditionally required document. Source: GAO analysis of SSA data and best practices. Notes: SSA's Office of Systems' project management directive required a "post release review" for development projects (i.e., a control document that included lessons learned). Current guidance calls for a conditionally required control document specifically named "lessons learned" that is optional for all projects. [A] SSA, Office of Systems Project Manager Orientation, May 2013. [B] SSA, Project Resource Guide, May 2013. [C] Software Engineering Institute, Capability Maturity Model® Integration for Development (CMMI-DEV), Version 1.3 (November 2010). [D] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. [E] SSA, QA2 System Release Certification Instructions (Dec. 11, 2009). [F] While SSA officials noted that the intent of the systems release certification document is not to identify defects and problems, the agency's instructions for completing the system release certification note that if the software is released with known problems, the validator should record a comment on the problems and risks associated with those problems should also be explained (SSA, QA2 System Release Certification Instructions, Dec. 11, 2009). [G] SSA, Office of Systems Procedure for Conducting a Quality Assurance Process Review (July 27, 2012). [End of table] * Internet 3441. The Internet 3441 project is a maintenance type project. The project was focused on maintaining a web application that electronically gathered SSA form 3441[Footnote 69] information from the public, and was a 4.74 work-year project that was completed in fiscal year 2010. The system development for this project was intended to, in part, provide the form's information electronically and online and was started in October 2002. The maintenance efforts began in October 2003 and continued until September 2010, when agency documentation[Footnote 70] noted it as completed. However, according to the project manager, this effort was combined with another similar maintenance project. * For this project, two control documents were required by SSA's Project Management Directive and three were conditionally required. [Footnote 71] SSA developed the two required documents, and both had limitations. Table 4 describes the two required documents, whether or not they were developed, and any limitations that we identified. Table 4: Status of Control Documents for Internet 3441: Required control document: Service request; GAO evaluation: Lack of traceability; Description: Service requests were required to be developed for the project. This control document should show that the requirements associated with new or revised software code have been appropriately authorized and tested before it is moved into production.[A, B] These requirements should be traceable to help ensure the changes made to a system over time are documented and understood to facilitate future development, enhancements, or modifications, such as any enhancements or legislative changes, among others, required for the disability function. According to CMMI, requirements and related documentation should be traceable to each other, and internal control standards state how documentation should facilitate tracing transactions through processing.[C, D] The service requests were not traceable to the project release because SSA's management system--the service request system--does not have the capability to capture the release number associated with the maintenance project. GAO evaluation: Inaccuracies; Description: Moreover, SSA's numbering of its maintenance releases was inaccurate and out of sync (2.2 to 2.4). For example, this maintenance project's service requests did not account for release number 2.3. Accounting for the number of releases is an important control to ensure different versions of system changes are accurately documented. Project officials said that they try to go in order but sometimes release numbers are missed. In addition, about 1,000 work hours charged to the project code were not accounted for in the service request records for the appropriate project. Project officials could not identify what the charges represented, but indicated that they probably supported a related project. Required control document: System release certification; GAO evaluation: Lack of traceability; Description: System release certifications were required for this project. This control document should ensure that software developed for a service request has been completed, fully tested, and is acceptable for release into production.[A,B] The document should note if the software is released with known problems and the risks associated with those problems.e It is also important that these transactions--the code released into production, the results of testing, and any associated defects--are clearly linked, documented, and traceable to ensure that requirements have been fully developed. Traceability between requirements and related testing documents is part of requirements management, which is fundamental to a controlled and disciplined engineering process.c However, the system release certifications noted that there were known problems and defects in testing, but these problems were not identified nor was their resolution traceable to the system release certification. Nonetheless, the agency's instructions for completing the system release certification note that if the software is released with known problems, the validator should record a comment on the problems, and the risks associated with those problems should also be explained.e While the project officials identified defects in a separate system, outside the system release certification system, that were stated to be related to the Internet 3441 maintenance project, the agency could not link or provide full traceability from these defects to the project-related system release certification number. Source: GAO analysis of SSA data and best practices. [A] SSA, Office of Systems Project Manager Orientation (May 2013). [B] SSA, Project Resource Guide (May 2013). [C] Software Engineering Institute, Capability Maturity Model® Integration for Development (CMMI-DEV), Version 1.3 (November 2010). [D] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. [E] SSA, QA2 System Release Certification Instructions (Dec. 11, 2009). [End of table] * Medicare Annual project. The Medicare Annual project was a cyclical project that identified up to 400,000 Medicare beneficiaries to request information to verify continued eligibility or any changes in benefits.[Footnote 72] The project was part of a larger effort to update the Medicare Part D systems.[Footnote 73] The Medicare Annual project was a 0.83 work-year project completed in fiscal year 2010.[Footnote 74] Six control documents were relevant for the Medicare Annual project-- five required and one conditionally required.[Footnote 75] SSA developed four of the six documents but did not develop two. Moreover, three of the four control documents that were developed had limitations. Table 5 describes the six documents, and whether or not they were developed and any limitations with the documents. Table 5: Status of Control Documents for Medicare Annual: Required control document: Risk management; GAO evaluation: No document developed; Description: This project was required to manage risks. IT investments need a continuous focus on interim results and successful risk management strategies to ensure that project objectives are achieved. [A, B, C] While the project manager stated that risks were in the project oversight system, he could not provide the document because that system was no longer operational. Agency officials stated the historical project status information, including risks, was not transferred to the newer project management oversight system if the project was no longer active. While SSA management noted that risks for this specific project, Medicare Annual, were identified as part of a larger effort, the agency did not provide specific examples of risks that were associated with the cyclical project or the larger effort. Required control document: Project schedule; GAO evaluation: No document developed; Description: This project was required to have a project schedule, which is an essential management document and is important to control and monitor resources within a given project.[A, B, C] However, project officials could not provide evidence that a project schedule was developed. While the project manager provided a schedule related to the larger effort this project is associated with, it was too high- level and not detailed enough to show how it related to the project. In addition, it was not clear how the milestones fit in with this project and it did not identify specific tasks as called for in best practices. While the agency indicated that cyclical projects are small and routine in nature, a project schedule for this type of project is still important to provide better information of what tasks were expected to be completed and in order to measure the effectiveness of its implementation. Required control document: Business process description*; GAO evaluation: Lack of traceability; Description: A business process description was conditionally required for this project. Among other things, business processes and associated requirements should be traceable to the scope and objectives of the project, such as legislative requirements to help ensure that the project is effectively implemented.[B, C] However, the business process description provided was not traceable to the project or its specific functionality changes and it was not located in the agency's information system that stores pertinent historical data, further hindering traceability in the requirements management process. While project officials subsequently provided a business process description that described changes required by legislation, the document did not describe what the existing Medicare redetermination selection processes were and it did not include a unique identifier, such as the project number called for in the agency's business process description guidance, to link it to the project. Further, the changes in the business process description were not clearly linked to the scope and objectives of the project and while the project manager said that it is a routine, cyclical project, without a clear and understandable business process description, the agency relies on the project manager's knowledge to interpret the information. Required control document: Project scope agreement; GAO evaluation: No limitation identified; Description: The agency developed a project scope agreement for this project and no limitation was identified. Required control document: Service request; GAO evaluation: Inaccuracies; Description: A service request was required for this project. This control document is essential to show that the requirements associated with new or revised code have been appropriately authorized and tested before it is moved into production.[A, B] According to CMMI, requirements and related documentation should be traceable to each other, and internal control standards state how documentation should facilitate tracing transactions through processing.[C, D] Initially, SSA provided service requests covering 2008 and 2009 that did not align the Medicare Annual project because the project was initiated and completed in 2010. Subsequently, SSA provided another service request that did represent work which occurred in 2010; however, it was linked to a different project number that was related to similar Medicare work. While SSA officials noted that this service request document supported the 2010 Medicare Annual effort, because the project number was inaccurate, it was unclear that control document supported the correct project. GAO evaluation: Lack of traceability; Description: In addition to the project number being incorrect, there were differences in the project name and project type. For example, the service request identified the project name as "redeterminations" instead of Medicare Annual and the project type as "software development/enhancement" instead of cyclical. While the project manager may have understood that this service request was the appropriate documentation to support the Medicare Annual cyclical project, the inaccurate project number and differences in project name and project type hinder traceability. Moreover, the service requests were not traceable to the project release because SSA's management system--the service request system--does not have the capability to capture the release number associated with the cyclical project. Required control document: System release certification; GAO evaluation: Lack of traceability; Description: For each service request, this project was required to develop a system release certification to control and manage software placed in production and includes management approval certifying that code is completed, fully tested, and validated.[A, B] Because the system release certification is linked to the service request and the service request was linked to an inaccurate project number and was associated with differing project names and project types, traceability was hindered in the system release certification control document as well. GAO evaluation: Inaccuracies; Description: As with the service request control document, the system release certifications that SSA initially provided were for prior years (2008 and 2009), not the correct year in which the project was performed, 2010. Subsequently, SSA provided a system release certification control that did represent 2010 work efforts, but contained an incorrect project number that referred to a different Medicare project. These inaccuracies and inconsistencies in project identification make it unclear what work was actually performed for what project. * Conditionally required document. Source: GAO analysis of SSA data and best practices. [A] SSA, Office of Systems Project Manager Orientation (May 2013). [B] SSA, Project Resource Guide (May 2013). [C] Software Engineering Institute, Capability Maturity Model® Integration for Development (CMMI-DEV), Version 1.3 (November 2010). [D] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. [End of table] * ECMS Release 3. ECMS Release 3 is a planning and analysis only project that was performed by SSA in fiscal year 2011. Planning and analysis only projects are defined by the agency as projects that entail planning and analysis activities to determine the feasibility of the project, from both a business and technical perspective. As such, according to project officials, this particular release focused on assessing the existing SSA business processes related to updating the master earnings file. ECMS Release 3 was a part of a larger Earnings Redesign initiative intended to improve data integrity by reconciling earnings posted to SSA's master earnings file with earnings reported to the Internal Revenue Service by fiscal year 2015. [Footnote 76] For this project, five control documents were required by SSA's Project Management Directive, and one was conditionally required.[Footnote 77] SSA developed three of the control documents but did not develop the other three. Further, the three control documents that were developed had limitations. Table 6 describes the control documents, whether or not they were developed, and any limitations. Table 6: Status of Control Documents for ECMS Release 3: Required control document: IT proposal; GAO evaluation: Lack of traceability; Description: The IT proposal provides information supporting management decisions on approving and selecting a project. SSA's guidance calls for this activity to be completed because the document includes, among other things, the proposal's business case and performance expectations.[A, B, C] While SSA developed the required IT proposal, it was not clear what the business case or performance expectations were for the planning and analysis only portion of the project and the project manager described the scope differently than the proposal, hindering traceability. Specifically, the project manager stated the goal of the effort was identifying the business process, but this was not stated in the proposal. Instead, it described the scope for the larger development effort that was to be completed by fiscal year 2015. It is important to reference the specific scope of each project so that the benefits of the project can be measured. Traceability of the project's scope to the IT proposal was further hindered because a project scope agreement was not developed that could have clearly defined the project's goals. Required control document: Risk management; GAO evaluation: No document developed; Description: No control document was developed. While SSA's systems management directive requires this control document for planning and analysis only projects, SSA officials stated that the oversight system that contained risk management documents was replaced and the agency did not transfer the historical data. However, if SSA saved individual risks for this project, it could have preserved institutional knowledge about the project. Further, risks associated with one project may also be present in others and by identifying these, the agency could have been in a better position to reduce or mitigate the threats. Required control document: Cost-benefit analysis; GAO evaluation: No document developed; Description: No control document was developed. The agency's Capital Planning and Investment Control guidance states that a cost-benefit analysis is used to inform the investment selection decision prior to project approval[C] and while the agency's project management directive required a cost-benefit analysis to be completed by the planning and analysis only project, no document was developed. The Associate Commissioner for Office of Earnings, Enumeration, and Administrative systems stated that a cost-benefit analysis could only have been completed later in the planning and analysis cycle--after the business process description was developed. The assessment provided by the cost-benefit analysis allows project managers to account for all projects' costs and explore alternative ways for achieving the objective of a project relative to the associated resources and workloads. Required control document: Project schedule; GAO evaluation: Inaccuracies; Description: Despite the importance of project scheduling in helping managers oversee projects,[A, B, C, D] the ECMS release 3 schedule was not kept up to date in the oversight systems. Further, the project schedule was inaccurate and incomplete. For example, key tasks were noted as complete, such as control (test plans); yet, according to the project manager, this was outside of the scope of the project and was in error. Further, even though the project manager told us that the project was "complete" and the business process description product produced from the planning and analyses only project was the planned scope of the work, the schedule noted the business process description as "not appropriate" in the schedule. Required control document: Business process description; GAO evaluation: Lack of traceability; Description: The business process description should describe the business processes as they currently exist or will exist and any changes to those processes should be documented and understood by stakeholders to effectively implement a project.[B, D] According to CMMI, requirements and related documentation should be traceable to each other, and internal control standards state how documentation should facilitate tracing transactions through processing.[D, E] To SSA's credit, the agency developed this control document for ECMS Release 3; however, the control document could not be fully linked, or traced, to other key project documentation, such as a project scope agreement, because that document was not developed and it could have identified which requirements and processes in the document were applicable to the planning and analysis only project. Further, while the project manager noted that this document--the business process description--was the end goal of the ECMS Release 3 planning and analysis only project, and stated that the project was "complete," project status documentation from one of the agency's key oversight systems noted the project as "withdrawn." Disparities in projects' status, such as complete versus withdrawn in key oversight systems, could hinder the agency's ability to effectively oversee projects. Required control document: Project scope agreement*; GAO evaluation: No document developed; Description: SSA did not develop a project scope agreement or project feasibility analysis for this project.[F] This control document could have clearly defined the scope of the project to include project- specific functionality, business goals, and requirements.[A, B] Without this critical information on the project scope, it is difficult to determine if the agency had met its goals. Further, the scope agreement is also important to determine missing functionalities that may not satisfy the business goals and customer needs. The project manager said that the project scope agreement was not required for the ECMS Release 3 planning and analysis only project. However, this is inconsistent with the agency's project management directive, a primary source for determining the required control documents, regardless of its classification or project type. If SSA had developed a project scope agreement, it would have helped identify the intention of the specific project which the project manager stated was the development of the business process description. Without these controls, no matter how small or large the project's scope is, managers are left to be the primary source of historical project information. Best practices call for more rigorous, disciplined, and documented approaches to help support and guide effective IT investments.[D] * Conditionally required document. Source: GAO analysis of SSA data and best practices. [A] SSA, Office of Systems Project Manager Orientation (May 2013). [B] SSA, Project Resource Guide (May 2013). [C] SSA, Capital Planning & Investment Control (May 2012). [D] Software Engineering Institute, Capability Maturity Model® Integration for Development (CMMI-DEV), Version 1.3 (November 2010). [E] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. [F] The agency guidance calls for either a project scope agreement or a similar control document, called a project feasibility analysis, to be developed. In this case, the agency did not develop either control document. [End of table] * Help Desk Activities. The Help Desk Activities is an active, ongoing National Computer Center project. As discussed earlier, this type of project is one of five major SSA categories for IT Investments. In fiscal year 2012, approximately 100 work years were devoted to the project. This project supports SSA's infrastructure and is within SSA's Office of Telecommunications and Systems Operations. Among other things, the project provides a call center to answer users' questions, ensures continued operation of the local area network, and provides hardware and software support. The Help Desk Activities also provide a support team that coordinates and resolves programmatic software system issues, such as slow response times associated with systems. * Of the 11 types of control documents we reviewed for this project, SSA's Office of Systems' Project Management Directive identified 3 that were applicable--2 required and 1 conditionally required. [Footnote 78] Of the 2 required controls, both were developed and had limitations. Table 7 describes the 2 required control documents, whether or not they were developed, and any limitations they had. Table 7: Status of Control Documents for Help Desk Activities Project: Required control document: IT proposal; GAO evaluation: Inaccuracies; Description: The IT proposal provides information supporting management decisions on approving and selecting a project. SSA's guidance calls for this activity to be completed because it includes an explanation of how a project will support SSA's strategic objectives, the business case, and an analysis of costs and benefits.[A, B, C] The IT proposal did not present the expected performance of the project. Further, the project manager stated that the proposal contained a $2 million error in estimated costs. While according to agency officials the IT proposal is not used to support the budget submissions, the information in the IT proposal is critical because it supports investment decisions and provides critical information for the SITAR process used to select investments. Required control document: Cost-benefit analysis; GAO evaluation: Lack of traceability; Description: SSA's guidance calls a cost-benefit analysis that provides a clear and descriptive analysis of the expected return on its IT investments and to determine which projects best serve and benefit the agency given limited resources[.B, C] While SSA did provide a high-level cost-benefit analysis for its data center investment, and the Help Desk Activities' project costs and benefits were stated to be part of this "rolled up" data center cost-benefit analysis, the Help Desk Activities project contributions were not traceable or identifiable in the analysis. As such, the control document lacked traceability or linkage to other project costs and benefits identified in other documentation such as the IT proposal. Without costs and benefits that are traceable to other key project documentation, such as the IT proposal, SSA lacks assurance that critical investment decision information, like cost-benefit analyses, accounts for all the project's identified costs and benefits. Source: GAO analysis of SSA data and best practices. [A] SSA, Office of Systems Project Manager Orientation (May 2013). [B] SSA, Project Resource Guide (May 2013). [C] SSA, Capital Planning & Investment Control (May 2012). [End of table] [End of section] Appendix III: Comments from the Social Security Administration: Social Security: Office of the Commissioner: Social Security Administration: Baltimore, MD 21235-0001: May 1, 2014: Ms. Valerie C. Melvin: Director, Information Management and Technology Resources Issues: United States Government Accountability Office: 441 G Street, NW: Washington, DC 20548: Dear Ms. Melvin: Thank you for the opportunity to review the draft report, "Information Technology: SSA Needs to Address Limitations in Management Controls and Human Capital Planning to Support Modernization Efforts" (GAO-14- 308). Please see our enclosed response. If you have any questions, please contact me at (4 10) 966-90 14. Your staff may contact Gary Hatcher, Senior Advisor for Records Management and Audit Liaison Staff; at (410) 965-0680. Sincerely, Signed by: Katherine Thornton: Deputy Chief of Staff: Enclosure: Comments On The Government Accountability Office Draft Report, "Information Technology: SSA Needs To Address Limitations In Management Controls And Human Capital Planning To Support Modernization Efforts" (GAO-14-308): Recommendation 1: Perform effective oversight to ensure that key management control documents for ongoing and future projects are developed, complete, accurate, and readily accessible in oversight systems to better support management, traceability, and project analysis IT investments. Response: We agree. We continue to make progress toward a more consistent and disciplined approach to managing information technology (IT) in vestments that enable us to support our future business. As stated in the draFt report, the Government Accountability Office based this recommendation on its review of a very limited, non-generalizable sample of IT projects From 2010 through 2012. While the findings are accurate for the projects reviewed (5 of the total universe of approximately 1,950 projects), in general, our existing governance structure ensures effective oversight of management controls. This governance structure includes activities performed by: the Associate Commissioner Software Engineering Process Group, Architecture Review Board, the Management Steering Committee, the PortFolio Executive Board, and our Strategic IT Assessment and Review Board. We continue to improve our governance process. For example, we have enhanced our quality assurance review process by requiring both process and product reviews of selected IT projects. We especially emphasize the criticality of the project manager maintaining complete and correct documentation as the project goes through our rigorous systems development life cycle. Recommendation 2: Assess control documents supporting the selection of investments, such as IT proposals, to ensure that they fully identify specific performance measures and base lines to gauge project success. Response: We agree. We continue to enhance our investment management process, while balancing the realized value from better performance measures with the cost of establishing the measures and maintaining the reporting systems. We will continue our work to improve in identifying performance measures. Recommendation 3: Identify future IT needs, including skills needed to support long-term goals and priorities, in the agency's updated human capital operating plan and associated analysis. Response: We agree. We will continue to work on activities in support of creating our IT Human Capital Plan that will be a comprehensive assessment of IT human capital needs, including IT succession planning. Our IT Human Capital Plan will align with our Agency Strategic Plan goals and focus on building and developing a highly- skilled, future-ready IT workforce with the right mix of experience, competencies, and skills. In addition, we will update our Skills Inventory to assess competencies and identify skill gaps through fiscal year 2016. When we complete the IT Human Capital Plan, we will add it as an addendum to the Human Capital Operating Plan. [End of section] Appendix IV: GAO Contact and Staff Acknowledgments: GAO Contact: Valerie C. Melvin, (202) 512-6304 or melvinv@gao.gov: Staff Acknowledgments: In addition to the contact named above, Christie Motley, Assistant Director; Michael Alexander; Neil Doherty; Rebecca Eyler; Torrey Hardee; David Hong; Lee McCracken; Christy Tyson; and Charles Youman made key contributions to this report. [End of section] Footnotes: [1] IT infrastructure refers to the computer hardware and software, telecommunications, data, and technology-governance components that underlie the agency's entire enterprise. [2] Software Engineering Institute, Capability Maturity Model® Integration for Development (CMMI-DEV), Version 1.3 (November 2010); GAO, Information Technology Investment Management: A Framework for Assessing and Improving Process Maturity (Supersedes AIMD-10.1.23), [hyperlink, http://www.gao.gov/products/GAO-04-394G] (Washington, D.C.: Mar. 1, 2004); and Standards for Internal Control in the Federal Government, [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1] (Washington D.C.: November 1999). [3] 42 U.S.C. Chapter 7. [4] This program was established by Title II of the Social Security Act and is sometimes referred to as "Title II." [5] There are two separate Social Security trust funds: (1) the Old- Age and Survivors Insurance Trust Fund pays retirement and survivors' benefits, and (2) the Disability Insurance Trust Fund pays disability benefits. [6] The Supplemental Security Income program was established by Title XVI of the Social Security Act and is sometimes referred to as "Title XVI." [7] For beneficiaries found eligible for disability benefits, SSA periodically conducts medical disability reviews to determine whether beneficiaries are still medically eligible for benefits. SSA also conducts reviews of beneficiaries' earnings and work activity, as appropriate, to determine if they are still financially eligible to receive disability benefits. [8] SSA conducts periodic reviews to determine if beneficiaries are still eligible to receive Supplemental Security Income payments based on the beneficiary's income, living arrangement, and other non-medical factors--these reviews are referred to as Supplemental Security Income redeterminations. [9] According to the Deputy Commissioner of the Office of Systems/CIO, these systems largely represent software applications that have been developed; however, the agency has continued to make technical and business enhancements to these applications. [10] COBOL is a business application programming language that was introduced in the 1960s. Gartner has reported that organizations using COBOL, a third-generation language, should consider replacing the language and in 2010 noted that there should be a shift in focus to using more modern languages for new projects. Gartner added that third- generation languages' procurement and operating costs will steadily rise, and businesses should seek alternative approaches to fulfilling their needs. See Gartner, IT Market Clock for Application Development, August 2010. In another report, Gartner noted that COBOL is an aging language, with declining skill sets. See IT Modernization the Changing Technology of Batch Processing, August 2010. [11] National Research Council of the National Academies, Social Security Administration Electronic Service Provision, A Strategic Assessment, September 2007. See also, SSA, Office of Inspector General, The Social Security Administration's Software Modernization and Use of Common Business Oriented Language, A-14-11-11132 (Baltimore, Md.: May 17, 2012). [12] Forrester Research, Inc., Assessment of The Social Security Administration's Use of COBOL and Mainframes (February 2011). [13] The Master Data Access Method database is used to support the storage and retrieval of SSA's major program master files. The database was developed in the early 1980s and was written using a programming language that is no longer widely used. SSA is in the process of converting its mission support data from its Master Data Access Method file to a more modern database management system, Database 2, which is a relational database product and includes a range of application development and management tools. [14] According to the agency, the data analysis for the Master Beneficiary Record file conversion was completed in July 2013. SSA planned to complete additional process analysis by March 2014, software development by October 2015, testing by July 2016, and the conversion by the end of fiscal year 2016. The agency expects the legacy system will be run in parallel into fiscal year 2017. [15] SSA reports having a total of 4,027 IT positions, of which 624 are within other Deputy Commissioner level components--primarily the 10 regional offices. SSA reported in its IRM Strategic Plan for Fiscal Years 2012-2016 that in May 2012, 94 percent of the agency's positions were non-IT and that 6 percent were IT-related. [16] SSA manages its IT human capital needs at the agency level as well as within the Office of Systems. Specifically, SSA's Office of the Deputy Commissioner, Human Resources, is responsible for directing the administration of a comprehensive agency-wide human resource program, including personnel management. This office reports directly to the Commissioner of SSA. [17] The Clinger-Cohen Act (40 U.S.C. §§ 11301-11331) provides a framework for effective IT management that encompasses systems integration planning and investment. [18] SSA's investment management process also includes a pre-select phase in which IT project proposals are assessed to determine the degree to which the investment supports the agency's operating plan and mission. The pre-select phase concludes with the decision of whether or not to include the proposal for consideration or defer it to a future year. [19] SSA has eight portfolios: (1) Core Services supports the development of major business processes and includes online services; (2) Disability Process includes automation of the disability adjudicative process; (3) Hearings Process includes IT projects to eliminate the backlog; (4) High Performing Workforce uses IT to improve productivity, efficiency, and quality of the agency's human resource systems; (5) Program Integrity serves to ensure the privacy and security of personal information; (6) Social Security Number Process addresses efforts associated with card issuance as well as the maintenance of accurate records; (7) Cross Cutting addresses those projects that cross multiple business areas; and (8) SSA Infrastructure assures the sustained operation of current IT systems. [20] SSA stated that it also relies on its multi-year Initiative Roadmaps which are for major proposed investments and capture strategic information that goes beyond 2 years. These Initiative Roadmaps are intended to help the portfolio teams and agency executives understand the life cycle of the investments and inform SITAR board decisions. [21] SSA does not require all control documents to be included in EGADS. For example, project schedules are not required. [22] Prism replaced the Vital Signs & Observations Report (VISOR) system in August 2012. [23] Other project release type labels include support, administrative, and management. [24] According to the agency, maintenance can include adding new functionality as long as doing so does not exceed 160 hours of effort. SSA does not indicate what type of project the effort would be categorized as if it exceeds 160 hours. [25] The Office of Systems accounts for its staffs' time within these five major project types, as well as other categories, such as administrative, management, and support. [26] Of the 1,950 projects from fiscal years 2010 to 2012, 1,088 projects consumed less than 2 work years and accounted for about 630 out of the approximately 10,739 work years during that period. [27] While these projects represent major categories of development, maintenance, and the National Computer Center's operations, they do not include acquisitions of products and services such as engineering support services, network infrastructure, and mainframe capacity infrastructure. [28] The project management directive applies to Office of Systems' projects. After the project is selected and approved by the SITAR board, the project management directive is used by Office of Systems to oversee, monitor, and control projects. [29] Conditionally required control documents need to be developed by project managers if certain "conditions," as defined in the agency's project management directive, are met. [30] The number of control documents required and conditionally required varies depending on the type of project. Specifically, for development, 45; planning and analyses only, 13; cyclical, 33; maintenance, 19; and National Computer Center, 7 are required or conditionally required. The remaining artifacts, outside of the 11, are important to development of a project as they include activities such as change management, test plans, peer reviews, privacy impact assessment, security testing, and customer satisfaction indicator. [31] The project management directive identified a range of about 46 to 48 documents from 2009 to 2013--covering the time frame of the projects selected for our review. SSA first defined the control documentation requirements for each of its project types in its January 2009 Office of Systems' project management directive. [32] The 11 types of documents in this table represent what SSA required in its project management directive beginning in 2009 and accounts for changes in the guidance that have occurred since then until 2013. [33] Two of the controls--the function point estimate and quality assurance reviews--were conditionally required based on whether the project was designated as Executive Oversight. Ready Retirement Release 1 was designated as an Executive Oversight project, and as such the control documents were required and developed. [34] One of the controls--the business process description--was required if one did not already exist. If the description did exist but changes or new business processes were created, then an update to the business process description control document was required. For our selected project, the manger provided the control document, which was developed based on changes to the business process, and we included it in our review. [35] Medicare Part D provides coverage of outpatient prescription drugs to beneficiaries who choose to enroll in this optional benefit. About 60 percent of eligible Medicare beneficiaries are currently enrolled in Part D. Some beneficiaries with limited income and resources may qualify for the low-income subsidy, which provides assistance with their Part D premiums, cost-sharing, and other out-of- pocket expenses. SSA is responsible for conducting outreach efforts to identify and notify individuals of the subsidy's availability, taking applications, making subsidy eligibility determinations, resolving appeals, and ensuring continued subsidy eligibility. The law requires that SSA periodically re-determine an individual's continuing eligibility for a subsidy and the amount of the subsidy. [36] One of the controls--the project scope agreement--was conditionally required based on the condition that either a project scope agreement or a project feasibility analysis could be completed. Since the requirement was to complete one or the other and SSA did not have a project feasibility analysis for this project, we included this control in our review. [37] According to SSA, this project will help ensure that accurate wage information is posted to workers' earnings records and that the agency's employees would be able to make timely, accurate, and efficient corrections to earnings records, minimizing improper payments. [38] In fiscal year 2012, it accumulated approximately 100 of the 800 work years. The project remained active in fiscal year 2013. [39] EVM is a project management tool that integrates the technical scope of work with schedule and cost elements for investment planning and control. It compares the value of work accomplished in a given period with the value of the work expected in that period. Differences in expectations are measured in both cost and schedule variances. The Office of Management and Budget requires agencies to use EVM for major IT projects in their performance-based management systems for the parts of an investment in which development effort is required or system improvements are under way. [40] Office of the Inspector General, Social Security Administration, The Social Security Administration's Implementation of Earned Value Management Systems, A-14-06-26085 (Baltimore, Md.: Sept. 18, 2006). [41] GAO, Information Technology: SSA Has Taken Key Steps for Managing Its Investments, but Needs to Strengthen Oversight and Fully Define Policies and Procedures, [hyperlink, http://www.gao.gov/products/GAO-08-1020] (Washington, D.C.: Sept. 12, 2008). [42] GAO, Social Security Administration: Improved Planning and Performance Measures Are Needed to Help Ensure Successful Technology Modernization, [hyperlink, http://www.gao.gov/products/GAO-12-495] (Washington, D.C.: Apr. 26, 2012). [43] Office of the Inspector General, Social Security Administration, Cost Savings Planned and Achieved through the Social Security Administration's Information Technology Development Initiatives, A-14- 13-13042 (Baltimore, Md.: Oct. 1, 2013). [44] The IT Dashboard is a public website that reports performance and supporting data for major IT investments. [45] GAO, IT Dashboard: Agencies Are Managing Investment Risk, but Ratings Need to Be More Accurate and Available, [hyperlink, http://www.gao.gov/products/GAO-14-64] (Washington, D.C.: Dec. 12, 2013). [46] Software Engineering Institute, Capability Maturity Model® Integration for Development (CMMI-DEV), Version 1.3 (November 2010). [47] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. [48] CMMI-DEV and [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. [49] Software Engineering Institute, Capability Maturity Model® Integration for Development (CMMI-DEV), Version 1.3 (November 2010). [50] GAO, Auditing and Financial Management: Standards for Internal Control in the Federal Government, [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1] (Washington, D.C.: Nov. 1, 1999). [51] Of the five projects we selected, Ready Retirement Release 1 was an Executive Oversight development project--a specific designation that means the project represents the agency's highest priority and should receive additional staff, budget, and senior management oversight--and the only project that had quality assurance reviews completed. Medicare Annual was a cyclical project but was under 2 work years. Over half of the projects that SSA performed from fiscal year 2010 to 2012 were under 2 work years. [52] SSA quality assurance officials stated that National Computer Center projects will not be included in the agency's new control document audits because they require fewer controls than its other project types and are focused on its IT infrastructure. [53] [hyperlink, http://www.gao.gov/products/GAO-04-394G]. [54] SSA, Information Resources Management (IRM) Strategic Plan FY 2012-2016 (May 2012). [55] The two projects that were not required to develop an IT proposal were Internet 3441 and Medicare Annual. According to the agency's project management directive, the requirement for this control document had not been in place for the selected maintenance and cyclical project at the time that the projects were performed; however, in January 2010, the agency changed its policy to require an IT proposal to set performance baselines and establish proposed benefits to be delivered by each investment. [56] GAO, A Model of Strategic Human Capital Management, [hyperlink, http://www.gao.gov/products/GAO-02-373SP] (Washington, D.C.: Mar. 15, 2002); Human Capital: Key Principles for Effective Strategic Workforce Planning, [hyperlink, http://www.gao.gov/products/GAO-04-39] (Washington, D.C.: Dec. 11, 2003); and Information Technology: FDA Needs to Establish Key Plans and Processes for Guiding Systems Modernization Efforts, [hyperlink, http://www.gao.gov/products/GAO-09-523] (Washington, D.C.: June 2, 2009). [57] [hyperlink, http://www.gao.gov/products/GAO-09-523]. [58] In 2002, the Office of Personnel Management released a Human Capital Assessment and Accountability Framework that identifies five human capital systems that together provide a consistent, comprehensive representation of human capital management for the federal government. [59] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. [60] COBOL is a business application programming language that has enabled SSA to support large transaction volumes and meet complex regulatory, benefit, and reporting requirements. Java is a programming language used especially to create interactive applications running over the Internet. [61] These domains are data management; software/applications; business intelligence; big data; computing platforms; network infrastructure; IT operations/data centers; information security; information dissemination, privacy, and disclosure; shared services; and diversity, inclusion, and management of Section 508 of the Rehabilitation Act. [62] GAO, Social Security Administration: Long-Term Strategy Needed to Address Key Management Challenges, [hyperlink, http://www.gao.gov/products/GAO-13-459] (Washington, D.C.: May 29, 2013). [63] These controls were (1) IT proposal, (2) risk management, (3) cost-benefit analysis, (4) project schedule, (5) business process description, (6) project scope agreement, (7) service requests, (8) system release certifications, (9) function point estimates, (10) lessons learned, and (11) quality assurance. [64] Software Engineering Institute, Capability Maturity Model® Integration for Development (CMMI-DEV), Version 1.3 (November 2010); [hyperlink, http://www.gao.gov/products/GAO-04-394G]; and [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. [65] [hyperlink, http://www.gao.gov/products/GAO-02-373SP]; [hyperlink, http://www.gao.gov/products/GAO-04-39]; [hyperlink, http://www.gao.gov/products/GAO-09-523]; [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. [66] The Medicare Only update for Ready Retirement Release 1 allows benefit applicants to apply for Medicare only benefits. Month of Election allows benefit applicants to decide when to start receiving benefits. The Month of Election update for Ready Retirement Release 1 provided more accurate Month of Election options. [67] SSA's goal for the Ready Retirement initiative was to support the agency in handling the millions of baby boomers who are becoming eligible for benefits. Ready Retirement was to enable cases handled by the field to be processed more efficiently and effectively and increase use of Internet services. [68] SSA's Office of Systems Project Management Directive "conditionally" requires selected control documents. For example, in this case, function point estimate and quality assurance reviews were conditionally required for development projects. The condition was whether the project was designated as Executive Oversight then the control document was required. Ready Retirement Release 1 was designated as an Executive Oversight project, and as such the control documents were required. [69] The SSA Form 3441 is used by SSA clients to appeal a disability case. [70] Oversight system documentation and Resource Accounting System documentation noted that the project was completed in September 2010. However, the project manager requested in April 2010 that the project be merged with a similar maintenance effort, without further explanation. [71] The three documents conditionally required by SSA's project management directive were risk management (based on management judgment), project schedule (based on the functionality involved for maintenance projects), and a business process description (which should be completed or updated if a new business process is created). SSA did not develop these control documents for the project and did not document this decision. [72] The project was to select a portion of all subsidy-eligible individuals for an eligibility redetermination based on profiling to select cases more likely to have a change in the subsidy amount, for example, a couple with different filing dates or different subsidy amounts. [73] Medicare Part D provides coverage of outpatient prescription drugs to beneficiaries who choose to enroll in this optional benefit. About 60 percent of eligible Medicare beneficiaries are currently enrolled in Part D. Some beneficiaries with limited income and resources may qualify for the low-income subsidy, which provides assistance with their Part D premiums, cost-sharing, and other out-of- pocket expenses. SSA is responsible for conducting outreach efforts to identify and notify individuals of the subsidy's availability, taking applications, making subsidy eligibility determinations, resolving appeals, and ensuring continued subsidy eligibility. The law requires that SSA periodically re-determine an individual's continuing eligibility for a subsidy and the amount of the subsidy. A change in income, resources, and/or household composition can affect an individual's eligibility for the subsidy and the amount of the subsidy. [74] While the cyclical project was noted as completed in 2010, the project was, according to project officials, folded into an overall maintenance project in 2011 and 2012, and in 2013, the annual activities were then established again as a cyclical project. The agency officials said the change back to a cyclical project was to provide more effective oversight for the time employees were spending on the project. [75] The business process description control document was a conditional requirement. The condition to complete the document is based on whether a business process description already exists. If one does not already exist, it should be developed. If it does exist but changes or new business processes are created, then an update to the business process description control document is required. Additionally, Medicare Annual had two conditionally required documents- -a function point estimate and quality assurance reviews--that were not completed because they did not meet the condition of being an Executive Oversight cyclical project specified in SSA's project management directive. [76] Currently, SSA's overall Earnings program accepts both electronic and paper wage data and posts those records to the master earnings file. The reconciliation process is the result of comparing employer earnings report data processed by SSA to employer's tax report data processed by the Internal Revenue Service, by Employer Identification Number and Tax Year, to determine if the reports agree. The earnings reconciliation program was established to ensure that employers submit correct wage and tax withholding information to both agencies so that employees' Social Security records are properly credited, and the proper tax withholding is collected from employers. [77] SSA's project management directive required the planning and analysis only project, ECMS Release 3, to complete a project scope agreement or its alterative, a project feasibility analysis, and SSA did not complete either document. [78] SSA's project management directive conditionally required the National Computer Center project, Help Desk Activities, to complete risk management based on management's judgment, and SSA did not develop this control document or document this decision. [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's website [hyperlink, http://www.gao.gov]. Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Phone: The price of each GAO publication reflects GAO's actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO's website, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO: Connect with GAO on facebook, flickr, twitter, and YouTube. Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts. Visit GAO on the web at [hyperlink, http://www.gao.gov]. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; E-mail: fraudnet@gao.gov; Automated answering system: (800) 424-5454 or (202) 512-7470. Congressional Relations: Katherine Siggerud, Managing Director, siggerudk@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, DC 20548. Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, DC 20548. [End of document]