Recommendation: We recommend that the Secretary of the Navy categorize and report the Navy's software sustainment costs, in accordance with DOD policy on the Depot Maintenance Core Capabilities Determination Process. [Recommendation 1]

Agency: Department of Defense: Department of the Navy
Status: Open

Comments: The PAO reported that OSD has established a cross-Service team to develop a framework for accounting and reporting core requirements and capabilities specific to weapon system software that recognizes Service-unique attributes, and is applicable across all of DoD. When developed, the framework will clarify how to identify software sustainment costs, and data sources in accordance with DOD policy. When we confirm what additional actions DOD has taken in response to this recommendation, we will provide updated information.

Recommendation: We recommend that the Secretary of Defense ensure that the Director for Cost Assessment and Program Evaluation complete its evaluation and select the most effective system to obtain cost and software data from DOD software centers, and develop an implementation plan that includes time frames for key milestones to execute and monitor the centers' submission of required data. [Recommendation 2]

Agency: Department of Defense
Status: Open

Comments: PAO reported contacting the office of the Director for Cost Assessment and Program Evaluation and provided a template for reporting progress implementing recommendations 2 and 4 with a suspense of 31 July 2020. When we confirm what actions DOD has taken in response to this recommendation, we will provide updated information.

Recommendation: We recommend that the Secretary of Defense ensure that the Director for Cost Assessment and Program Evaluation takes steps to prioritize the respective military departments' obtaining and reporting of complete operating and support costs for software sustainment through its Visibility and Management of Operating and Support Costs systems. [Recommendation 3]

Agency: Department of Defense
Status: Open

Comments: PAO reported contacting the office of the Director for Cost Assessment and Program Evaluation and provided a template for reporting progress implementing this recommendation with a suspense of 31 July 2020. When we confirm what actions DOD has taken in response to this recommendation, we will provide updated information.

Recommendation: To help DOD address key risks to F-35 affordability and operational readiness, and to improve the reliability of its O&S cost estimates for the life cycle of the program, the Secretary of Defense should direct the F-35 Program Executive Officer, to enable DOD to better identify, address, and mitigate performance issues with the Autonomic Logistics Information System (ALIS) that could have an effect on affordability, as well as readiness, to establish a performance-measurement process for ALIS that includes, but is not limited to, performance metrics and targets that (1) are based on intended behavior of the system in actual operations and (2) tie system performance to user requirements.

Agency: Department of Defense
Status: Open
Priority recommendation

Comments: DOD concurred with our recommendation. According to DOD officials, as of July 2018, plans are emerging between the services and the Joint Program Office on a path forward for ALIS, focusing on both the current iteration of ALIS and the future state. Going forward, the services and the Joint Program Office are developing plans for the necessary re-architecture of ALIS. Once these current improvements and future requirements are finalized, appropriate performance metrics, tying system performance in operations environments to user requirements, will be incorporated. As of January 2020, DOD officials stated that there was no update to this status. Although DOD has a way ahead as it relates to developing performance metrics for ALIS, DOD has yet to develop any metrics that are based on intended behavior of the system and tie system performance to user requirements. Until DOD takes this action, our recommendation will remain open.

Recommendation: To help DOD address key risks to F-35 affordability and operational readiness, and to improve the reliability of its O&S cost estimates for the life cycle of the program, the Secretary of Defense should direct the F-35 Program Executive Officer, to develop a high level of confidence that the aircraft will achieve its R+M goals, to develop a software reliability and maintainability (R+M) assessment process, with metrics, by which the program can monitor and determine the effect that software issues may have on overall F-35 R+M issues.

Agency: Department of Defense
Status: Open

Comments: DOD concurred with our recommendation. According to DOD officials, as of July 2018, the department and the Joint Program Office, as part of their focus on agile software development, are working to incorporate software reliability and maintainability metrics into future software development and sustainment contracts. Some of the proposed metrics under consideration include: change failure rate; number of errors in developmental/user/operational testing; time to fix on critical errors; and mean time to restore. As of September 2019, DOD officials stated that there was no update to this status. Although attention is being paid to software Reliability & Maintainability, until DOD develops a process focused on software and its effects on overall Reliability & Maintainability issues, this recommendation will remain open.

Recommendation: To help DOD address key risks to F-35 affordability and operational readiness, and to improve the reliability of its O&S cost estimates for the life cycle of the program, the Secretary of Defense should direct the F-35 Program Executive Officer, to promote competition, address affordability, and inform its overarching sustainment strategy, to develop a long-term Intellectual Property (IP) Strategy to include, but not be limited to, the identification of (1) current levels of technical data rights ownership by the federal government and (2) all critical technical data needs and their associated costs.

Agency: Department of Defense
Status: Open
Priority recommendation

Comments: DOD concurred with our recommendation. According to DOD officials, as of January 2020, in the updated F-35 Life Cycle Sustainment Plan (LCSP) issued in January 2019, "Secure Use of Appropriate Technical Data" was one of the identified elements of success necessary to improve F-35 readiness and reduce sustainment costs. As part of the ongoing Plans of Action & Milestones (POAM) implementation process for the LCSP success elements, the F-35 Joint Program Office is working with the OEMs to determine the data rights the government already has, and to determine the specific technical data the government needs, and what it needs that data to accomplish. Significant progress has been made on both fronts with the prime contractor. We acknowledge that progress surrounding technical data rights is being made; however, until an Intellectual Property strategy is developed and released, this recommendation will remain open.

Recommendation: To provide the Congress with the necessary insight into program planning and affordability, and to decrease the risk of cost and schedule overruns, NASA's Administrator should direct the Human Exploration and Operations Mission Directorate to take the following action: To allow for a continued assessment of progress and affordability, NASA should structure each future increment of SLS capability with a total cost exceeding the $250 million threshold for designation as a major project as a separate development effort within the SLS program. In doing so, NASA should require each increment to complete both the technical and programmatic reviews required of other major development projects, per the agency's acquisition and system engineering policies.

Agency: National Aeronautics and Space Administration
Status: Open
Priority recommendation

Comments: NASA agreed with this recommendation. In January 2020, NASA stated that it planned to establish an agency baseline commitment for capability upgrades (e.g., Block 1B upgrades, such as Mobile Launcher-2 and Exploration Upper Stage) above the $250 million threshold. A joint confidence level analysis will be performed at key decision points and will include the cost and schedule range estimates for each of these upgrades. To fully implement this recommendation, however, NASA needs to provide evidence that each capability upgrade is designated a major project and is required to complete the technical and programmatic reviews required of other major development projects.

Recommendation: To provide the Congress with the necessary insight into program planning and affordability, and to decrease the risk of cost and schedule overruns, NASA's Administrator should direct the Human Exploration and Operations Mission Directorate to take the following action: Provide decision makers with an informed basis for making investment decisions regarding the SLS program, NASA should identify a range of possible missions for each future SLS variant that includes cost and schedule estimates and plans for how those possible missions would fit within NASA's funding profile.

Agency: National Aeronautics and Space Administration
Status: Open
Priority recommendation

Comments: NASA agreed with this recommendation. In January 2020, NASA stated that new leadership at the Human Exploration and Operations Mission Directorate is conducting an internal independent assessment to assess the current schedule and technical approach for achieving a lunar landing by 2024, including the utilization of SLS for Artemis missions. With the insights gained from this assessment, NASA will move forward with planning and executing these missions through the annual budgeting process. To fully address this recommendation, NASA will need to identify cost and schedule estimates for possible SLS missions beyond its first exploration mission, now known as Artemis I, and how its planned missions would fit within NASA's funding profile.

Recommendation: To ensure effective management and modernization of HUD's IT environment, the Secretary of Housing and Urban Development should direct the department's Chief Information Officer to define the scope, implementation strategy, and schedule of its overall modernization approach, with related goals and measures for effectively overseeing the effort.

Agency: Department of Housing and Urban Development
Status: Open
Priority recommendation

Comments: In July 2018, the Office of the Chief Information Officer (OCIO) reported that the goal of the Chief Technology Officer's technical assessment of HUD's IT environment was to identify gaps and develop an implementation strategy and approach to establish a modernization roadmap. As of March 2020, OCIO reported that it had completed the technical assessment to identify gaps in IT. The department has also taken action to define an overall modernization approach, including the scope, implementation strategy, and schedule for modernizing its IT environment and systems. However, as of March 2020, HUD had not yet established measures for overseeing its modernization efforts.