This is the accessible text file for GAO report number GAO-14-152 entitled 'DOD Business Systems Modernization: Air Force Business System Schedule and Cost Estimates' which was released on March 10, 2014. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States Government Accountability Office: GAO: Report to Congressional Requesters: February 2014: DOD Business Systems Modernization: Air Force Business System Schedule and Cost Estimates: GAO-14-152: GAO Highlights: Highlights of GAO-14-152, a report to congressional requesters. Why GAO Did This Study: The Department of Defense (DOD) has stated that the development and implementation of DEAMS is critical to the department's goal of producing auditable financial statements by September 2017. In October 2010, GAO reported that although the Air Force had developed a cost estimate that met best practices, it had not developed a schedule that met best practices for implementing DEAMS. GAO has published guides that identify the characteristics and associated best practices for developing reliable schedule and cost estimates. GAO was asked to review the schedule and cost estimates for selected DOD systems. This report addresses the extent to which the current schedule and cost estimates for DEAMS were prepared in accordance with GAO's Schedule and Cost Guides. Specifically, GAO's review focused on the schedule and cost estimates that supported DOD's February 2012 Milestone B decision, which determined that investment in DEAMS was justified. GAO assessed the schedule and cost estimates and supporting documentation. GAO also assessed an updated schedule dated October 2012. GAO interviewed DEAMS program officials, lead schedulers, and cost estimators. What GAO Found: The Air Force's schedule that supported the February 2012 Milestone B decision for the Defense Enterprise Accounting and Management System (DEAMS) did not meet best practices. The cost estimate did meet best practices, but the issues associated with the schedule could negatively affect the cost estimate. GAO found that the schedule supporting the Air Force's decision to invest in DEAMS partially or minimally met the four characteristics for developing a high-quality and reliable schedule. For example, the schedule did not reflect all government and contractor activities, and resources were not assigned to specific activities. The schedule also lacked a valid critical path, preventing management from focusing on the activities most likely to cause critical program delays if they are not completed as planned. In addition, a schedule risk analysis was not conducted to predict a level of confidence in meeting the program's completion date. Table: Extent DEAMS Schedule Met Best Practices: Characteristic: Comprehensive; Assessment: Partially met. Characteristic: Well-constructed; Assessment: Partially met. Characteristic: Credible; Assessment: Minimally met. Characteristic: Controlled; Assessment: Partially met. Source: GAO analysis based on information provided by the Air Force. [End of table] GAO found that the October 2012 updated schedule estimate was not comprehensive, well-constructed, and credible, and contained weaknesses similar to those found in the previous schedule. In May 2013, program officials provided a third schedule that they said contained some improvements but acknowledged that issues remained that prevented the schedule from meeting best practices. GAO found that the DEAMS cost estimate fully or substantially met the four characteristics of a high-quality and reliable cost estimate. For example, the cost estimate included both government and contractor costs for the program over its life cycle and provided for an independent assessment and reconciliation. Because the cost estimate relies on dates derived from the schedule and GAO is questioning the reliability of the schedule, the credibility of the cost estimate could be affected. Characteristic: Comprehensive; Assessment: Fully met. Characteristic: Well-documented; Assessment: Substantially met. Characteristic: Accurate; Assessment: Substantially met. Characteristic: Credible; Assessment: Substantially met. Source: GAO analysis based on information provided by the Air Force. [End of table] What GAO Recommends: GAO recommends that the Secretary of the Air Force update the cost estimate as necessary after implementing GAO's prior recommendation to adopt scheduling best practices. DOD concurred with the recommendation. View [hyperlink, http://www.gao.gov/products/GAO-14-152]. For more information, contact Asif A. Khan at (202) 512-9869 or khana@gao.gov and Nabajyoti Barkakati at (202) 512-4499 or barkakatin@gao.gov. [End of section] Contents: Letter: Scope and Methodology: Background: DEAMS Program Schedule Did Not Meet Best Practices, but the Cost Estimate Did Meet Best Practices: Conclusions: Recommendation for Executive Action: Agency Comments and Our Evaluation: Appendix I: Assessment of the Defense Enterprise Accounting and Management System Program's Schedule Estimate: Appendix II: Assessment of the Defense Enterprise Accounting and Management System Program's Cost Estimate: Appendix III: Comments from the Department of Defense: Appendix IV: GAO Contacts and Staff Acknowledgments: Tables: Table 1: Extent DEAMS Schedule Met Best Practices: Table 2: Extent DEAMS Cost Estimate Met Best Practices: Table 3: Assessment of DEAMS Schedule Estimate Compared to Best Practices: Table 4: Assessment of DEAMS Cost Estimate Compared to Best Practices: Abbreviations: DEAMS: Defense Enterprise Accounting and Management System: DOD: Department of Defense: ERP: enterprise resource planning: [End of section] United States Government Accountability Office: GAO: 441 G St. N.W. Washington, DC 20548: February 7, 2014: Congressional Requesters: The Department of Defense (DOD) invests billions of dollars to develop and implement enterprise resource planning (ERP) systems, which it considers critical to transforming the department's business operations and addressing some of its long-standing weaknesses, including those related to financial management and business systems modernization.[Footnote 1] DOD has stated that the development and implementation of the Air Force's Defense Enterprise Accounting and Management System (DEAMS) is critical to the department's goal of producing auditable financial statements by September 2017,[Footnote 2] as called for by the National Defense Authorization Act for Fiscal Year 2010.[Footnote 3] In October 2010, we reported that although the Air Force met best practices in developing a cost estimate, it did not meet best practices in developing the schedule estimate for implementing DEAMS. [Footnote 4] Having such a schedule is crucial to the Air Force's ability to reliably estimate the program completion date. To support Congress's continuing oversight of DOD's progress in implementing its ERP systems, you asked us to review the schedule and cost estimates for selected DOD ERP systems. The objective of this review was to determine the extent to which the current schedule and cost estimates for DEAMS were prepared in accordance with GAO's Schedule and Cost Guides.[Footnote 5] We reviewed the most current schedule and cost estimates that supported DOD's February 2012 Milestone B decision, which determined that investment in DEAMS was justified.[Footnote 6] Scope and Methodology: We assessed the DEAMS schedule[Footnote 7] using the GAO Schedule Guide to determine whether it was comprehensive, well-constructed, credible, and controlled.[Footnote 8] To assess the schedule, we obtained and reviewed documentation, including the integrated master plan and work breakdown structure.[Footnote 9] In assessing the program's cost estimate, we used the GAO Cost Guide to evaluate the DEAMS Program Management Office's estimating methodologies, assumptions, and results to determine whether the cost estimate was comprehensive, well-documented, accurate, and credible. We obtained and reviewed documentation, including the program office estimate, software cost model, independent cost estimate, and risk and uncertainty analysis.[Footnote 10] We also interviewed key program officials, such as the Program Manager, lead schedulers, and cost estimators, to obtain information, such as explanations to resolve identified discrepancies. After we briefed DEAMS program officials on the results of our assessment, they provided an updated schedule dated October 2012. For this updated schedule, we determined the extent to which it met certain best practices for the comprehensive, well-constructed, and credible characteristics, because not implementing these best practices would affect the reliability of the entire schedule. In May 2013, program management officials provided another updated DEAMS schedule, which they acknowledged contained issues that prevented the schedule from meeting best practices. Although we did not independently assess the May 2013 schedule, we did confirm that it included certain information needed for long-term planning. We conducted this performance audit from May 2012 to February 2014 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Background: DEAMS was initiated in August 2003 and is intended to provide the Air Force with the entire spectrum of financial management capabilities, including collections, commitments and obligations, cost accounting, general ledger, funds control, receipt and acceptance, accounts payable and disbursement, billing, and financial reporting for the general fund.[Footnote 11] In February 2012, the DOD Deputy Chief Management Officer granted Milestone B approval for DEAMS to enter the Engineering Development Phase of the acquisition life cycle, which is considered the official start of the program. DEAMS program functionality is intended to be implemented across the Air Force in a series of releases in two increments--Increment 1 will include six releases and Increment 2 will include two releases.[Footnote 12] DOD has approved the funding for the Air Force to proceed with the acquisition of the functionality for the first increment of DEAMS. This funding is approximately $1.6 billion, with deployment scheduled to occur during the fourth quarter of fiscal year 2016. The Air Force reported that it had spent about $427.5 million as of September 30, 2013, on the program. As stated earlier, in October 2010, we reported that although the Air Force met best practices in developing a cost estimate, it did not meet best practices in developing the schedule estimate for implementing DEAMS.[Footnote 13] In particular, the Air Force had not developed a fully integrated master schedule that reflected all government and contractor activities. We recommended that the Air Force develop an integrated master schedule that fully incorporated best practices, such as capturing all activities, sequencing all activities, integrating activities horizontally and vertically,[Footnote 14] establishing the critical path for all activities, identifying float between activities,[Footnote 15] conducting a schedule risk analysis, and updating the schedule using logic and durations to determine dates. DOD concurred with our recommendation, and we discuss later in this report the status of DOD's efforts to address this recommendation. GAO's Cost and Schedule Guides: In March 2009, we published the Cost Guide to address a gap in federal guidance about processes, procedures, and practices needed for ensuring reliable cost estimates.[Footnote 16] The Cost Guide provides a consistent methodology based on best practices that can be used across the federal government to develop, manage, and evaluate capital program cost estimates. The methodology is a compilation of characteristics and associated best practices that federal cost estimating organizations and industry use to develop and maintain reliable cost estimates throughout the life of an acquisition program. In May 2012, we issued the Schedule Guide as a companion to the Cost Guide.[Footnote 17] A consistent methodology for developing, managing, and evaluating capital program cost estimates includes the concept of scheduling the necessary work to a timeline, as discussed in the Cost Guide. Simply put, schedule variances are usually followed by cost variances. Because some program costs, such as labor, supervision, rented equipment, and facilities, cost more if the program takes longer, a reliable schedule can contribute to an understanding of the cost impact if the program does not finish on time. In addition, management tends to respond to schedule delays by adding more resources or authorizing overtime. Further, a schedule risk analysis allows for program management to account for the cost effects of schedule slippage when developing the life-cycle cost estimate. A cost estimate cannot be considered fully credible if it does not account for the cost effects of schedule slippage. A well-planned schedule is a fundamental management tool that can help government programs use public funds effectively by specifying when work will be performed in the future and measuring program performance against an approved plan. Moreover, as a model of time, an integrated and reliable schedule can show when major events are expected to occur as well as the completion dates for all activities leading up to them, which can help determine if the program's parameters are realistic and achievable. A program's success depends in part on the quality of its schedule. DEAMS Program Schedule Did Not Meet Best Practices, but the Cost Estimate Did Meet Best Practices: We found that the schedule for the DEAMS program did not meet best practices. The cost estimate did meet best practices, but the issues associated with the schedule could negatively affect the cost estimate. Specifically, the DEAMS schedule supporting the February 2012 Milestone B decision partially or minimally met the four characteristics for developing a high-quality and reliable schedule-- it was not comprehensive, well-constructed, credible, or controlled. In addition, our assessment of the October 2012 updated schedule found that it was not comprehensive, well-constructed, and credible and thus was also not reliable.[Footnote 18] In contrast, the DEAMS cost estimate fully or substantially met the four characteristics of a high- quality and reliable cost estimate--it was comprehensive, well- documented, accurate, and credible. However, because the cost estimate is based on the schedule, the unreliability of the schedule could affect the cost estimate. For example, if there are schedule slippages, the costs for the program could be greater than currently estimated. Program Schedule Was Not Developed in Accordance with Best Practices: Our analysis found that the DEAMS program partially met three and minimally met one of the characteristics of a reliable schedule estimate and therefore did not provide the information needed to support the February 2012 Milestone B decision (see table 1). Appendix I contains our detailed analysis of the DEAMS schedule estimate. The success of any program depends on having a reliable schedule of the program's work activities that will occur, how long they will take, and how the activities are related to one another. As such, the schedule not only provides a roadmap for systematic execution of a program, but also provides the means by which to gauge progress, identify and address potential problems, and promote accountability. Table 1: Extent DEAMS Schedule Met Best Practices: Characteristic: Comprehensive; Assessment: Partially met. Characteristic: Well-constructed; Assessment: Partially met. Characteristic: Credible; Assessment: Minimally met. Characteristic: Controlled; Assessment: Partially met. Source: GAO analysis based on information provided by the Air Force. Note: GAO's methodology includes five levels of compliance with its best practices. "Not met" means the program provided no evidence that satisfies any of the criterion. "Minimally met" means the program provided evidence that satisfies a small portion of the criterion. "Partially met" means the program provided evidence that satisfies about half of the criterion. "Substantially met" means the program provided evidence that satisfies a large portion of the criterion. "Fully met" means the program provided evidence that completely satisfies the criterion. [End of table] Comprehensive. A schedule should reflect all activities as defined in the program's work breakdown structure, including activities to be performed by the government and the contractor; the resources (e.g., labor, materials, and overhead) needed to do the work; and how long each activity will take. We found that the schedule used to support the Milestone B decision included the activities to be performed by both the government and contractor for Releases 1 through 3 of Increment 1. However, the schedule did not reflect activities to be performed for Releases 4 through 6 of Increment 1 and for Releases 1 and 2 of Increment 2. The DEAMS Program Manager stated that a comprehensive schedule for Increment 1 that included the activities for all six releases would not be completed until mid-2014. The Program Manager also stated that Increment 2 had not been included because program officials did not know the detailed activities to be performed that far in advance. To address this issue, the DEAMS program office developed a roadmap depicting Releases 1 through 6 of Increment 1 and Releases 1 and 2 of Increment 2 with a full deployment date of fiscal year 2017.[Footnote 19] However, the program office did not provide a schedule that supported the estimated dates in the roadmap. A comprehensive schedule should reflect all of a program's activities and recognize that uncertainties and unknown factors in schedule estimates can stem from, among other things, data limitations. As such, a schedule incorporates different levels of detail depending on the information available at any point in time. That is, near-term effort will be planned in greater detail than long-term effort. Effort beyond the near term that is less well defined is represented within the schedule as long-term planning packages. Planning packages are a summarization of the work to be performed in the distant future with less specificity. Planning packages are planned at higher levels such that a single activity may represent several months of effort, generic work to be accomplished, or even a future contract or phase. Planning packages can be used as long as they are defined and estimated as well as possible. By not including all work for all deliverables for both increments and all releases, the DEAMS program could incur difficulties resulting from an incomplete understanding of the plan and what constitutes a successful conclusion for the program. DEAMS program officials provided a draft of the Schedule Management Plan that documented their intent to use a planning package approach when updating the DEAMS schedule in the future. Resources were identified in the schedule; however, the resources were not assigned to specific activities in the schedule. Although our analysis determined that activity durations were manageable and reasonably estimated, resource availability affects estimates of work and its duration, as well as resources that will be available for subsequent activities. DEAMS program management officials told us that government resource allocations are determined by management as needed. These officials told us that management does not necessarily take into consideration the resource information captured in the schedule when determining resource allocations. However, DEAMS officials did not provide any documentation that specific resources were being mapped to the schedule. As mentioned above, the estimates of work required and duration for an activity are tied to the availability of resources; therefore, the lack of such information could hinder management's ability to compute total labor and equipment hours, calculate total project and per-period cost, resolve resource conflicts, and establish the reasonableness of the plan. Well-constructed. A schedule should be planned so that critical project dates can be met. To meet this objective, all activities should be logically sequenced--that is, listed in the order in which they are to be carried out. In particular, activities that must finish prior to the start of other activities (i.e., predecessor activities), as well as activities that cannot begin until other activities are completed (i.e., successor activities), should be identified and their relationships established. The establishment of a critical path is necessary for examining the effects of any activity slipping along this path. The calculation of a critical path determines which activities drive the project's earliest completion date. The schedule should also identify total float so that the schedule's flexibility can be accurately determined. We found that the majority of logic used to sequence the activities within the schedule was generally error-free with a minimal use of lags,[Footnote 20] clearly indicating to program management the order of activities that must be accomplished. Although we found few missing logic relationships for Release 3 of Increment 1,[Footnote 21] approximately 25 percent of the remaining activities for Releases 1 and 2 of Increment 1 were missing logic relationships. Because interdependencies among activities were not identified, the DEAMS program management officials' ability to properly calculate dates and predict changes in the future is impaired. We found a significant number of constraints for activities throughout the schedule.[Footnote 22] A schedule is intended to be a dynamic, proactive planning and risk mitigation tool that models the program and can be used to track progress toward important milestones. Schedules with constrained dates can portray an artificial or unrealistic view of the project. Constraints should be minimized because they can create false dates in a schedule. Further, the schedule did not have a valid critical path and identified critical activities more by their constraints than by logic. Rather than relying on constraints, the schedule should use logic and durations in order to reflect realistic start and completion dates for activities. Successfully identifying the critical path relies on several factors, such as capturing all activities, properly sequencing activities, and assigning resources, which, as noted earlier, had not been done. Without a valid critical path, management cannot focus on activities that will have detrimental effects on the key project milestones and deliveries if they slip. We found that total float was not reasonable, and that in some instances unreasonable float was a direct result of improper sequencing or missing logic. Releases 1 and 2 of Increment 1 showed that 25 percent of program activities had total float equal to or greater than 392 working days, meaning that those activities could slip almost 2 working years and not affect the end date of the program. Without knowledge of the reason float exists for a program activity, management cannot determine the flexibility of tasks and therefore cannot properly reallocate resources from tasks that can safely slip to tasks that cannot slip without adversely affecting the estimated program completion date. Credible. A schedule should be horizontally and vertically integrated. A horizontally integrated schedule links products and outcomes with other associated sequenced activities, which helps verify that activities are arranged in the right order to achieve aggregated products or outcomes. A vertically integrated schedule ensures that the start and completion dates for activities are aligned with such dates on subsidiary schedules supporting tasks and subtasks. Such mapping or alignment among subsidiary schedules enables different groups--such as government teams and contractors--to work to the same master schedule, and provides assurance that the representation of the schedule to different audiences is consistent and accurate. A schedule risk analysis should also be performed using statistical techniques to predict the level of confidence in meeting a program's completion date. We found that Release 3 of Increment 1 exhibited horizontal integration, but Releases 1 and 2 of Increment 1 did not because date constraints prevented forecasted dates from being calculated realistically for future activities. If the schedule lacks horizontal integration, activities whose durations are greatly extended will have no effect on key milestones reflected in the schedule. We further found that Releases 1 and 2 of Increment 1 did not demonstrate vertical integration. For example, we found instances where the start dates for the same activities differed by 1 day, 1 week, and 1 month between the government and contractor schedules. Unless the schedule is vertically integrated, lower-level schedules will not be consistent with upper-level schedule milestones, affecting the integrity of the entire schedule and the ability of different teams to work to the same schedule expectations. DEAMS program management officials stated that a schedule risk analysis had not been conducted because the schedule had not been approved to be used as a baseline schedule--the target schedule against which program performance can be measured, monitored, and reported. These officials stated that although this analysis had not been conducted, they were collecting best-case and worst-case durations from the contractor with their periodic schedule delivery. These data can be used by program management to calculate more reliable estimates of durations for future activities. However, we found that the schedule did not contain best-or worst-case duration data for 600 of 605 detailed activities. For the five instances where duration data were contained in the schedule, we determined that four were questionable because two activities were already completed and two had already exceeded the worst-case estimate. If a schedule risk analysis is not conducted, program management cannot determine the likelihood of the project's completion date, how much schedule risk contingency is needed to provide an acceptable level of certainty for completion by a specific date, risks most likely to delay the project, how much contingency reserve each risk requires, and the paths or activities that are most likely to delay the project. As discussed later, the lack of a schedule risk analysis can affect the credibility of the cost estimate. Controlled. A schedule should be continuously updated using logic, durations, and actual progress to realistically forecast dates for program activities. A schedule narrative should accompany the updated schedule to provide decision makers and auditors a log of changes and their effect, if any, on the schedule time frame. The schedule should be analyzed continuously for variances to determine when forecasted completion dates differ from planned dates. This analysis is especially important for those variations that affect activities identified as being in a program's critical path and that can affect a scheduled completion date. A baseline schedule should be used to manage the program scope, the time period for accomplishing it, and the required resources. We found that DEAMS program management met weekly to discuss proposed schedule changes and updated the schedule's progress. However, a schedule narrative was not prepared by DEAMS program management. In addition, we found a number of date anomalies throughout the schedule, including activities with planned start dates scheduled to occur in the past and activities with actual finish dates scheduled to occur in the future. We also found a number of out-of-sequence activities in the schedule--activities that started before their predecessors finished, in contradiction to the planned sequence. If the schedule is not continually monitored to determine when forecasted completion dates differ from planned dates, then it cannot be used to determine whether schedule variances will affect work needed to be accomplished at a future date. We also found that there was no baseline schedule that could be used to measure program performance. DEAMS program management officials did maintain a schedule narrative document that contained a list of custom fields and assumptions; however, the document did not explain ground rules and assumptions, justifications for logic, and other unique features of the schedule. These officials stated that other process documents were being developed. Without a formally established baseline schedule to measure performance against, management cannot identify or mitigate the effect of unfavorable performance. Updated Schedule Also Did Not Follow Best Practices: Our assessment of the updated schedule dated October 2012 found that it was not comprehensive, well-constructed and credible.[Footnote 23] Although the DEAMS Program Manager stated that the government and contractor activities for Releases 1 through 3 of Increment 1 had been integrated in the October 2012 schedule, this schedule was not comprehensive. Specifically, it excluded activities for both the government and contractor related to Releases 4 through 6 of Increment 1 and Releases 1 and 2 of Increment 2. If activities are missing from the schedule, then other best practices will not be met. The schedule was also missing relationships for a significant number of the remaining milestones and activities. In addition, the October 2012 schedule included a significant number of date constraints with little or no justification for their use in the schedule. Similar to the previous schedule, the updated schedule presented unreasonable float throughout and did not include a schedule risk analysis. As a result of these shortcomings, the updated schedule was not reliable. Further, program officials could not rely on this schedule as a baseline to effectively manage and monitor program performance. In May 2013, program management officials provided another updated DEAMS schedule that they stated included some improvements, but they acknowledged that it contained issues that prevented the schedule from meeting best practices. For example, these officials stated that the May 2013 schedule included long-term planning packages for activities related to Releases 4 through 6 of Increment 1 and Releases 1 and 2 of Increment 2, integrated government and contractor activities, and reduced the number of constraints and out-of-sequence activities in the schedule. However, the officials acknowledged that several outstanding issues remained related to, for example, vertical and horizontal integration, missing logic relationships, and the lack of a schedule risk analysis. Although we did not independently assess the May 2013 schedule to determine whether it met the four schedule characteristics, we did confirm that it included long-term planning packages, which are needed to create a complete picture of the program from start to finish and to allow the monitoring of a program's critical path. The results of our analyses of the schedule that supported the February 2012 Milestone B decision and October 2012 DEAMS schedule reflect similar weaknesses to those we reported in October 2010.[Footnote 24] Therefore, given the findings of this review, our prior recommendation for improving the DEAMS schedule remains valid. Program Cost Estimate Met Best Practices: We found that the DEAMS program fully or substantially met the four characteristics of a reliable cost estimate to support the Milestone B decision, as shown in table 2. However, because the cost estimate relies on dates derived from the schedule and we are questioning the reliability of the forecasted program dates, the credibility of the cost estimate can be affected. Appendix II contains our detailed analysis of the DEAMS cost estimate. A reliable cost estimate is critical to the success of any program. Such an estimate provides the basis for informed investment decision making, realistic budget formulation and program resourcing, meaningful progress measurement, proactive course correction when warranted, and accountability for results. Table 2: Extent DEAMS Cost Estimate Met Best Practices: Characteristic: Comprehensive; Assessment: Fully met. Characteristic: Well-documented; Assessment: Substantially met. Characteristic: Accurate; Assessment: Substantially met. Characteristic: Credible; Assessment: Substantially met. Source: GAO analysis based on information provided by the Air Force. Note: GAO's methodology includes five levels of compliance with its best practices. "Not met" means the program provided no evidence that satisfies any of the criterion. "Minimally met" means the program provided evidence that satisfies a small portion of the criterion. "Partially met" means the program provided evidence that satisfies about half of the criterion. "Substantially met" means the program provided evidence that satisfies a large portion of the criterion. "Fully met" means the program provided evidence that completely satisfies the criterion. [End of table] Comprehensive. A cost estimate should include costs of the program over its full life cycle, provide a level of detail appropriate to ensure that cost elements are neither omitted nor double-counted, and document all cost-influencing ground rules and assumptions. The cost estimate should also completely define the program and be technically reasonable. We found that the cost estimate for DEAMS was comprehensive. The cost estimate included both government and contractor costs of the program over its life cycle--from the inception of the program through design, development, deployment, and operation and maintenance--as outlined in the roadmap prepared by program officials. As stated earlier, the roadmap provided an overall summary of the program's key phases (increments and releases) and the expected milestones for completion. The cost estimate also included an appropriate level of detail, which provided assurance that cost elements were neither omitted nor duplicated, and included documentation of all cost-influencing ground rules and assumptions. The cost estimate documentation included the purpose of the cost estimate, a technical description of the program, and technical risks (e.g., the resolution for any identified deficiencies). Well-documented. A cost estimate should be supported by detailed documentation that describes how it was derived and how the expected funding will be spent in order to achieve a given objective. Therefore, the documentation should capture such things as the source data used, the calculations performed, the results of the calculations, the estimating methodology used to derive each work breakdown structure element's cost, and evidence that the estimate was approved by management. We found that the cost estimate for DEAMS was well-documented. The cost estimate captured such things as the source data used, the calculations performed and the results of the calculations, and the rationale for choosing a particular estimating methodology. This information was captured in such a way that the data used to derive the estimate can be traced back to, and verified against, the sources so that the estimate can be easily replicated. However, there was no discussion of efforts taken, if any, to ensure the reliability of the data used. The DEAMS Program Management Office presented evidence of receiving approval of the estimate through briefings to management. Accurate. A cost estimate should be based on an assessment of most likely costs (adjusted properly for inflation), updated to reflect significant changes and grounded in a historical record of cost estimating and actual experiences on other comparable programs. We found that the cost estimate for DEAMS was accurate. The cost estimate provided results that were substantially unbiased, and the cost model detailed the calculations and inflation indexes underlying the estimate. Calculations within the model could be traced back to supporting documentation. The cost estimate was updated regularly to reflect significant changes in the program and updated annually to incorporate actual costs expended in prior fiscal years. Further, the cost estimate was based on historical data. However, the cost estimate did not discuss variances between planned and actual costs, which would enable estimators to assess how well they are estimating program costs and to identify lessons learned. Credible. A cost estimate should discuss any limitations of the analysis because of uncertainty or biases surrounding data or assumptions. In addition, the estimate's results should be cross- checked and reconciled to an independent cost estimate to determine whether other estimating methods produce similar results. We found that the cost estimate was credible. The DEAMS Program Management Office conducted a risk and uncertainty analysis by identifying the cost elements with the greatest degree of uncertainty, determining the cost drivers for the program, and identifying the impact of changing major ground rules and cost driver assumptions. An independent cost estimate developed by the Air Force Cost Analysis Agency was reconciled to the program's estimate. However, a sensitivity analysis was not completed for each of the major cost drivers. As a result, the cost estimator will not have a clear understanding of how each major cost driver is affected by a change in a single assumption and thus which scenario most affects the cost estimate. Further, as discussed previously, because a schedule risk analysis was not performed as required by best practices, the cost estimate does not include a contingency amount to account for any schedule slippage that could occur. To the extent that a schedule slippage does occur, there could ultimately be an impact on the cost estimate. Conclusions: The Air Force did not meet best practices in developing a schedule for the DEAMS program. As a result, this raises questions about the credibility of the deadline for acquiring and implementing DEAMS to provide needed functionality for financial improvement and audit readiness. Because of these questions, the cost estimate, while following best practices, may not fully capture all costs associated with the program, particularly if there is significant schedule slippage. Moreover, Air Force management did not have a reliable schedule estimate when making its decision to invest in the DEAMS program. It is critical to correct the deficiencies identified with the schedule estimate to help ensure that the projected spending for this program is being used in the most efficient and effective manner. Recommendation for Executive Action: To help provide for the successful implementation of DEAMS, we recommend that the Secretary of the Air Force direct the Under Secretary of the Air Force, in his capacity as the Chief Management Officer, to consider and make any necessary adjustments to the DEAMS cost estimate after addressing our prior recommendation to adopt scheduling best practices. Agency Comments: We provided a draft of this report to DOD for review and comment. In its written comments, reprinted in appendix III, DOD concurred with our recommendation. DOD also provided a technical comment, which we incorporated. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of Defense; the Secretary of the Air Force; the Assistant Secretary of Defense (Acquisition); the Deputy Chief Management Officer; the Under Secretary of Defense (Comptroller); the Under Secretary of the Air Force, in his capacity as the Chief Management Officer of the Air Force; and the Program Manager for DEAMS. In addition, the report will be available at no charge on the GAO website at [hyperlink, http://www.gao.gov]. If you or your staffs have any questions about this report, please contact Asif A. Khan at (202) 512-9869 or khana@gao.gov or Nabajyoti Barkakati at (202) 512-4499 or barkakatin@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff members who made key contributions to this report are listed in appendix IV. Signed by: Asif A. Khan: Director: Financial Management and Assurance: Signed by: Nabajyoti Barkakati: Chief Technologist: Applied Research and Methods: Center for Science, Technology, and Engineering: List of Requesters: The Honorable Thomas R. Carper: Chairman: The Honorable Tom Coburn: Ranking Member: Committee on Homeland Security and Governmental Affairs: United States Senate: The Honorable Claire McCaskill: Chairman: Subcommittee on Financial and Contracting Oversight: Committee on Homeland Security and Governmental Affairs: United States Senate: The Honorable John McCain: Ranking Member: Permanent Subcommittee on Investigations: Committee on Homeland Security and Governmental Affairs: United States Senate: [End of section] Appendix I: Assessment of the Defense Enterprise Accounting and Management System Program's Schedule Estimate: This appendix provides the results of our analysis of the extent to which the Defense Enterprise Accounting and Management System (DEAMS) schedule estimate supporting the February 2012 Milestone B decision [Footnote 25] met the characteristics of a high-quality, reliable schedule. Table 3 provides the detailed results of our analysis. GAO's methodology includes five levels of compliance with its best practices.[Footnote 26] "Not met" means the program provided no evidence that satisfies any of the criterion. "Minimally met" means the program provided evidence that satisfies a small portion of the criterion. "Partially met" means the program provided evidence that satisfies about half of the criterion. "Substantially met" means the program provided evidence that satisfies a large portion of the criterion. "Fully met" means the program provided evidence that completely satisfies the criterion. Table 3: Assessment of DEAMS Schedule Estimate Compared to Best Practices: Characteristic: Comprehensive; Best practice: The schedule should reflect all activities as defined in the project's work breakdown structure, which defines in detail the work necessary to accomplish a project's objectives, including activities to be performed by both the government and contractors. The schedule should reflect the resources (e.g., labor, materials, and overhead) needed to do the work, whether those resources will be available when needed, and whether any funding or time constraints exist. The schedule should realistically reflect how long each activity will take, allowing for discrete progress measurement with specific start and finish dates; Assessment: Partially met; Key examples of rationale for assessment: The schedule reflected the activities to be performed by both the government and the contractor for Increment 1, Releases 1 through 3, of the program. However, it did not reflect activities for Releases 4 through 6 of Increment 1 and for Releases 1 and 2 of Increment 2 of the program. To be comprehensive, the program schedule should reflect all effort necessary to successfully complete the program. In addition, the program schedule was not fully integrated. Specifically, although the schedule narrative noted that Release 2 of Increment 1 included contractor activities and that integration with government activities was in process, there were no links between the government and contractor activities for this release. For the most part, resources had been identified; however, the resources had not been assigned to specific activities in the schedule. The amount of available resources affects estimates of work and its duration and resources that will be available for subsequent activities. Information on resource needs and availability assists the program office in forecasting the likelihood that activities will be completed as scheduled. Because the current schedule did not allow insight into current or projected assignment of resources, the risk of program slippage was significantly increased; The project duration was determined by a target date and the majority of activity durations in the schedule met the standard best practice for activity duration (44 days). Specifically, 98 percent of the activities were less than 44 days. Characteristic: Well-constructed; Best practice: The schedule should be planned so that critical project dates can be met. All activities should be logically sequenced--that is, listed in the order in which they are to be carried out--with predecessor and successor logic. Date constraints and lags should be minimized and justified to help ensure that the interdependence of activities that collectively lead to the completion of events or milestones can be established and used to guide work and measure progress. The schedule should identify the program critical path--the path of longest duration through the sequence of activities--to determine the program's earliest completion date. The schedule should identify total float--the amount of time by which a predecessor activity can slip before the delay affects the program's estimated finish date--so that the schedule's flexibility can be accurately determined. As a general rule, activities along the critical path have the least amount of float; Assessment: Partially met; Key examples of rationale for assessment: The schedule was not planned to ensure that critical project dates could be met. The majority of logic used to sequence the activities within the schedule was generally error-free with a minimal use of lags--the passage of time between two activities--and leads--negative lags used to accelerate a successor activity, clearly indicating which activities must finish. Our analysis found that the schedule for Increment 1, Release 3, accounted for predecessor and successor logic on 97 percent of remaining activities and milestones. However, our analysis of the schedule for Releases 1 and 2 of Increment 1 showed that for the remaining activities and milestones 10 percent were missing predecessor links, 17 percent were missing successor links, and 2 percent were missing both. That is, approximately one-fourth of all remaining detail activities and milestones were missing some logic link. Missing predecessors or successors reduce the credibility of the calculated dates. If an activity that has no logical successor slips, the schedule will not reflect the effect on the critical path, float, or scheduled start dates of downstream activities. In addition, there were a small percentage of activities with dangling relationships in both schedules. A dangling relationship is a scheduling relationship that is not properly tied to an activity's start or end date. Each activity's start date--other than the start and finish milestones--must be driven by a predecessor activity, and each activity's finish date must drive a successor activity's start or finish. The schedule for Releases 1 and 2 of Increment 1 had 5 remaining activities whose start dates were not being driven by a predecessor activity and 11 activities that had finish dates that did not drive successor activities. The schedule for Release 3 of Increment 1 had 1 remaining activity whose start date was not being driven by a predecessor activity and 24 activities that had finish dates that did not drive successor activities. Dangling relationships, a form of incomplete relationships, can interfere with the valid forecasting of scheduled activities. Both schedules had date constraints on about one-third of the remaining activities. A customer- mandated date, including contractual obligations, does not constitute a legitimate reason to constrain an activity. A schedule is intended to be a dynamic, proactive planning and risk mitigation tool that models the program and can be used to track progress toward important milestones. Schedules with constrained dates can portray an artificial or unrealistic view of the project. Our analysis found that both schedules did not have a valid critical path. For example, there were numerous critical activities scheduled to start before the project status date had started. Successfully identifying the critical path relies on capturing all activities, properly sequencing activities, horizontal traceability, the reasonableness of float, accurate status updates, and--if there are resource limitations--assigning resources. Until the schedule can produce a true critical path, the program office will not be able to provide reliable timeline estimates or identify when problems or changes may occur and their effect on downstream work. Also, without a valid critical path, management cannot focus on activities that will have detrimental effects on the key project milestones and deliveries if they slip. Our analysis found that float calculations within the schedules were not reliable and that in some instances unreasonable float was a direct result of improper sequencing or missing logic. The schedule for Releases 1 and 2 of Increment 1 had average total float for the remaining detail and milestone activities of 239 business days. Twenty- five percent of activities had total float equal to or greater than 392 days, meaning they could slip almost 2 years and not affect the end date of the project. In addition, 77 activities had negative float, ranging from -1 to -18 days. Negative float stems from constraining one or more activities or milestones in the network. Negative float indicates critical path effort that may require management action such as overtime or resequencing of work. The constraint should be examined and justified, and the resulting negative float should be evaluated for reasonableness. The schedule for Release 3 of Increment 1 also had an average total float for the remaining detail and milestone activities of 289 business days; 25 percent of remaining activities had total float equal to or greater than 489 days. There were no activities with negative float. Without reasonable float estimates, management may be unable to identify activities that could be permitted to slip and thus release and reallocate resources to activities that require more resources to be completed on time. Characteristic: Credible; Best practice: The schedule should be horizontally integrated, meaning that it should link products and outcomes associated with other sequenced activities. These links are commonly referred to as "handoffs" and serve to verify that activities are arranged in the right order to achieve aggregated products or outcomes. The schedule should also be vertically integrated, meaning that the dates for starting and completing activities in the integrated master schedule should be aligned with the dates for supporting tasks and subtasks. Such mapping or alignment among levels enables different groups to work to the same master schedule. A schedule risk analysis should be performed using statistical techniques to predict the level of confidence in meeting a program's completion date, determine the time contingency needed for a level of confidence, and identify high-priority risks and opportunities. The analysis focuses not only on critical path activities but also on activities near the critical path, since they can affect the program's status; Assessment: Minimally met; Key examples of rationale for assessment: The schedule for Release 3 of Increment 1 exhibited horizontal traceability; however, the schedule for Releases 1 and 2 of Increment 1 did not because date constraints prevented forecasted dates from being calculated realistically for future activities. Unless the schedule is horizontally traceable, activities whose durations are greatly extended will have no effect on key milestones. Our analysis did not find vertical traceability within the schedule for Releases 1 and 2 of increment 1--the ability to consistently trace work breakdown structure activities between detailed, intermediate, and master schedules. For example, we traced three activities between the government schedule and the underlying prime contractor schedule and in each case found mismatching start dates that differed by a day, a week, and a month. Vertical traceability provides assurance that the representation of the schedule to different audiences is consistent and accurate. Unless the schedule is vertically traceable, lower-level schedules will not be consistent with upper-level schedule milestones, affecting the integrity of the entire schedule and the ability of different teams to work to the same schedule expectations. The program office had not conducted a schedule risk analysis. If a schedule risk analysis is not conducted, program management cannot determine the likelihood of meeting the program's completion date, how much schedule risk contingency is needed to provide an acceptable level of certainty for completion by a specific date, risks most likely to delay the program, how much contingency reserve each risk requires, and the paths or activities that are most likely to delay the program. Unless a statistical simulation is run, calculating the completion date from schedule logic and the most likely duration distributions will tend to underestimate the program's overall critical path duration. Characteristic: Controlled; Best practice: The schedule should be continuously updated using logic and durations to realistically forecast dates for program activities. A schedule narrative should accompany the updated schedule to provide decision makers and auditors a log of changes and their effect, if any, on the schedule timeframe. The schedule should be analyzed continuously for variances to determine when forecasted completion dates differ from planned dates. This analysis is especially important for those variations that impact activities identified as being on a program's critical path and can impact a scheduled completion date. A baseline schedule is the basis for managing the program scope, the time period for accomplishing it, and the required resources. The baseline schedule is designated the target schedule, and is subject to a configuration management control process, against which program performance can be measured, monitored, and reported. A corresponding baseline document explains the overall approach to the program; defines custom fields in the schedule file; details ground rules and assumptions used in developing the schedule; and justifies constraints, lags, long activity durations, and any other unique features of the schedule; Assessment: Partially met; Key examples of rationale for assessment: Management met weekly to discuss proposed schedule changes and update schedule progress. The program management office followed general guidelines for updating the schedule; however, a schedule narrative did not accompany the updates, and we found a significant number of date anomalies in the schedules provided for Releases 1 through 3 of Increment 1. For example, we found instances of activities with start dates in the past, activities with planned finish dates in the past, activities with actual start dates in the future, and activities with actual finish dates in the future. We also found a number of out-of-sequence activities, that is, activities that started before their predecessors finished. A schedule is a fundamental program management tool that specifies when work will be performed in the future and how well the program is performing against an approved plan. If the schedule is not updated, then it is impossible to tell what activities have been completed, are in progress, are late, and are planned to start on time. In addition, if a schedule is not updated to reflect what is actually occurring on the program, it may have inaccurate completion dates and critical paths. When this is the case, management cannot use the schedule to monitor progress and make decisions regarding risk mitigation and resource allocations. There was no baseline schedule that could be used to measure program performance. As a result, it was not possible to track variances to past baselines using the schedules provided for Increment 1, Releases 1 through 3. The program management office did maintain a baseline schedule narrative document that contained a list of custom fields and assumptions. However, this document did not explain the overall program approach, ground rules and assumptions, justifications for logic, and other unique features of the schedule. Without a formally established baseline schedule to measure performance against, management cannot identify or mitigate the effect of unfavorable performance. A well- documented schedule is essential for validating and defending a baseline schedule, analyzing changes in the program schedule, and identifying the reasons for variances between estimates. Source: GAO analysis of Air Force data. [End of table] [End of section] Appendix II: Assessment of the Defense Enterprise Accounting and Management System Program's Cost Estimate: This appendix provides the results of our analysis of the extent to which the Defense Enterprise Accounting and Management System (DEAMS) cost estimate supporting the February 2012 Milestone B decision [Footnote 27] met the characteristics of a high-quality cost estimate. Table 4 provides the detailed results of our analysis. GAO's methodology includes five levels of compliance with its best practices.[Footnote 28] "Not met" means the program provided no evidence that satisfies any of the criterion. "Minimally met" means the program provided evidence that satisfies a small portion of the criterion. "Partially met" means the program provided evidence that satisfies about half of the criterion. "Substantially met" means the program provided evidence that satisfies a large portion of the criterion. "Fully met" means the program provided evidence that completely satisfies the criterion. Table 4: Assessment of DEAMS Cost Estimate Compared to Best Practices: Characteristic: Comprehensive; Best practice: The cost estimate includes all life-cycle costs. The cost estimate completely defines the program, reflects the current schedule, and is technically reasonable. The cost estimate work breakdown structure is product oriented, traceable to the statement of work, and at an appropriate level of detail to ensure that cost elements are neither omitted nor double-counted. The estimate documents all cost- influencing ground rules and assumptions; Assessment: Fully met; Key examples of rationale for assessment: The DEAMS cost estimate included both government and contractor costs of the program over its life cycle. The documentation provided a description of the purpose of the estimate; a technical description of the program including functional relationships; and technical risks (e.g., hours per test activity and the resolution for any identified deficiencies). The work breakdown structure clearly outlined the end product and major work to be performed. The program provided supporting documentation that showed the ground rules and assumptions, such as the total number of users, use of a commercial-off-the-shelf product, and licensing requirements. The estimate was based on a cost element structure as stated in the Department of Defense Automated Information System Economic Analysis Guide. Characteristic: Well-documented; Best practice: The documentation should capture the source data used, the reliability of the data, and how the data were normalized. The documentation describes in sufficient detail the calculations performed and the estimating methodology used to derive each work breakdown structure element's cost. The documentation describes step by step how the estimate was developed so that a cost analyst unfamiliar with the program could understand what was done and replicate it. The documentation discusses the technical baseline description and the data in the baseline are consistent with the estimate. The documentation provides evidence that the cost estimate was reviewed and accepted by management; Assessment: Substantially met; Key examples of rationale for assessment: The documentation identified the source of the data used and that the data were normalized. Major cost drivers were also identified, such as personnel costs, support costs, and system development. However, there was no discussion of efforts taken, if any, to ensure the reliability of the data used. The documentation identified the methodology used for each cost element in sufficient detail to replicate it. The document also contained the rationale for selecting the methodology. Technical descriptions in the cost model were consistent with the Cost Analysis Requirement Description, which contained a system overview outlining the incremental approach being used to develop DEAMS. The DEAMS Program Management Office presented evidence of receiving approval of the estimate through briefings to management although there was no reference in the briefing to the risk and uncertainty analysis that was performed. Characteristic: Accurate; Best practice: The cost estimate results are unbiased, not overly conservative or optimistic and based on an assessment of most likely costs. The estimate has been adjusted properly for inflation. The estimate contains few, if any, minor mistakes. The cost estimate is regularly updated to reflect significant changes in the program so that it is always reflecting current status. The estimate is based on a historical record of cost estimating and actual experiences from other comparable programs. Variances between planned and actual costs are documented, explained, and reviewed; Assessment: Substantially met; Key examples of rationale for assessment: The program provided a risk and uncertainty analysis that documented the range of all possible costs. This analysis showed that the results were unbiased and reflected the most likely costs. The DEAMS cost model detailed the calculations and inflation indexes underlying the estimated costs. Calculations within the model can be traced back to supporting documentation. The estimate did not contain any mistakes. In addition, the cost model was updated annually to incorporate actual costs expended in prior fiscal years. The estimate was based on historical data as appropriate. However, the cost estimate did not discuss variances between planned and actual costs, which would enable estimators to assess how well they are estimating program costs and identify lessons learned. Characteristic: Credible; Best practice: A risk and uncertainty analysis was conducted that quantified the imperfectly understood risks and identified the effects of changing key cost driver assumptions and factors. Major cost elements were cross-checked to see whether results were similar. An independent cost estimate was conducted by a group outside the acquiring organization to determine whether other estimating methods produce similar results. The cost estimate includes a sensitivity analysis that identifies a range of possible costs based on varying major assumptions, parameters, and data inputs; Assessment: Substantially met; Key examples of rationale for assessment: The program management office provided a risk and uncertainty analysis that quantified the risks for the major cost drivers (e.g., personnel, deployment software and integrated logistics support) based on changes in the assumptions (e.g., pending negotiations and level of knowledge about support concepts). DEAMS program officials stated that some cross-checking was performed using cost models; however, the results of this analysis were not documented. The Air Force Cost Analysis Agency developed an independent cost estimate that was reconciled to the program's cost estimate. However, a sensitivity analysis was not completed for each of the major cost drivers. As a result, the cost estimator will not have a clear understanding of how each major cost driver is affected by a change in a single assumption and thus which scenario most affects the cost estimate. Source: GAO analysis of Air Force data. [End of table] [End of section] Appendix III: Comments from the Department of Defense: Assistant Secretary of Defense: Acquisition: 3015 Defense Pentagon: Washington, DC 20301-3015: January 16, 2014: Mr. Asif A. Khan: Director: Financial Management Assurance: U.S. Government Accountability Office: 441 G Street, N.W. Washington, DC 20548: Dear Mr. Khan: This is the Department of Defense (DoD) response to the Government Accounting Office (GAO) Draft Report, GAO-14-152, "DoD Business System Modernization: Air Force Business System Schedule and Cost Estimates" dated November 26, 2013. The Department's comments are enclosed. Sincerely, Ms. Karrina McFarland: Assistant Secretary of Defense (Acquisition): Enclosure: As stated: GAO-14-152 (GAO Code 197120): "DOD Business Systems Modernization: Air Force Business System Schedule and Cost Estimates" Department of Defense Comments to The GAO Recommendation: Recommendation: The GAO recommends that the Secretary of the Air Force direct the Under Secretary of the Air Force, in his capacity as the Chief Management Officer, to consider an make any necessary adjustments to the DEAMS cost estimate after addressing our prior recommendation to adopt scheduling best practices. DoD Response: Concur with the GAO recommendation. [End of section] Appendix IV: GAO Contacts and Staff Acknowledgments: GAO Contacts: Asif A. Khan, (202) 512-9869 or khana@gao.gov: Nabajyoti Barkakati, (202) 512-4499 or barkakatin@gao.gov: Staff Acknowledgments: In addition to the contacts named above, Cynthia Jackson (Director), Karen Richey (Assistant Director), Beatrice Alff, Jennifer Echard, Patrick Frey, and Jason Lee made key contributions to this report. [End of section] Footnotes: [1] An ERP system is an automated system using commercial off-the- shelf software consisting of multiple, integrated functional modules that perform a variety of business-related tasks, such as general ledger accounting, payroll, and supply chain management. [2] For a complete list of the ERP systems that DOD has identified as essential to transforming its business operations, see GAO, DOD Financial Management: Reported Status of Department of Defense's Enterprise Resource Planning Systems, [hyperlink, http://www.gao.gov/products/GAO-12-565R] (Washington, D.C.: Mar. 30, 2012). [3] Pub. L. No. 111-84, § 1003(a), (b), 123 Stat. 2190, 2439-40 (Oct. 28, 2009). [4] GAO, DOD Business Transformation: Improved Management Oversight of Business System Modernization Efforts Needed, [hyperlink, http://www.gao.gov/products/GAO-11-53] (Washington, D.C.: Oct. 7, 2010). [5] GAO, GAO Schedule Assessment Guide: Best Practices for Project Schedules--Exposure Draft, [hyperlink, http://www.gao.gov/products/GAO-12-120G] (Washington, D.C.: May 2012), and GAO Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program Costs, [hyperlink, http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: March 2009). The guides document the specific characteristics and associated best practices for preparing schedule and cost estimates. [6] Milestone B is considered the official start of the program according to the Defense Acquisition Management System Framework. The framework is intended to translate mission needs and requirements into stable, affordable, and well-managed acquisition programs. [7] Although the Air Force provided us with its integrated master schedule for the DEAMS program, we determined that it did not meet the characteristics of an integrated master schedule in accordance with GAO's Schedule Guide. Specifically, an integrated master schedule is a document that integrates the planned work by the government, contractor, and other key parties; the resources necessary to accomplish that work; and the associated budget. Therefore, for purposes of this report, we use "schedule" to refer to the Air Force's integrated master schedule. [8] A schedule is controlled if it is updated periodically by trained schedulers to realistically forecast dates for program activities and compared against a designated baseline schedule to measure, monitor, and report the project's progress. [9] An integrated master plan provides an event-based hierarchy of program events, with each event supported by accomplishments and each accomplishment associated with specific criteria to be satisfied for its completion. The plan is normally part of the contract and is therefore contractually binding. A program's work breakdown structure defines in detail the work necessary to accomplish a program's objectives, including activities to be performed by both the government and contractors. [10] An independent cost estimate is another estimate based on the same technical information that is used to validate and cross-check the cost estimate, but is prepared by a person or organization that has no stake in the approval of the program. A risk and uncertainty analysis recognizes the potential for error and attempts to quantify it by identifying the effects of changing key cost drivers (e.g., costs associated with personnel or software development). [11] An agency's general fund accounts are those accounts in the U.S. Treasury holding all federal money administered by an agency that is not allocated by law to any other fund account. [12] An increment is a useful and supportable operational capability that can be developed, tested, produced, deployed, and sustained. It may consist of multiple capability releases to facilitate delivery of the system. [13] [hyperlink, http://www.gao.gov/products/GAO-11-53]. [14] A horizontally integrated schedule links products and outcomes with other associated sequenced activities, and a vertically integrated schedule ensures that the start and completion dates for activities are aligned with such dates on subsidiary schedules supporting tasks and subtasks. [15] Float is the amount of time by which a predecessor activity can slip before the delay affects the program's estimated finish date. [16] [hyperlink, http://www.gao.gov/products/GAO-09-3SP]. [17] [hyperlink, http://www.gao.gov/products/GAO-12-120G]. [18] We did not assess the extent to which the updated October 2012 schedule was controlled. [19] A roadmap is a planning document that briefly outlines the program's key phases (increments and releases) and the expected milestones for completion. A roadmap does not include specific activities or a detailed work breakdown structure, which would be included in the schedule that is prepared to support the roadmap. [20] A lag in a schedule denotes the passage of time between two activities. Lags have a specific use in scheduling but may be misused to force activities to begin on specific dates. [21] The purpose of a logic relationship, or dependency, is to depict the sequence in which activities occur. Such relationships state when activities are planned to start and finish in relation to the start and finish of other activities. A logic relationship therefore models the effect of an on-time, delayed, or accelerated activity on subsequent activities. [22] A constraint predefines the start, finish, or both dates of an activity. [23] Program officials stated that the updated schedule represented the DEAMS program baseline as of October 2012. The updated schedule included the period May 2, 2011, to December 17, 2015, and encompassed Releases 1 through 3 of Increment 1. [24] [hyperlink, http://www.gao.gov/products/GAO-11-53]. [25] Milestone B is considered the official start of the program according to the Defense Acquisition Management System Framework. The framework is intended to translate mission needs and requirements into stable, affordable, and well-managed acquisition programs. [26] GAO, GAO Schedule Assessment Guide: Best Practices for Project Schedules--Exposure Draft, [hyperlink, http://www.gao.gov/products/GAO-12-120G] (Washington, D.C.: May 2012). [27] Milestone B is considered the official start of the program according to the Defense Acquisition Management System Framework. The framework is intended to translate mission needs and requirements into stable, affordable, and well-managed acquisition programs. [28] GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program Costs, [hyperlink, http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: March 2009). [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's website [hyperlink, http://www.gao.gov]. Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to [hyperlink, http://www.gao.gov] and select “E-mail Updates.” Order by Phone: The price of each GAO publication reflects GAO's actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO's website, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO: Connect with GAO on facebook, flickr, twitter, and YouTube. Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts. Visit GAO on the web at [hyperlink, http://www.gao.gov]. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; E-mail: fraudnet@gao.gov; Automated answering system: (800) 424-5454 or (202) 512-7470. Congressional Relations: Katherine Siggerud, Managing Director, siggerudk@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, DC 20548. Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, DC 20548. [End of document]