This is the accessible text file for GAO report number GAO-14-333 
entitled 'Architect of the Capitol: Incorporating All Leading 
Practices Could Improve Accuracy and Credibility of Projects' Cost 
Estimates' which was released on March 25, 2014. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as 
part of a longer term project to improve GAO products' accessibility. 
Every attempt has been made to maintain the structural and data 
integrity of the original printed product. Accessibility features, 
such as text descriptions of tables, consecutively numbered footnotes 
placed at the end of the file, and the text of agency comment letters, 
are provided but may not exactly duplicate the presentation or format 
of the printed version. The portable document format (PDF) file is an 
exact electronic replica of the printed version. We welcome your 
feedback. Please E-mail your comments regarding the contents or 
accessibility features of this document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

United States Government Accountability Office: 
GAO: 

Report to Congressional Committees: 

March 2014: 

Architect of the Capitol: 

Incorporating All Leading Practices Could Improve Accuracy and 
Credibility of Projects' Cost Estimates: 

GAO-14-333: 

GAO Highlights: 

Highlights of GAO-14-333, a report to congressional committees. 

Why GAO Did This Study: 

AOC is responsible for the maintenance, renovation, and new 
construction of the U.S. Capitol complex, which comprises more than 
four dozen facilities. Reliable cost estimates for projects are 
crucial to AOC's capital-planning and construction processes. The 
House Appropriations Committee report accompanying the fiscal year 
2014 Legislative Branch Appropriations bill mandated that GAO review 
AOC's cost-estimating methodology. 

This report addresses the extent to which AOC's policies and guidance 
for developing cost estimates adheres to leading practices. GAO 
analyzed AOC's cost-estimating guidance, interviewed AOC officials, 
and compared AOC's cost-estimating guidance and documentation and two 
projects' cost estimates to leading practices in GAO's Cost Guide. 
When most or all of the practices associated with each characteristic 
of a high-quality, reliable estimate are followed, GAO considers the 
characteristic to be “fully” or “substantially” met. When, in turn, 
all four characteristics are at least “fully” or “substantially” met, 
GAO considers a cost estimate to be reliable. 

What GAO Found: 

GAO's Cost Estimating and Assessment Guide (Cost Guide) defines 12 
leading practices that are associated with four characteristics—-
comprehensive, well documented, accurate, and credible-—that are 
important to developing high-quality, reliable project-cost estimates. 
Using the Cost Guide, GAO determined that the Architect of the 
Capitol's (AOC) cost-estimating guidance conforms to leading practices 
for developing estimates that are, in general, comprehensive and well-
documented. However, AOC's guidance does not substantially conform to 
leading practices related to developing cost estimates that are 
accurate and credible. For example, pertaining to the credible 
characteristic, AOC's guidance does not require determining the 
confidence level of estimates or quantifying the extent to which a 
project's costs could vary due to changes in key assumptions. GAO 
found the strengths and weaknesses of AOC's guidance generally 
reflected in the cost estimates for AOC's Cannon House Office 
Building's (Cannon Building) renewal project ($753 million) and 
Capitol Dome's restoration project ($125 million). 

* Cannon Building renewal—GAO found the estimate is substantially 
comprehensive, well documented, and accurate, but several factors that 
affect its credibility are lacking. For example, AOC's risk analysis 
does not allow for determination of which risks have the greatest 
influence on project costs and may overstate the effect of the risks. 

* Capitol Dome restoration—GAO found the estimate is substantially 
comprehensive and well documented, but lacking key analysis that 
support accurate and credible estimates. For example, AOC did not use 
actual costs from completed phases to update its estimates and did not 
complete a risk and uncertainty analysis. 

Overall, AOC's cost-estimating guidance may not enable fully reliable 
estimates because it incorporates some, but not all, leading 
practices. Without reliable cost estimates that convey their 
confidence levels, AOC's projects risk experiencing cost overruns or 
budget surpluses, missed deadlines, and performance shortfalls. 
Potential limitations in the reliability of AOC's estimates may make 
it difficult for Congress to make well-informed funding decisions and 
affect how AOC allocates resources across competing projects in its 
capital portfolio. 

Table: GAO's Assessment of AOC's Cost-Estimating Characteristics in 
Its Guidance for Two Project Cost Estimates: 

Characteristic: Comprehensive; 
AOC's guidance: Fully Meets; 
Cannon Office Building: Substantially Meets; 
Capitol Dome: Substantially Meets. 

Characteristic: Well documented; 
AOC's guidance: Fully Meets; 
Cannon Office Building: Substantially Meets; 
Capitol Dome: Substantially Meets. 

Characteristic: Accurate; 
AOC's guidance: Partially Meets; 
Cannon Office Building: Substantially Meets; 
Capitol Dome: Partially Meets. 

Characteristic: Credible; 
AOC's guidance: Minimally Meets; 
Cannon Office Building: Partially Meets; 
Capitol Dome: Minimally Meets. 

Source: GAO analysis of AOC documents and data. 

Note: A characteristic is fully met when the associated tasks of 
underlying leading practices are completely satisfied; substantially 
met when a large portion of the associated tasks are satisfied; 
partially met when about half of the associated tasks are satisfied; 
minimally met when a small portion of the associated tasks are 
satisfied; and not met when none of the associated tasks are satisfied. 

[End of table] 

What GAO Recommends: 

GAO recommends that AOC incorporate additional leading practices into 
its cost-estimating guidance and submit the confidence levels of 
project estimates to Congress. AOC concurred with the recommendations 
and provided context and clarification on its cost-estimating guidance 
and policies. 

View [hyperlink, http://www.gao.gov/products/GAO-14-333]. For more 
information, contact Lorelei St. James at (202) 512-2834 or 
stjamesl@gao.gov. 

[End of section] 

Contents: 

Letter: 

Background: 

AOC's Policies and Guidance Conform to Leading Practices for Two 
Characteristics of Reliable Cost Estimates, but Fall Short in Two 
Others: 

Conclusions: 

Recommendations: 

Agency Comments and Our Evaluation: 

Appendix I: The Leading Practices for Developing High-Quality Cost 
Estimates and Associated Tasks: 

Appendix II: Objectives, Scope, and Methodology: 

Appendix III: GAO's Summary Assessments of AOC's Project Cost 
Estimates for the Cannon Building's Renewal and Capitol Dome's 
Restoration: 

Appendix IV: Comments from the Architect of the Capitol: 

Appendix V: GAO Contact and Staff Acknowledgments: 

Tables: 

Table 1: GAO's Cost-Estimating Characteristics and Leading Practices: 

Table 2: Phases and Budget of the Capitol Dome's Restoration Project, 
as of February 2014: 

Table 3: GAO Summary Assessment of AOC Cost-Estimating Guidance and 
Policies: 

Figures: 

Figure 1: Cannon House Office Building, Washington D. C. 

Figure 2: Scope and Phases of AOC's Renewal Project for the Cannon 
Building, as of February 2014: 

Figure 3: Damage to Capitol Dome, Outside Left, Inside Right (2013 
Photographs): 

Abbreviations: 

AOC: Architect of the Capitol: 

EAC: estimate at completion: 

EVM: earned value management: 

FCA: facility condition assessment: 

ICE: independent cost estimate: 

LCCE: life-cycle cost estimate: 

WBS: work breakdown structure: 

[End of section] 

United States Government Accountability Office: 
GAO:
441 G St. N.W. 
Washington, DC 20548: 

March 25, 2014: 

The Honorable Jeanne Shaheen: 
Chairwoman: 
The Honorable John Hoeven: 
Ranking Member: 
Subcommittee on Legislative Branch: 
Committee on Appropriations: 
United States Senate: 

The Honorable Tom Cole: 
Chairman: 
The Honorable Debbie Wasserman Schultz: 
Ranking Member: 
Subcommittee on Legislative Branch: 
Committee on Appropriations: 
House of Representatives: 

The Architect of the Capitol (AOC) is responsible for the maintenance, 
renovation, and new construction of the U.S. Capitol complex in 
Washington, D. C., which comprises more than four dozen facilities. 
Among the major projects involving these facilities, AOC has begun 
work restoring the dome atop the U.S. Capitol (Capitol Dome) and 
designing the renewal of the Cannon House Office Building (Cannon 
Building). AOC currently estimates the Capitol Dome and Cannon 
Building project costs to be $125 million and $753 million, 
respectively.[Footnote 1] 

Reliable cost estimates are crucial for supporting AOC's capital 
planning and construction processes. Without a reliable cost estimate, 
an AOC project is at risk of experiencing cost overruns, budget 
surpluses, missed deadlines, and performance shortfalls. Furthermore, 
AOC's ability to generate reliable cost estimates is important to 
informing Congress' funding decisions and enabling AOC to effectively 
allocate resources across competing projects in its capital program. 
The House Appropriations Committee report accompanying the fiscal year 
2014 Legislative Branch Appropriations Bill (H.R. 2792) mandated that 
we review AOC's cost-estimating methodology to ensure that AOC is 
accounting for all of the variables that should contribute to a 
project's cost estimates.[Footnote 2] This report addresses the extent 
to which AOC's policies and guidance for developing cost estimates 
conform to leading practices identified in GAO's Cost Estimating and 
Assessment Guide[Footnote 3] and provide a reliable basis to support 
funding and capital program decisions. This report also examines 
whether the estimates for the Capitol Dome and the Cannon Building 
projects reflect leading practices. 

In conducting our study, we reviewed AOC's policies and guidance for 
developing cost estimates and interviewed AOC officials. To determine 
the extent to which AOC's policies and guidance comply with practices 
for cost estimating, we compared them to the leading practices set 
forth in GAO's Cost Estimating and Assessment Guide (Cost Guide). 
[Footnote 4] The Cost Guide identifies 12 leading practices that 
represent work across the federal government and private sector and 
that have been found to be the basis for a high-quality, reliable 
project cost estimate. An estimate created using the leading practices 
exhibits four broad characteristics: it is comprehensive, well 
documented, accurate and credible. That is, each characteristic is 
associated with a specific set of leading practices. In turn, each 
leading practice is made up of a number of specific tasks. (See 
appendix I for a listing of the tasks that make up each of the 12 
leading practices.) When the tasks associated with the leading 
practices that define a characteristic are mostly or completely 
satisfied, we consider the characteristic to be "substantially" or 
"fully" met. When all four characteristics are at least "substantially 
met"--we consider a cost estimate to be reliable.[Footnote 5] 

We also examined AOC's cost estimates for two major capital projects: 
the Cannon Building renewal and the Capitol Dome restoration. We 
selected these projects based on their significance to the Capitol 
complex, comparatively high costs, and public visibility. In addition, 
our review of the Cannon Building's renewal estimate is responsive to 
another mandate in the House Appropriations Committee report 
accompanying the fiscal year 2010 Legislative Branch Appropriations 
Bill (H.R. 2918) requiring us to monitor the progress of the project. 
[Footnote 6] We selected versions of the estimates to review that had 
the most complete information for our assessments. We assessed the 
reliability of data used in developing the estimates and found them to 
be sufficiently reliable for the purposes of this report. For example, 
while some source data were unavailable to us, we were able to assess 
AOC's process for building its estimates with the data and check for 
errors. We also interviewed AOC staff and its technical consultant 
about these projects and observed existing conditions at the Cannon 
Building and Capitol Dome. While our review of these projects' cost 
estimates provides key insights and illustrates recent products of 
AOC's cost-estimating policies and guidance, the results of our review 
should not be used to make generalizations about all AOC project cost 
estimates. 

We conducted our work from September 2013 to March 2014 in accordance 
with all sections of GAO's Quality Assurance Framework that are 
relevant to our objectives. The framework requires that we plan and 
perform the engagement to obtain sufficient and appropriate evidence 
to meet our stated objectives and to discuss any limitations in our 
work. We believe that the information and data obtained, and the 
analysis conducted, provide a reasonable basis for any findings and 
conclusions in this product. Appendix II provides a more detailed 
description of our scope and methodology. 

Background: 

Cost-Estimating Leading Practices: 

The GAO Cost Estimating and Assessment Guide[Footnote 7] explains how 
four characteristics of a high-quality, reliable cost estimate can be 
understood in relation to 12 leading practices. The extent to which an 
agency meets the leading practices underlying each characteristic 
determines its performance for that characteristic.[Footnote 8] For 
example, we consider the comprehensive characteristic to be 
"substantially met" if the organization substantially meets the 
underlying leading practices of (1) developing an estimating plan and 
(2) determining an estimating structure. Because the leading practices 
are separate and discrete, an agency's performance in each of the 
characteristics can vary. For example, an organization's cost 
estimating methodology could be found to be comprehensive and well 
documented, but not accurate or credible, resulting in the 
organization producing cost estimates of limited reliability. Table 1 
illustrates the relationship of the 12 leading practices to the four 
characteristics of a high-quality, reliable estimate. 

Table 1: GAO's Cost-Estimating Characteristics and Leading Practices: 

Four characteristics of a high-quality, reliable cost estimate: 
Comprehensive when estimate accounts for all possible costs associated 
with a program, is structured in sufficient detail to ensure that 
costs are neither omitted nor double counted, and documents all cost-
influencing assumptions; 
12 leading practices to develop high-quality, reliable estimates: 
Develop the estimating plan; 
Determine the estimating structure. 

Four characteristics of a high-quality, reliable cost estimate: 
Well documented when supporting documentation explains the process, 
sources, and methods used to create the estimate; contains the 
underlying data used to develop the estimate; and is adequately 
reviewed and approved by management; 
12 leading practices to develop high-quality, reliable estimates: 
Define the estimate's purpose; 
Define the program's characteristics; 
Identify ground rules and assumptions; 
Obtain the data; 
Document the estimate; 
Present the estimate to management for approval. 

Four characteristics of a high-quality, reliable cost estimate: 
Accurate when estimate is not overly conservative or optimistic, is 
based on an assessment of the costs most likely to be incurred, and is 
regularly updated so that it always reflects the program's current 
status; 
12 leading practices to develop high-quality, reliable estimates: 
Develop the "point estimate"[A]; 
Update the estimate to reflect actual costs/changes. 

Four characteristics of a high-quality, reliable cost estimate: 
Credible when (1) any limitations of the analysis because of 
uncertainty or sensitivity surrounding data or assumptions are 
discussed, (2) the estimate's results are cross-checked, and (3) an 
independent cost estimate is conducted by a group outside the 
acquiring organization to determine whether other estimating methods 
produce similar results; 
12 leading practices to develop high-quality, reliable estimates: 
Compare the point estimate to an independent cost estimate[A]; 
Conduct sensitivity analysis; 
Conduct a risk and uncertainty analysis. 

Source: GAO. 

[A] As described in the GAO Cost Estimating and Assessment Guide, 
developing the point estimate and comparing it to an independent cost 
estimate--shown separately in the table--are separate parts of a 
single leading practice. For purposes of assessing the extent to which 
a cost estimate achieves the characteristics of a high-quality cost 
estimate, developing the point estimate contributes to accuracy, while 
comparing the point estimate to an independent cost estimate 
contributes to credibility. A "point estimate" is the most likely 
value for the cost estimate, given the underlying data. 

[End of table] 

AOC's Project Development: 

AOC project development consists of three stages: planning, design, 
and construction. According to AOC's guidance, AOC refines 
requirements and updates cost estimates as projects progress through 
these stages. AOC has guidance and requirements for cost estimates at 
each stage of a project's development. In general, as projects develop 
over time and requirements are refined, the accuracy of cost estimates 
is expected to increase. 

Given the current fiscal environment and existing building conditions, 
AOC must prioritize projects in its capital program and decide to 
either request funding for a project or defer it while mitigating 
potential facility issues. For example, in fiscal year 2014, AOC 
requested almost $155 million for 17 projects that AOC deemed urgent, 
while deferring 46 projects estimated to cost about $172 million. 
[Footnote 9] Two of AOC's largest and most recognizable projects that 
it considers urgent are the Cannon Building renewal and the Capitol 
Dome restoration. 

Cannon Building's Renewal: 

The Cannon Building, completed in 1908, is the oldest congressional 
office building and is occupied by members of the House of 
Representatives and their staffs. (See figure 1.) The building houses 
142 office suites, 5 conference rooms, 4 hearing rooms, and the Caucus 
Room, which can accommodate large meetings. The building also provides 
space for a library, food service, and a health unit. 

Figure 1: Cannon House Office Building, Washington D.C. 

[Refer to PDF for image: photograph] 

Source: Architect of the Capitol. 

[End of figure] 

According to AOC planning studies, the building is plagued with 
serious safety, health, environmental, and operational issues that are 
worsening. For example, without action, essential systems for heating 
and cooling that are located behind walls and in mechanical rooms will 
continue to deteriorate, potentially negatively affecting Members of 
Congress, staff, and constituents. AOC has been developing the scope 
of the Cannon Building's renewal project since approximately 2004 when 
AOC's consultant conducted a facility condition assessment (FCA) that 
identified the building's deficiencies.[Footnote 10] In 2009, we 
reviewed AOC's progress in developing the Cannon Building project and 
found that while it had followed a reasonable process to plan the 
building's renewal such as through updating the FCA, it was important 
that AOC continue, as planned, to refine the project's scope and cost 
estimate.[Footnote 11] Since our 2009 review, AOC has proceeded with 
the design phase of the project, based upon its budget of $753 million 
for the planning, design, and construction phases. The project is 
currently expected to be completed by 2025. 

According to AOC officials and current design documents, AOC plans to 
correct most of the Cannon Building's identified deficiencies and to 
address requirements such as energy conservation, physical security, 
hazardous materials abatement, and historic preservation. The project 
is to involve substantial reconfiguration of interior and exterior 
spaces to include reconstructing the building's top floor, which now 
partially consists of storage space, and landscaping the courtyard. 
The project is also expected to provide refurbished windows and a new 
roof. Additional work is intended to preserve and repair the 
building's stone exterior. The project is also expected to allow for 
complete replacement of all plumbing, heating and cooling, fire 
protection, electrical, and alarm systems; refurbish restrooms and 
make them more accessible to people with disabilities; and provide new 
wall and floor finishes in some areas. In addition, the project 
includes removing asbestos that may be contained in plaster ceilings 
and walls. AOC plans to conduct the work in phases corresponding to 
the four sections of the building and including an initial phase for 
utility work as shown in figure 2. Tenants displaced during 
construction of each section are to move to temporary offices while 
other occupants will remain in building sections not affected by 
construction. 

Figure 2: Scope and Phases of AOC's Renewal Project for the Cannon 
Building, as of February 2014: 

[Refer to PDF for image: illustration and accompanying data] 

AOC intends to conduct construction in discrete phases affecting one 
section of the building at a time. The plan shown to the right 
illustrates the planned construction phasing. 

Project scope includes: 
* Demolition; 
* Repairs to heating and cooling, security, electrical, plumbing, and 
fire protection systems; 
* Reconfiguration of interior spaces and restroom refurbishment; 
* Roof replacement; 
* Stone preservation. 

Project schedule: 
* Five total phases (Phase 0, 1, 2, 3, 4); 
* Phase 0 incorporates building utility work to relocate such items as 
electrical, plumbing, and heating and cooling equipment to prepare for 
the work in the four building sections; 
* Two years per building section phase (2017–2025). 

Source: Architect of the Capitol. 

[End of figure] 

Capitol Dome Restoration: 

The U.S. Capitol Dome, an important symbol of American democracy and 
an architectural icon, was constructed of cast iron more than 150 
years ago. According to AOC, the dome has not undergone a complete 
restoration since 1960, and due to age and weather is now plagued by 
more than 1,000 cracks and deficiencies that are causing it to 
deteriorate. Figure 3 shows cracks at the exterior column base and 
deteriorating interior paint. 

Figure 3: Damage to Capitol Dome, Outside Left, Inside Right (2013 
Photographs): 

[Refer to PDF for image: 2 photographs] 

Source: GAO. 

[End of figure] 

The project is intended to stop deterioration in the dome's cast iron 
structure as well as to ensure the protection of the interior of the 
dome and rotunda. In the 1960 restoration, the dome was stripped of 
its paint so the ironwork could be repaired, primed with a rust 
inhibitor, and then repainted. As part of the current project, AOC is 
undertaking similar restorative work to include removing old paint, 
repairing the cast iron, and repainting. 

The project has proceeded in phases as shown in table 2, with phase 
IIA restoration work currently in progress. In earlier phases, AOC 
completed interim painting, revalidated the project's design, and 
restored the base--or skirt--of the dome. To complete the project, AOC 
is seeking appropriations of about $20 million for construction and 
other costs for phase IIC work in fiscal year 2015. As of February 
2014, AOC estimated the total cost of the Capitol Dome restoration 
project at about $125 million. 

Table 2: Phases and Budget of the Capitol Dome's Restoration Project, 
as of February 2014: 

Phase: IA; 
Description: Interim painting 2002; 
Status: Completed 01/03/2003; 
Total cost: Not included. 

Phase: IB; 
Description: Interim painting 2010; 
Status: Completed 12/20/2010; 
Total cost: $2,500,000. 

Phase: IC; 
Description: Skirt restoration; 
Status: Completed 09/01/2012; 
Total cost: $19,959,000. 

Phase: Phase II; 
Description: Design revalidation; 
Status: Completed 12/12/2012; 
Total cost: $3,996,000. 

Phase: Phase IIA; 
Description: Dome restoration exterior estimated 24-month duration; 
Status: Contract Awarded 09/30/2013; 
Total cost: $58,070,000. 

Phase: Phase IIB; 
Description: Dome restoration interior estimated 9-month duration; 
Status: Option in phase IIA contract; 
(fiscal year 2014 appropriations received, option not yet exercised); 
Total cost: $15,400,000. 

Phase: Phase IIC; 
Description: Dome restoration Rotunda estimated 1-month duration; 
Status: Unfunded option in phase IIA contract (estimated to be awarded 
in fiscal year 2015); 
Total cost: $20,220,000. 

Phase: Other Costs; 
Description: Study, design, inspection, US Capitol Police overtime; 
Status: Miscellaneous over phases; 
Total cost: $5,228,000. 

Phase: Total; 
Total cost: $125,373,000. 

Source: GAO analysis of AOC data. 

Note: We focused our assessment on estimates for work currently in 
progress (phase IIA) or planned to be completed (phases IIB and IIC). 

[End of table] 

AOC's Policies and Guidance Conform to Leading Practices for Two 
Characteristics of Reliable Cost Estimates, but Fall Short in Two 
Others: 

As previously discussed, the GAO Cost Estimating and Assessment Guide 
[Footnote 12] defines 12 leading practices related to four 
characteristics--comprehensive, well documented, accurate, and 
credible--that are important to developing high-quality, reliable 
estimates.[Footnote 13] Our analysis determined how well AOC met a 
characteristic based on our assessment of AOC's conformance to the 
leading practices related to that characteristic. We discuss 
characteristics using five rating categories--does not meet, minimally 
meets, partially meets, substantially meets, or fully meets.[Footnote 
14] As shown in table 3, our analysis found that AOC's cost-estimating 
policies and guidance contribute to estimates that are comprehensive 
and well documented in that AOC fully meets most of the tasks that 
underlie the leading practices associated with these two 
characteristics. We found that AOC's cost-estimating policies and 
guidance partially met the accuracy characteristic and minimally met 
the credible characteristic based on their conformance to the leading 
practices associated with those characteristics. Table 3 also contains 
key examples of our rationale for our assessment of each leading 
practice and characteristic. 

Table 3: GAO Summary Assessment of AOC Cost-Estimating Guidance and 
Policies: 

GAO assessment: Fully meets; 
Characteristic: Comprehensive; 

Leading Practice: Develop the estimating plan; 
Key examples of rationale for assessment: AOC has a formal process 
that develops the estimating plan, including describing responsible 
parties and defining specific tasks. 

Leading Practice: Determine the estimating structure; 
Key examples of rationale for assessment: The estimating structure 
follows a work breakdown structure (WBS)[A] that appears to be 
standard in the construction industry. 

GAO assessment: Fully meets; 
Characteristic: Well documented; 

Leading Practice:Define estimate's purpose; 
Key examples of rationale for assessment: AOC uses different types of 
cost estimates and has defined when each applies and what each entails. 

Leading Practice: Define the program's characteristics[B]; 
Key examples of rationale for assessment: AOC has formally defined 
program characteristics. 

Leading Practice: Identify ground rules and assumptions; 
Key examples of rationale for assessment: AOC points to supporting 
documents that, when developed, should contain ground rules and 
assumptions specific to a given project. 

Leading Practice: Obtain the data; 
Key examples of rationale for assessment: AOC guidance is specific 
about sources of data. These include commercially available 
construction cost databases as well as market and industry knowledge, 
staff knowledge, and historical data. 

Leading Practice: Document the estimate; 
Key examples of rationale for assessment: AOC's guidance explains how 
to document the estimate. 

Leading Practice: Present estimate to management for approval; 
Key examples of rationale for assessment: AOC's guidance requires 
management approval of estimates. 

GAO assessment: Partially Meets; 
Characteristic: Accurate; 

Leading Practice: Develop the point estimate[C]; 
Key examples of rationale for assessment: AOC develops point estimates 
as projects progress through planning and design phases to support 
budget requests. 

Leading Practice: Update the estimate to reflect actual costs/changes; 
Key examples of rationale for assessment: AOC does not update 
estimates with actual costs as projects progress. 

GAO assessment: Minimally Meets; 
Characteristic: Credible; 

Leading Practice: Compare the point estimate to an independent cost 
estimate (ICE)[C]; 
Key examples of rationale for assessment: ICE's defined by AOC's 
estimating process have a limited degree of independence. 

Leading Practice: Conduct a sensitivity analysis; 
Key examples of rationale for assessment: AOC guidance discusses some 
sensitivity analysis, but it skips many steps identified by GAO in its 
leading practices. 

Leading Practice: Conduct a risk and uncertainty analysis; 
Key examples of rationale for assessment: AOC's guidance is to set 
aside a fixed amount for contingency.[D] This is not as beneficial as 
quantitative-risk and uncertainty analysis to assess the variability 
in the estimate and the level of confidence associated with the cost 
estimate. 

Source: GAO analysis of AOC data. 

Fully Meets: AOC provided complete evidence that satisfies the 
associated tasks of the leading practices: 

Partially Meets: AOC provided evidence that satisfies about half of 
the associated tasks of the leading practices: 

Minimally Meets: AOC provided evidence that satisfies a small portion 
of the associated tasks of the leading practices: 

[A] A WBS deconstructs a program's end product into successive levels 
with smaller specific elements until the work is subdivided to a level 
suitable for management control. AOC requires use of the ASTM 
UNIFORMAT II structure, which is organized according to systems or 
assemblies that are common to most buildings, for estimates. 

[B] In the context of this report, a program refers to a group of 
projects or functions associated with delivering a capital asset. For 
example, the Cannon House Office Building program refers to projects 
or functions--completed via contract or by AOC staff--for planning, 
designing, and constructing all phases of the renewal work and 
providing for ongoing operations and maintenance of the facility. 

[C] As described in the GAO's Cost Estimating and Assessment Guide, 
developing the point estimate and comparing it to an independent cost 
estimate are separate parts of a single leading practice. For purposes 
of assessing the extent to which a cost estimate achieves the 
characteristics of a high-quality cost estimate, developing the point 
estimate contributes to accuracy, while comparing the point estimate 
to an independent cost estimate contributes to credibility. 

[D] AOC applies additional percentages to the items in the cost 
estimate for contingencies at a series of milestones in completing 
projects' design and construction documents as established by AOC 
policy. For example, early in the projects' design, AOC establishes a 
contingency range for design at 20 to 25 percent and construction at 5 
to 20 percent. When the design is completed, the design contingency is 
removed. 

[End of table] 

AOC Conforms to the Leading Practices Associated with Comprehensive 
and Well Documented Cost Estimates: 

Comprehensive: 

In reference to the leading practices underlying the characteristics 
of a comprehensive cost estimate, we found that AOC: 

* has a formal process for developing estimating plans, and: 

* follows an estimating approach incorporating a work breakdown 
structure (WBS) that is widely used in the construction industry. 
[Footnote 15] 

In developing estimating plans, we found that AOC's policies and 
guidance that describe the project-planning process provide the 
framework for meeting this leading practice. In the planning stage, 
AOC project managers establish project scope and outline roles and 
responsibilities for planning, design, and construction stages. To 
facilitate these efforts, AOC project managers are to use particular 
documents. One document, the Project Development Form, provides a 
basis for preparing requirements studies that help to define scope and 
estimate costs. AOC then is to use this information in developing 
Project Management Plans that are to describe other key components of 
project delivery. These components include defining the strategy for 
providing design and construction services, identifying project team 
members, establishing a communications plan, and setting project 
controls such for managing project changes. AOC also considers cost 
reporting as a component of the project delivery process.[Footnote 16] 
For example, AOC typically requires its design contractors to provide 
cost reports at the same time incremental design submittals are made. 
Because AOC's project development process accounts for cost reporting, 
we determined that it satisfies the intent of the leading practice 
encouraging use of a formal process for developing estimating plans. 

Well Documented: 

With regard to leading practices related to producing well-documented 
estimates, we found that AOC: 

* requires formal definitions of program[Footnote 17] characteristics, 
such as the program's purpose; 

* develops ground rules and assumptions from supporting documents; 

* requires the use of cost data sources that are specific to the 
construction industry (i.e., obtain the data); 

* requires documenting the estimate and supporting documents; and: 

* requires management approval of the cost estimate. 

AOC fully meets the requirements of most of these leading practices. 
[Footnote 18] For example, AOC's policies and guidance for the project 
development process provide the structure to enable an adequate 
understanding of program characteristics--such as key design features, 
technical definitions, and the acquisition strategy--that will 
comprise the cost estimate. In addition, AOC guidance establishes 
requirements for use of industry-accepted cost-data sources and allows 
for application of staff knowledge and historical data in developing 
estimates.[Footnote 19] AOC guidance also provides for estimates to be 
documented to show important parameters, assumptions, descriptions, 
methods, and calculations used to derive the estimate. However, while 
AOC requires management approval of its cost estimates, it falls short 
of fully meeting the requirements of this leading practice. In 
particular, AOC's briefings do not include the level of information 
defined by our leading practice, particularly information related to 
risks associated with the underlying data and methods. This is due, in 
part, to AOC's policies and guidance not requiring a risk and 
uncertainty analysis in developing estimates. We discuss this leading 
practice in the following section of this report. 

AOC Partially and Minimally Conforms to the Leading Practices 
Associated with Accurate and Credible Cost Estimates, Respectively: 

We found that AOC policies and guidance do not require enough detail 
to suggest that resulting estimates would be accurate and credible. 

Accurate: 

In reference to accuracy, we found that AOC's policies and guidance 
partially meet the respective underlying leading practices pertaining 
to this characteristic. According to AOC's policies and guidance, AOC 
is to develop estimates as projects progress through planning and 
design phases that are based on sufficiently detailed documentation of 
construction requirements. AOC is to then use these estimates to 
support budget requests for construction funding. However, AOC's 
guidance does not require that cost estimates be updated with actual 
costs during a project's construction phase--a leading practice. 
According to the Cost Guide, updating estimates to reflect actual 
costs as the project progresses allows agencies to review variances 
between planned and actual costs and provides insight as to how the 
project changed over time. 

Credible: 

In reference to credibility, we found that AOC's policies and guidance 
minimally meet the underlying leading practices pertaining to this 
characteristic. For example, we found that AOC's guidance does not 
require: 

* following all steps for conducting a "sensitivity analysis"[Footnote 
20] and determining the estimate's reasonableness, and: 

* conducting a risk and uncertainty analysis.[Footnote 21] 

While AOC guidance does discuss conducting some sensitivity analysis, 
it skips many associated tasks of leading practices we identified, 
such as identifying key cost drivers, ground rules and assumptions for 
sensitivity testing, and evaluating the results to determine which 
drivers most affect the cost estimate. Similarly, while AOC guidance 
provides that contingencies be added to estimates to account for risk 
and uncertainty, AOC's guidance does not provide documented reasons 
explaining how the actual budgeted amounts for unforeseen costs were 
developed.[Footnote 22] AOC officials told us that while their 
policies and guidance do not require sensitivity or "quantitative-risk 
and uncertainty analyses," they perform such assessments qualitatively 
to establish budget contingency. However, the leading practice to 
determine whether a program is realistically budgeted is to perform a 
quantitative-risk and uncertainty analysis,[Footnote 23] so that the 
probability associated with achieving its point estimate can be 
determined. The results of a quantitative-risk and uncertainty 
analysis--the range of costs around a point estimate--can be useful to 
decision makers because it conveys the confidence level in achieving 
the most likely cost and informs them about cost, schedule, and 
technical risks.[Footnote 24] Not having an understanding of an 
estimate's confidence level limits AOC's ability to determine the 
appropriate level of contingency that is needed to address risks and 
uncertainty for a particular project and could lead AOC to 
ineffectively allocate resources across competing projects if 
contingency levels are either overstated or understated. In addition, 
absent information on estimates' confidence levels, Congress will not 
have critical information for making well-informed funding decisions. 

In addition, pertaining to the leading practice of developing a point 
estimate and comparing it to an independent cost estimate (ICE)--which 
affects both accurate and credible characteristics--we found that the 
independent estimates defined by AOC's estimating process have a 
limited degree of independence and focus solely on proposed contractor 
costs rather than on the entire cost estimate, including both 
government and contractor efforts. According to our leading practices, 
an ICE, conducted by an organization outside the program office, 
provides an objective and unbiased assessment of whether the agency's 
program estimate can be achieved. However, because AOC's project 
management and cost-estimating functions are performed within the same 
organizational group, the estimates could potentially be influenced by 
AOC's project managers and therefore could be susceptible to bias. 

Because of the weaknesses in AOC's policies and guidance pertaining to 
two of the four characteristics that are important to developing high-
quality, reliable estimates, the project cost estimates that AOC 
produces may not always be reliable. Without reliable cost estimates, 
AOC's projects risk experiencing cost overruns or budget surpluses, 
missed deadlines, and performance shortfalls. Furthermore, potential 
limitations in the reliability of its estimates may impair Congress's 
ability to make well-informed funding decisions and affect how AOC 
allocates resources across competing projects in its capital program. 

Strengths and Weaknesses in AOC's Policies and Guidance Are Reflected 
in the Cannon Building and Capitol Dome Projects: 

In comparing the Cannon Office Building's renewal and Capitol Dome's 
restoration cost estimates to GAO's leading practices, we found 
strengths and weaknesses that generally correspond to our assessment 
of AOC's overall policies and guidance for developing cost 
estimates.[Footnote 25] We initially determined that both estimates 
were comprehensive while lacking, to a varying extent, in their 
documentation, accuracy, and credibility.[Footnote 26] Following our 
initial determination, AOC provided further documentation that 
resulted in improvements to each estimate's assessment. Our final 
assessment found that: 

* the Cannon Building renewal estimate to be substantially 
comprehensive, well documented, and accurate while lacking in elements 
affecting its credibility, and: 

* the Capitol Dome restoration estimate to be substantially 
comprehensive and well documented while lacking in areas pertaining to 
accuracy and credibility. 

In aggregate, because of weaknesses pertaining to characteristics that 
are important to the development of high-quality, reliable estimates, 
AOC's cost estimates for the Cannon Building's renewal and Capitol 
Dome's restoration may not be fully reliable.[Footnote 27] Appendix 
III provides additional details on our comparison of these two cost 
estimates to our leading practices. 

Cannon Building's Renewal Estimate: 

In reference to the comprehensive characteristic, we found that the 
estimate substantially met the underlying leading practices.[Footnote 
28] In particular, the estimate uses a WBS[Footnote 29] that is 
standard in the construction industry and allows for tracking of cost 
and schedule performance by defined elements of work. In addition, AOC 
conducted a life-cycle-cost analysis of the heating and cooling system 
alternatives it considered to include analyzing their energy 
consumption for the purpose of comparing operational costs. However, 
the estimate did not fully meet the characteristic as it did not 
include full life-cycle costs, from inception through design, 
construction, operation, and maintenance. Without a life-cycle cost 
analysis that captures the total cost of the project, AOC cannot 
evaluate design alternatives on a total-cost basis. 

In terms of being well documented, we found that the estimate 
substantially met associated leading practices. In general, the 
information provided by AOC describes how the estimate was built up 
from engineering drawings, specifications, and design documents. In 
addition, there was evidence of documented management approval. 
However, the documentation AOC provided did not contain source data, 
such as from contractor bids or cost estimating databases. While AOC 
officials said they could obtain this information for our review, to 
the extent that it was available,[Footnote 30] our leading practices 
indicate that an estimate's documentation should be detailed enough so 
that the derivation of each cost element can be traced to all sources 
allowing for the estimate to be easily replicated and updated. Because 
AOC could not readily provide the actual source data for the estimate, 
we determined that it did not fully meet the requirements of this 
characteristic. 

For the accuracy characteristic, we found the estimate substantially 
met leading practices. AOC's acquisition approach to the building's 
renewal involves its contracting with an architect, construction 
manager, and construction contractor, each of whom produced separate 
estimates. These estimates have a limited degree of independence 
because they were conducted for the same program office and focus only 
on the proposed contractor cost. However, this array of estimates has 
enabled AOC to make comparisons among them, determine similarities and 
differences, and develop a reasonably accurate assessment of estimated 
costs.[Footnote 31] 

For the credibility characteristic, we found the estimate partially 
met leading practices. During the course of our review, AOC conducted 
a risk and uncertainty analysis in accordance with a key leading 
practice for this characteristic.[Footnote 32] However, we found 
several issues affecting the quality of the analysis that AOC provided 
to us. For example, AOC's analysis concluded that the Cannon 
Building's renewal estimate had a confidence level that exceeded 90 
percent--meaning there is a greater than a 90 percent probability that 
actual costs will be equal to or less than AOC's estimate--which may 
be unreasonably high.[Footnote 33] However, an AOC official said that 
the agency will be reconsidering the confidence level once the design 
progresses further. In addition, we found that the method AOC used to 
model the project's risks it identified (1) resulted in an unusually 
narrow range of estimated costs across the confidence intervals and 
(2) provides managers limited ability to understand the effects of 
individual risks.[Footnote 34] Because AOC aggregated risks for the 
purposes of its analysis as opposed to modeling them separately, AOC 
cannot identify relationships between risk elements and determine 
which risks have the greatest influence on project costs. As a result, 
AOC is limited in its ability to manage the risks. In addition, 
modeling the aggregated risks likely contributes to the overly narrow 
range of estimated costs over the confidence levels, which implies 
that AOC's analysis overstates the effect of the risks.[Footnote 35] 

Capitol Dome's Restoration Estimate: 

In reference to the comprehensive characteristic, we found that the 
estimate substantially met leading practices. In particular, the 
estimate uses an industry-standard WBS format and contains sufficient 
detail of the technical characteristics of the project. However, the 
estimate did not fully meet leading practices, in part because we did 
not find a consolidated list of ground rules and assumptions or 
descriptions of how these affected the estimate. Contract documents 
provide indications of ground rules and assumptions affecting the 
project. For example, contract specifications and drawings for the 
recently-awarded phase II work describe assumptions pertaining to 
availability of funding for contract options and accessibility of the 
site during the workday. However, we could not determine if these 
represented a comprehensive assessment of all ground rules and 
assumptions. According to the Cost Guide, because ground rules and 
assumptions can significantly affect cost and introduce risk, it is 
important that they are clearly documented in the estimate to enable 
areas of potential risk to be identified and resolved. 

For the well documented characteristic, we determined the estimate as 
having substantially met leading practices. We found, for example that 
cost elements from the estimate can be compared to information in the 
drawings and specifications that define the contract for the phase II 
restoration. This linkage enables a good understanding of key 
characteristics of the estimate. In addition, AOC attested that the 
estimate had been briefed to and approved by management. However, 
because records describing briefings of the estimate to management are 
not well documented, it is difficult to trace management's 
recommendations for changes, feedback, and key decisions affecting the 
project. While AOC officials told us that its project staff and 
management communicate routinely as part of their normal business 
functions, documenting management briefings is important because some 
key personnel have changed over the life of the Capitol Dome 
restoration project, which originated more than 15 years ago. By not 
having well documented records of management briefings over the life 
of the project, AOC risks loss of continuity in its oversight. 

Considering the accuracy characteristic, we determined that the 
estimate partially met leading practices. We found, for example, that 
the estimating software and database used to construct the estimate 
included standard information, such as costs for labor, materials, and 
equipment. However, AOC did not consider the actual cost of previously 
completed restoration phases in updates to its estimate for the phase 
II work. While AOC used information on the price it paid for certain 
items in the phase I restoration work that are also part of the phase 
II project, it did not determine the contractor's actual cost to 
perform the work. AOC officials said that having information on actual 
contractor costs for items may be of limited usefulness because market 
conditions and other factors determine what contractors bid and AOC 
ultimately pays. However, not having information on actual costs makes 
it difficult for AOC to assess the difference between the price it 
paid and its contractors' costs to determine the estimate's 
reasonableness. 

For the credibility characteristic, we determined that the estimate 
minimally met leading practices. We found, in contrast to the approach 
it took in developing the Cannon Building renewal estimate, that AOC 
did not conduct a risk and uncertainty analysis of the Capitol Dome 
restoration estimate. While AOC officials told us they have not 
conducted a quantitative-risk and uncertainty analysis of the 
estimate, they said they have taken steps to qualitatively assess and 
mitigate project risks. For example, AOC structured the phase II 
contract solicitation to have contractors include a base bid (phase 
IIA) and pre-priced options for later stages (phases IIB and IIC) of 
the work. According to AOC officials, by structuring the contract this 
way, AOC is protected from price escalations affecting the future 
phases. In addition, AOC established unit prices and quantity 
allowances for some restoration tasks that are of indefinite quantity, 
such as repairing cracks. According to AOC officials, this will help 
to control project costs because, while their quantity estimates may 
be imprecise, knowing the unit prices associated with the work should 
mitigate some cost risk. While these actions are encouraging, they are 
not documented within the context of a quantitative-risk and 
uncertainty analysis. As a result, we do not know how these actions 
relate to other risk mitigation efforts that may have been considered 
and why AOC chose these actions over other efforts. Moreover, not 
having a quantitative-risk and uncertainty analysis precludes 
establishing a level of confidence associated with achieving the 
estimated cost and limits AOC management's ability to determine an 
appropriate level of contingency reserves that may be necessary to 
address risks that AOC has identified. 

Conclusions: 

AOC project's cost-estimating guidance policy and two resulting 
project estimates that we reviewed may not be fully reliable because 
they incorporated some, but not all, leading practices in cost 
estimating. Because AOC's project cost estimates inform Congress's 
funding decisions and affect AOC's ability to effectively allocate 
resources across competing projects in its capital program, there is a 
risk that funding decisions and resource allocations could be made 
based on information that is not reliable. We recognize that 
incorporating GAO's cost-estimating best practices into AOC's cost-
estimating policy and guidance may involve additional costs--such as 
for conducting a risk and uncertainty analysis for projects and 
conveying the confidence level of the estimate to Congress and AOC 
managers. However, without investing in these practices, Congress 
risks making funding decisions and AOC management risks making 
resource allocation decisions without the benefit that a robust 
analysis of levels of risk, uncertainty, and confidence provide 
decision makers. 

Recommendations: 

To improve the Architect of the Capitol's project-cost-estimating 
process, enhance the transparency of its related process, and allow 
for more informed decision making related to projects' costs, we 
recommend that the Architect of the Capitol take the following two 
actions, to the extent that the benefits exceed the costs: 

* incorporate leading practices we identified as lacking for cost 
estimating in the AOC's cost-estimating guidance and policies, and: 

* for ongoing and future projects, submit the confidence level derived 
from risk and uncertainty analyses along with budget documentation to 
appropriate congressional decision makers, so that Congress is aware 
of the range of likely costs and AOC's associated confidence levels. 

Agency Comments and Our Evaluation: 

We provided a draft of this report to the Architect of the Capitol for 
review and comment. AOC agreed with our recommendations and provided 
us with additional context and information on specific actions that 
the AOC has taken or intends to take to more fully address our 
recommendations. For example, AOC said that it planned to revise its 
policies and guidance to require quantitative-risk and uncertainty 
analysis be done, as specified in the Cost Guide, for high-dollar-
value projects prior to requesting construction funding. In addition, 
AOC said that it would explore the most effective approach for 
communicating to congressional decision makers the confidence level 
derived from risk and uncertainty analyses along with budget 
documentation. We made no changes to our draft based upon AOC's 
comments. AOC's comments are reprinted in appendix IV, followed by our 
response to AOC's detailed comments. 

We are sending copies of this report to appropriate congressional 
committees and the Architect of the Capitol. In addition, the report 
is available at no charge on the GAO website at [hyperlink, 
http://www.gao.gov]. 

If you or your staff have any questions about this report, please 
contact me at (202) 512-2834 or stjamesl@gao.gov. Contact points for 
our Offices of Congressional Relations and Public Affairs may be found 
on the last page of this report. GAO staff who made key contributions 
to this report are listed in appendix V. 

Signed by: 

Lorelei St. James: 
Director, Physical Infrastructure Issues: 

[End of section] 

Appendix I: The Leading Practices for Developing High-Quality Cost 
Estimates and Associated Tasks: 

Step: 1; 
Leading practice: Define estimate's purpose; 
Associated task: 
* Determine estimate's purpose, required level of detail, and overall 
scope; 
* Determine who will receive the estimate. 

Step: 2; 
Leading practice: Develop estimating plan; 
Associated task: 
* Determine the cost-estimating team and develop its master schedule; 
* Determine who will do the independent cost estimate; 
* Outline the cost-estimating approach; 
* Develop the estimate's timeline. 

Step: 3; 
Leading practice: Define program characteristics; 
Associated task: 
* In a technical baseline-description document, identify the program's 
purpose and its system and performance characteristics and all system 
configurations; 
* Any technology implications; 
* Its program acquisition schedule and acquisition strategy; 
* Its relationship to other existing systems, including predecessor or 
similar legacy systems; 
* Support (manpower, training, etc.) and security needs and risk items; 
* System quantities for development, test, and production; 
* Deployment and maintenance plans. 

Step: 4; 
Leading practice: Determine estimating structure; 
Associated task: 
* Define a work breakdown structure (WBS) and describe each element in 
a WBS dictionary (a major automated-information system may have only a 
cost element structure); 
* Choose the best estimating method for each WBS element; 
* Identify potential cross-checks for likely cost and schedule drivers; 
* Develop a cost-estimating checklist. 

Step: 5; 
Leading practice: Identify ground rules and assumptions; 
Associated task: 
* Clearly define what the estimate includes and excludes; 
* Identify global and program-specific assumptions, such as the 
estimate's base year, including time phasing and life cycle; 
* Identify program schedule information by phase and program 
acquisition strategy; 
* Identify any schedule or budget constraints, inflation assumptions, 
and travel costs; 
* Specify equipment the government is to furnish as well as the use of 
existing facilities or new modification or development; 
* Identify prime contractor and major subcontractors; 
* Determine technology refresh cycles, technology assumptions, and new 
technology to be developed; 
* Define commonality with legacy systems and assumed heritage savings; 
* Describe effects of new ways of doing business. 

Step: 6; 
Leading practice: Obtain data; 
Associated task: 
* Create a data collection plan with emphasis on collecting current 
and relevant technical, programmatic, cost, and risk data; 
* Investigate possible data sources; 
* Collect data and normalize them for cost accounting, inflation, 
learning, and quantity adjustments; 
* Analyze the data for cost drivers, trends, and outliers and compare 
results against rules of thumb and standard factors derived from 
historical data; 
* Interview data sources and document all pertinent information, 
including an assessment of data reliability and accuracy; 
* Store data for future estimates. 

Step: 7; 
Leading practice: Develop point estimate and compare it to an 
independent cost estimate; 
Associated task: 
* Develop the cost model, estimating each WBS element, using the best 
methodology from the data collected, and including all estimating 
assumptions; 
* Express costs in constant year dollars; 
* Time-phase the results by spreading costs in the years they are 
expected to occur, based on the program schedule; 
* Sum the WBS elements to develop the overall point estimate; 
* Validate the estimate by looking for errors like double counting and 
omitted costs; 
* Compare estimate against the independent cost estimate and examine 
where and why there are differences; 
* Perform cross-checks on cost drivers to see if results are similar; 
* Update the model as more data become available or as changes occur 
and compare results against previous estimates. 

Step: 8; 
Leading practice: Conduct sensitivity analysis; 
Associated task: 
* Test the sensitivity of cost elements to changes in estimating input 
values and key assumptions; 
* Identify effects on the overall estimate of changing the program 
schedule or quantities; 
* Determine which assumptions are key cost drivers and which cost 
elements are affected most by changes. 

Step: 9; 
Leading practice: Conduct risk and uncertainty analysis; 
Associated task: 
* Determine and discuss with technical experts the level of cost, 
schedule, and technical risk associated with each WBS element; 
* Analyze each risk for its severity and probability; 
* Develop minimum, most likely, and maximum ranges for each risk 
element; 
* Determine type of risk distributions and reason for their use; 
* Ensure that risks are correlated; 
* Use an acceptable statistical analysis method (e.g., Monte Carlo 
simulation) to develop a confidence interval around the point estimate; 
* Identify the confidence level of the point estimate; 
* Identify the amount of contingency funding and add this to the point 
estimate to determine the risk-adjusted cost estimate; 
* Recommend that the project or program office develop a risk 
management plan to track and mitigate risks. 

Step: 10; 
Leading practice: Document the estimate; 
Associated task: 
* Document all steps used to develop the estimate so that a cost 
analyst unfamiliar with the program can recreate it quickly and 
produce the same result; 
* Document the purpose of the estimate, the team that prepared it, and 
who approved the estimate and on what date; 
* Describe the program, its schedule, and the technical baseline used 
to create the estimate; 
* Present the program's time-phased life-cycle cost; 
* Discuss all ground rules and assumptions; 
* Include auditable and traceable data sources for each cost element 
and document for all data sources how the data were normalized; 
* Describe in detail the estimating methodology and rationale used to 
derive each WBS element's cost (prefer more detail over less); 
* Describe the results of the risk, uncertainty, and sensitivity 
analyses and whether any contingency funds were identified; 
* Document how the estimate compares to the funding profile; 
* Track how this estimate compares to any previous estimates. 

Step: 11; 
Leading practice: Present estimate to management for approval; 
Associated task: 
* Develop a briefing that presents the documented life-cycle cost 
estimate; 
* Include an explanation of the technical and programmatic baseline 
and any uncertainties; 
* Compare the estimate to an independent cost estimate (ICE) and 
explain any differences; 
* Compare the estimate (life-cycle cost estimate (LCCE)) or 
independent cost estimate to the budget with enough detail to easily 
defend it by showing how it is accurate, complete, and high in quality; 
* Focus in a logical manner on the largest cost elements and cost 
drivers; 
* Make the content clear and complete so that those who are unfamiliar 
with it can easily comprehend the competence that underlies the 
estimate results; 
* Make backup slides available for more probing questions; 
* Act on and document feedback from management; 
* Request acceptance of the estimate. 

Step: 12; 
Leading practice: Update the estimate to reflect actual costs and 
changes; 
Associated task: 
* Update the estimate to reflect changes in technical or program 
assumptions or keep it current as the program passes through new 
phases or milestones; 
* Replace estimates with EVM, and independent estimate at completion 
(EAC) from the integrated EVM system; 
* Report progress on meeting cost and schedule estimates; 
* Perform a post mortem and document lessons learned for elements 
whose actual costs or schedules differ from the estimate; 
* Document all changes to the program and how they affect the cost 
estimate. 

Source: GAO, DHS, DOD, DOE, NASA, SCEA, and industry. 

[End of table] 

[End of section] 

Appendix II: Objectives, Scope, and Methodology: 

The House Appropriations Committee report accompanying the fiscal year 
2014 Legislative Branch Appropriations Bill (H.R. 2792) mandated that 
we review the Architect of the Capitol's (AOC) cost estimating 
methodology to ensure that AOC is accounting for all of the variables 
that should contribute to project cost estimates.[Footnote 36] This 
report addresses the extent to which AOC's policies and guidance for 
developing cost estimates conform to leading practices identified in 
GAO's Cost Estimating and Assessment Guide[Footnote 37] and provide a 
reliable basis to support funding and capital program decisions. This 
report also examines whether the estimates for the Capitol Dome and 
the Cannon Building projects reflect leading practices. 

To determine the extent to which AOC's policies and guidance comply 
with leading practices for cost estimating and provide a reliable 
basis to support funding and program decisions, we compared AOC's 
documents to leading practices set forth in GAO's Cost Estimating and 
Assessment Guide.[Footnote 38] The Cost Guide identifies 12 leading 
practices that represent work across the federal government and are 
the basis for a high-quality, reliable cost estimate. An estimate 
created using the leading practices exhibits four broad 
characteristics: it is accurate, well documented, credible, and 
comprehensive. That is, each characteristic is associated with a 
specific set of leading practices. In turn, each leading practice is 
made up of a number of specific tasks. (See app, I for a listing of 
the tasks that make up each of the 12 leading practices.) When the 
tasks associated with the leading practices that define a 
characteristic are mostly or completely satisfied, we consider the 
characteristic to be "substantially" or "fully" met. When all four 
characteristics are at least "substantially met"--we consider a cost 
estimate to be reliable.[Footnote 39] We shared our analysis with AOC 
officials to review, comment, and provide additional information, and 
we adjusted our analysis where appropriate. 

In reference to our examination of the estimates for the Capitol Dome 
and Cannon Building projects, we selected these projects based on 
their significance to the Capitol Complex, comparatively high costs, 
and public visibility. In addition, our review of the Cannon Building 
estimate is responsive to a mandate in the House Appropriations 
Committee report accompanying the fiscal year 2010 Legislative Branch 
Appropriations Bill (H.R. 2918) requiring us to monitor the progress 
of the project.[Footnote 40] We selected versions of the estimates to 
review that had the most complete information for our assessments. We 
assessed the reliability of data used in developing the estimates and 
found them to be sufficiently reliable for the purposes of this 
report. For example, while some source data were unavailable to us, we 
were able to assess AOC's process for building its estimates with the 
data and check for errors. We also interviewed AOC's staff and its 
technical consultant about these projects and observed existing 
conditions at the Cannon Building and Capitol Dome. While our review 
of these projects' cost estimates provides key insights and 
illustrates products of AOC's cost-estimating policies and guidance, 
the results of our review should not be used to make generalizations 
about all AOC project-cost estimates. We shared our analysis with AOC 
officials to review, comment, and provide additional information, and 
we adjusted our analysis where appropriate. 

We conducted our work from September 2013 to March 2014 in accordance 
with all sections of GAO's Quality Assurance Framework that are 
relevant to our objectives. The framework requires that we plan and 
perform the engagement to obtain sufficient and appropriate evidence 
to meet our stated objectives and to discuss any limitations in our 
work. We believe that the information and data obtained, and the 
analysis conducted, provide a reasonable basis for any findings and 
conclusions in this product. 

[End of section] 

Appendix III: GAO's Summary Assessments of AOC's Project Cost 
Estimates for the Cannon Building's Renewal and Capitol Dome's 
Restoration: 

Comprehensive: 

Estimating leading practice characteristic and rationale for 
assessment: The cost estimate should include both government and 
contractor costs of the program over its full life cycle, from 
inception of the program through design, development, deployment, 
operation and maintenance, to retirement of the program. It should 
also completely define the program, reflect the current schedule, and 
be technically reasonable. Comprehensive cost estimates should be 
structured in sufficient detail to ensure that cost elements are 
neither omitted nor double counted. Specifically, the cost estimate 
should be based on a product-oriented work breakdown structure (WBS) 
that allows a program to track cost and schedule by defined 
deliverables, such as hardware or software components. Finally, where 
information is limited and judgments must be made, the cost estimate 
should document all cost-influencing ground rules and assumptions. 

GAO assessment: Substantially Meets; 
Estimating leading practice characteristic and rationale for 
assessment: The Cannon Building cost estimate did not include all life 
cycle costs as AOC guidance did not require these life cycle costs to 
be identified. The estimate tracked well with the work required, as it 
had a WBS structure consistent with this type of effort, and clearly 
laid out and appeared to update ground rules and assumptions based on 
evolving understanding of the scope of work. 

GAO assessment: Substantially Meets; 
Estimating leading practice characteristic and rationale for 
assessment: The Capitol Dome cost estimate covers the construction 
phase of the dome's restoration. The cost estimate relies on 
specifications and drawings that exist in separate files. The cost 
estimate uses a standard WBS structure but does not include the 
underlying data. AOC submitted an extract of the data for our review. 
While the specifications include ground rules and assumptions, the 
cost estimate does not specify them. 

Well documented: 

Estimating leading practice characteristic and rationale for 
assessment: A good cost estimate--while taking the form of a single 
number--is supported by detailed documentation that describes how it 
was derived and how the expected funding will be spent in order to 
achieve a given objective. Therefore, the documentation should capture 
in writing such things as the source data used, the calculations 
performed and their results, and the estimating methodology used to 
derive each WBS element's cost. Moreover, this information should be 
captured in such a way that the data used to derive the estimate can 
be traced back to, and verified against, their sources so that the 
estimate can be easily replicated and updated. The documentation 
should also discuss the technical baseline description and how the 
data were normalized. Finally, the documentation should include 
evidence that the cost estimate was reviewed and accepted by 
management. 

GAO assessment: Substantially Meets; 
Estimating leading practice characteristic and rationale for 
assessment: The Cannon Building cost estimate documentation did not 
provide source data. It did provide a reasonable explanation for how 
the cost estimates were prepared, what type of data were used, and the 
basic estimating methodologies indicating some step by step processes. 
The documentation provided consisted of tables of unit values, 
quantities, and extended values. While extended costs were identified, 
it was difficult to follow the roll-up of these costs. Without a copy 
of or access to the cost model, we were unable to verify the accuracy 
of escalation calculations and we could not trace all of the logic 
from the source data to the resulting estimate. The calculations and 
changes to the baseline were not well documented but subsequent data 
and explanations highlighted the process for the changes. 
Additionally, soft costs and contingencies, comprising a substantial 
portion of total program costs, were only superficially addressed. 
There was evidence of management approval of the estimate. 

GAO assessment: Substantially Meets; 
Estimating leading practice characteristic and rationale for 
assessment: The Capitol Dome cost estimate captures all source 
documents used. The cost estimate relies on a WBS and shows how the 
calculations were performed. A cost analyst unfamiliar with the 
program could develop the cost estimate, although in places it is not 
clear. The cost estimate has a brief reference to the source documents 
used. Costs are presented in constant-year and then-year dollars. 

Accurate: 

Estimating leading practice characteristic and rationale for 
assessment: The cost estimate should provide for results that are 
unbiased, and it should not be overly conservative or optimistic. An 
estimate is accurate when it is based on an assessment of most likely 
costs, adjusted properly for inflation, and contains few, if any, 
minor mistakes. In addition, a cost estimate should be updated 
regularly to reflect significant changes in the program--such as when 
schedules or other assumptions change--and actual costs, so that it is 
always reflecting current status. During the update process, variances 
between planned and actual costs should be documented, explained, and 
reviewed. Among other things, the estimate should be grounded in a 
historical record of cost estimating and actual experiences on other 
comparable programs. 

GAO assessment: Substantially Meets; 
Estimating leading practice characteristic and rationale for 
assessment: The Cannon Building cost estimate included an uncertainty 
analysis indicating what confidence level the budget fell within, 
which was greater than 90 percent. Additionally, in the absence of 
having the cost model used to prepare the estimate, we were only able 
to validate the accuracy of a very small sample of WBS elements. 
Furthermore, while the estimate documentation identified inflation 
adjustments, because of missing calculation and conversion factors, we 
were unable to determine if the estimate has been adjusted properly 
for inflation. However, the documentation did provide a discussion of 
what and how the inflation adjustments were applied. 

GAO assessment: Partially Meets; 
Estimating leading practice characteristic and rationale for 
assessment: The Capitol Dome cost estimate did not include a risk and 
uncertainty analysis indicating what confidence level for the estimate 
the budget was set at. The estimate is shown in constant-year and then-
year dollars using indices for inflation. The indices used to adjust 
for inflation appear to be out-of-date. The estimate has few errors. 
AOC updated its cost estimate prior to beginning the project, but once 
the project began, AOC did not update the estimate. AOC did not 
document variances between planned and actual costs. AOC relied on 
historical records and on an industry database to support its 
estimate. The estimate uses an engineering build-up throughout, which 
appears appropriate for this project. 

Credible: 

Estimating leading practice characteristic and rationale for 
assessment: GAO assessment¢Ý: The cost estimate should discuss any 
limitations of the analysis because of uncertainty or biases 
surrounding data or assumptions. Major assumptions should be varied, 
and other outcomes recomputed to determine how sensitive they are to 
changes in the assumptions. Risk and uncertainty analysis should be 
performed to determine the level of risk associated with the estimate. 
Further, the estimate's cost drivers should be crosschecked, and an 
independent cost estimate conducted by a group outside the acquiring 
organization should be developed to determine whether other estimating 
methods produce similar results. 

GAO assessment: Partially Meets; 
Estimating leading practice characteristic and rationale for 
assessment: The Cannon Building cost estimate includes analyses of 
sensitivity, risk, and uncertainty. However, the sensitivity analysis 
was performed after the fact and was not used to inform the budget or 
provided for management consideration of risk prior to project start. 
AOC's risk and uncertainty analysis indicates that the budget for the 
project is set above the 90% confidence level, which may be 
unreasonably high. AOC has received multiple estimates from different 
sources (architect, construction manager, pre-construction contractor) 
as a cross-check for reasonableness. While these estimates are not 
necessarily outside the AOC program office's influence, they do appear 
to represent independent estimates based on each company's 
interpretation of the project's requirements and helped to validate 
the reliability of the estimates. 

GAO assessment: Minimally Meets; 
Estimating leading practice characteristic and rationale for 
assessment: The Capitol Dome cost estimate does not include a 
sensitivity, risk, and uncertainty analysis. AOC did not provide 
documented reasons explaining how the actual budgeted contingency 
amounts for unforeseen costs were developed. To mitigate risk, AOC 
negotiated unit costs to account for uncertainties in effort, such as 
the number of inches of stitching that would be needed to repair 
cracks. AOC does not appear to cross-check major cost elements, but 
AOC requested and obtained an independent cost estimate. AOC 
subsequently compared the point estimate and the independent cost 
estimate, and the two numbers were a close match. 

Source: GAO analysis of AOC documents. 

Fully Meets: AOC provided complete evidence that satisfies the 
associated tasks of the leading practices. 

Substantially Meets: AOC provided evidence that satisfies a large 
portion of the associated tasks of the leading practices. 

Partially Meets: AOC provided evidence that satisfies about half of 
the associated tasks of the leading practices. 

Minimally Meets: AOC provided evidence that satisfies a small portion 
of the associated tasks of the leading practices. 

Does Not Meet: AOC provided no evidence that satisfies the associated 
tasks of the leading practices. 

[End of table] 

[End of section] 

Appendix IV: Comments from the Architect of the Capitol: 

Note: GAO's comments supplementing those in the report's text appear 
at the end of this appendix. 

Architect of the Capitol: 
U.S. Capitol, Room SB-15: 
Washington, DC 20515: 
202-228-1793: 
[hyperlink, http://www.aoc.gov] 

March 14, 2014: 

Ms. Lorelei St. James: 
Director, Physical Infrastructure Issues: 
U.S. Government Accountability Office: 
Washington, D.C. 20548: 

Dear Ms. St. James: 

We are pleased to have the opportunity to comment on your draft report 
to House and Senate Committees on Appropriations regarding our 
construction project cost estimating process. We also are appreciative 
of your staff's willingness to explain your cost estimating guide as 
well as your recognition that given the varying nature and size of our 
projects, a cost-benefit approach is appropriate for deciding which 
steps in the guide should be applied. 

The Architect of the Capitol (AOC) has a systematic, thorough, and 
independent cost estimating process that we believe produces reliable 
estimates, and that our cost estimates for the Capitol Dome 
Restoration and Cannon Building Renewal projects are reliable as well. 
We appreciate that your draft report clearly notes that while your 
review of the Dome Restoration and Cannon Building Renewal projects' 
cost estimates provided key insights and illustrated "recent products 
of AOC's cost estimating policies and guidance," the results of your 
review "should not be used to make generalizations about all AOC 
project costs estimates." At the same time, we concur that 
incorporating leading practices you recommended in a cost-beneficial 
manner may make our cost estimates stronger as well as provide 
additional precision when setting contingency amounts. The strengths 
of our estimating process, as well as the steps we have taken or plan 
to take to implement your recommendations are summarized below. 

Estimating Process Produces Reliable Results: 

The AOC has a well-structured, detailed, and robust cost estimating 
process, carried out by skilled and experienced staff and contractors, 
that provides reliable cost estimates used to formulate our annual 
budget requests, and in particular, our Line Item Construction 
Projects. As your draft report establishes, our cost estimating 
process incorporates many leading practices in whole or in part. In
addition, we use best practices from a number of industry leaders and 
other federal agencies to ensure that appropriated dollars are 
invested in Congressional facilities wisely and effectively. In
those instances in which our process does not follow leading practices 
to the letter, it generally addresses the leading practice in a 
meaningful manner. For example, although the guide calls for 
quantitative sensitivity and risk and uncertainty analyses, we perform 
these qualitatively in a way that assists us in identifying risks and 
uncertainties and setting contingencies as well as effectively
managing risks during construction. In addition, our cost estimates 
are not produced by the organization benefiting from the project. 
Instead, they are developed by contractors who are overseen by an 
organization separate from the benefiting organization, and are 
rigorously reviewed by a group separate from the project team and 
benefiting jurisdiction. When appropriate, we obtain a second estimate 
from a different contractor. Cost reports are prepared, reviewed, and 
approved by management. In many instances, management briefings occur 
and estimates are accepted. However, we recognize the need to improve 
the level of documentation kept as a result of these briefings, and
other process steps to meet leading practices. 

Enhancing the AOC's Cost Estimating Process: 

The vast majority of the AOC's work addresses necessary repairs and 
maintenance to Congressional facilities or historic preservation 
projects as opposed to new construction. Many of these projects are
not high dollar projects such as those exemplified in your guide. 
Consequently, it would not be practical or cost effective to follow 
the Government Accountability Office's (GAO's) guide to the letter. We 
are pleased that you took costs and benefits into consideration in 
developing your recommendations. We have taken, or plan to take, the 
following actions to modify our cost estimating process to more fully 
address practices described in GAO's guide that will be cost effective 
and add value to our estimates. 

* Conduct sensitivity and risk and uncertainty analyses. We currently 
perform these analyses qualitatively, but will revise our policies and 
guidance to require that they be done quantitatively as specified in 
your guide for high dollar value projects prior to requesting 
construction funding. We have completed a quantitative sensitivity and 
risk and uncertainty analyses for the Cannon Building Renewal project 
and plan to conduct an integrated cost-schedule risk and uncertainty 
analysis when the design is complete. You also noted that neither our 
policies and guidance required, nor our cost estimate reports for the 
Dome Restoration project contained the reasons for contingency 
amounts. We will revise our policies and guidance to require this 
information going forward. Although we have received contractor 
pricing for the Dome Restoration project's options one and two, and 
thereby face less cost risk, we are considering performing a risk and 
uncertainty analysis for one or both of these options. [See comment 1] 

* Compare the point estimate to an independent cost estimate. GAO's 
guide appears to contemplate one estimate being performed by the 
organization benefiting from the project, and a second estimate being 
done by a separate, independent organization. In our process, the
benefiting organization does not perform the primary cost estimate. It 
is performed by a contractor being overseen by the AOC's Project 
Management Division. The Project Management Division is part of AOC's 
Planning and Project Management (PPM) organization, which is separate 
and apart from the benefiting jurisdiction. Our Cost Estimating Group, 
which is in PPM's Technical Support Division, is responsible for
reviewing cost estimates. Thus, while cost estimates are overseen and 
reviewed within PPM, these functions are carried out by staff who work 
under different supervisors. When we obtain a second estimate - 
performed by a different contractor and subcontractors - for specific 
projects, the organizational set up is the same. While we believe that 
this arrangement provides reasonable separation, we will revise our 
policies and guidance to require second estimates for all high dollar 
value projects, and ensure that our Cost Estimating Group has 
sufficient capacity to review the estimates. [See comment 2] 

Another issue discussed in the report is that our cost estimate 
reports do not include all costs. That is, our estimating contractor 
estimates construction contract costs, but not other costs, such as 
AOC management costs, life-cycle costs, or contingencies. Our policies 
and guidance did not require life-cycle costs, and we provide the 
estimating contractor percentage ranges for estimating the other 
costs. The December 2013 update to our Project Management Manual 
requires the use of life-cycle cost analysis. We will explore 
alternative approaches for estimating non-construction contractor 
costs. 

* Update estimates to reflect actual costs and changes. We update our 
estimates to reflect changes in scope or design prior to construction 
contract award. We also perform independent estimates for changes to 
contracts. However, as a practice, we have not updated estimates with 
actual cost data once construction begins. This is for two reasons. 
First, we do not receive actual costs from our contractors because our 
contracts are firm fixed price. Recognized as most appropriate for 
construction because the price is not subject to any adjustment on the 
basis of the contractor's actual costs, a firm fixed price contract 
places the maximum risk and responsibility for all costs and resulting 
profit or loss on the contractor. The invoices we receive are based on 
price, not costs. During the course of your review, your staff 
indicated that this information may be obtainable if we were to pay
contractors for it. Second, we do not believe having actual costs for 
projects underway or completed will usually be cost beneficial for 
future estimates because the prices we receive are market driven. 
However, there are certain instances in which having actual cost
information may be cost beneficial. We will explore those types of 
situations for which having actual cost will be beneficial and various 
approaches for obtaining these data. In addition, we will revise our 
policies and guidance to require comparison of invoice data to our 
cost estimates for new, high dollar value projects at the aggregate 
cost element level, such as for construction cost, AOC management 
costs, and contingencies. Although we currently use earned value 
analysis and cash flow curves to track costs against contract prices
for large projects based on contractor payments, we have not tracked 
such costs against our estimates. 

* Present estimates to management. Our policies and guidance require a 
written report to management on cost estimates, but do not require a 
management briefing, which would be consistent with the leading 
practice in your guide. Your draft report also noted that management 
briefings on the Dome Restoration project's estimated costs were not 
well documented, despite our efforts to demonstrate that numerous 
briefings - to both internal AOC management and Congressional decision 
makers - were held about this very high-profile project. We will 
revise our policies and guidance to require management briefings as
well as written reports on cost estimates for high dollar value 
projects. We will also specify the content of these briefings to be 
consistent with your guide, and will require the briefings to be 
documented along with management's decisions related to the estimates. 

* Identify ground rules and assumptions. Although our policies and 
guidance require ground rules and assumptions to be listed in estimate 
reports, they do not require that the risk impacts be identified if 
assumptions tum out to be incorrect, and the estimate reports for the
Dome Restoration project did not list assumptions or their risk 
impacts. For the Dome Restoration project, the assumptions were 
converted to contract specifications when the construction contract 
was developed. The risk impacts were addressed, but this process was
not documented. Our policies and guidance will be revised to emphasize 
the importance of listing the assumptions and require identifying the 
risk impacts in the cost estimate reports even if they are shown in 
the contract specifications. 

Regarding your recommendation that we provide confidence levels 
derived from risk and uncertainty analyses we perform on projects for 
which appropriated funding is being requested, we will explore 
developing a framework of best practices for performing sensitivity, 
risk, and uncertainty analyses to serve as the basis for setting 
contingency amounts on high dollar value projects. We will continue to 
prepare our cost estimates with the appropriate contingency levels
based on project risks and complexity, and will explore the most 
effective approach for communicating this information to Congressional 
decision makers. 

Cannon Project Estimation Process: 

We appreciate the recognition of the thoroughness of our estimating 
process for the Cannon Building Renewal project. There are, however, 
four issues on which we would like to provide additional comments. 

First, your draft report discusses three matters that are said to 
affect the quality of the risk and uncertainty analysis performed on 
the Cannon Building Renewal project's cost estimate. While the 
conditions your draft report discusses are correct, we believe they 
relate to the context of the conditions existing at the time of the 
analysis rather than its quality. 

1. The confidence level exceeded 90 percent, which may be unreasonably 
high. Given the need to maintain a $753 million budget, the project's 
early stage of design, and the significant risks associated with the 
project, we believe that the 90 percent confidence level is reasonable 
at this time. Risks include a significant amount of work scope that is 
still being defined, the known presence of hazardous materials, the 10-
year construction duration period, having to perform the work in an 
occupied building, and the historic nature of the building. We will 
reconsider the desired confidence level after the design is complete. 
[See comment 3] 

2. The distribution of estimated costs at various confidence levels 
was unusually narrow. According to our construction management 
contractor, MBP-AECOM, this situation is not unusual for construction 
projects. In construction projects, cost components can be broken
into labor, equipment and installed material. Due to the competitive 
market, the cost variation between the optimistic and pessimistic 
values (uncertainty modeling) can be narrow, whether it is related to 
the labor, equipment, or material. For the Cannon Building Renewal
project, at this stage of design development, one of the key variables 
is confidence in the "quantity take off," which is a function of the 
level of design development, which in turn is a function of scope 
definition. These uncertainties in the estimate were factored in the 
cost risk analysis, including external risks as identified in the 
project risk register. [See comment 4] 

3. Risk impacts were not analyzed individually, thereby preventing a 
determination as to which risks have the greatest impact and limiting 
AOe's ability to manage risks. Because of the project's early design 
stage and the associated uncertainties, our contractor, MBP-AECOM, did 
not believe that it would be meaningful or cost beneficial to apply the
risks individually at this time. However, it does plan to do this when 
it performs the next risk and uncertainty analysis when the design is 
complete. With regard to AOC's ability to manage risks on the project, 
we have a structured, systematic risk-management program in place for 
the Cannon Building Renewal project. We maintain a project risk 
register that prioritizes risks based on the probability of occurrence 
and the severity of impacts on cost and schedule. These risk items are 
updated monthly as to how the mitigation strategies are progressing. 
Additionally, the entire project team meets on a quarterly basis to 
review the existing risks and identify additional risks. Therefore, we 
do not believe that the way information from the risk register was 
used or quantified in performing the cost risk analysis has impaired 
the project team's effectiveness in managing risks. In fact, the 
project team has been proactively identifying and managing risk from 
the onset of this project. [See comment 4] 

Second, your draft report notes that you could not determine if the 
cost estimate was adjusted properly for inflation because all the 
information necessary to do this was not located in our files. 
Although this is correct, our contractors that performed the 
escalation analyses were willing to make this information available to 
GAO at the AOe's offices, and we did make this offer to your staff. To
address the issue you raised, we will reassess our policies and 
guidance in this area to determine whether it is practical and cost 
beneficial to maintain this information at the AOC. [See comment 5] 

Third, your draft report notes that the estimate did not include full 
life-cycle costs. We presume this meant life-cycle costs for all 
elements of the scope of work. Regarding the modernization of existing
historic buildings, life-cycle cost analysis for major building 
elements, such as structure and fasade cladding is not applicable. As 
GAO noted, our design contractor performed important life-cycle cost
analysis regarding the selection of an appropriate mechanical system. 
During the Program of Requirements phase of the project, the 
contractor performed a comprehensive analysis of eight mechanical 
system options that included conceptual layouts and extensive energy 
modeling. The recommended mechanical system achieved the best balance 
of fixed costs, ongoing maintenance costs, and other important 
selection criteria. Also, it is important to note that the AOC and our 
design contractor, Construction Manager as Agent (CMa), and 
Construction Manager as Constructor (CMc) partners are performing 
ongoing value analysis of alternative strategies for materials and 
systems regarding each submission as the design documents are further 
developed. [See comment 6] 

Fourth, in Appendix III, your draft report notes that soft costs and 
contingencies, comprising a substantial portion of total program 
costs, were only superficially addressed. We provided GAO with a 
narrative description of our soft cost rationale and a project budget 
with tabs of detailed information regarding soft cost line items. 
Although soft costs were not addressed in a similar manner as 
construction costs (reconciliation of multiple estimates), the AOC and 
our CMa have spent considerable time reducing soft cost risk. Most 
importantly, we have firm fixed price contracts from design 
development through project completion for a large portion of the soft 
costs including design costs, project management costs (CMa), and 
contractor preconstruction services (CMc). We have developed an AOC 
staffing plan through project completion as well as the related cost 
estimate. We have coordinated with the u.s. Capitol Police regarding 
estimates for construction security. We are working closely with the 
House Chief Administrative Officer in refining estimates for furniture,
fixtures, and equipment. We have a dedicated team working on swing 
space that has developed cost estimates. Under our CMc contracting 
method, we will be awarding a Guaranteed Maximum Price contract and 
will be reviewing both our construction contractor's and 
subcontractors' pricing before contract award and when we receive 
monthly invoices. Finally, we have established contingencies based on 
industry best practices relative to the unique risks associated with 
the Cannon Building Renewal project. 

Dome Project Estimation Process: 

Your draft report also contained a number of statements on the Capitol 
Dome Restoration project's cost estimating process that we believe 
warrant additional discussion. 

* Your draft report notes that we did not conduct quantitative 
sensitivity or risk and uncertainty analyses, use actual contractor 
cost data from previous projects, or document the reasons for the 
contingency amounts. We did, however, perform these analyses 
qualitatively and considered costs we paid for the Dome Skirt 
Restoration project - completed in September 2012 - and used this 
information in developing our cost estimate and contingency. We also 
used this information to limit our risks in the contract documents.
Consequently, the contract prices for the remaining Dome Restoration 
construction work are within 10 percent of our cost estimate. Without 
having a quantitative risk and uncertainty analysis, the draft report 
says that you could not see how the actions we took related to other
risk mitigation efforts and why we chose the ones we did over others, 
and that we could not determine the appropriate contingency level. We 
addressed the risks we identified as having the greatest potential 
effect on cost based on the Dome Skirt Restoration work and our
experience on other projects. We believe this gave us a reasonable 
basis for setting the contingency. We acknowledge that we did not 
always rigorously document this work and that a quantitative risk and 
uncertainty analysis may further strengthen our estimates. To address 
the issues you raised, we plan to revise our policies and guidance to 
emphasize the importance of adequately documenting our estimation 
steps, including setting contingencies and further documenting 
briefings to management. [See comment 7] 

* We did not use actual costs from completed phases to update our 
estimate. We used information from the Dome Skirt Restoration work in 
the estimation process. This enabled us to build on our experience 
from the most recent Dome Restoration work, specifically as it related 
to identifying cost drivers, market impacts, and risk management. We 
acknowledge that the use of these data was not well documented in our 
files. [See comment 8] 

* We did not update our estimate with actual costs as the work 
progressed. Doing this is not part of our estimation process. To 
address this issue, we will revise our policies and guidance to 
require comparison of our estimates to actual invoices we receive for 
high dollar value projects. We will also explore obtaining actual cost 
data for selected projects where we believe it will be cost 
beneficial. We believe it is necessary to exercise prudence on this 
issue because obtaining actual cost data on firm fixed price contracts 
will likely add cost to the project but may have limited value in 
determining the reasonableness of estimates because the market will 
likely drive the cost we eventually pay. [See comment 8] 

* Indicies used for escalation appeared outdated. As the draft report 
indicates, we estimate costs at various stages of the design process 
and the estimate reports from contractors are often revised after the 
AOC reviews them. In this case, we inadvertently did not initially
provide the most recent version of the estimate report with the 
correct inflation data. However, we did provide it during the course 
of your field work. [See comment 9] 

* The cost estimate report did not include a comprehensive list of 
assumptions. This is because we converted the assumptions made during 
the design process into contract specifications, such as enumerating 
the allowable times construction work could be carried out. We will 
revise our policies and guidelines to emphasize the importance of 
including this list and the associated risk impacts in the estimate 
report even when assumptions are incorporated into contract documents. 

Again, thank you for the opportunity to review and comment on this 
draft report. We believe that implementing the recommendations will 
further improve our cost estimating process. At the same time, 
although we have not always fully documented all the steps we take, we 
have a robust cost estimating process that embraces industry best 
practices and produces credible and reliable results. This is amply 
demonstrated by the recent on time and within budget completion of the 
Capitol Dome Skirt Restoration and the Inaugural Stand construction 
projects. Please don't hesitate to contact me should you have any 
questions on our comments. 

Sincerely, 

Signed by: 

Stephen T. Ayers, FAIA, LEED AP: 
Architect of the Capitol: 

Doc No. 140314-16-01. 

The following are GAO's comments on the Architect of the Capitol's 
letter dated March 14, 2014. 

GAO Comments: 

1. We continue to believe that a quantitative risk and uncertainty 
analysis enables more effective oversight than can be obtained by 
taking a qualitative approach. A quantitative risk and uncertainty 
analysis conveys the confidence level in achieving the most likely 
cost and provides actionable information about cost, schedule, and 
technical risks that cannot be obtained qualitatively. 

2. While AOC's contractors act independently, AOC's Project Management 
Division is responsible for the estimates prepared via contract, and 
AOC's Cost Estimating Group is responsible for reviewing them. Because 
AOC's Project Management Division and Cost Estimating Group staff are 
part of the same Planning and Project Management organization, the 
Cost Estimating Group staff could be biased because of organizational 
influences and unable to provide a fully objective review of costs to 
the Project Management Division. 

3. While our leading practices do not specify target confidence 
levels, experts we consulted with in developing our leading practices 
agreed that program cost estimates should be budgeted to at least the 
55 percent confidence level and potentially as high as 80 percent. 
While AOC has good reasons for its interest in maintaining the 
project's budget at a high confidence level, we believe that the 
confidence level for the project's budget--exceeding 90 percent--may 
be excessive. 

4. AOC identified 60 risks to the project and modeled them in 
aggregate, as opposed to individually, in conducting its quantitative 
risk and uncertainty analysis. Modeling the aggregated risks precludes 
AOC from identifying relationships among risk elements and determining 
which risks have the greatest influence on project costs. We continue 
to believe that modeling the risks in aggregate likely contributes to 
the overly narrow range of estimated costs over the confidence levels, 
and can overstate the effect of the risks. 

5. Our leading practices indicate that an estimate's documentation 
should be detailed enough so that the derivation of each cost element 
can be traced to all sources allowing the estimate to be easily 
replicated and updated. Because some information was not readily 
available in documents maintained by AOC, we were unable to determine 
if the estimate has been adjusted properly for inflation. 

6. AOC has taken positive steps in conducting a life-cycle cost 
analysis that considered fixed costs and ongoing maintenance costs to 
inform the AOC's selection of mechanical systems for heating and 
cooling the Cannon Building. However, AOC did not conduct life-cycle 
analyses for other components of the project, such as the building's 
roof and alarm system. This precludes AOC from capturing the total 
cost of the project and evaluating design alternatives on a total-cost 
basis. 

7. AOC takes positive steps to qualitatively identify risks and 
uncertainty and assess sensitivity. While this analysis is useful in 
setting contingencies and managing risks during construction, a 
quantitative risk and uncertainty analysis enables more effective 
oversight because it conveys the confidence level in achieving the 
most likely cost and provides actionable information about cost, 
schedule, and technical risks that cannot be obtained qualitatively. 

8. While AOC has used some information from completed phases of the 
Capitol Dome's restoration project to inform development of the 
estimate for later phases, not having information on actual costs 
makes it difficult for AOC to assess the difference between the price 
it paid and its contractors' costs to determine the estimate's 
reasonableness. 

9. Our leading practices indicate that an estimate's documentation 
should be detailed enough so that the derivation of each cost element 
can be traced to all sources allowing the estimate to be easily 
replicated and updated. Because some information was out-of-date at 
the time of our review, we were unable to determine if the estimate 
had the proper escalation adjustments. 

[End of section] 

Appendix V: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Lorelei St. James, (202) 512-2834 or stjamesl@gao.gov: 

Staff Acknowledgments: 

In addition to the contact named above Michael Armes, Assistant 
Director; Karen Richey, Assistant Director; George Depaoli, Analyst-in-
Charge; Laura Erion; Emile Ettedgui; Colin Fallon; Geoffrey Hamilton; 
James Manzo; Faye Morrison; and Vanessa Welker made key contributions 
to this report. 

[End of section] 

Footnotes: 

[1] For the purposes of this report, a cost estimate is the summation 
of individual cost elements, using established methods and valid data, 
to estimate the future costs of a program, based on what is known 
today. AOC's estimates for the Capitol Dome restoration and Cannon 
Building renewal include costs for planning, design, construction, and 
implementation. Implementation includes costs for construction 
contract administration, testing, inspection, and quality control. 

[2] H. R. Rep. No. 113-173, at 11 (2013). 

[3] GAO, GAO Cost Estimating and Assessment Guide: Best Practices for 
Developing and Managing Capital Program Costs, [hyperlink, 
http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: March 2009). 

[4] [hyperlink, http://www.gao.gov/products/GAO-09-3SP]. 

[5] We established five descriptions for our assessments of leading 
practices and cost estimate characteristics: Fully Meets, 
Substantially Meets, Partially Meets, Minimally Meets, and Does Not 
Meet. We consider a leading practice to be fully met when the 
associated tasks are completely satisfied; substantially met when a 
large portion of the associated tasks are satisfied; partially met 
when about half of the associated tasks are satisfied; minimally met 
when a small portion of the associated tasks are satisfied; and not 
met when none of the associated tasks are satisfied. Our assessment 
method weights each leading practice equally and bases the assessment 
of each characteristic on the average score of underlying leading 
practices. We assign each description a numerical value (5 for Fully 
Meets to 1 for Does Not Meet) and round scores to the higher numerical 
value (i.e., a score of 4.5 would round up to 5). 

[6] H. R. Rep. No. 111-160, at 21 (2009). 

[7] [hyperlink, http://www.gao.gov/products/GAO-09-3SP]. 

[8] See footnote 5. 

[9] Architect of the Capitol, Budget Justification Fiscal Year 2014 
(Washington, D.C.). 

[10] Condition assessments provide information on a facility's state 
of repair and are developed from inspections of structural, 
electrical, mechanical, plumbing, and other building systems. FCA 
reports typically catalog all of the deficiencies identified during 
the inspections along with the estimated cost to correct the problems. 

[11] GAO, Architect of the Capitol: Plans for Renovating the Cannon 
House Office Building and Garages, [hyperlink, 
http://www.gao.gov/products/GAO-09-673T] (Washington, D.C.: May, 6, 
2009). 

[12] [hyperlink, http://www.gao.gov/products/GAO-09-3SP]. 

[13] See Appendix I for the associated tasks within the 12 leading 
practices. 

[14] See footnote 5. 

[15] A WBS deconstructs a program's end product into successive levels 
with smaller specific elements until the work is subdivided to a level 
suitable for management control. For estimates, AOC requires use of 
the ASTM UNIFORMAT II structure, which is organized according to 
systems or assemblies that are common to most buildings. 

[16] AOC's policies and guidelines define cost reports that are 
relevant to various points of a project's development. Reports range 
from "exploratory cost assessments," conducted at a pre-planning 
stage, to a "construction cost estimate," based on completed and 
approved construction documents. 

[17] In the context of this report, a program refers to a group of 
projects or functions associated with delivering a capital asset. For 
example, the Cannon House Office Building program refers to projects 
or functions--completed via contract or by AOC staff--for planning, 
designing, and constructing all phases of the renewal work and 
providing for ongoing operations and maintenance of the facility. 

[18] We determined that AOC fully meets the well-documented 
characteristic even though it does not fully meet the requirements of 
the six underlying leading practices. Our assessment method weights 
each leading practice equally and bases the category-level assessment 
on the average score of underlying leading practices. Because the 
average score of all leading practices contributing to the well 
documented characteristic is at the fully meets level, we say that AOC 
fully meets the requirements of the well-documented characteristic. 

[19] Industry sources include the R.S. Means Cost Database. 

[20] According to the Cost Guide, a "sensitivity analysis" addresses 
some of the estimating uncertainty by testing discrete cases of 
assumptions and other factors that could change. By examining each 
assumption or factor independently, while holding all others constant, 
the cost estimator can evaluate the results to discover which 
assumptions or factors most influence the estimate. A sensitivity 
analysis also requires estimating the high and low uncertainty ranges 
for significant cost driver input factors. To determine what the key 
cost drivers are, a cost estimator needs to determine the percentage 
of total cost that each cost element represents. The major 
contributing variables within the highest percentage cost elements are 
the key cost drivers that should be varied in a sensitivity analysis. 

[21] "Quantitative-risk and uncertainty analysis" provide a way to 
assess the variability in the estimate. Using this type of analysis, a 
cost estimator can model such effects as schedules slipping, missions 
changing, and proposed solutions not meeting user needs, allowing for 
a known range of potential costs. 

[22] AOC applies additional percentages to the items in the cost 
estimate for contingencies at a series of milestones in completing the 
design and construction documents of the project as established by AOC 
policy. For example, early in the projects' design, AOC establishes a 
contingency for design at 20 to 25 percent and construction at 5 to 20 
percent. When the design is completed, the design contingency is 
removed. 

[23] The analysis produces a cumulative probability distribution, more 
commonly known as an S curve--usually derived from a simulation such 
as Monte Carlo--and is useful in portraying the uncertainty 
implications of various cost estimates. 

[24] A "point estimate" is the most likely value for the cost 
estimate, given the underlying data. The "level of confidence for the 
point estimate" is the probability that the point estimate will 
actually be met. For example, if the confidence level for a point 
estimate is 80 percent, there is an 80 percent chance that the final 
cost will be at or below the point estimate and a 20 percent chance 
that costs will exceed the point estimate. 

[25] The strengths and weakness in the Cannon Building' renewal and 
Capitol Dome's restoration estimates are generally consistent with the 
strengths and weaknesses we found in reviewing other agencies' 
estimates. Estimates that we have reviewed include those from the 
departments of Agriculture, Commerce, Defense, Energy, Homeland 
Security, Transportation, and Veterans Affairs; the Internal Revenue 
Service; and the Missile Defense Agency. 

[26] Our initial assessments were based upon documentation provided by 
AOC over a period of 3 months from October to December 2013. 

[27] While our review of these projects' cost estimates provides key 
insights and illustrates products of AOC's cost-estimating 
methodology, the results of our review cannot be generalizable to the 
universe of all AOC project cost estimates. 

[28] Appendix III provides additional information on our assessment of 
these estimates compared to the characteristics of high-quality, 
reliable estimates. 

[29] See footnote 15. 

[30] AOC indicated that source data for one of the estimates we 
reviewed derives from a proprietary system maintained by its 
contractor. As such, AOC said that the source information contained in 
this system could not be provided to us. 

[31] We did not assess the cost of Cannon Building's renewal project 
estimate for one of the leading practices for accuracy. As 
construction has not begun on the project, AOC has not had the 
opportunity to update the estimate with actual costs. 

[32] While AOC's policies and guidance for developing cost estimates 
do not require conducting a risk and uncertainty analysis, AOC 
officials determined during the course of our review that it would be 
beneficial to perform a quantitative-risk and uncertainty analysis on 
the cost estimate for the Cannon Building's renewal. AOC provided a 
draft of its analysis in January 2014, and our report discusses our 
assessment of AOC's draft analysis. 

[33] A project funded to an unreasonably high confidence level may be 
over-budgeted. While our leading practices do not specify target 
confidence levels, experts we consulted in developing our leading 
practices agreed that program cost estimates should be budgeted to at 
least the 55 percent confidence level and potentially as high as 80 
percent. 

[34] AOC identified 60 risks to the project in conducting its risk and 
uncertainty analysis and estimated that their potential cost impact 
was $161 million. AOC used the aggregated risks in creating a 
distribution of construction costs using a Monte Carlo simulation. 

[35] AOC's analysis shows, for example, the difference between the 
estimate at the 50 percent confidence level and 90 percent confidence 
level is about $25 million, approximately 3 percent of the total 
estimate. Modeling the risks individually would likely have resulted 
in a greater spread between the 50 and 90 percent confidence levels 
and given a more accurate indication of expected costs and of which 
risks were most significant. 

[36] H.R. Rep. No. 113-173, at 11 (2013). 

[37] GAO, GAO Cost Estimating and Assessment Guide: Best Practices for 
Developing and Managing Capital Program Costs, [hyperlink, 
http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: March 2009). 

[38] [hyperlink, http://www.gao.gov/products/GAO-09-3SP]. 

[39] We established five descriptions for our assessments of leading 
practices and cost estimate characteristics: Fully Meets, 
Substantially Meets, Partially Meets, Minimally Meets, and Does Not 
Meet. We consider a leading practice to be fully met when the 
associated tasks are completely satisfied; substantially met when a 
large portion of the associated tasks are satisfied; partially met 
when about half of the associated tasks are satisfied; minimally met 
when a small portion of the associated tasks are satisfied; and not 
met when none of the associated tasks are satisfied. Our assessment 
method weights each leading practice equally and bases the assessment 
of each characteristic on the average score of underlying leading 
practices. We assign each description a numerical value (5 for Fully 
Meets to 1 for Does Not Meet) and round scores to the higher numerical 
value (i.e. a score of 4.5 would round up to 5). 

[40] H. R. Rep. No. 111-160, at 21 (2009). 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation, and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the 
performance and accountability of the federal government for the 
American people. GAO examines the use of public funds; evaluates 
federal programs and policies; and provides analyses, recommendations, 
and other assistance to help Congress make informed oversight, policy, 
and funding decisions. GAO's commitment to good government is 
reflected in its core values of accountability, integrity, and 
reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's website [hyperlink, http://www.gao.gov]. Each 
weekday afternoon, GAO posts on its website newly released reports, 
testimony, and correspondence. To have GAO e-mail you a list of newly 
posted products, go to [hyperlink, http://www.gao.gov] and select 
"E-mail Updates." 

Order by Phone: 

The price of each GAO publication reflects GAO's actual cost of 
production and distribution and depends on the number of pages in the 
publication and whether the publication is printed in color or black 
and white. Pricing and ordering information is posted on GAO's 
website, [hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card, 
MasterCard, Visa, check, or money order. Call for additional 
information. 

Connect with GAO: 

Connect with GAO on facebook, flickr, twitter, and YouTube.
Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts.
Visit GAO on the web at [hyperlink, http://www.gao.gov]. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 
Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; 
E-mail: fraudnet@gao.gov; 
Automated answering system: (800) 424-5454 or (202) 512-7470. 

Congressional Relations: 

Katherine Siggerud, Managing Director, siggerudk@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, DC 20548. 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, DC 20548. 

[End of document]