Technology Assessment Design Handbook
GAO-20-246G: Published: Dec 4, 2019. Publicly Released: Dec 4, 2019.
The Technology Assessment Design Handbook offers both GAO’s own staff and other interested users tools and approaches to think about when designing technology assessments. The handbook helps users analyze the impact of technology and make complex issues more easily understood and useful to policymakers.
The handbook outlines:
what to think about when designing assessments
what to think about when including policy options in assessments
examples of design and methodology
potential challenges and some ways to address them
We expect to update the handbook based on public comments and additional experiences. Send comments to TAHandbook@gao.gov.
The front of GAO Headquarters
What GAO Found
The Technology Assessment Design Handbook identifies tools and approaches GAO staff and others can consider in the design of robust and rigorous technology assessments (TAs). The handbook underscores the importance of TA design (Chapter 1), outlines the process of designing TAs (Chapter 2), and describes approaches for mitigating selected TA design and implementation challenges (Chapter 3). While the primary audience of this handbook is GAO staff, we expect that other organizations engaged or interested in TAs will find portions of this handbook useful. We anticipate modifying and refining this handbook, as needed, based on experience and public comments received. We will accept comments on this handbook at TAHandbook@gao.gov for approximately 1 year after publication.
The handbook identifies three general design phases, as appropriate, as shown in the figure below. The handbook also highlights seven cross-cutting considerations for designing TAs: the iterative nature of TA design, the requester’s interests, resources, independence, stakeholder engagement, potential challenges, and communication. In addition, the handbook provides a high-level process for developing policy options, as a tool for analyzing and articulating what a policymaker could do in the context of a given technology and policy goal. Steps in developing policy options include, as applicable: formulating initial policy options to consider; gathering evidence, determining relevant dimensions to analyze, and analyzing the policy options; and presenting the results of the policy analysis.
Summary of Key Phases of Technology Assessment Design
We found that GAO TAs have and can use a variety of design approaches and methods. The handbook provides TA design and methodology examples, including related to objectives commonly found in GAO TAs, such as: describe a technology, assess opportunities and challenges of a technology, and assess policy considerations. One example provided is: some GAO TAs include an objective related to describing the status and feasibility of a technology, which GAO teams have done by using methodologies such as expert panels, interviews, literature and document reviews, site visits, and determining the Technology Readiness Level.
Also included in the handbook are examples of TA design and implementation challenges we found, along with possible mitigation strategies. We identified four general categories of challenges, including: (1) ensuring TA products are useful for Congress and others; (2) determining policy goals and measuring impact; (3) researching and communicating complicated issues; and (4) engaging all relevant stakeholders. An example of a potential mitigation strategy to the specific challenge of writing simply and clearly about technical subjects includes: allowing sufficient amount of time for writing, including reviewing and revising writing.
Why GAO Did This Study
In January 2019, at the direction of Congress, GAO formed the Science, Technology Assessment, and Analytics team to expand its work on cutting-edge science and technology issues, and to provide oversight, insight, and foresight for science and technology. TAs can be used to strengthen decision-making, enhance knowledge and awareness, and provide early insights into the potential impacts of technology. TA design can enhance TA quality, credibility, and usefulness; ensure its independence; and ensure effective use of resources.
Under the Comptroller General authority, we developed this handbook using the format of the 2012 GAO methodology transfer paper, Designing Evaluations. Below is a summary of the approach we used to affirm and document TA design steps and considerations for this handbook.
- Reviewed select GAO documents, including Designing Evaluations (GAO-12-208G), published GAO TAs, select GAO products that presented policy options, and other GAO reports
- Reviewed select Office of Technology Assessment reports
- Reviewed select Congressional Research Service reports
- Reviewed select literature on TAs and related to development and analysis of policy options
- Held an expert forum to gather experts’ input on TA design
- Considered experiences of GAO teams that have successfully assessed and incorporated policy options into GAO products, as well as GAO teams that are currently incorporating policy options into their TA design
- Collected input from GAO staff who provided key contributions to GAO TAs, regarding challenges to TA design and implementation and possible solutions
For more information, contact Timothy Persons at (202) 512-6888 or PersonsT@gao.gov.