This is the accessible text file for GAO report number GAO-14-306 
entitled 'Office of Personnel Management: Agency Needs to Improve 
Outcome Measures to Demonstrate the Value of Its Innovation Lab' which 
was released on April 30, 2014. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as 
part of a longer term project to improve GAO products' accessibility. 
Every attempt has been made to maintain the structural and data 
integrity of the original printed product. Accessibility features, 
such as text descriptions of tables, consecutively numbered footnotes 
placed at the end of the file, and the text of agency comment letters, 
are provided but may not exactly duplicate the presentation or format 
of the printed version. The portable document format (PDF) file is an 
exact electronic replica of the printed version. We welcome your 
feedback. Please E-mail your comments regarding the contents or 
accessibility features of this document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

United States Government Accountability Office: 
GAO: 

Report to Congressional Requesters: 

March 2014: 

Office of Personnel Management: 

Agency Needs to Improve Outcome Measures to Demonstrate the Value of 
Its Innovation Lab: 

GAO-14-306: 

GAO Highlights: 

Highlights of GAO-14-306, a report to congressional requesters. 

Why GAO Did This Study: 

Organizations from around the globe are emphasizing that strategies 
promoting innovation are vital to solving complex problems. To try to 
instill a culture of innovation in its agency, OPM followed the lead 
of a number of private sector companies, nonprofit organizations, and 
government bodies by creating an innovation lab. GAO was asked to 
examine the lab. 

Specifically, GAO, 1) described the lab's start-up costs, staffing and 
organization, activities, and policies governing the lab's use, and 2) 
assessed how OPM's innovation lab compares to other organizations' 
innovation labs, including how it uses benchmarks and metrics and how 
it addresses challenges to innovation. GAO reviewed cost, staffing, 
and performance information. GAO also reviewed relevant literature on 
innovation and interviewed officials from public, private, and 
nonprofit organizations with innovation facilities similar to OPM's 
lab. 

What GAO Found: 

In March 2012, the Office of Personnel Management (OPM) opened its 
innovation lab, a distinct physical space with a set of policies for 
engaging people and using technology in problem solving. The goals of 
OPM's innovation lab are to provide federal workers with 21st century 
skills in design-led innovation, such as intelligent risk-taking to 
develop new services, products, and processes. OPM's lab was built at 
a reported cost of $1.1 million, including facility upgrades and 
construction, equipment and training, and other personnel costs. The 
lab employs approximately 6 full-time equivalents, including a 
director, and in fiscal year 2013, the lab's operating costs were 
approximately $476,000, including salaries. 

Figure: View of OPM's Innovation Lab: 

[Refer to PDF for image: photograph] 

Source: OPM. 

[End of figure] 

OPM's innovation lab is similar in mission and design to other 
innovation labs GAO reviewed, and OPM has incorporated some of the 
prevalent practices that other labs use to sustain their operations. 
Specifically, OPM is using its lab for a variety of projects, 
including as a classroom for building the capacity to innovate in the 
federal government. Lab staff indicated that they plan to begin long-
term immersion projects—complex projects with diverse users—within a 
few months. OPM plans to develop and implement evaluation plans 
specific to each immersion project that will help them track cost 
benefits or performance improvement benefits associated with the 
projects.
Starting in March 2013, OPM lab staff began work on a program 
evaluation framework to more systematically measure the lab's progress 
toward meeting its overarching goals. In addition, lab staff members 
are tracking lab activities, such as classes and workshops, and are 
surveying lab users about the quality of their experience in the lab. 
However, they have not developed performance targets or measures 
related to project outcomes, and without a rigorous evaluation 
framework that can help OPM track the lab's performance, it will be 
hard to demonstrate that the lab is operating as originally envisioned.
While labs provide a physical space where innovators can convene, 
federal agencies are not fully aware of their growing community. 
However, OPM is taking steps to ensure work done in the lab is shared 
across OPM and with other federal innovators—for example, by hosting 
weekly training sessions in the lab on best practices. Studies show 
that information sharing and interorganizational networks can be a 
powerful driver supporting innovation. 

What GAO Recommends: 

Among other things, GAO recommends that the Director of OPM should 
direct lab staff to 1) develop a mix of performance targets and 
measures to help them monitor and report on progress toward lab goals, 
and 2) build on existing efforts to share information with other 
agencies that have innovation labs. OPM generally concurred with GAO's 
recommendations; in addition, they described the steps being taken and 
planned to refine their ongoing evaluation efforts and to further 
leverage other federal innovation labs. 

View GAO-14-306. For more information, contact Seto J. Bagdoyan at 
(202) 512-4749 or bagdoyans@gao.gov. 

[End of section] 

Contents: 

Letter: 

Background: 

OPM Lab Activities Are Intended to Build Capacity for Innovation and 
Support Project-Based Problem Solving: 

OPM's Lab Is Similar in Mission and Design to Other Innovation Labs, 
but OPM Needs to Systematically Evaluate the Lab's Performance: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Objectives, Scope, and Methodology: 

Appendix II: Innovation Labs Share Similar Design Elements and Problem-
Solving Methods: 

Appendix III: Office of Personnel Management's Innovation Lab Non-
Design Programmed Activities, Calendar Years 2012-13: 

Appendix IV: Comments from the Office of Personnel Management: 

Appendix V: GAO Contact and Staff Acknowledgments: 

Tables: 

Table 1: Breakdown of OPM's Lab Start-Up Costs (Fiscal Year 2012): 

Table 2: Description of OPM's Lab Staff for Fiscal Year 2013: 

Table 3: OPM's Innovation Lab Sessions with Human-Centered Design 
Programming, Calendar Years 2012 and 2013: 

Table 4: List of Innovation Labs with Facilities Similar to OPM: 

Figures: 

Figure 1: OPM's Lab: 

Figure 2: Innovation Labs Share Similar Design Elements and Problem-
Solving Methods: 

Figure 3: Five-Step Framework for Guiding Innovation in the Census and 
HUD Labs: 

Abbreviations: 

CAT: Center for Applied Technology: 

Census: U.S. Census Bureau: 

CFPB: Consumer Financial Protection Bureau: 

FCAT: Fidelity Center for Applied Technology: 

FDA: Food and Drug Administration: 

FTE: Full-time equivalent: 

GSA: General Services Administration: 

HUD: Housing and Urban Development: 

KSC: Kennedy Space Center: 

NASA: National Aeronautics and Space Administration: 

NERD: New England Research and Development: 

OPM: Office of Personnel Management: 

STEM: Science, technology, engineering, and mathematics: 

TACSI: Australian Center for Social Innovation: 

[End of section] 

United States Government Accountability Office: 
GAO:
441 G St. N.W. 
Washington, DC 20548: 

March 31, 2014: 

The Honorable Darrell Issa: 
Chairman: 
Committee on Oversight and Government Reform: 
House of Representatives: 

The Honorable Blake Farenthold: 
Chairman: 
Subcommittee on Federal Workforce, U.S. Postal Service and the Census: 
Committee on Oversight and Government Reform: 
House of Representatives: 

Organizations from around the globe are emphasizing that strategies 
promoting innovation are vital to solving complex problems and that 
future growth will increasingly depend on productivity-raising 
innovation.[Footnote 1] To try to instill a culture of innovation in 
its agency, in March 2012, the Office of Personnel Management (OPM) 
followed the lead of a number of private sector companies, nonprofit 
organizations, and U.S. and foreign government bodies and created an 
innovation lab. An innovation lab is generally defined as a distinct 
physical space with a set of policies for engaging people and using 
technology in problem solving. OPM's lab, located in the sub-basement 
of its Washington, D.C. headquarters, uses a problem-solving approach 
called "human-centered design," which is described as the discipline 
of generating concrete solutions and ideas driven by the needs, 
behaviors, and context of the people for whom they are designed. This 
approach also uses iterative testing of new services, products, and 
processes and is intended to encourage faster learning at the front 
end of the design process rather than large, expensive failures upon 
implementation.[Footnote 2] The premise behind OPM's lab was that the 
new space and a new approach to problem solving would support efforts 
to address some complex issues and eventually lead to a more 
innovative agency culture. 

You requested that we review OPM's innovation lab. This report (1) 
describes the OPM innovation lab's start-up and operating costs, 
staffing and organization, activities, and policies governing the 
lab's use, and (2) assesses how OPM's innovation lab compares to other 
organizations' innovation labs, including how it uses benchmarks and 
associated metrics and how it addresses potential challenges to 
innovation. 

To address our first objective, we reviewed documentation on OPM's 
start-up and operating costs, staffing, and policies regarding the lab 
and interviewed OPM staff overseeing the lab and its activities. To 
address the second objective, we conducted a comprehensive literature 
search--including research conducted by global consulting firms and 
academic institutions--and interviewed representatives familiar with 
innovation facilities. We reviewed documentation and interviewed 
officials from the U.S. Census Bureau (Census), the Department of 
Housing and Urban Development (HUD), and the National Aeronautics and 
Space Administration (NASA) innovation labs. In addition, we met with 
an official from the Consumer Financial Protection Bureau (CFPB). 
While CFPB lacks a dedicated innovation lab, the agency has a 
reputation among federal agencies as a leader in innovative website 
development. We also interviewed representatives from other 
government, private sector, and nonprofit organization innovation 
facilities, such as Denmark's MindLab, the Fidelity Center for Applied 
Technology, and UNICEF's innovation labs.[Footnote 3] To select 
organizations for comparison, we identified those with innovation 
facilities similar to OPM's lab, including those with a dedicated 
physical space and using similar problem-solving methods. We examined 
documentation and interviewed OPM lab staff regarding identifying any 
outcomes--such as cost reductions, performance improvements, or other 
results from projects--undertaken by the lab since its inception. We 
also conducted a detailed literature search to identify benchmarks and 
associated metrics applicable to the development and use of innovation 
facilities in the public and private sectors. Based on our literature 
search, we identified common challenges that can hamper organizations' 
efforts to use labs as innovation vehicles and prevalent practices 
that can support labs' success and sustainability. In addition, we 
reviewed our interview records to identify commonly recurring 
challenges and prevalent practices that can support labs' success and 
sustainability. We verified that the challenges and prevalent 
practices we identified during our literature search were also those 
more often cited during the interviews. 

We conducted this performance audit from July 2013 to March 2014 in 
accordance with generally accepted government auditing standards. 
Those standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe 
that the evidence obtained provides a reasonable basis for our 
findings and conclusions based on our audit objectives. More detailed 
information on our scope and methodology appears in appendix I. 

Background: 

Innovation is a dynamic process through which problems and challenges 
are defined, new and creative ideas are developed, and new solutions 
are selected and implemented.[Footnote 4] It is also a complex process 
that involves taking iterative steps to solve problems. Innovation 
requires an environment that encourages participants to challenge 
traditional practices without fear of repercussions. Ideally, 
innovation participants are empowered to be creative and make 
mistakes, and appropriate risk-taking is not only tolerated but 
encouraged. 

Some federal leaders are trying various innovation tools, including on-
line idea submission programs, competitions, and prizes, as ways of 
unleashing employee creativity. For example, in a memorandum issued in 
March 2010, the administration urged federal agencies to use 
challenges and prizes to crowdsource[Footnote 5] innovative approaches 
to government initiatives and programs.[Footnote 6] At relatively low 
costs, crowdsourcing initiatives can garner valuable and creative 
solutions that may not have come through traditional means. As another 
example, the Presidential Innovation Fellows program pairs top 
innovators from the private sector, nonprofit organizations, and 
academia with top innovators in government to collaborate during 
focused six-to thirteen-month periods. The program aims to develop 
solutions that can save lives, save taxpayer money, and fuel job 
creation. For example, the goal of one of the program's projects is to 
identify information critical to saving lives and mitigating damage in 
a disaster. 

Even with the efforts of some federal leaders to encourage innovation, 
federal government-wide scores tracking how agencies foster and reward 
employee innovation dropped in 2013 for the second year in a row. 
OPM's 2013 Federal Employee Viewpoint Survey, released in November 
2013, found that only 35 percent of federal workers believe that 
creativity and innovation are rewarded, with positive responses in 
this area showing a steady decline of six percentage points over the 
past three years. Research suggests that half of all innovations are 
not initiated by organizational leaders. Instead, research shows that 
it is important to have processes for gathering stakeholders' and 
front-line workers' views to identify areas for possible improvement. 
[Footnote 7] 

As an innovation tool, labs are based on the idea that the 
competencies needed for systematic innovation--such as intelligent 
risk-taking to develop new services, products and processes--are not 
the same as those required for daily operations. Innovation labs seek 
to provide approaches, skills, models, and tools beyond those that 
most employees are trained in and use to do their work. In addition, 
public sector innovation labs can be viewed as attempts to create an 
organizational response to a range of challenges to innovation, as 
innovation efforts face unique obstacles in the public sector. For 
example, funding for new public ventures is limited, and the risks of 
innovation are high in government. A defining characteristic of the 
public sector is that it is subject to broad scrutiny, so that when an 
innovation fails or is less than a complete success, there is the 
prospect of political consequences. 

With constrained budgets expected to continue into the foreseeable 
future, innovation in our public services is a necessity. In the last 
decade, many public sector organizations around the globe have set up 
facilities with the explicit purpose of supporting innovation efforts. 
For example, Denmark's MindLab, started in 2002, is a cross-
governmental innovation unit that is part of the country's Ministry of 
Business and Growth, the Ministry of Education, the Ministry of 
Employment, and Odense Municipality, and which collaborates with the 
Ministry for Economic Affairs and the Interior. The group covers broad 
policy areas including areas such as entrepreneurship, digital self-
service, education, and employment. OPM's lab was modeled, in part, on 
Denmark's MindLab. 

OPM officials, consistent with other innovation lab representatives we 
interviewed, maintained that, unlike a typical conference room, 
innovation labs can be easily reconfigured for large groups and 
smaller breakout sessions. They allow users to write on walls and 
preserve visual artifacts more easily than typical cubicles and 
traditional office space. This can be done with very low-tech tools 
such as markers and a whiteboard. Figure 1 shows a view of OPM's lab. 

Figure 1: OPM's Lab: 

[Refer to PDF for image: photograph] 

Source: OPM. 

[End of figure] 

Organizations with different missions are pursuing a lab-based 
strategy to foster innovation. For example, organizations--including 
OPM--use their labs as a space where participants can conceptualize 
and prototype new products or processes outside their normal 
environment. Many also use their labs as a teaching space where 
participants can exchange ideas and information through classes, 
workshops, presentations, or other events. Figure 2 shows how 
innovation labs we surveyed from the public, private, and nonprofit 
sectors share common design elements, and how these different 
organizations generally use their labs for multiple and similar 
purposes. 

Figure 2: Innovation Labs Share Similar Design Elements and Problem-
Solving Methods: 

[Refer to PDF for image: Interactive graphic] 

Directions: 

Click each lab organization's PHOTO to see more information regarding 
how labs are used then. 

Click The Highlighted Name To Close. 

To print text version of this graphic, go to appendix II. 

Sources: GAO analysis of lab documents and interviews with lab 
representatives. Photos provided by lab representatives; photo of 
Harvard i-lab taken by Neal Doyle, photo of MindLab taken by MindLab 
staff, photo of Microsoft New England Research & Development (NERD) 
Center taken by Dana J. Quigley Photography, photo of the Australian 
Center for Social Innovation Lab taken by Margaret Frazer, and photo 
of UNICEF Innovation Lab in Uganda taken by UNICEF CC/A/SA. 

[End of figure] 

OPM Lab Activities Are Intended to Build Capacity for Innovation and 
Support Project-Based Problem Solving: 

Based on OPM documents, the innovation lab's start-up costs totaled 
approximately $1.1 million including facility upgrades and 
construction, equipment, and training and other personnel costs. (See 
table 1 for a breakdown of costs.) In building the lab, OPM worked 
with the General Services Administration (GSA) and contracted with 
both design and architectural firms to renovate a former storage room 
in the sub-basement of its headquarters building.[Footnote 8] The 
3,000 square foot renovated space presents an open layout with a 
meeting area for up to two dozen people and is surrounded by breakout 
areas and team rooms. The physical renovation of the facility was 
completed in March 2012, after the installation of final technology 
equipment, asbestos abatement, and enhancements to ventilation and 
life-safety systems. According to OPM, to make the space useful for 
any purpose, much of the funding for the improvements and construction 
of the space would have been required. 

Table 1: Breakdown of OPM's Lab Start-Up Costs (Fiscal Year 2012): 

Item: Facility upgrade/space construction (includes asbestos 
abatement, etc.); 
Cost: $428,226[A]. 

Item: Concept design/construction/furniture; 
Cost: $299,712[A]. 

Item: Project fee paid to GSA; 
Cost: $25,315. 

Item: Equipment/information technology; 
Cost: $247,867. 

Item: Travel; 
Cost: $14,533. 

Item: Training; 
Cost: $75,000. 

Item: Salaries and benefits; 
Cost: $30,000. 

Item: Supplies; 
Cost: $14,410. 

Item: Total; 
Cost: $1,135,063. 

Source: OPM. 

[A] GSA funded $457,820 of the total of these start-up costs. 

[End of table] 

OPM officials said that in fiscal year 2013, the lab's total operating 
budget, including all contracting costs, was $476,000, which supported 
a build up to 5.5 full-time equivalent (FTE) employees over the last 
seven months of fiscal year 2013.[Footnote 9] Officials expect this 
amount will remain stable in the coming fiscal years, proportional to 
a full fiscal year. 

OPM's Lab Is Managed and Operated By a Core Group of Staff: 

Operational responsibility for the innovation lab has been assigned to 
OPM's Employee Services Division and is managed by the agency's Deputy 
Associate Director of Strategic Workforce Planning.[Footnote 10] 
According to OPM officials, since February 2013, the lab has grown 
from 1 FTE to roughly 6 FTEs. Specifically, as of the end of summer 
2013, day-to-day operations in the lab are carried out by 4 FTE staff 
members, 1 FTE intern, 1 part-time intern, and 1 part-time staff 
member whose time is divided between the innovation lab and OPM's 
Resource Management Office. A core group of staff from OPM's Employee 
Services Division have been trained in human-centered design and also 
contribute up to 15 percent of their time in the lab. According to OPM 
officials, the lab reached its maximum fiscal year 2013 funding level 
of 5.5 FTEs in July 2013. A brief description of each position is 
provided in table 2. 

Table 2: Description of OPM's Lab Staff for Fiscal Year 2013: 

Lab position: Lab director; 
Job description: Delivers on and ensures the integrity of the 
innovation lab's activities, outputs, and outcomes. Directs the 
program's strategies, in consultation with the lab team and with the 
executive director of the lab (the Deputy Associate Director of 
Strategic Workforce Planning). 

Lab position: Lead program analyst; 
Job description: Leads lab program evaluation efforts; provides course 
instruction, design support on projects, and facilitation as needed; 
prepares reports; and assists on managing lab use and other activities. 

Lab position: Graphic designer; 
Job description: Produces designs for lab projects and for groups that 
engage with the lab, including executive staff leadership. 

Lab position: Program analyst; 
Job description: Manages lab calendar, supports delivery of courses, 
conducts some facilitation, and supports the lab director and program 
analyst in ongoing review of lab activities. 

Lab position: Student intern; 
Job description: Provides ongoing support for lab operations and 
services, with a focus on program evaluation support and outreach to 
the OPM community. Conducts research, provides administrative support, 
and transcribes outputs. 

Lab position: Student intern; 
Job description: Provides ongoing support for lab operations and 
services, with a focus on supporting lab technology and graphic 
designing. Conducts research, provides administrative support, and 
transcribes outputs. 

Lab position: Business analyst; 
Job description: Supports project scoping as well as delivery of 
courses, operations, individual projects, and lead facilitators; 
manages budget and other financial-related reporting. 

Lab position: Other staffing; 
Job description: A core group of staff from OPM's Employee Services 
Division have been trained in human-centered design and also 
contribute up to 15 percent of their time in the lab. 

Source: OPM. 

[End of table] 

OPM Employed a Phased Approach to Lab Development Initially Focusing 
on Capacity Building: 

OPM has taken a phased approach to developing the lab programming 
(activities taking place in the lab) and the policies governing lab 
use (such as priority-setting policies for lab projects). According to 
OPM documents, each phase has incorporated an element of 
experimentation, review, and a shift in strategy based on lessons 
learned. 

Phase I lasted from March through June 2012. During these first 4 
months after the lab was built, OPM made the space available to the 
OPM workforce for meetings and events. OPM leadership also used this 
time to investigate an appropriate problem-solving approach to pair 
with the lab that would be consistent with approaches used by other 
labs; they determined that a human-centered design approach and 
curriculum would complement OPM employees' technical expertise and 
analytic competencies. OPM also began to recruit interested staff from 
the Employee Services Division to be trained in human-centered design 
fundamentals: this staff would then support project sessions in the 
lab as part of their collateral duties. 

Phase II lasted from July 2012 through March 2013 and consisted mostly 
of facilitated sessions with OPM project teams. During these sessions, 
Employee Services staff worked with project teams to generate ideas to 
long-standing problems through exercises such as project or strategic 
planning, brainstorming sessions, or stakeholder mapping aimed at 
discussing and testing potential solutions. Topics discussed during 
these sessions involved a variety of initiatives directed at improving 
OPM processes and addressing government-wide human resources 
challenges. OPM officials noted that innovation lab projects have 
included, among others, designing an implementation plan with other 
federal agencies to collect valid, accurate, and timely data on the 
federal cyber security workforce; updating the government-wide 
strategy for veterans recruitment; attracting and retaining 
individuals with talent in science, technology, engineering, and 
mathematics disciplines; and specific challenges unique to individual 
agencies. 

Phase III lasted from April through November 2013. In Phase III, OPM 
continued to provide facilitated design sessions and in some cases, 
follow-on coaching to program offices from within OPM and OPM-led 
projects. For example, a facilitated design session included lab staff 
working with the Food and Drug Administration's (FDA) Battery Working 
Group to more effectively engage with the group's external 
stakeholders. According to an OPM case study about the lab, eight FDA 
employees attended the Fundamentals of Human-Centered Design course. 
Following the course, lab staff provided planning support for a public 
workshop with over 200 participants from stakeholder groups including 
medical device and battery manufacturers, other regulatory groups, and 
hospital staff; lab staff also attended the Battery-Powered Medical 
Device workshop to support the FDA team in their use of design 
methods. According to an FDA participant, their collaboration with the 
lab helped them engage in stakeholder dialogue that would not have 
been otherwise possible. OPM lab staff also began to offer classes in 
the lab designed to develop mission-critical skills federal workers 
need to become better problem solvers. The OPM lab offers courses such 
as Human-Centered Design Fundamentals, Prototyping in the Public 
Sector, and Communicating Visually, among other topics. These classes 
are available to OPM staff and other federal workers. Staff also made 
the lab available for federal communities of practice to convene. 
According to lab staff, the lab is becoming a hub for a number of 
standing meetings of a growing community of federal innovators and 
innovation communities of practice. Table 3 presents a summary of 
OPM's human-centered design lab activities since its inception. 
[Footnote 11] 

Table 3: OPM's Innovation Lab Sessions with Human-Centered Design 
Programming, Calendar Years 2012 and 2013: 

Quarter (calendar year): Q1 2012; 
Approximate number of sessions: NA[B]; 
Approximate number of participating organizations[A]: NA[B]; 
Number of participants per meeting (ranges): NA[B]; 
Examples of topics included: [Empty]. 

Quarter (calendar year): Q2; 
Approximate number of sessions: 13; 
Approximate number of participating organizations[A]: 25; 
Number of participants per meeting (ranges): 8-40; 
Examples of topics included: Attracting and retaining science, 
technology, engineering, and mathematics (STEM) talent; 
Presidential Management Fellows re-design; cyber security. 

Quarter (calendar year): Q3; 
Approximate number of sessions: 33; 
Approximate number of participating organizations[A]: 27; 
Number of participants per meeting (ranges): 8-35; 
Examples of topics included: Domestic violence policy; veterans' 
hiring and retention; human capital reports consolidation. 

Quarter (calendar year): Q4; 
Approximate number of sessions: 12; 
Approximate number of participating organizations[A]: 29; 
Number of participants per meeting (ranges): 6-40; 
Examples of topics included: FDA's Entrepreneurs in Residence program; 
retirement benefits; closing critical skills gaps government-wide. 

Quarter (calendar year): Q1 2013; 
Approximate number of sessions: 43; 
Approximate number of participating organizations[A]: 16; 
Number of participants per meeting (ranges): 4-95; 
Examples of topics included: USAJOBS; Chief Financial Officers' action 
planning; international affairs communications. 

Quarter (calendar year): Q2; 
Approximate number of sessions: 66; 
Approximate number of participating organizations[A]: 24; 
Number of participants per meeting (ranges): 8-40; 
Examples of topics included: Interagency community of practice on 
innovation; usability training; new employee on-boarding. 

Quarter (calendar year): Q3; 
Approximate number of sessions: 57; 
Approximate number of participating organizations[A]: 7; 
Number of participants per meeting (ranges): 8-40; 
Examples of topics included: Diversity and inclusion; human-centered 
design workshops; knowledge sharing. 

Source: GAO analysis of OPM data. 

Note: Data as of September 30, 2013. OPM made the lab available to 
other agencies for a variety of purposes, such as targeted design 
support, classes, and follow-on coaching. Some organizations used the 
lab more than once over the six quarters reported in the table. 

[A] Participating agencies for both human-centered design and non-
design activities included the following: Departments of Agriculture, 
Commerce, Defense, Education, Energy, Health and Human Services, 
Homeland Security, Housing and Urban Development, Interior, Justice, 
Labor, State, Transportation, Treasury, and Veterans Affairs; Chief 
Human Capital Officers Council; Director of National Intelligence; 
Environmental Protection Agency; Executive Office of the President; 
Federal Aviation Administration; Food and Drug Administration; Forest 
Service; General Services Administration; Institute of Museum and 
Library Services; Internal Revenue Service; National Aeronautics and 
Space Administration; National Institute of Health; National Institute 
of Standards and Technology; National Oceanic and Atmospheric 
Administration; National Science Foundation; Nuclear Regulatory 
Commission; Office of Management and Budget; Office of Science and 
Technology Policy; Small Business Administration; Social Security 
Administration; Transportation Security Administration; U.S. Agency 
for International Development; United States Postal Service; and the 
White House Office for Science and Technology. Not all participating 
organizations were at all the meetings. 

[B] The lab was opened in March 2012, but no programmed human-centered 
design activity was reported during this period. 

[End of table] 

Lab staff report that in the future they intend to expand from their 
session-based work and targeted design support projects, such as those 
consultative sessions that took place in the lab during Phase III. 
While some of these more episodic projects may continue to occur, the 
lab's focus will be on creating and establishing large-scale projects 
typically involving stakeholders from either wholly within OPM or 
across different agencies that are working on crosscutting issues. As 
discussed later in the report, projects appropriate for this design 
method would have diverse users, be more complex, and be called 
immersion projects. These would be the most structured activities 
undertaken in the lab, characteristically being longer-term activities 
that could take up to six months of intense collaboration with project 
owners and a diverse group of stakeholders. 

OPM's Lab Is Similar in Mission and Design to Other Innovation Labs, 
but OPM Needs to Systematically Evaluate the Lab's Performance: 

We identified a common set of challenges that can undermine 
organizations' efforts to use innovation labs and a set of prevalent 
practices that the organizations employ to address these challenges 
and support their labs' success and sustainability. OPM has 
incorporated some of these practices, such as pairing a distinct space 
with a structured approach to problem-solving, but has not implemented 
others, such as developing meaningful performance measures. Although 
OPM has begun reaching out to other federal innovators, the agency has 
not fully leveraged the experience of other agencies employing similar 
approaches. 

OPM Has Dedicated Physical Space, a Problem-Solving Approach, and 
Plans to Undertake Long-Term Immersion Projects: 

As a prevalent practice for encouraging and supporting greater 
innovation, both the literature and representatives from the 
organizations we reviewed stressed the benefits of pairing a dedicated 
physical space with a structured framework rooted in design-thinking 
principles. Many of the lab representatives said building or 
establishing a distinct space carries an important symbolic value as 
it signals an organization-level commitment to a culture that supports 
innovation. However, simply building a lab is not sufficient to change 
an organization's culture; it is necessary to also introduce a new 
framework for problem solving. Although these organizations use 
different terminology to describe their selected frameworks, such as 
agile development and human-centered design, the general principles 
are similar.[Footnote 12] They include placing users at the center of 
the desired solution--research on successful innovation practices 
shows the importance of engaging customers and understanding their 
needs. Further, they include extensive collaboration with relevant 
stakeholders, experimentation, prototyping, and iterative steps to 
find a solution. A primary objective of this approach is to allow for 
failure in the beginning of the design cycle, so that organizations 
can manage and learn from early mistakes, rather than try to recover 
from an expensive, comprehensive failure upon implementation. 

For example, Census and HUD have similar problem-solving frameworks 
for lab use. As figure 3 shows, the Census Center for Applied 
Technology and the HUD Innovation Lab rely on a five-step framework to 
guide innovation in their labs. 

Figure 3: Five-Step Framework for Guiding Innovation in the Census and 
HUD Labs: 

[Refer to PDF for image: illustration] 

Solicit: 
Solicit innovative ideas, concepts, and initiatives that require 
innovative solutions from agency business units. 

Review and select: 
Review ideas and decide which ideas are aligned with the agency's 
strategic plan or priority initiatives and should be moved to the lab. 

Design: 
Research and develop a conceptual solution or model that identifies 
constraints, restraints, resources, and projected benefits of the idea. 

Prototype: 
Produce a working prototype of the solution. 

Adopt: 
Adopt a working prototype, refine the solution, and implement the idea 
into production. 

Source: GAO analysis of HUD and Census protocols for lab use. 

[End of figure] 

As another example, although CFPB does not have a dedicated physical 
space, it used a similar framework to develop its on-line mortgage 
disclosure form. According to CFPB's Creative Director for Technology 
and Innovation, designers interacted with end-users including mortgage 
applicants, prototyped different forms, and made refinements based on 
continual feedback before launching the new form. On its website, CFPB 
describes its design process in detail, including prototyping and 
feedback sessions with consumers, lenders, and mortgage brokers. 
[Footnote 13] 

OPM's lab provides a menu of design services to meet the specific 
needs of various projects. The lab's larger-scale immersion project 
work will involve taking a complex problem through OPM's problem-
solving framework, which encompasses steps for problem framing to 
learning about users to analysis to concept development, testing, and 
rapid iterative steps. This problem-solving framework is similar to 
those employed by other innovation labs. 

As discussed earlier in this report, similar to other organizations 
with labs, OPM is using its lab for a variety of purposes including as 
a learning space for classes on human-centered design principles and 
techniques and as a meeting space for interagency task teams and 
communities of practice. As originally envisioned in its strategic and 
performance plans, the lab was designed to host a mix of activities 
rooted in the human-centered design approach, including longer-term 
design challenges. Moreover, OPM lab staff asserted that to gain an 
organization's confidence and to instill a culture of innovation, it 
is necessary for a successful innovation lab to have an array of 
sufficiently compelling projects that demonstrate how the lab approach 
can lead to performance improvements. 

Based on our interviews with public and private sector organizations 
with similar innovation facilities, larger-scale problem-solving 
projects were common activities in their innovation labs' service 
portfolios. As an example, Denmark's MindLab has contributed to 
tackling several pressing social issues including simplifying the 
process for managing claims related to industrial accidents and 
shortening the time before injured workers return to the market. 
Opportunities to showcase a new approach to problem solving reduce the 
likelihood that the lab might lose its distinction as different from a 
traditional meeting space or classroom usually associated with 
training facilities. 

Consistent with what staff from other labs told us, OPM officials said 
they needed the past two years to first introduce agency staff to 
human-centered design concepts and applications before they could 
initiate an immersion project. Lab staff said this phased approach was 
necessary for several reasons. Targeted design support sessions 
allowed lab staff to expose lab users to design methods and provided 
opportunities for collaboration. These sessions also allowed lab staff 
to quickly show value for program offices in response to a specific 
need. For example, OPM lab staff members were able to help FDA staff 
plan and engage with over 200 different stakeholders at a conference. 
They also said targeted design support is a critical way for emerging 
design practitioners to develop and hone their own skills before 
applying them to a longer term, and higher stakes, project engagement. 
Lab staff said overall these sessions benefited both the users of the 
lab, who developed new skills to take to their home offices, such as 
problem framing and engaging with stakeholders, as well as lab 
employees, who continue to grow and refine their human-centered design 
skills. In a December 2013 document, OPM staff stated they intend to 
create and establish these longer-term immersion projects and evaluate 
their impact during the next phase of the lab's development. 

OPM Has Not Fully Developed Meaningful Performance Measures: 

Measuring the long-term outcomes of innovation labs is a prevalent 
practice for building acceptance and demonstrating the value of the 
labs. Consistent with our literature review, several representatives 
we interviewed from other innovation labs concurred with the director 
of innovation at Denmark's innovation lab, MindLab, who said that 
innovation labs need to know how much they are spending and their 
outcomes. According to the director, the labs must also be able to 
attribute where the change happens based on their work. In addition, 
lab staff must be prepared to present a narrative of their work. He 
acknowledged that innovation labs are risky because they look 
different, and they have a different focus than other government 
entities. The director said that, as a result, innovation lab 
officials need to show where the funds are going along with the 
benefits and results of those investments. 

Representatives from newer labs--i.e. those operating less than three 
years--stated they primarily rely on output measures to gauge their 
initial efforts such as number of users, ways in which the lab is 
being used, classes or events held in the lab, and anecdotal evidence. 
Developing outcome measures is more challenging for several reasons. 
Appropriate outcome measures are often not obvious at the onset of a 
project. Moreover, agencies may not have appropriate measures or 
baseline data when they start using an innovation lab as a problem-
solving tool, and the role of the lab in driving a successful 
innovation may not always be clear. 

Given these challenges to accurately measuring innovation and the 
value of an innovation lab, lab managers from labs that have been 
operating for a longer period of time told us they focus on developing 
meaningful milestones and measures applicable to different phases of 
the innovation lifecycle, such as problem generation, idea generation, 
and skills development. For example, the UNICEF Innovation Unit--which 
has been helping member-country offices set up innovation labs since 
2006--and several European initiatives developed a set of benchmarks 
intended to help them measure the value of public sector labs and 
identify ways in which the lab's performance can be improved.[Footnote 
14] The benchmarks UNICEF developed span across six categories, such 
as problem definition and idea generation, internal and external 
collaboration, and secondary effects. Within each category, they 
include a list of questions intended to assess their strengths and 
weaknesses. For example, they want to know whether labs are helping 
employees define problems and generate ideas, strengthen internal 
collaborations, and build external partnerships. They also measure the 
extent to which work done in the labs results in new team or staff 
capacity, excitement and goodwill toward the organization, and an 
increase in leverage and influence in their field. OPM is undertaking 
a similar effort to establish benchmarks that will help lab staff 
gauge the extent to which lab users are learning and applying many of 
these same skills, but the lab is not mature enough to have results. 

OPM documents state that the goal of OPM's innovation lab is to 
provide federal workers with 21st century skills in design-led 
innovation, and the intended purpose of the lab is to provide a 
physical space for project-based problem solving. The documents also 
note that the value of the lab can be measured, in part, by how well 
it helps develop the mission-critical competencies to improve the 
federal workforce's ability to solve problems and deliver results. 

In its strategy document, OPM laid out the following high-level goals 
for the innovation lab: 

* Employees assigned to the innovation lab should go back to their 
home organization with an understanding of, and an appreciation for, 
the power of innovative approaches to problem solving. 

* Employees should be equipped to implement similar methodologies in 
their home organizations on future projects. 

* As the innovation lab matures, and as more and more projects are 
completed, the notion of using innovation to tackle complex problems 
will gain traction across the organization. 

* Eventually, leaders and employees across OPM will vie to get their 
issues sent to the innovation lab for resolution. 

* This in turn will contribute to a decrease in organizational silos, 
and a concurrent increase in cross-organizational teams addressing one 
organization's issues. 

In the same document, OPM officials also described an evaluation 
strategy resembling an agile approach.[Footnote 15] Specifically, OPM 
described these goals as moving targets which would be achieved 
through an evolving and self-correcting process. Lab staff immediately 
started to track lab activities and outputs, such as number of 
participating people and agencies, and how participants used the lab, 
such as consultative sessions, follow-on coaching, training classes, 
or as a meeting space. Five months after the lab opened, they also 
started to survey users who participated in day-long facilitated 
sessions. For example, there was a one-page evaluation that asked 
respondents to rate the appropriateness of the environment and quality 
of the facilitators. The surveys also asked whether users would 
recommend the lab to colleagues and whether human-centered design 
problem-solving tools can be used as an effective tool government-
wide. The responses were generally positive--about 82 percent of 
respondents (84 out of 103) said they would recommend the lab to 
someone else, providing a baseline for subsequent survey findings. 
According to lab staff, they periodically reviewed the available data 
and adjusted their strategy for operating the lab. 

Starting in March 2013, a year after the lab opened, OPM lab staff 
began work on a program evaluation framework to more systematically 
measure the lab's progress toward meeting their overarching goals. To 
evaluate the extent to which lab participants are learning and 
applying innovative approaches, lab staff intends to measure the lab's 
performance along three overarching categories: service experience, 
skill development, and project outcome. According to the framework, 
resources dedicated to evaluation efforts will reflect the resources 
needed to host lab-sponsored events. Episodic events such as 
consulting sessions will correspond to a "light-touch" follow-up 
effort, such as immediately surveying all participants on their 
session experience and skills development. More long-term, resource-
intensive efforts such as immersion projects will employ a more robust 
follow-up effort that, in addition to assessing the session experience 
and skills development, will also address project-specific outcomes. 
Collection of assessment data in all three areas will include the 
administration of surveys to participants both before and right after 
a session, and some services will involve the administration of 
surveys to participants before a service and subsequent periodic check-
ins. Depending on the nature of the lab session, information on skill 
development and outcomes will also be obtained from session clients in 
pre-session scoping conversations and periodic, post-session check-ins 
using either surveys, or interviews. For one type of session, 
assessment of participant skill development will also include a survey 
of participant supervisors. 

Lab staff has used a series of surveys to measure participant 
experience and skills development, and to capture specific project-
related outcomes for the different services they offer. However, the 
survey instruments are unlikely to yield data that would be of 
sufficient capacity, credibility, and relevance to indicate the nature 
and extent to which the lab is achieving what it intends to accomplish 
or its value to those who use the lab space. Although there are 
several items across all surveys that are reasonably aligned with 
generally accepted questionnaire and item design principles, there are 
limitations associated with many items where language is ambiguous, 
where the intent of the question is not clear, and directions are 
lacking. For example, phrases such as "changed behavior" or "tangible 
outputs you can move forward" are open to numerous interpretations and 
are likely to engender an array of responses that range from being 
relevant to not at all relevant or relatable to the purposes or 
objectives of the session. In addition, some of the items may be more 
likely to engender responses with a greater likelihood of being 
subject to a respondent's social desirability bias. For example, the 
respondent may want to provide answers that are socially desirable, 
maintain the status quo, or make a good impression. 

While some customization is to be expected, the surveys did not 
indicate any approach to evaluate some core aspects of the lab and its 
value using a consistently presented set of the same questions. For 
example, the question asking participants about the likelihood that 
they would recommend the lab to someone else is the type of item that 
could, with revision, be incorporated in all of the surveys. Analyses 
of a core set of items by type of lab event or service would enable 
lab staff to discern and compare where participants were more and less 
engaged in lab activities and curricula. Consequently, these survey 
instruments and the items on them may be susceptible to various types 
of question and respondent bias and could, when the responses are 
analyzed, produce results that would be difficult to interpret or link 
to expected participant effects, or to the intent or activities of the 
workshop session. Moreover, lab staff has not developed outcome 
measures or milestones related to customer experience and skills 
development. 

The evaluation framework being developed by OPM does not include 
interim performance targets or measures. Best practices state that new 
initiatives benefit when managers set time-bounded, quantifiable 
interim goals, establish related performance measures, collect data, 
and use that information to assess and adjust their 
performance.[Footnote 16] To evaluate the overall performance of 
MindLab, the director said he develops an annual work plan, which 
describes the number and types of projects and other activities the 
lab will undertake, as well as the relative resource allocation to 
those projects and activities. He said his staff also conducts an 
annual review of the budget and actual expenditures with the board. 
OPM lab staff has been tracking outputs--such as number of 
participants and number and type of activities--meaning that they have 
baseline data which could inform realistic, meaningful targets and 
measures related to lab use and activities for the upcoming year. 
Although they continue to refine their surveys, they could use the 
results from earlier versions to establish targets and measures 
related to customer experience and skills development. Meaningful 
measures or milestones could help them assess their progress toward 
improving participants' ability to solve problems and accurately 
measure the effect of working in the lab on services, products, and 
processes. 

As mentioned previously, the lab plans to host the more resource-
intensive immersion projects. To demonstrate that the lab is operating 
as originally intended, evaluation plans will be needed for specific 
immersion projects that can help track cost-benefits and performance 
improvement outcomes. OPM stated that evaluation plans will be 
prepared for each immersion project to account for project outcomes. 
They indicated that they wanted to host their first immersion project 
within the next several months. 

OPM Has Started Efforts to Connect with the Federal Innovation 
Community, but Has Not Fully Leveraged Other Agencies' Efforts: 

Another prevalent practice we identified included leveraging other 
innovation labs' efforts to try to increase the value of the lab 
approach. Studies show that information sharing and 
interorganizational networks can be a powerful driver supporting 
innovation. One study showed that interorganizational networks of 
innovators help members develop new products at a faster rate with 
lower investment commitments, due in large part to the information 
sharing that takes place.[Footnote 17] Sharing information can help 
mitigate the risks and uncertainty that typically characterize 
innovation ventures. Best practices state the importance of 
establishing channels of communication and other mechanisms that 
facilitate knowledge-sharing and building networks of like-minded 
communities to help agencies achieve crosscutting objectives.[Footnote 
18] For example, the Census Bureau's Chief Technology Officer 
suggested a way in which innovation leaders could share information 
and pool resources. Specifically, instead of each agency creating its 
own technology innovation lab with its own hardware, software, and 
associated maintenance, they could use a common innovation 
infrastructure service in the public cloud.[Footnote 19] Every agency 
could still have their own branded offering and could still provide 
access in their own facility or at their regional offices. However, an 
outside vendor could provide the infrastructure. For example, if an 
agency wanted to experiment with some unique visual analytic tools, 
they could purchase what they need on a subscription service; this 
would eliminate agencies buying all the tools themselves. 

While labs provide a physical space where innovators can convene, 
federal agencies are not fully aware of their growing community. As of 
June 2013, OPM was unaware that other agencies such as Census, HUD, 
and NASA were pursuing a lab approach to promote innovation. Moreover, 
the lab directors at these agencies were not aware or only marginally 
aware of OPM's lab and its resources or other federal innovation labs. 
OPM's efforts to develop an innovation lab occurred around the same 
time or pre-dated those of other agencies we interviewed. 

According to OPM officials, during its first year of operations, OPM 
lab staff focused their efforts on promoting awareness of the lab and 
its resources internally to OPM staff. In their second year, OPM lab 
staff planned more activities intended to promote the lab and its 
resources externally to connect with federal agencies' innovation 
efforts. Staff noted the OPM lab is the hub of various interagency 
networks of innovation practitioners. For example, an interagency 
community of practice on idea generation meets in the lab on a monthly 
basis. OPM's lab staff also reported that they host weekly trainings 
in the lab on best practices, including webinars about measuring the 
success of enterprise-level design efforts and the value of 
visualizing information. These training sessions include case study 
presentations from other federal agencies, such as GSA, and non-
federal entities. In addition, OPM has shared best practices with 
other public sector design labs across the globe by participating in a 
number of conferences. OPM is also collaborating with a current 
Presidential Innovation Fellow, who is building an innovation toolkit. 
Although projects in the lab are currently managed and for the most 
part delivered by OPM employees, staff noted that they are 
increasingly looking to leverage detailees, short-term assignments, 
and other ways to harness the potential of talent from other agencies. 
OPM staff said they also give regular tours of their innovation lab 
for other government entities already supporting innovation 
initiatives or developing them. In addition to these activities, OPM 
hosted a convening of federal innovators to compare various agencies' 
innovation communication efforts across the agencies. 

Several federal officials we interviewed said they would welcome the 
opportunity to communicate as the need arose with a community of peers 
to exchange information and ideas and trouble-shoot problems related 
to the start-up and maintenance of their labs. For example, CFPB's 
Creative Director for Technology and Innovation said it would be 
helpful to find out what other bureaus and departments are doing to 
incorporate design principles, so that she could exchange ideas and 
information. An official from NASA's Swamp Works noted that it would 
be beneficial to show that others in the federal sector are also 
looking at innovation labs. To that end, simply knowing the innovation 
community exists and how agency staff leading innovation efforts can 
initiate a conversation related to a specific topic would likely be 
beneficial and would help avoid the risk of a fragmented innovation 
community. 

OPM's 2014 through 2018 Strategic Plan Indicates It Intends to Use the 
Lab to Advance OPM Priorities: 

Because innovation necessarily entails culture change, 
experimentation, periodic setbacks, and often resource investments, 
another prevalent practice necessary to sustain a lab includes 
leadership support. Innovation labs are one tool that agencies can use 
to foster innovation. Agency officials and lab directors we 
interviewed said leaders must be willing to embrace experimentation 
within the lab and understand that smart failures--failures that 
result from trial and error, where the alternative would be to do 
something truly risky due to lack of evidence--are part of the design 
process. For example, the Census Bureau's Chief Technology Officer 
noted that support by Census Bureau leadership is critical to ensure 
staff participation and the continued availability of funds to drive 
innovation in its Center for Applied Technology lab. Other lab 
directors highlighted several strategies they use to balance the risks 
and failures that accompany a problem-solving methodology rooted in a 
more experimental approach. These include accelerated timelines of 
three to six months, which allows organizations to quickly shelve 
projects that are without merit. Some lab leaders also noted that a 
quick win or early success can give new labs the underlying support 
they need to take on riskier projects. 

In March 2014, OPM released its 2014 through 2018 strategic plan, 
which states that the agency plans to seek new, innovative ways to 
accomplish its work of advancing human resource management in the 
federal government. In the strategic plan, OPM indicates that, among 
other things, it intends to use the innovation lab and human-centered 
design methods to address OPM's operational challenges. 

Conclusions: 

For OPM and the rest of the federal government, finding more efficient 
and effective ways of doing business to help meet rising citizen 
demands for public services is critical, particularly in an era of 
continued fiscal and budgetary constraints. OPM's innovation lab is 
one such tool intended to give rise to solutions of complex problems 
facing the federal government. 

Consistent with other innovation labs, development of performance and 
outcome measures, tools to assess performance, and further leveraging 
the experience of other organizations undertaking similar efforts will 
also be critical. Having clear and specific outcome measures will help 
OPM track and evaluate the extent to which the lab is meeting its 
original intent and over time, to make any necessary adjustments. 
Otherwise, OPM's innovation efforts may not be able to demonstrate the 
types of results initially envisioned. 

Recommendations for Executive Action: 

We recommend that the Director of OPM take the following actions to 
help substantiate the lab's original goals of enhancing skills in 
innovation and supporting project-based problem solving: 

* Direct lab staff to develop a mix of performance targets and 
measures to help them monitor and report on their progress toward lab 
goals. Output targets could include number and type of lab activities 
over the next year. Outcome targets and measures should correspond to 
the lab's overarching goals to build organizational capacity to 
innovate and achieve specific innovations in concrete operational 
challenges. 

* Direct lab staff to review and refine the set of survey instruments 
to ensure that taken as a whole, they will yield data of sufficient 
credibility and relevance to indicate the nature and extent to which 
the lab is achieving what it intends to accomplish or is demonstrating 
its value to those who use the lab space. For example, lab staff 
should consider the following actions: 

- Developing a standard set of questions across all service offerings. 

- Revising the format and wording of existing questions related to 
skills development to diminish the likelihood of social desirability 
bias and use post-session questions that ask, in a straight-forward 
way, about whether, or the extent to which, new information was 
acquired. 

- Replacing words or phrases that are ambiguous or vague with defined 
or relevant terminology (e.g., terms actually used in the session) so 
that the respondent can easily recognize a link between what is being 
asked and the content of the session. 

* Direct lab staff to build on existing efforts to share information 
and knowledge within the federal innovation community. For example, 
OPM lab staff could reach out to other agencies with labs such as 
Census, HUD, and NASA's Kennedy Space Center to share best practices 
and develop a credible evaluation framework. 

Agency Comments and Our Evaluation: 

We provided a draft of this report to the Director of OPM for review 
and comment. The director provided written comments, which we have 
reprinted in appendix IV. In summary, OPM generally concurred with our 
recommendations and described ongoing and planned steps to refine 
evaluation efforts and further leverage other federal innovation labs. 

For the recommendations on evaluating performance, the director 
described a competency-based skills gap pilot the lab is undertaking, 
based on targets from pre-and post-testing of participants in lab 
activities. We acknowledge that this is an important step in 
developing performance measures, and OPM will also need targets and 
measures to demonstrate the lab's value in achieving specific 
innovations in concrete operational challenges. 

For the recommendation on leveraging other federal agency innovation 
efforts, the director noted OPM's work seeking out information and 
contacts from other innovation endeavors, including lab-based ones. We 
acknowledge OPM's more recent emphasis in this area, including 
participating in an interagency community of practice on innovation. 
Federal officials we interviewed said they would welcome the 
opportunity to communicate as the need arose with a community of peers. 

To clarify, the report recognizes sustained organizational leadership 
as a prevalent practice for the success of innovation labs. However, 
this was not a specific report recommendation, but an acknowledgment 
regarding the general role leadership plays in ensuring the success of 
innovation labs. In her response, the director stated that OPM 
recently released its 2014 through 2018 strategic plan, which she said 
demonstrates OPM leadership's commitment to the advancement of work in 
the lab. Accordingly, we updated the report to reflect the most 
current information available at the time of our publication. 

As agreed with your offices, unless you publicly announce the contents 
of this report earlier, we plan no further distribution until 30 days 
from the report date. At that time, we will send copies of this report 
to the Director of OPM and appropriate congressional committees. This 
report will also be available at no charge on our website at 
[hyperlink, http://www.gao.gov]. 

If you or your staff members have any questions about this report, 
please contact me at (202) 512-4749 or bagdoyans@gao.gov. Contact 
points for our Offices of Congressional Relations and Public Affairs 
may be found on the last page of this report. GAO staff members who 
made major contributions to this report are listed in appendix V. 

Signed by: 

Seto J. Bagdoyan: 
Acting Director: 
Strategic Issues: 

[End of section] 

Appendix I: Objectives, Scope, and Methodology: 

This appendix provides information on the scope of work and the 
methodology used to (1) describe the Office of Personnel Management 
(OPM) innovation lab's start-up and operating costs, staffing and 
organization, activities, and policies governing the lab's use, and 
(2) assess how OPM's innovation lab compares to other organizations' 
innovation labs, including how it uses benchmarks and associated 
metrics and how it addresses potential challenges to innovation. 

To address the first objective, we reviewed documentation and met 
several times with OPM staff overseeing the lab and its activities. We 
reviewed the lab's construction and operations' budget, including the 
funding sources for the lab, and interviewed agency officials 
knowledgeable about the lab's budget. Based on interviews and e-mail 
exchanges with knowledgeable OPM and General Services Administration 
staff and reviewed documents, we found OPM's lab expense data to be 
sufficiently reliable for the purposes of our report. We reviewed 
spreadsheets maintained by lab staff tracking lab outputs, such as 
workshops hosted in the lab and number of attendees.[Footnote 20] We 
also reviewed lab performance materials such as the lab's performance 
plan, user surveys, and the results of those surveys. Survey 
specialists in our Center for Design, Methods, and Analysis reviewed 
the lab user surveys using internal review guidance that is typically 
performed on draft GAO surveys as part of our development process and 
required before deployment of a survey. In addition to reviewing with 
lab staff the documents they provided us, we interviewed them about 
OPM's process for identifying and selecting a lab strategy, lab 
staff's approach to implementing a human-centered design curriculum, 
and their goals for the lab. 

To address the second objective, we conducted a detailed literature 
search of material from academic institutions, global management 
consultants, professional associations, think tanks, news outlets, and 
various other organizations. We also reviewed literature documenting 
public, private, and academic innovation efforts and associated 
positive and negative outcomes. Our literature search helped us 
identify benchmarks and associated metrics applicable to the 
development and use of innovation facilities in the public, private, 
and nonprofit sectors. We also interviewed OPM lab staff on how they 
intend to identify outcomes--such as cost reductions, performance 
improvements, or other results--from projects undertaken by OPM since 
the inception of the lab. 

We used the findings from our literature review to identify 
organizations with innovation facilities having a dedicated physical 
space and using problem-solving methods similar to OPM's lab. We 
selected a mix of 11 public, nonprofit, and private organizations to 
visit or interview. In addition, we met with an official from the 
Consumer Financial Protection Bureau (CFPB). While CFPB lacks a 
dedicated innovation lab, the agency has a reputation among federal 
agencies as a leader in innovative website development. Table 4 lists 
the organizations we visited in person or interviewed their 
representatives by telephone. 

Table 4: List of Innovation Labs with Facilities Similar to OPM: 

Innovation lab: Census Bureau's Center for Applied Technology; 
Sector: Public. 

Innovation lab: Housing and Urban Development's Innovation Lab; 
Sector: Public. 

Innovation lab: National Aeronautics and Space Administration Kennedy 
Space Center's Swampworks; 
Sector: Public. 

Innovation lab: Denmark's MindLab; 
Sector: Public. 

Innovation lab: UNICEF's Innovation Labs; 
Sector: Nonprofit. 

Innovation lab: Harvard University's Harvard i-lab; 
Sector: Nonprofit. 

Innovation lab: The Australian Centre for Social Innovation (TACSI); 
Sector: Nonprofit. 

Innovation lab: Microsoft's New England Research & Development (NERD) 
Center; 
Sector: Private. 

Innovation lab: Fidelity Investment's Fidelity Center for Applied 
Technology (FCAT); 
Sector: Private. 

Innovation lab: Staples' Velocity Lab; 
Sector: Private. 

Innovation lab: Deloitte's Govlab; 
Sector: Private. 

Source: GAO. 

[End of table] 

At every lab we visited or contacted, we interviewed lab 
representatives about the history of the lab, including why they 
decided to pursue a lab strategy; how the lab is used; the protocols 
for engaging participants; how lab directors measure the performance 
of the lab; challenges to promoting innovation within the 
organization; and practices for addressing those challenges. Based on 
our literature search, we identified common challenges that can hamper 
organizations' efforts to use labs as innovation vehicles and 
prevalent practices that can support labs' success and sustainability. 
In addition, we reviewed our interview records to identify commonly 
recurring challenges and prevalent practices that can support labs' 
success and sustainability. We verified that the challenges and 
prevalent practices we identified during our literature search were 
also those more often cited during the interviews. 

We also interviewed representatives from two management consultancies 
that promote problem-solving approaches rooted in design-thinking 
principles, IDEO and Luma. OPM contracted with Luma to help design the 
lab and implement human-centered design programming. We spoke with 
their representatives to understand the challenges their clients face 
in changing organizational culture and the benchmarks and metrics they 
advise their clients to adopt to measure the performance of new labs 
and problem-solving methods. 

In addition, we interviewed officials from three public-sector 
organizations--San Francisco Mayor's Office of Civic Innovation, 
Canada's Public Policy Forum, and United Kingdom Behavioural Insights 
Team--that are pursuing strategies to promote innovation in their 
organizations but opted not to build innovation labs. They spoke to us 
about the challenges that prevented them from building labs and the 
steps they are taking to incorporate human-centered design-like 
problem-solving methods without a physical lab. 

We conducted this performance audit from July 2013 to March 2014 in 
accordance with generally accepted government auditing standards. 
Those standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe 
that the evidence obtained provides a reasonable basis for our 
findings and conclusions based on our audit objectives. 

[End of section] 

Appendix II: Innovation Labs Share Similar Design Elements and Problem-
Solving Methods: 

This table contains the same text portrayed in figure 2 and shows how 
innovation labs we surveyed from the public, private, and nonprofit 
sectors generally use their labs for multiple and similar purposes. 

Organization: Lab@OPM; 
Sector: Public; 
Purpose of lab: To help deliver on President Obama's vision of a more 
effective and efficient government for the American people by 
supporting a government-wide community of innovators; 
How is it used: Dedicated space where OPM and interagency teams can 
take a problem through a full design cycle. This includes problem 
framing, learning about users, analysis, concept development, testing, 
and rapid iteration; Capacity building--lab hosts classes and 
workshops; Meeting space for OPM staff and other federal workers and 
interagency communities of practice. 

Organization: Census Center for Applied Technology (CAT); 
Sector: Public; 
Purpose of lab: To provide a "safe zone" where Census staff can 
explore new technology solutions without impact to production 
operations; 
How is it used: Dedicated space where lab and other Census staff can 
develop, test, and implement ideas into production outside the 
standard production environment; Capacity building--lab hosts 
presentations of new technologies and solutions. 

Organization: NASA Kennedy Space Center's (KSC) Swampworks; 
Sector: Public; 
Purpose of lab: To provide a dedicated space where NASA KSC engineers 
can quickly resolve problems related to deep-space exploration; 
How is it used: Space where 20 NASA engineers and scientists prototype 
and test emerging technologies; Swamp Works has contracts with other 
NASA centers worth about $7 million a year. 

Organization: Housing and Urban Development (HUD) Innovation Lab; 
Sector: Public; 
Purpose of lab: Provide a space that looks different from the 
traditional office environment, where HUD employees can accelerate the 
development of solutions more efficiently than other available 
approaches; 
How is it used: Dedicated space where HUD lab staff and mission area 
leads can develop, test, and implement ideas into production within a 
compressed timeframe. 

Organization: Denmark's MindLab; 
Sector: Public; 
Purpose of lab: Provide a neutral space for government ministries to 
work with citizens and businesses to create new solutions for society; 
How is it used: Ministry officials from Danish Ministries of Business 
and Growth, Education, and Employment use the MindLab space and 
resources to take a problem through the full design cycle; Capacity 
building--lab hosts classes, conferences, and workshops. 

Organization: UNICEF Innovation Labs; 
Sector: Nonprofit; 
Purpose of lab: Help the organization become more flexible, agile, and 
better prepared for global changes by providing spaces where 135 
country offices can collaborate with local partners; 
How is it used: Dedicated spaces where UNICEF country office staff and 
their local partners can take a problem through a full design cycle. 

Organization: Harvard i-lab; 
Sector: Nonprofit; 
Purpose of lab: Provides space, practical skill building, and 
programming for Harvard students, faculty, staff, alumni, and others 
engaged in new ventures, nonprofit creation, product or service 
innovation, small business development, and related educational and 
research activities; 
How is it used: Incubator, workspace, and programming for start-up 
ventures involving Harvard students, and their partners; Teaching 
space for Harvard students; Meet-up space for local community. 

Organization: The Australian Centre for Social Innovation (TACSI); 
Sector: Nonprofit; 
Purpose of lab: To tackle tough social issues, such as family 
breakdown and social inequality, by building Australia's social 
innovation capability; 
How is it used: Laboratory for co-designing new social programs for 
vulnerable populations--this includes generating ideas, conducting 
ethnographic research, and prototyping potential solutions; Meet-up 
space: TACSI hosts events, workshops, and conferences for social 
change community; Capacity building for other social innovators. 

Organization: Staples Velocity Lab; 
Sector: Private; 
Purpose of lab: Provide a dedicated physical space in the heart of the 
Cambridge, MA tech sector that can accelerate the speed of product 
development, increase collaboration, and attract new talent; 
How is it used: Office space for moving new and emerging technologies 
through the development pipeline; Meet-up space for local tech 
community; Recruitment tool for top tech talent. 

Organization: Microsoft's New England Research & Development (NERD) 
Center; 
Sector: Private; 
Purpose of lab: To build a strong and permanent research and 
development presence in Cambridge, MA where Microsoft researchers and 
programmers can build relationships with local universities, biotech, 
and healthcare companies; 
How is it used: Office space for moving new and emerging technologies 
through the development pipeline; Meet-up space for local tech 
community; Recruitment tool for top tech talent. 

Organization: Fidelity Center for Applied Technology (FCAT); 
Sector: Private; 
Purpose of lab: Provide a dedicated physical space where Fidelity 
executives explore how emerging technologies can improve products and 
processes for internal business units; 
How is it used: Laboratory where FCAT staff can identify solutions and 
develop new products and processes for Fidelity business units; Hosts 
conferences, workshops, and social events related to technological and 
social innovation. 

Organization: Deloitte GovLab; 
Sector: Private; 
Purpose of lab: Provide a dedicated physical space that exemplifies 
how environment can promote creativity and collaboration where 
Deloitte's future leaders can hypothesize, research, and test new 
ideas; 
How is it used: Leadership development institute for Deloitte's 
highest performing consultants; A think tank where fellows develop 
innovative yet practical strategies governments can use to transform 
the way they deliver their services and prepare for the challenges 
ahead; Capacity building--GovLab educates Deloitte account teams on 
these emerging trends. 

Source: GAO analysis of lab documents and interviews with lab 
representatives. 

[End of table] 

[End of section] 

Appendix III: Office of Personnel Management's Innovation Lab Non-
Design Programmed Activities, Calendar Years 2012-13: 

Quarter (calendar year): Q1 2012; 
Approximate number of meetings: 7; 
Approximate number of participating organizations[A]: 12; 
Number of participants per meeting[B] (ranges): 9-25; 
Examples of topics included: Recruitment policy and outreach; 
civil service innovation cohort; Chief Learning Officers' meeting. 

Quarter (calendar year): Q2; 
Approximate number of meetings: 37; 
Approximate number of participating organizations[A]: 8; 
Number of participants per meeting[B] (ranges): 8-30; 
Examples of topics included: Acquisition planning; retirements 
calculator challenge; leadership and talent management. 

Quarter (calendar year): Q3; 
Approximate number of meetings: 7; 
Approximate number of participating organizations[A]: 2; 
Number of participants per meeting[B] (ranges): unknown; 
Examples of topics included: Performance improvement review; 
Presidential Innovation Fellows coaching session. 

Quarter (calendar year): Q4; 
Approximate number of meetings: 21; 
Approximate number of participating organizations[A]: 4; 
Number of participants per meeting[B] (ranges): unknown; 
Examples of topics included: MyGov platform; RFP-EZ; health exchanges. 

Quarter (calendar year): Q1 2013; 
Approximate number of meetings: 14; 
Approximate number of participating organizations[A]: 4; 
Number of participants per meeting[B] (ranges): 5-47; 
Examples of topics included: HealthCare.gov summit; retirement service 
summit; Science, Technology, Engineering, and Mathematics (STEM) event. 

Quarter (calendar year): Q2; 
Approximate number of meetings: 4; 
Approximate number of participating organizations[A]: 3; 
Number of participants per meeting[B] (ranges): 10-15; 
Examples of topics included: State e-Diplomacy; speaker series; 
U.S. Department of Agriculture brainstorming. 

Quarter (calendar year): Q3; 
Approximate number of meetings: 21; 
Approximate number of participating organizations[A]: 6; 
Number of participants per meeting[B] (ranges): 5-25; 
Examples of topics included: Graphic Facilitators Community of 
Practice; State's global partnership initiative retreat; 
USAJOBS training session with Environmental Protection Agency students. 

Source: GAO analysis of OPM data. 

Note: Data as of September 30, 2013. 

[A] Participating agencies for both human-centered design and non-
design activities included the following: Departments of Agriculture, 
Commerce, Defense, Education, Energy, Health and Human Services, 
Homeland Security, Housing and Urban Development, Interior, Justice, 
Labor, State, Transportation, Treasury, and Veterans Affairs; Chief 
Human Capital Officers Council; Director of National Intelligence; 
Environmental Protection Agency; Executive Office of the President; 
Federal Aviation Administration; Food and Drug Administration; Forest 
Service; General Services Administration; Institute of Museum and 
Library Services; Internal Revenue Service; National Aeronautics and 
Space Administration; National Institute of Health; National Institute 
of Standards and Technology; National Oceanic and Atmospheric 
Administration; National Science Foundation; Nuclear Regulatory 
Commission; Office of Management and Budget; Office of Science and 
Technology Policy; Small Business Administration; Social Security 
Administration; Transportation Security Administration; U.S. Agency 
for International Development; United States Postal Service; and the 
White House Office for Science and Technology. Not all participating 
organizations were at all the meetings. 

[B] The ranges provided are estimates based on OPM data. 

[End of table] 

[End of section] 

Appendix IV: Comments from the Office of Personnel Management: 

United States Office of Personnel Management: 
The Director: 
Washington, DC 20415: 

March 26, 2014: 

Mr. Seto Bagdoyan: 
Acting Director, Strategic Issues: 
U.S. Government Accountability Office: 
441 G Street, NW: 
Washington, DC 20548: 

Dear Mr. Bagdoyan: 

Thank you for giving the U.S. Office of Personnel Management (OPM) the 
opportunity to comment on the Government Accountability Office (GAO) 
draft report, "Agency Needs to Improve Outcome Measures to Demonstrate 
the Value of its Innovation Lab." 

We appreciate the thoughtfulness with which you conducted the research 
and drafted the report on OPM's Lab, in particular your consideration 
of the growing community of labs and innovation environments in public 
and private sector organizations around the world. OPM is proud to be 
a part of this growing community of innovation practices, and draw on 
these best practices for inspiration as we work to further develop the 
Lab's quality programming. As demonstrated in the documentation and 
information we have provided, the Lab's program plans include long-
term projects with measurable outcomes, sustained thought leadership, 
a community of practice, and a more mature overall evaluative 
framework. 

OPM values and accepts the report's recommendation that we must 
improve and increase the consistency of our program evaluation 
efforts. Recognizing this need from the beginning, we have continued 
to develop our evaluation strategy by drawing on OPM's internal 
expertise in program evaluation and organizational psychology. 

The GAO found that the Lab staff has experimented with a number of 
evaluative instruments as part of our effort to design the strongest 
possible measurement strategy for the Lab. This observation is 
correct, but we would like to emphasize, in particular, that we have 
drawn on international best practices while focusing on designing a 
measurement strategy appropriate for the unique value that the Lab is 
providing. As GAO points out, OPM has captured qualitative and 
quantitative feedback since the beginning of Phase II of the Lab's 
existence. As noted on page 22 of the report, we have consistently 
used the "Net Promoter Score," captured immediately after people's 
experiences with the Lab, to determine the likelihood of visitors 
recommending engaging with the Lab to others. 

The GAO report also indicates on page 22 that the Lab did not 
undertake to measure skill development over time with measures and 
milestones. For clarification, we note that we do have a competency-
based skills gap pilot the Lab is undertaking. OPM is serving as a pilot
agency to test the Lab's impact on participants' problem-solving, 
strategic thinking, and data analysis proficiency levels. 

The Lab's Skills Gap Pilot has the following targets for each 
competency being assessed: 

I-year: 
Problem Solving: 10% increase in proficiency level from pre-test to 
post-test; 
Strategic Thinking: 10% increase in proficiency level from pre-test to 
post-test. 

2-year: 
Problem Solving: 20% increase; 
Strategic Thinking: 20% increase; 
Data Analysis: 10% increase. 

We additionally appreciate and accept the report's recommendation that 
we do more to foster a government-wide community of innovators. It is 
for this reason that we have used the Lab to convene cross-agency 
communities of innovation practitioners. There is an enormous untapped 
potential in bringing ongoing innovation efforts together to share 
with and learn from each other. We would clarify, though, the report's 
characterization of OPM's efforts to coordinate the Lab's work with 
other Federal efforts. Throughout the Lab's development, OPM's senior 
leadership, Employee Services leadership, and the Lab's Executive 
Director have consistently sought out information and contacts from 
other Federal innovation efforts, including lab-based ones. We did 
this by actively participating in an interagency community of practice 
on innovation, convening meetings in the Lab with other civil service 
SES and senior management innovation practice leads, and using our 
growing network of innovation practitioners to map innovation efforts 
across the Federal government. We have consistently found, however, 
that existing labs are primarily focused on technology innovation, as 
opposed to the OPM focus on human-centered design as a specifically 
employee-focused innovation practice. We believe that there is a 
unique need for public sector labs to use human-centered design to 
drive more customer-centric government policies and services; and that 
OPM's Lab was the first of its kind in the United States Federal 
government. 

With respect to your recommendation that sustained organizational 
leadership plays a pivotal role in the Lab's success, and to further 
demonstrate how OPM is leveraging the Lab for sustained program and 
management initiatives, I would like to direct your attention to OPM's
recently released 2014-2018 Strategic Plan, which demonstrates OPM's 
commitment to the advancement of the work of the Lab, available at: 
[hyperlink, http://www.opm.gov/about-us/budgetperformance/strategic-
plans/2014-2018-strategic-plan.pdf]. This document clearly presents
the Lab as a strategic asset contributing significantly to Goal 1, 
creating a diverse and inclusive OPM Workforce, and Goal 2, improving 
the delivery of OPM customer service. These initiatives, in addition 
to other work we have launched, will give the Lab the opportunity to 
provide measurable results that demonstrate the Lab's value. 

Again, thank you for the opportunity to comment. 

Sincerely, 

Signed by: 

Katherine Archuleta: 
Director: 

[End of section] 

Appendix V: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Seto J. Bagdoyan, (202) 512-4749 or bagdoyans@gao.gov. 

Staff Acknowledgments: 

In addition to the contact named above, Thomas Gilbert, Assistant 
Director, and Judith Kordahl, Analyst-in-Charge, supervised the 
development of this report. Jessica Nierenberg and Anthony Patterson 
made significant contributions to all aspects of this report. Other 
important contributors included Thomas Beall, Karin Fangman, Donna 
Miller, and Robert Robinson. 

[End of section] 

Footnotes: 

[1] The top 1,000 global innovation companies, including Intel, 
Microsoft, Samsung, and Volkswagen, spent an estimated $638 billion in 
2013 on their research and development activities, a key input driving 
innovation. See Barry Jaruzelski, John Loehr, and Richard Holman, "The 
2013 Global Innovation 1000 Study: Navigating the Digital Future," 
Booz & Company, Issue 73 (New York, New York: Booz and Company Inc., 
Winter 2013). 

[2] An iterative process arrives at a decision or a desired result by 
repeating rounds of analysis or a cycle of operations. The objective 
is to bring the decision or result closer to the desired outcome with 
each repetition. 

[3] See figure 2 for a complete list of the innovation labs we 
surveyed. 

[4] OPM defines innovation as the process of improving, adapting, or 
developing a product, system, or service to deliver better results. 

[5] Crowdsourcing is the practice of obtaining needed services, ideas, 
or content by soliciting contributions from a large group of people--
especially from the online community--rather than from traditional 
employees or suppliers. 

[6] Office of Management and Budget, Guidance on the Use of Challenges 
and Prizes to Promote Open Government, OMB Memorandum M-10-11 
(Washington, D.C.: 2010). 

[7] Geoff Mulgan and David Albury, Innovation in the Public Sector, 
Ver.1.9 (London, England: Strategy Unit, October 2003) p. 13. 

[8] OPM leases its entire headquarters building and pays GSA at a rate 
based on square footage. 

[9] An FTE is the number of total hours worked divided by the maximum 
number of compensable hours in a work year. For example, if the work 
year is defined as 2,080 hours, then one worker occupying a paid full-
time job all year would consume one FTE. Two persons working for 1,040 
hours each would consume one FTE between the two of them. 

[10] OPM's Employee Services Division provides policy direction and 
leadership in designing, developing, and implementing government-wide 
human resources systems and programs for recruitment, pay, leave, 
performance management and recognition, employee development, work/ 
life/wellness programs, and labor and employee relations. They also 
provide technical support to agencies for human resources management 
policies and practices, including veterans' employment and evaluation 
of agencies' human resources programs; the division is also 
responsible for managing the operation of OPM's internal human 
resources program. 

[11] See appendix III for a summary of OPM's non-design lab 
activities, which include more traditional meetings such as 
Presidential Innovation Fellows coaching sessions, since the 
innovation lab's inception. 

[12] For example, agile software development focuses on keeping 
software code simple, testing often, and delivering functional bits of 
the application as soon as they are ready. The goal of agile 
development is to build upon small parts as the project progresses, as 
opposed to delivering one large application at the end of the project. 
See GAO, Software Development: Effective Practices and Federal 
Challenges in Applying Agile Methods, [hyperlink, 
http://www.gao.gov/products/GAO-12-681] (Washington, D.C.: Jul. 27, 
2012). 

[13] See [hyperlink, 
http://www.consumerfinance.gov/knowbeforeyouowe/#2]. 

[14] For more information on the work that Nesta, an innovation 
organization in the United Kingdom, and the Danish Agency for Science, 
Technology and Innovation are doing to evaluate innovation in the 
public sector, see Hugo Thenint, "Innovation in the Public Sector," 
Mini Study 10 (INNO Global Review of Innovation Intelligence and 
Policy Studies, Feb. 2010). 

[15] The agile approach was first articulated in 2001 and is still 
used today. With this approach, project managers establish their 
higher-level, strategic goals but are more likely to respond to change 
rather than follow a set plan. This approach recognizes that long-term 
project plans are not adaptable, whereas short-term plans provide more 
flexibility in responding to change. It is considered more effective 
to devise a detailed plan for a shorter period and a general plan for 
a longer period. 

[16] GAO, The Results Act: An Evaluator's Guide to Assessing Agency 
Annual Performance Plans, [hyperlink, 
http://www.gao.gov/products/GAO/GGD-10.1.20] (Washington, D.C.: April 
1998) and GAO, Streamlining Government: Questions to Consider When 
Evaluating Proposals to Consolidate Physical Infrastructure and 
Management Functions, [hyperlink, 
http://www.gao.gov/products/GAO-12-542] (Washington, D.C.: May 23, 
2012). 

[17] Jennifer Lewis Priestly and Subhashish Samaddar, "Information 
Sharing in Innovation Networks," Encyclopedia of Information Science 
and Technology, Second Edition (Hershey, PA: IRM Press, 2006). 

[18] GAO, Managing for Results: Key Considerations for Implementing 
Interagency Collaborative Mechanisms, [hyperlink, 
http://www.gao.gov/products/GAO-12-1022] (Washington, D.C.: Sept. 27, 
2012); Interagency Collaboration: State and Army Personnel Rotation 
Programs Can Build on Positive Results with Additional Preparation and 
Evaluation, [hyperlink, http://www.gao.gov/products/GAO-12-386] 
(Washington, D.C.: Mar 9, 2012) and Results-Oriented Government: 
Practices That Can Help Enhance and Sustain Collaboration among 
Federal Agencies, [hyperlink, http://www.gao.gov/products/GAO-06-15] 
(Washington, D.C.: Oct. 21, 2005). 

[19] A public cloud is a set of computers and computer network 
resources based on the standard cloud computing model, in which a 
service provider makes resources, such as applications and storage, 
available to the general public over the Internet. 

[20] Outputs as used here include measurement of the lab's services 
and level of participation in lab activities. 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation, and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the 
performance and accountability of the federal government for the 
American people. GAO examines the use of public funds; evaluates 
federal programs and policies; and provides analyses, recommendations, 
and other assistance to help Congress make informed oversight, policy, 
and funding decisions. GAO's commitment to good government is 
reflected in its core values of accountability, integrity, and 
reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's website [hyperlink, http://www.gao.gov]. Each 
weekday afternoon, GAO posts on its website newly released reports, 
testimony, and correspondence. To have GAO e-mail you a list of newly 
posted products, go to [hyperlink, http://www.gao.gov] and select 
"E-mail Updates." 

Order by Phone: 

The price of each GAO publication reflects GAO's actual cost of 
production and distribution and depends on the number of pages in the 
publication and whether the publication is printed in color or black 
and white. Pricing and ordering information is posted on GAO's 
website, [hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card, 
MasterCard, Visa, check, or money order. Call for additional 
information. 

Connect with GAO: 

Connect with GAO on facebook, flickr, twitter, and YouTube.
Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts.
Visit GAO on the web at [hyperlink, http://www.gao.gov]. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 
Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; 
E-mail: fraudnet@gao.gov; 
Automated answering system: (800) 424-5454 or (202) 512-7470. 

Congressional Relations: 

Katherine Siggerud, Managing Director, siggerudk@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, DC 20548. 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, DC 20548. 

[End of document]