This is the accessible text file for GAO report number GAO-10-159T 
entitled 'Information Security: Concerted Effort Needed to Improve 
Federal Performance Measures' which was released on October 29, 2009.

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately.

Mr. Chairman and Members of the Subcommittee:

Thank you for the opportunity to participate in today’s hearing on how 
agencies can establish cost-effective cyber defense. My statement today 
is based on our report titled Information Security: Concerted Effort 
Needed to Improve Federal Performance Measures, which is being released 
at this hearing.[Footnote 1] 

Cyber security is a critical consideration for any organization that 
depends on information systems and computer networks to carry out its 
mission or business. Organizations are faced with a variety of 
information security threats, such as fraudulent activity from cyber 
criminals, unauthorized access by disgruntled or dishonest employees, 
and denial-of-service attacks and other disruptions. The recent 
dramatic increase in reports of security incidents, the wide 
availability of hacking tools, and steady advances in the 
sophistication and effectiveness of attack technology all contribute to 
the urgency of ensuring that adequate steps are taken to protect the 
federal government’s information and the systems that contain and 
process it. 

The Federal Information Security Management Act (FISMA), which was 
enacted in 2002, sets forth a comprehensive framework for ensuring the 
effectiveness of security controls over information resources that 
support federal operations and assets. The act assigns specific 
responsibilities to federal agencies, the Office of Management and 
Budget (OMB), and the National Institute of Standards and Technology 
(NIST). It also requires agencies and OMB to annually report on the 
adequacy and effectiveness of agency information security programs and 
compliance with the provisions of the act. To help meet these 
requirements, OMB established a uniform set of information security 
measures that all federal agencies report on annually. 

Mr. Chairman, in light of questions about whether or not agencies are 
measuring the right things in securing their systems, you requested 
that GAO examine how organizations develop and use metrics to assess 
the performance and effectiveness of information security activities. 
In response to your request, our report and my statement provide (1) a 
description of key types and attributes of performance measures; (2) 
information about the practices of leading organizations for developing 
and using measures to guide and monitor information security control 
activities;[Footnote 2] (3) information on the measures used by federal 
agencies to guide and monitor information security control activities 
and how they are developed; and (4) an assessment of the effectiveness 
of the measures-reporting practices that the federal government uses to 
inform Congress on the effectiveness of information security programs. 
In conducting this work, we collected and analyzed information from 
leading organizations, security experts, NIST, 24 major federal 
agencies, and OMB.[Footnote 3] Our work for this report was performed 
in accordance with generally accepted government auditing standards. 

In brief, Mr. Chairman, leading organizations and experts have 
identified different types of measures that are useful in helping to 
achieve information security goals. While officials categorized these 
types using varying terminology, we concluded that they generally fell 
into three types: (1) compliance, (2) control effectiveness, and (3) 
program impact. These types are consistent with those laid out by NIST 
in its information security performance measurement guide.[Footnote 4] 
In addition, while information security measures can be grouped into 
these three major types, organizations and experts reported that all 
such measures generally have certain key characteristics, or 
attributes. These attributes include being (1) measurable, (2) 
meaningful, (3) repeatable and consistent, and (4) actionable.[Footnote 

Further, these organizations and experts indicated that the successful 
development of information security measures depends on adherence to a 
number of key practices, including focusing on risks, involving 
stakeholders, assigning accountability, and linking to business goals. 
Additional practices are critical to ensuring that the measures are 
useful in effectively conveying information to operational managers, 
executives, and oversight officials. These include tailoring measures 
to the audience; correlating data; and capturing progress, trends, and 
weaknesses. Figure 1 illustrates the interrelationship of these key 
practices with the key characteristics. 

Figure 1: Measures Development and Use Cycle: 

[Refer to PDF for image: circle chart] 


* Focus on risks; 
* Involve stakeholders; 
* Assign accountability; 
* Link to business goals; 

Measures characteristics: 


Control effectiveness; 
Program impact; 



Effective reporting: 

* Tailor to the audience; 
* Correlate data; 
* Capture progress, trends, and weaknesses; 

Source: GAO. 

[End of figure] 

We determined that federal agencies have not always followed key 
practices identified by leading organizations for developing 
information security performance measures. While agencies have 
developed measures that fall into each of the three major types (i.e. 
compliance, control effectiveness, and program impact), on balance they 
have relied primarily on compliance measures, which have a limited 
ability to gauge program effectiveness. Agencies stated that, for the 
most part, they predominantly collected measures of compliance because 
they were focused on measures associated with OMB’s FISMA reporting 
requirements. In addition, while most agencies have developed some 
measures that include the four key attributes identified by leading 
organizations and experts, these attributes were not always present in 
all agency measures. Further, agencies have not always followed key 
practices in developing measures, such as focusing on risks. 

Lastly, we determined that OMB’s measures did not address the 
effectiveness of several key areas of information security controls, 
including, for example, agency security control testing and evaluation 
processes. There is no measure of the quality of agencies’ test and 
evaluation processes or results that demonstrate the effectiveness of 
the controls that were evaluated.[Footnote 6] In addition, OMB’s report 
to Congress does not fully employ key practices for reporting and thus 
provides limited information about the effectiveness of agency 
information security programs. 

We made five recommendations to OMB to assist federal agencies in 
developing and using measures that better address the effectiveness of 
their information security programs: 

* issue revised guidance to chief information officers for developing 

* direct chief information officers to ensure that measures exhibit key 

* direct chief information officers to employ the key practices for 
developing a measure as identified by leading organizations; 

* revise annual FISMA reporting guidance to agencies; and: 

* revise the annual FISMA report to Congress to provide better status 
information on the security posture of the federal government. 

Implementing these recommendations will help to focus attention on 
activities that will enhance the effectiveness of agency information 
security controls and improve the cyber defense of federal computer 
systems and information. In providing oral comments on a draft of the 
report, representatives of OMB's Office of E-Government and Information 
Technology stated that they generally agreed with the contents and 
recommendations of the report. 

Mr. Chairman, this concludes my prepared statement. I would be pleased 
to respond to any questions that you or other members of the 
subcommittee may have. 

For questions about this statement, please contact Gregory C. Wilshusen 
at (202) 512-6244 or Individuals making key 
contributors to this testimony include Ashley Brooks, John de Ferrari, 
Season Dietrich, Neil Doherty, Ronalynn Espedido, Min Hyun, Anjalique 
Lawrence, Joshua Leiling, Lee McCracken, and David Plocher. 

[End of section] 


[1] [hyperlink,] 
(Washington, D.C.: Sept. 14, 2009). 

[2] For the purposes of this report, “leading organizations” refers to 
prominent, nationally known organizations, academic institutions, and 
state agencies that have implemented comprehensive enterprisewide 
information security programs. 

[3] The 24 major federal agencies are the Agency for International 
Development; the Departments of Agriculture, Commerce, Defense, 
Education, Energy, Health and Human Services, Homeland Security, 
Housing and Urban Development, the Interior, Justice, Labor, State, 
Transportation, the Treasury, and Veterans Affairs; the Environmental 
Protection Agency; the General Services Administration; the National 
Aeronautics and Space Administration; the National Science Foundation; 
the Nuclear Regulatory Commission; the Office of Personnel Management; 
the Small Business Administration; and the Social Security 

[4] National Institute of Standards and Technology, Performance 
Measurement Guide for Information Security, NIST Special Pub. 800-55 
Revision 1 (Gaithersburg, Md.: July 2008). 

[5] Although we focused on identifying attributes and practices for 
measuring the performance of information security programs, our 
findings conformed closely to our prior work on effective performance 
measurement and reporting practices for the federal government in 
general. See, for example, GAO, Managing for Results: Enhancing Agency 
Use of Performance Information for Management Decision Making, 
[hyperlink,] (Washington, 
D.C.: Sept. 9, 2005). 

[6] OMB does require agency inspectors general to assess agencies’ 
certification and accreditation process; however, the assessment may or 
may not include an assessment of security control testing and 
evaluation processes. Further, OMB does not provide a transparent 
depiction of how an assessment of an agency’s security control testing 
and evaluation process contributes to the overall certification and 
accreditation quality rating.

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink,]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink,] 
and select "E-mail Updates." 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO’s Web site, 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional 

To Report Fraud, Waste, and Abuse in Federal Programs: 


Web site: [hyperlink,]: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Ralph Dawn, Managing Director, 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Chuck Young, Managing Director, 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: