This is the accessible text file for GAO report number GAO/OIG-9-3 entitled 'Four People Performance Measures: Many Attributes of Successful Measures Met; Opportunities Exist for Further Enhancements' which was released on August 31, 2009. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Office of the Inspector General: United States Government Accountability Office: GAO: August 2009: Four People Performance Measures: Many Attributes of Successful Measures Met; Opportunities Exist for Further Enhancements: GAO-OIG-9-3: GAO/OIG Highlights: Objectives: As part of the Office of Inspector General’s (OIG) commitment to periodically review the reliability and validity of GAO’s performance measures, OIG evaluated four of the agency’s people measures—staff development, staff utilization, leadership, and organizational climate. OIG based its evaluation largely on an assessment of whether these measures met nine specific attributes that earlier GAO work cited as key to successful performance measures. Findings: OIG’s evaluation showed that, for fiscal year 2007, GAO accurately calculated the four performance measures reviewed. Of the nine attributes of successful performance measures identified by previous GAO work, staff utilization and organizational climate met all of the attributes, and staff development and leadership met many of the attributes. All four measures’ scores are derived from GAO’s Employee Feedback Survey. For the staff development and leadership measures, OIG found that GAO had not fully disclosed in its annual performance report that (1) it calculates these scores by excluding survey respondents that answered “ no basis to judge/not applicable” (no basis/NA) and (2) the exclusion of these respondents has the effect of changing the two measures’ scores. A more complete interpretation of the scores would be possible with additional disclosures in the report’s tables. In addition, the score for the leadership measure was significantly affected by the large number of no basis/NA respondents for one of its 10 questions. In fiscal year 2007, 45 percent of surveyed respondents answered no basis/NA to the question about supervisors’ effectiveness in handling equal employment opportunity (EEO) and discrimination issues. GAO officials stated this large number most likely reflects the relatively few formal discrimination cases and the safeguarding of private information related to these cases. GAO’s employee survey has recently added other questions on diversity that could provide more useful information and do not have large numbers of no basis/NA respondents. Further, although GAO’s leadership measure captures information about employees’ satisfaction with their immediate supervisors, it does not include the employees’ survey responses about satisfaction with the strategic leadership of GAO. As a result, the measure’s name differs from what is being measured. Finally, GAO was not timely in reporting changes made to its staff development measure that resulted in performance data no longer being comparable. In response to OIG’s work, GAO made this disclosure in its fiscal year 2008 annual performance report. However, the agency does not have written procedures that would help ensure the timely reporting of future changes to measures. Recommendations: OIG recommends that GAO take the following four actions: * Disclose in its annual performance report’s tables that the four measures’ scores are calculated by excluding no basis/NA respondents and that this approach, when the number of these respondents is large, has the effect of changing the scores for two measures. * Determine for the leadership measure whether the current EEO and discrimination question should be retained or replaced. * Consider (1) a more descriptive name for the leadership measure or (2) incorporating survey results about GAO’s strategic leadership. * Develop written procedures to ensure changes made to measures and any effects on performance data comparability are promptly reported. [End of section] GAO/OIG: Memorandum: Date: August 31, 2009: To: Acting Comptroller General – Gene L. Dodaro: From: [Signed by] Inspector General – Frances Garcia: Subject: Four People Performance Measures: Many Attributes of Successful Measures Met; Opportunities Exist for Further Enhancements: GAO relies on a talented, multidisciplinary workforce to deliver accurate, high-quality results and fulfill its mission of helping Congress improve the performance and ensure the accountability of the federal government. Salaries and benefits for this workforce represent a significant investment—almost 80 percent of GAO’s budget. In addition, recent retirements and changing demographics in the agency are creating a younger, less experienced workforce, as well as giving rise to knowledge and skill gaps at the middle management and senior levels. For such reasons, GAO continues to identify human capital as one of its most important management challenges and has committed to creating better and more comprehensive performance measures to improve its human capital management. Congress enacted the Government Performance and Results Act of 1993 (GPRA) to help resolve the long-standing management problems that undermined the federal government’s efficiency and effectiveness and to provide greater accountability for results.[Footnote 1] GPRA requires federal agencies to develop strategic plans with long-term, outcome- oriented goals and objectives, annual goals linked to the long-term goals, and annual reports on the results achieved. As a legislative branch agency, GAO is not required to comply with GPRA, but generally does follow the intent of this law. GAO uses a strategic planning and management process based on strategic goals and objectives, as well as performance goals identified for the agency. One of its strategic goals— maximizing the value of GAO by being a model federal agency and world- class professional services organization—addresses human capital management. To monitor how well the agency is managing its human capital, GAO has developed eight “people measures” and, similar to executive branch agencies, reports their results as part of its annual Performance and Accountability Report. As part of our commitment to periodically review the reliability and validity of GAO’s performance measures, we evaluated four of the agency’ s people measures—staff development, staff utilization, leadership, and organizational climate.[Footnote 2] We based our evaluation largely on an assessment of whether these measures met nine specific attributes that earlier GAO work cited as key to successful performance measures. Table 1 is a summary of the nine attributes, including the potentially adverse consequences if they are not met. [Footnote 3] Table 1: Nine Key Attributes of Successful Performance Measures: Attribute: 1. Linkage; Definition: Measure is aligned with division and agencywide goals and mission and clearly communicated throughout the organization; Potentially adverse consequences of not meeting attribute: Behaviors and incentives created by measure do not support achieving division or agencywide goals or mission. Attribute: 2. Clarity; Definition: Measure is clearly stated and the name and definition are consistent with the methodology used to calculate it; Potentially adverse consequences of not meeting attribute: Data could be confusing and misleading to users. Attribute: 3. Measurable target; Definition: Measure has a numerical goal; Potentially adverse consequences of not meeting attribute: Cannot tell whether performance is meeting expectations. Attribute: 4. Objectivity; Definition: Measure is reasonably free from significant bias or manipulation; Potentially adverse consequences of not meeting attribute: Performance assessments may be systematically over- or understated. Attribute: 5. Reliability; Definition: Measure produces the same result under similar conditions; Reported performance data is inconsistent and adds uncertainty. Attribute: 6. Core program activities; Definition: Measures cover the activities that an entity is expected to perform to support the intent of the program; Potentially adverse consequences of not meeting attribute: Not enough information is available in core program areas to managers and stakeholders. Attribute: 7. Limited overlap; Definition: Measures should provide new information beyond that provided by other measures; Potentially adverse consequences of not meeting attribute: Manager may have to sort through redundant, costly information that does not add value. Attribute: 8. Balance; Definition: Balance exists when a suite of measures ensures an organization’s various priorities are covered; Potentially adverse consequences of not meeting attribute: Lack of balance could create skewed incentives when measures over-emphasize some goals. Attribute: 9. Governmentwide priorities; Definition: Each measure should cover a priority, such as quality, timeliness, and cost of service; Potentially adverse consequences of not meeting attribute: A program’s overall success is at risk if all priorities are not addressed. Source: GAO. [End of table] For our assessment, we reviewed prior GAO reports and guidance regarding government agencies’ performance measures and spoke with GAO experts and an external expert on developing performance measures and using performance information. We interviewed GAO staff to determine how they used performance information and to identify agency practices that may facilitate or hinder the use of these measures. We also reviewed relevant standard operating procedures and internal controls for developing and publicly reporting these measures. In addition, for the annual Employee Feedback Survey that provides data regarding staff satisfaction with these measures, we consulted with GAO survey design methodologists to understand how the survey was created and related issues of survey design. To assess the accuracy and method of calculation for data reported in GAO’s Performance and Accountability Report for fiscal year 2007,[Footnote 4] we analyzed survey data for questions pertaining to these four people measures. We conducted this performance audit from August 2008 to August 2009 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Results in Brief: Our evaluation showed that performance scores for the staff development, staff utilization, leadership, and organizational climate people measures were accurately calculated according to GAO’s established methodology and that these measures met many of the nine attributes of successful performance measures. For example, staff utilization and organizational climate met all nine attributes. In addition, all four measures met the linkage, measurable target, core program activities, limited overlap, balance, and governmentwide priority. All attributes are not equal and failure to have a particular attribute does not necessarily indicate that there is a weakness in that area or that the measure is not useful; rather, it indicates an opportunity for further refinement. Two measures—leadership and staff development—did not fully meet all of the attributes. Specifically: Clarity attribute. We found that the leadership measure does not fully meet this attribute because it provides information about employees’ opinions about their immediate supervisors and captures information on the agency’s top managers–who set agency’s policies, priorities, and goals–only in their role as supervisors. The clarity attribute requires a measure’s name and definition to be consistent with the methodology used to calculate it. GAO has several options to better meet this attribute. For example, the agency could adopt a more descriptive name reflecting that the measure is based on opinions about immediate supervisors. Another option would involve the agency revising its leadership measure to include employee survey results regarding GAO’s Executive Committee, team management leadership, or both. GAO’s Chief Human Capital Officer noted that it is common in both the public and private sectors to include information about senior leadership as part of a leadership measure. Objectivity attribute. Our evaluation also showed that the leadership and staff development measures could better meet this attribute with more disclosure about their performance scores in GAO’s annual performance reports. Greater disclosure would help assure that GAO is meeting the objectivity attribute requirement that performance assessments are not misinterpreted as over- or understated. Agency performance scores generally constitute the percentage of survey respondents that chose two favorable responses (such as strongly agree and generally agree) on a five-point scale. GAO has reported that it calculates these scores by excluding respondents that answered “no basis to judge/not applicable” (hereafter referred to as no basis/NA respondents) in its detailed description of these measures, but not in tables presenting the actual performance scores. GAO has made such disclosures in tables presenting performance scores in its own audit and evaluation reports.[Footnote 5] The disclosure in table notes is particularly important, for example, because GAO has used these tables to report on its performance in its annual testimonies before its appropriations subcommittees. In addition, GAO has not disclosed in its annual performance reports that its approach of excluding no basis/NA respondents has the effect of changing the leadership and staff development measures’ scores, by 6 percentage points and 13 percentage points, respectively. According to our technical advisers, such disclosure is needed to provide a complete interpretation of the scores and for transparency. This disclosure is important when questions constituting a measure have relatively large numbers of no basis/NA respondents. With smaller numbers of no basis/NA respondents, the effect on scores is negligible and the importance of disclosure is diminished. In addition, our work raises a concern about whether the leadership measure’s question about how effectively immediate supervisors deal with equal employment opportunity (EEO) and discrimination issues is providing useful information. This question has the largest number of no basis/NA respondents among the 21 questions constituting the four measures we reviewed. In addition, it is the primary reason for the change in the leadership performance score since almost half of this question’s respondents answered no basis/NA. According to GAO officials, the high number of no basis/NA respondents most likely reflects the relatively few formal discrimination cases and the steps taken to safeguard the private information related to these cases. GAO has recognized the need for better information to monitor its progress in creating an inclusive culture that values differences and consequently added three questions to its Employee Feedback Survey in 2008. These questions compose the Support for Diversity index developed by the Partnership for Public Service. This index is part of their Best Places to Work in the Federal Government ranking, which is used to evaluate employee satisfaction across the federal government in 279 agencies and subcomponents, such as inspector general offices. The agency’s top diversity officials and Chief Human Capital Officer said that they believe the new survey questions could provide more useful information to the leadership measures than the current EEO and discrimination question. Reliability attribute. We found that the staff development measure could better meet this attribute through more timely disclosures about changes made to the measure. GAO published two annual performance reports for fiscal years 2006 and 2007 without disclosing that it had changed this measure or describing limitations in the comparability of the measure’s 5-year trend data resulting from these changes. The reliability attribute, as defined by GAO, refers to a measure producing the same results under similar conditions. We are reporting not that the data are not reliable, but rather that the agency could better meet this attribute by more timely disclosure of changes to this measure. After we brought this matter to responsible officials’ attention, the fiscal year 2008 Performance and Accountability Report was revised to clarify the measure had been changed and that the 5-year trend data were not comparable. We identified that the agency does not have written procedures to ensure that changes to measures and any effects on comparability of performance data are disclosed in a timely manner. GAO’s guidance for evaluating agency’s performance reporting states that when standard operating procedures are not used, changes to measures that result in noncomparable data can occur.[Footnote 6] To improve GAO’s performance measures, this report recommends that GAO * consider adopting a more descriptive name, or revising the leadership measure to incorporate survey questions about team management, the Executive Committee, or both; * more fully disclose in tables in its annual performance report that the four measures’ scores are calculated by excluding no basis/NA respondents and that this approach, when the number of these respondents are relatively large, has the effect of changing the scores for two measures; * determine for leadership measure whether the current EEO and discrimination question should be retained or if one or more of the survey’s three diversity and inclusiveness questions would provide more useful information; and; * develop written procedures to ensure changes made to measures and any effects on the comparability of performance data are promptly reported. Background: In its audit and evaluation work, GAO has reported that effective performance measures can be a key tool in assessing how well an agency is managing its human capital.[Footnote 7] Effective measures can provide practical information that alerts managers to the existence of problems, helps managers take timely corrective action, and suggests effective problem-solving approaches. GAO has also reported that to fully realize the benefits of this tool, performance measures must be used by decision makers at all levels and that improving the usefulness of performance information and measures is a key practice in enhancing agency performance. In addition to its performance measures, GAO has other information to identify areas for improving its human capital management. For example, employee and management concerns about the agency’s performance appraisal system have been obtained through the Employee Advisory Committee, its annual customer satisfaction survey, International Federation of Professional and Technical Employee Union representatives, its recent African-American Performance Assessment Study,[Footnote 8] and regular management meetings. Based on this information, GAO is involved in a comprehensive review of its performance appraisal system. In addition to the staff development performance measure, GAO collects and analyzes direct participant feedback on the course materials and instructors for all classes taught through GAO’s Learning Center (internal training). One way that GAO measures how well it is doing and identifies areas for improvement is through an annual Employee Feedback Survey. This Web- based survey is conducted by an outside contractor to ensure respondent confidentiality and is administered to our employees once a year. GAO developed the survey using a methodically rigorous approach that included extensive focus groups and pretests to decrease survey errors. The total survey consists of more than 100 questions and allows employees to indicate what they think about the agency’s overall operations, work environment, and organizational culture, as well as how they rate three levels of managers—their immediate supervisor, team or unit managers, and the Executive Committee—on key aspects of their leadership styles.[Footnote 9] From the Employee Feedback Survey, GAO uses the responses to 21 questions to provide the data for the four people measures we evaluated (see attachment 1). GAO selected the subset of questions used for each measure based on senior management’s judgment about the questions’ relevance to the measure and specialists’ knowledge about the development of indexes. To establish scores for the measures, GAO uses the responses of staff expressing opinions on the five-point scale for each measure’s questions. The agency excludes the number of survey respondents who (1) leave the question blank or check “no answer” and (2) check “no basis to judge/not applicable.” For example, as shown in figure 1, the five-point scale generally consists of two favorable responses, one neutral response, and two unfavorable responses. To compute a score, GAO divides the total number of respondents who express a favorable response by the total number of respondents who expressed an opinion on the five-point scale.[Footnote 10] These scores are calculated agencywide and for each individual team or unit. Figure 1: Example of Response Options to GAO’s Employee Survey Staff Utilization Question: [Refer to PDF for image: illustration] 1. During the last 12 months (less if you came to GAO more recently), how often did each of the following occur? a. My job made good use of my skills and abilities: Always or almost Always: Most of the time: About 1/2 of the time: Some of the time: Never or almost never: No basis to judge/Not applicable: No answer: Source: GAO. [End of figure] GAO Measures Met Many Attributes of Successful Performance Measures: Our evaluation showed that the staff development, staff utilization, leadership, and organizational climate people measures were accurately calculated according to GAO’s established methodology and that these measures met many of the nine attributes of successful performance measures. For example, as summarized in table 2, staff utilization and organizational climate met all nine attributes. Table 2: Inspector General Analysis of Attributes of Successful Performance Measures for Four GAO People Measures: Attribute: 1. Linkage; Measure: Staff development: [Check]; Measure: Staff utilization: [Check]; Measure: Leadership: [Check]; Measure: Organizational climate: [Check]. Attribute: 2. Clarity; Measure: Staff development: [Check]; Measure: Staff utilization: [Check]; Measure: Leadership: [Empty]; Measure: Organizational climate: [Check]. Attribute: 3. Measurable target; Measure: Staff development: [Check]; Measure: Staff utilization: [Check]; Measure: Leadership: [Check]; Measure: Organizational climate: [Check]. Attribute: 4. Objectivity; Measure: Staff development: [Empty]; Measure: Staff utilization: [Check]; Measure: Leadership: [Empty]; Measure: Organizational climate: [Check]. Attribute: 5. Reliability; Measure: Staff development: [Empty]; Measure: Staff utilization: [Check]; Measure: Leadership: [Check]; Measure: Organizational climate: [Check]. Attribute: 6. Core program activities; Measure: Staff development: [Check]; Measure: Staff utilization: [Check]; Measure: Leadership: [Check]; Measure: Organizational climate: [Check]. Attribute: 7. Limited overlap; Measure: Staff development: [Check]; Measure: Staff utilization: [Check]; Measure: Leadership: [Check]; Measure: Organizational climate: [Check]. Attribute: 8. Balance; Measure: Staff development: [Check]; Measure: Staff utilization: [Check]; Measure: Leadership: [Check]; Measure: Organizational climate: [Check]. Attribute: 9. Governmentwide priorities; Measure: Staff development: [Check]; Measure: Staff utilization: [Check]; Measure: Leadership: [Check]; Measure: Organizational climate: [Check]. [Check]: indicates measure met attribute. Source: IG analysis of GAO information. [End of table] Table 2 also shows that all four measures met the following attributes: linkage, measurable target, core program activities, limited overlap, balance, and governmentwide priority. * Linkage. We determined the measures met the linkage attribute because they align with GAO’s strategic goal of being a model federal agency and its objective of becoming a professional service employer of choice. Also at the team or unit level, we identified a clear linkage between the team’s scores on the four measures and efforts to improve their operations and the day-to-day activities of staff. Specifically, after reviewing their scores, managing directors used focus groups and other efforts to better understand their scores and take corrective actions. For example, to improve staff development scores, one team developed a separate survey to obtain more specific information on employee dissatisfaction with external training–the focus of one of the survey questions for this measure. Based on the results of this survey, the team made changes to better inform staff of external training opportunities. * Measurable target. GAO sets a quantifiable, numerical target for each measure. Performance data for each measure are reported as both a current year target and actual results. For example, for fiscal year 2007, the target for staff development was a 75 percent favorable response rate and the actual result was 76 percent. Quantifiable measures make it easier for managers to compare actual results to expected performance. * Core program activities. The measures generally cover the activities needed to provide managers with useful information and the survey questions constituting the measures can be considered significant core activities, given GAO’s mission. * Limited overlap. Our work found that each measure provides distinct information beyond that given by other measures. Specifically, a selected number of employee satisfaction survey questions are used to develop each measure. Questions for one measure do not overlap with questions from another measure. * Balance. We found that the four measures are part of a balanced suite of performance measures. In addition to the four people measures we reviewed, GAO has measures to assess the benefits resulting from its work, client satisfaction (i.e., Congress), and the effectiveness of internal administration services. * Governmentwide priority. These four measures focus on GAO’s human capital management challenge, which is also a governmentwide concern: ensuring the federal government has a highly skilled and knowledgeable workforce for the 21st century. Leadership Measure Could Better Meet the Clarity Attribute: We determined that the leadership measure does not fully meet the clarity attribute because the measure does not capture information about the strategic leadership of GAO. As defined by GAO, a successful performance measure has clarity when its name and definition are consistent with the methodology used to calculate it. In designing its Employee Feedback Survey, the agency recognized that leadership encompasses three different management levels—immediate supervisor, team management, and the Executive Committee. Currently, the leadership measure is based on survey responses to questions about immediate supervisors and not the strategic leadership that sets agency policies, priorities, and goals. The measure does capture information about Executive Committee and team managers but only in their role as immediate supervisors. Most immediate supervisors are not in these two management levels. Similarly, in evaluating employee satisfaction across federal agencies, the Partnership for Public Service in its Best Places to Work in the Federal Government also recognizes different levels of leadership, in particular senior leaders and supervisors. In conjunction with American University’s Institute for the Study of Public Policy Implementation, the partnership uses data from the Office of Personnel Management’s Federal Human Capital Survey[Footnote 11] to produce detailed rankings of employee satisfaction for best places to work and 10 workplace environment (“best in class”) categories such as effective leadership. The rankings are designed to inform a broad audience of job seekers, researchers, federal employees, and government leaders. In its 2009 rankings, the Partnership for Public Service also assessed which factors shape employees’ views of their leadership and found that, while conventional wisdom holds that the greatest influence on employee’s satisfaction is their immediate supervisor, it is actually the quality of an agency’s senior leadership that has the greatest bearing on employees’ job satisfaction. In addition, GAO’s Chief Human Capital Officer stated including senior leadership as part of a leadership performance measure is a common public and private sector practice. To better meet the clarity attribute, the agency has several options. It could adopt a more descriptive name that reflects the measure is based on opinions about immediate supervisors. For example, the Partnership for Public Service has two indices, Effective Leadership– Senior Leaders and Effective Leadership–Supervisors Another option would be to revise the measure to include employee survey results regarding GAO’s Executive Committee, team management leadership, or both. To make such revisions, GAO would need to consider that all 10 survey questions constituting this measure are asked of both immediate supervisors and team management, while only 7 of these 10 questions are currently asked about the Executive Committee. Three of the 10 questions were dropped because they had large percentages of staff responding that they had no basis to judge for these questions. The dropped questions asked whether the Executive Committee (1) gave the employee the opportunity to do what he/she does best, (2) made decisions in a timely manner, and (3) dealt effectively with EEO and discrimination issues. Additional Disclosure Would Help Two Measures Better Meet the Objectivity Attribute: Our work showed that the staff development and leadership measures could better meet the objectivity attribute with more disclosure that would allow a more complete interpretation of these performance scores. Specifically, the agency could be more transparent in reporting that these scores are based only on survey respondents who answered the 5- point scale and that this choice of not including no basis/NA respondents has the effect of changing the scores by 6 percentage points or more. In addition, our work showed that management attention is needed to determine if the leadership measure’s EEO and discrimination question is providing useful information. More Disclosure Is Needed to Interpret Performance Scores for Two Measures: For transparency and a complete interpretation of scores, GAO survey design methodologists and an external performance measurement expert who provided us technical advice told us that, at a minimum, the agency should disclose any significant effects associated with its approach for calculating performance scores. Because scores can be calculated either by consistently including or excluding no basis/NA respondents, our advisers said the approach taken and any related effects should be disclosed. For example, as the number of no basis/NA respondents increase the two approaches can result in different scores. In its annual performance reports GAO has publicly disclosed that it excludes no basis/NA respondents when calculating performance scores in the report’s detailed description of these measures, but not in the report’ s tables presenting the performance scores. At the same time, the agency has not disclosed anywhere in its annual performance reports that its approach of excluding no basis/NA respondents has the effect of changing the scores for these two measures. If the number of no basis/NA respondents is relatively large, disclosing such information in table notes is important so that the tables if removed and used in other publications provide complete information. For example, GAO used these tables to report on its performance in its annual testimonies before its appropriations subcommittees. By making such disclosures, greater transparency and a more complete interpretation of scores would be possible. The agency’s approach of not including no basis/NA respondents has the effect of changing performance scores for the staff development and leadership measures by 13 percentage points and 6 percentage points, respectively.To illustrate how these scores may be misinterpreted, for fiscal year 2007, GAO reported a satisfaction score of 76 percent for the staff development measure, which could be misinterpreted as 76 percent of all surveyed employees were satisfied. Instead, the score reflects (1) 63 percent of respondents to the measures’ questions answered they were satisfied on the 5-point scale and (2) the number of no basis/NA respondents for the measure’s three questions (12 percent for two questions and 28 percent for a third). For fiscal year 2008, the percentage of no basis/NA respondents for these questions remained the same or increased by about 2 percentage points. Similarly, for the leadership measure in fiscal year 2007, GAO reported a satisfaction score of 79 percent. This score reflects that 73 percent of surveyed employees who answered on the 5-point scale were satisfied. The difference in the percentages is primarily caused by excluding 45 percent of respondents who answered no basis/NA to one of the measure’s 10 questions, which is about EEO and discrimination issues.[Footnote 12] For fiscal year 2008, the percentage of no basis/NA respondents for this question increased to 49 percent. Leadership Measure’s EEO and Discrimination Question Warrants Further Management Attention: Our work raises a concern about whether the leadership measure’s EEO and discrimination question is obtaining useful employee feedback and should be retained as part of the measure. This question’s narrow focus on EEO legal issues is responsible for almost half of the respondents answering no basis/NA, according to agency officials. They explained that since GAO has so few formal discrimination cases and takes steps to keep information related to these cases confidential, many employees do not have direct knowledge about how supervisors and/or managers deal with such issues. While we could not find any guidance that specifically addressed acceptable no basis/NA response rates for survey data that are used to establish performance scores, we did identify GAO guidance that response rates of at least 70 percent are acceptable when individual questions (survey items) are expected to be key to a report message. [Footnote 13] While no basis/NA respondents are not item nonrespondents, GAO survey design methodologists we consulted said GAO’ s guidance for item nonresponse could reasonably serve as criteria for no basis/NA respondents. As a result, we believe GAO management should reconsider whether the EEO and discrimination question should continue to be used in its leadership measure since almost half of the respondents answered no basis/NA—which falls well below the 70 percent response rate. In addition, removing this question from the leadership measure would reduce (1) the impact on the performance score caused by excluding no basis/NA respondents in its calculations and (2) the likelihood for misinterpretation of this score. One alternative would be to substitute one or more of the recently added diversity and inclusiveness questions for the question on EEO and discrimination issues. Recognizing the need for additional information on diversity and inclusiveness, GAO in 2008 added the three questions to its Employee Feedback Survey that constitute the Support for Diversity index developed by the Partnership for Public Service as part of its Best Places to Work in the Federal Government ranking. The questions are: * Supervisors/team leaders in my work unit are committed to a workforce representative of all segments of society. * Policies and programs promote diversity in the workplace (for example, recruiting minorities and women, training in awareness of diversity issues, mentoring). * Managers/supervisors/team leaders work well with employees of different backgrounds. According to knowledgeable staff, these questions were selected to enable GAO to benchmark its results with federal governmentwide survey data and address a recommendation from the African-American Performance Assessment Study. The recommendation suggested that the agency use the Employee Feedback Survey to monitor the extent to which employees believe GAO has an inclusive culture that values differences. These new questions may provide more useful information than the current question and more staff should be able to respond. GAO’s Special Assistant to the Comptroller General for Diversity and the Acting Managing Director of GAO’s Office of Opportunity and Inclusiveness stated the new questions should offer more useful data since they directly address how supervisors and other managers support creating an inclusive and diverse culture and most staff should have direct knowledge and experiences that will enable them to provide an answer. For example, in the fiscal year 2008 survey between 10 percent and 16 percent of respondents answered “do not know” in response to the new questions while 49 percent of respondents answered no basis/NA to the EEO and discrimination question. In addition, GAO’s Chief Human Capital Officer also thought the new questions would provide more useful information than the current EEO question. Written Procedures Would Help Ensure Full and Timely Disclosure about Changes to Performance Measures: After making changes to its staff development measure in fiscal year 2006, GAO published two annual performance reports that did not disclose that this measure had changed. GAO’s detailed description of the measure did describe the revised questions and responses but not that they had changed from the prior year. The agency also did not disclose limitations in the comparability of the measure’s 5-year trend data resulting from these changes. As a result, there was no way to discern that the measure had been changed. In October 2008, after we notified GAO management of this issue, the agency reported changing this performance measure in its fiscal year 2008 Performance and Accountability Report[Footnote 14] and disclosed that the staff development performance data for fiscal years 2004 and 2005 are not comparable with the data for fiscal years 2006 to 2008. In fiscal year 2006, GAO made two changes to the staff development measure. Originally, the measure was composed of responses to four questions about internal training, computer-based training, external training, and on-the-job training. The first change, as shown in table 3, was to revise the responses to the internal training question from two to three favorable responses.[Footnote 15] According to agency officials, GAO’s Chief Learning Officer proposed this change because she believed the revision would better capture staff satisfaction with internal training, which was undergoing significant changes. The second change was to drop the computer-based training question because this training was a significant part of (and therefore included in) the measure’s other questions. Table 3: Changes in Staff Development—Internal Training Question’s Five- Point Response Scale: Original response scale: Very positive impact [highlighted]: Generally positive impact [highlighted]: Neither positive nor negative impact: Generally negative impact: Very negative impact: Revised response scale: Very greatly useful and relevant [highlighted]: Greatly useful and relevant [highlighted]: Moderately useful and relevant [highlighted]: Somewhat useful and relevant: Little or not useful and relevant: Source: IG analysis of GAO information. Note: Favorable responses are highlighted. [End of table] One cause for the delayed disclosures may be that the agency’s Office of Quality and Continuous Improvement does not have written procedures to ensure changes to performance measures and their effects to data comparability are disclosed in a timely manner. GAO’s guidance for evaluating agency’s performance reporting states that when standard operating procedures are not used, changes to measures can occur that result in noncomparable data.[Footnote 16] Further, GAO’s Standards for Internal Control in the Federal Government states that management is responsible for developing detailed procedures to ensure that they become an integral part of the agency’s operations. The standards also state that such internal controls should be clearly documented, and the documentation should be readily available for examination.[Footnote 17] Conclusions: GAO has taken significant steps in developing effective human capital performance measures and monitoring its human capital management activities, such as developing and implementing an annual Employee Feedback Survey. Consequently, two of the four measures met all of the agency’s attributes for successful performance measures, and two other measures met most attributes. We believe that adding greater clarity and transparency to the leadership and staff development measures and improving the processes for making changes to GAO’s performance measures would help avoid misinterpretations of data and further strengthen these measures’ usefulness. Recommendations for Executive Action: To improve the performance measures GAO uses to address its human capital management challenge and the processes for developing these measures, we recommend that the Acting Comptroller General take the following four actions: * To help the leadership measure fully meet the clarity attribute, adopt a more descriptive name that better aligns with its focus on immediate supervisors or incorporate into the measure, additional employee survey questions about team management, the Executive Committee leadership, or both. * To help the staff development and leadership performance measures fully meet the objectivity attribute, - disclose in tables presenting performance scores that the four measures’ scores are calculated by excluding no basis/NA respondents and that this approach, when the number of these respondents are relatively large, has the effect of changing the scores for two measures, and; - determine for leadership measure whether the current EEO and discrimination question should be retained or if one or more of the survey’s three diversity and inclusiveness questions would provide more useful information to gauge the agency’s progress in creating a more diverse work environment. * To help ensure the reliability attribute is fully met when performance measures are changed, direct the Managing Director, Office of Quality and Continuous Improvement, to develop written procedures to ensure that such changes and any effects on comparability of performance data are fully disclosed in a timely manner. Agency Comments: The Inspector General provided GAO with a draft of this report for review and comment. GAO generally agreed with our recommendations. The agency also provided technical comments that we incorporated, as appropriate. GAO’s written comments are reproduced in attachment 2. If you or your staff have any questions about this report, please contact me at (202) 512-5748 or garciaf@gao.gov. OIG staff who made significant contributions to this report are Cathy Helm (Deputy Inspector General), Gwendolyn Jaffe, and Kurt Kershow. Attachments - 3: [End of section] Attachment I: Four People Performance Measures That GAO Uses to Assess Its Human Capital Management: Following are the four people performance measures—organizational climate, staff development, staff utilization, and leadership—that GAO uses to assess its human capital management. Each measure includes selected questions and possible responses taken from GAO’s Employee Feedback Survey. 1. Organizational Climate: Thinking back over the last 12 months, how strongly do you agree or disagree with each of the following statements? 1. A spirit of cooperation and teamwork exists in my work unit. 2. I am treated fairly and with respect in my work unit. 3. My morale is good. 4. Sufficient effort is made in my work unit to get the opinions and thinking of people who work here. 5. Overall, I am satisfied with my job at GAO. 2. Staff Development: How much positive or negative impact did the following developmental activities have on your ability to do your job during the last 12 months? 1. External training/conferences. 2. On-the-job training that I received. 3. Internal (Learning Center) training courses. 3. Staff Utilization: During the last 12 months, how often did each of the following occur? 1. My job made good use of my skills and abilities. 2. GAO provided me with opportunities to do challenging work. 3. In general, I was utilized effectively. 4. Leadership: In your opinion, how often was each of the following behaviors or attributes exhibited by your immediate supervisor during the last 12 months? 1. Gave me the opportunity to do what I do best. 2. Treated me fairly. 3. Acted with honesty and integrity toward me. 4. Ensured that there was a clear link between my performance and recognition of it. 5. Gave me the sense that my work is valued. 6. Provided me meaningful incentives for high performance. 7. Made decisions in a timely manner. 8. Demonstrated GAO’s core values of accountability, integrity, and reliability. 9. Implemented change effectively. 10. Dealt effectively with equal employment opportunity and discrimination issues (e.g., ensures zero tolerance for discrimination). [End of section] Attachment II: Comments from the Office of the Comptroller General of the United States: GAO: Memorandum: Date: August 25, 2009: To: Inspector General - Frances Garcia: From: [Signed by] Gene L. Dodaro - Acting Comptroller General: Subject:Response to GAO Inspector General's Draft Report: Four GAO People Performance Measures: Attributes of Successful Measures Met, Opportunities Exist for Further Enhancements: Thank you for your draft report assessing GAO's people measures. We have made it a priority to enhance our human capital management practices and these measures help us to track our progress. We appreciate your recommendations and will consider how best to address them as we continue to improve our measurements going forward. While we intend to address these recommendations, we want to reiterate that GAO established its methodology for these measures based on the expert advice of the methodologists in our Applied Research and Methods team, which houses our experts on methodological approaches and analysis that help the agency address congressional requests. ARM also regularly solicits input from external survey experts to improve GAO's survey practices. GAO methodologists believed the measures, as constructed, are consistent with conventional survey practice and appropriate. They also believed the decision to exclude no basis/not applicable responses was a reasonable one, consistent with conventional survey practice and that it would provide a better measure because it represents only those employees who have an opinion on the questions. Nonetheless, in addressing your recommendations we will consider how best to report the results and, accordingly, will adapt our targets as appropriate. We will keep you informed of our progress in addressing your recommendations. cc: Sallyanne Harper, Chief Administrative Officer: Daniel I. Gordon, Acting General Counsel: [End of section] Related GAO Products: Government Performance: Lessons Learned for the Next Administration on Using Performance Information to Improve Results. [hyperlink, http://www.gao.gov/products/GAO-08-1026T]. Washington, D.C.: July 24, 2008. Managing for Results: Enhancing Agency Use of Performance Information for Management Decision Making. [hyperlink, http://www.gao.gov/products/GAO-05-927]. Washington, D.C.: September 9, 2005. Results-Oriented Government: GPRA Has Established a Solid Foundation for Achieving Greater Results. [hyperlink, http://www.gao.gov/products/GAO-04-38]. Washington, D.C.: March 10, 2004. Tax Administration: IRS Needs to Further Refine Its Tax Filing Season Performance Measures. [hyperlink, http://www.gao.gov/products/GAO-03- 143]. Washington, D.C.: November 22, 2002. The Results Act: An Evaluator’s Guide to Assessing Agency Annual Performance Plans. [hyperlink, http://www.gao.gov/products/GAO/GGD- 10.1.20]. Washington, D.C.: April 1998. [End of section] Footnotes: [1] GPRA, Pub. L. No. 103-62. [2] We reported the results of our evaluation of GAO’s other four people measures in our August 5, 2008, report, Fiscal Year 2007 New Hire, Acceptance, and Retention Rates Performance Measures Need Improvement and Alternative Measures Should be Considered. [3] GAO, Tax Administration: IRS Needs to Further Refine Its Tax Filing Season Performance Measures, [hyperlink, http://www.gao.gov/products/GAO-03-143] (Washington, D.C.: Nov. 22, 2002). [4] GAO, Performance and Accountability Report: Fiscal Year 2007, [hyperlink, http://www.gao.gov/products/GAO-08-1SP] (Washington, D.C.: Nov. 15, 2007). [5] GAO, Government Performance: Lessons Learned for the Next Administration on Using Performance Information to Improve Results, [hyperlink, http://www.gao.gov/products/GAO-08-1026T] (Washington, D.C.: July 24, 2008); and Results-Oriented Government: GPRA Has Established a Solid Foundation for Achieving Greater Results, [hyperlink, http://www.gao.gov/products/GAO-04-38] (Washington, D.C.: Mar. 10, 2004). [6] GAO, The Results Act: An Evaluator’s Guide to Assessing Agency Annual Performance Plans, [hyperlink, http://www.gao.gov/products/GAO/GGD-10.1.20] (Washington, D.C.: April 1998). [7] [hyperlink, http://www.gao.gov/products/GAO-08-1026T]. [8] Ivy Planning Group, African American Performance Assessment Study (Rockville, Md., April 2008). [9] GAO’s Executive Committee is currently composed of the Acting Comptroller General, Chief Administrative Officer/Chief Financial Officer, and Acting General Counsel. [10] GAO characterizes favorable responses to survey questions in four ways (1) “strongly agree” or “generally agree”; (2)“very positive impact” or “generally positive impact”; (3) “always or almost always” or “most of the time”; and (4)“very greatly useful and relevant,” “ greatly useful and relevant,” or “moderately useful and relevant.” [11] [hyperlink, http://www.fhcs.opm.gov]. [12] The difference is also due to the percentage of no basis/NA respondents—7 percent to 10 percent—for 3 other questions that are part of this measure. [13] GAO, Calculating and Reporting Response Rates and Addressing Nonresponse Issues (Washington, D.C.: Dec. 10, 2003). [14] GAO, Performance and Accountability Report: Fiscal Year 2008, [hyperlink, http://www.gao.gov/products/GAO-09-1SP] (Washington, D.C.: Nov. 15, 2008). [15] GAO’s other 20 people measure survey questions have two favorable responses. [16] [hyperlink, http://www.gao.gov/products/GAO/GGD-10.1.20]. [17] GAO, Standards for Internal Control in the Federal Government, [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1] (Washington, D.C.: November 1999). [End of section] Reporting Fraud, Waste, and Abuse in GAO’s Internal Operations: To report fraud, waste, and abuse in GAO’s internal operations, do one of the following. (You may do so anonymously.) * Call toll-free (866) 680-7963 to speak with a hotline specialist, available 24 hours a day, 7 days a week. * Send an e-mail to OIGHotline@gao.gov. * Send a fax to the OIG Fraud, Waste, and Abuse Hotline at (202) 512- 8361. * Write to: GAO Office of Inspector General: 441 G Street NW, Room 1808: Washington, DC 20548: Obtaining Copies of GAO/OIG Reports and Testimony: To obtain copies of OIG reports and testimony, go to GAO’s Web site: [hyperlink, www.gao.gov/about/workforce/ig.html].