Program Measurement Continues to Confront Data Reliability Issues
GAO-12-519, Apr 19, 2012
What GAO Found
To prepare for its reporting obligations under the Telework Enhancement Act of 2010, the Office of Personnel Management (OPM) assembled the Interagency Telework Measurement Group, consisting of officials from several federal agencies, to assist in revising the telework data callthe survey OPM has used since 2002 to collect telework data from executive agencies. This group standardized key terms such as telework, employee, and eligibility to promote a common reporting methodology among the agencies. The revised telework data call also included changes to the time period for which OPM requested agencies report telework data, and included more extensive training for respondents.
Because of changes made to the data call to allow OPM to meet requirements of the act and assist agencies in responding to the data call, OPM officials believe they will be able to provide to Congress an improved report on telework in June 2012. However, these changes also mean that OPM officials will not be able to use participation and frequency data from the 2011 data call to compare to data from previous years and across agencies. OPM officials have noted that this could limit OPMs ability to report agency progress in its first report to Congress. The ability to compare with previous years is affected by:
agencies use of methods of varying reliability to collect telework data, and some agencies made changes to their data collection systems for the 2011 data call. Executive agencies provide telework participation and frequency data by relying on estimates, counting telework agreements, or using automated time and attendance records.
modifications to the data call instrument, including changes to terminology and the time period during which telework data was requested. OPM officials said they expect these changes will improve the consistency of data. But if OPM reports progress based on data collected using changing terminology and from different time periods, the agency may reach erroneous conclusions.
Participants at the two data call training sessions may not have received the same reporting instructions, and uncertainty about whether all agency respondents attended training, created a risk that some respondents may be unaware of important terms and instructions. While some of the information provided at the two training sessions was similar, each session contained some new information, usually in response to questions raised at a previous session.
Future data call improvement efforts could result in a trade-off between the desire for maintenance of a consistent data series over time for comparison with previous data calls and a need to improve data collection. According to OPM, agencies will begin piloting automated telework data collection during 2012 and 2013. OPM expects that this method of data collection will provide it more reliable data than other methods. However, these efforts to standardize methods for tracking telework data may result in changes to agencies methods of data collection. The 2011 data call, notwithstanding its limitations, will be useful to help OPM identify and understand major changes in reported participation data that could occur during a transition to automated data collection.
Why GAO Did This Study
The Telework Enhancement Act of 2010 requires OPM to report to Congress on the degree of telework participation at executive agencies. To meet this requirement, OPM collects information on agency telework programs through an annual survey, which it refers to as the data call. However, concerns exist about the reliability of these data.
GAO was asked to assess OPMs: (1) actions in response to the requirements of the act and (2) handling of identified data reliability issues in the 2011 telework data call. To address these objectives GAO reviewed its previous reports addressing telework data reliability, and used the Office of Management and Budgets guidance for federal surveys to review OPMs (1) plans to collect telework participation data from agencies and (2) development of a data collection instrument. GAO interviewed key OPM officials about its implementation of the 2011 data call.
What GAO Recommends
GAO recommends that OPM (1) clearly report reliability limitations with the 2011 telework data call in its June 2012 report to Congress and (2) continue efforts to improve data collection and gather information to allow for the appropriate qualification of year-to-year comparisons and inform users about the effects of data collection changes going forward. OPM partially concurred with the first recommendation. However, GAO believes it should report limitations in its annual report. OPM fully concurred with the second. OPM provided a number of technical comments which GAO incorporated as appropriate.
For more information, contact Yvonne D. Jones at (202) 512-6806 or email@example.com.
- Closed - implemented
- Closed - not implemented
Recommendations for Executive Action
Recommendation: To improve OPM's annual reporting of telework to Congress, the OPM Director should ensure that the reliability limitations related to the 2011 telework data call are clearly reported in its June 2012 report to Congress by fully describing how existing measures of telework participation vary widely in validity and reliability and limit the capability of OPM to reliably report the actual level and frequency of telework participation.
Agency Affected: Office of Personnel Management
Status: Closed - Implemented
Comments: In its report, GAO noted that that comparisons to telework data from previous years are affected by several factors: (1) executive agencies use methods of varying reliability to collect telework data such as estimates, counting telework agreements, or using automated time and attendance records, and (2) modifications to the telework data call instrument. Since then, in its June 2012 annual report OPM stated that there are some limitations with respect to the participation and frequency findings that should be considered, and that agencies rely upon differing methodologies and data sources when gathering participation and frequency data, including time and attendance systems, counts of telework agreements, and surveys of employees. As a result, the final participation and frequency numbers may underreport telework with consequences for the reliability of reported results. Additionally, in the report's detailed methodology, OPM states that, while changes to the data call make some comparisons between previous years' data calls less appropriate, they were necessary in order to accurately gauge the changing nature of Federal telework programs.
Recommendation: To improve OPM's annual reporting of telework to Congress, the OPM Director should continue efforts to improve data collection and gather information that allows for the appropriate qualification of year-to-year comparisons and informs users about the effects of data collection changes going forward.
Agency Affected: Office of Personnel Management
Comments: OPM states the telework data call completed by agencies in October-November 2012 established a baseline for year-to-year comparisons in its 2013 report to Congress. The 2013 report will be reviewed regarding the comparisons it provides. The report is expected to be issued during the summer of 2013.