This is the accessible text file for GAO report number GAO-05-526 entitled 'Immigration Services: Better Contracting Practices Needed at Call Centers' which was released on June 30, 2005. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Report to Congressional Requesters: United States Government Accountability Office: GAO: June 2005: Immigration Services: Better Contracting Practices Needed at Call Centers: GAO-05-526: GAO Highlights: Highlights of GAO-05-526, a report to congressional requesters: Why GAO Did This Study: The U.S. Citizenship and Immigration Services (USCIS) bureau within the Department of Homeland Security (DHS) provides toll-free telephone assistance through call centers to immigrants, their attorneys, and others seeking information about U.S. immigration services and benefits. As the volume of calls increased—from about 13 million calls in fiscal year 2002 to about 21 million calls in fiscal year 2004—questions were raised about USCIS’s ability to ensure the reliability and accuracy of the information provided at call centers run by an independent contractor. This report analyzes: (1) the performance measures established by USCIS to monitor and evaluate the performance of contractor-operated call centers; (2) how performance measures were used to evaluate the contractor’s performance; and (3) any actions USCIS has taken, or plans to take, to strengthen call center operations. What GAO Found: USCIS developed seven performance measures intended to assess the performance and overall quality of responses provided by customer service representatives at contractor-operated call centers. These measures include how quickly calls were answered and the accuracy of information provided. The contract between USCIS and its contractor stipulated that the contractor could earn financial incentive awards if the average monthly performance met or exceeded the standards on a quarterly basis at each of four call centers. Conversely, financial deductions could be made if the standards were not met. USCIS did not finalize the terms regarding how the contractor’s actual performance would be calculated, or scored, before awarding the contract. This limited USCIS’s ability to exercise performance incentives (positive or negative) because the parties could not reach agreement on performance terms. USCIS suspended the use of financial incentives while the parties negotiated the issue. Agreement was not reached after 16 months, however, USCIS determined that the contractor had failed to meet standards for 4 of the 7 performance measures in the fourth quarter of 2004 and took action to reduce its payments for services. The contractor objected, citing the lack of agreement on the performance measurements and the impact of workload increases, but USCIS disagreed and stated it would reduce payment. In a separate but related matter, USCIS failed to meet contractual, regulatory, and GAO standards pertaining to how the contractor’s performance would be documented—especially with respect to any deficiencies. Finally, USCIS exercised its option to extend the call center contract through May 2006, to allow time to solicit and award new call center contracts. USCIS said it intends to finalize performance measurement terms in the new contracts. USCIS used contractor performance data it collected over the course of the contract to identify opportunities to improve customer service and call flow, among other things. Several initiatives were launched as a result. USCIS Call Flow and Call Volume for Fiscal Year 2004: [See PDF for image] [End of figure] What GAO Recommends: To improve USCIS’s evaluation of contractor performance, GAO recommends that USCIS take steps to ensure that performance measurement provisions are finalized before awarding new contracts and that performance evaluation records are properly maintained. DHS generally agreed with GAO’s recommendations and indicated USCIS was taking steps to implement them. www.gao.gov/cgi-bin/getrpt?GAO-05-526. To view the full product, including the scope and methodology, click on the link above. For more information, contact Paul Jones at (202) 512- 8777 or jonespl@gao.gov. [End of section] Contents: Letter: Results in Brief: Background: USCIS Established Performance Measures to Assess Contractor's Call Center Performance: USCIS Evaluated Contractor's Performance but Suspended Use of Financial Incentives for More Than 2 Years Due to Performance Measurement Dispute: USCIS Has Used Results of Monitoring Efforts to Identify Opportunities to Improve Customer Service and Call Flow at All Call Centers: Conclusions: Recommendation for Executive Action: Agency Comments and Our Evaluation: Appendix I: Scope and Methodology: Appendix II: Criteria and Methodology Used by Contractor to Measure Quality of Tier 1 Calls Monitored: Appendix III: Criteria and Methodology Used to Measure Customer Satisfaction with Customer Service Representatives: Appendix IV: Criteria and Methodology Used by Independent Consulting Firm to Measure Quality of Calls Monitored: Appendix V: Mystery Shopper Scenario Example: Appendix VI: Comments from the Department of Homeland Security: Appendix VII: Comments from Pearson Government Solutions: Appendix VIII: GAO Contact and Staff Acknowledgments: Tables: Table 1: Call Quality Monitoring or "Soft Skills" Table 2: Accuracy of Information Provided: Table 3: Accuracy of Capturing Information: Figures: Figure 1: Organization of USCIS's National Customer Service Center and Call Centers: Figure 2: Tier 1 Customer Service Representatives Processing Immigration-Related Calls at a Contractor-Operated Call Center: Figure 3: USCIS Call Flow and Call Volume for Fiscal Year 2004: Abbreviations: USCIS: U.S. Citizenship and Immigration Services: COTR: contracting officer's technical representatives: CSR: customer service representative: DHS: Department of Homeland Security: FAR: Federal Acquisition Regulation: IIO: immigration information officer: INS: Immigration and Naturalization Service: NCSC: National Customer Service Center: OFPP: Office of Federal Procurement Policy: PRS: Performance Requirements Summary: VA: Department of Veterans Affairs: United States Government Accountability Office: Washington, DC 20548: June 30, 2005: Congressional Requesters: The U.S. Citizenship and Immigration Services (USCIS) bureau within the Department of Homeland Security (DHS) provides telephone assistance to customers calling about U.S. immigration services and benefits. USCIS customers--immigrants and their attorneys or family members, employers, and others--can call toll-free for information on such topics as how to become U.S. citizens, obtain work visas, initiate the naturalization process, obtain legal forms, and determine the status of applications for benefits. This telephone assistance service helps customers avoid making visits to USCIS offices, which could require customers to take time off from work and travel long distances. Telephone calls for assistance to USCIS have increased in recent years, from about 13 million in fiscal year 2002 to about 21 million in fiscal year 2004. These calls are handled in a variety of ways: by an interactive voice response system;[Footnote 1] by four call centers managed by a private contractor (Pearson Government Solutions) engaged by USCIS; and by two call centers operated directly by USCIS. You expressed interest in how USCIS monitors and evaluates contractor- operated call centers to ensure that they operate reliably and provide accurate information to their growing number of customers. This report addresses the following: (1) What performance measures did USCIS establish to monitor and evaluate the performance of contractor- operated call centers? (2) How were these performance measures used to evaluate the contractor's performance? (3) What actions, if any, did USCIS take or plan to take to strengthen call center operations? To address these objectives, we obtained and analyzed information from USCIS officials in Washington, D.C., from contractor officials in Arlington, Virginia, and at one of four contractor-operated call centers.[Footnote 2] At these locations, we interviewed officials and collected and analyzed pertinent documentation, including descriptions of monitoring and evaluation programs, contract requirements for call center operations, and summaries of call center evaluation results. We assessed the reliability of telephone call volume data provided to USCIS by a telecommunications vendor, as well as USCIS and contractor staffing data. To carry out our data reliability assessments, we (1) reviewed information about the data, systems that produced the data, and data quality control procedures, and (2) interviewed USCIS and contractor officials knowledgeable about the data as necessary. We determined that these data were sufficiently reliable for purposes of this report. We conducted our work between May 2004 and May 2005 in accordance with generally accepted auditing standards. (See app. I for details about our scope and methodology.) Results in Brief: USCIS developed seven performance measures that were intended to assess the performance of and overall quality of responses provided by customer service representatives at contractor-operated call centers. These measures were described in a contract signed by both parties in January 2002. Two performance measures, for example, were to assess the accuracy of information provided to callers and how quickly calls were answered. Based on provisions in the contract, the contractor could earn financial incentive awards if certain standards or goals set for each performance measure were met or exceeded on a quarterly basis at each of the call centers--for instance, if callers waited an average of 30 seconds to 36 seconds for their calls to be answered. Conversely, the contract provided for negative financial incentives (penalties), whereby USCIS could deduct from payments owed to the contractor on a quarterly basis if the standards were not met. USCIS collected data on the contractor's performance on a monthly basis since June 2002, when the contract was awarded. However, these data were not used for the purpose of applying financial incentives (either positive or negative) until the fourth quarter of 2004. The use of financial incentives was suspended because USCIS awarded the contract without finalizing how the contractor's actual performance would be calculated, or scored, for the various performance measures. The two parties negotiated on this issue over a period of about 16 months after the contract was awarded before abandoning the effort. Thereafter, USCIS determined that, for the fourth quarter of 2004, the contractor had failed to meet standards for four of the seven performance measures, and announced its intention to reduce payments to the contractor for services. The contractor objected to USCIS's decision on the grounds that the performance measurements had not been finalized and that changes in call center workloads affected the basis for applying financial incentives. USCIS disagreed and stated it would reduce payment to the contractor. Although this dispute had not been resolved, USCIS exercised its option to extend the current call center contract through May 2006 to allow time to solicit and award new call center contracts. The exercise of this option has no effect on the contract's performance measurement terms, which is the source of the parties' disagreement. In a separate but related matter, USCIS failed to meet contractual, regulatory, and GAO standards pertaining to how the contractor's performance was documented--especially with respect to any deficiencies. For example, USCIS's failure to obtain the contractor's written acknowledgement of USCIS-identified performance deficiencies did not meet the notification procedures established by the contract, documentation requirements of the Federal Acquisition Regulations, and GAO's standards for internal control. Consequently, USCIS did not maintain a complete and reliable record of the contractor's performance needed to ensure accountability. USCIS used the contractor performance data it collected monthly over the course of the contract to identify opportunities to improve customer service, help better respond to customer inquiries, and manage the flow of calls into call centers. Several initiatives have been launched as a result. For example, USCIS has implemented new call- routing procedures designed, in part, to route customers' calls to service representatives more quickly and reduce delays (on-hold time) that arise when call volume is heavy. These and other initiatives were undertaken by USCIS concurrent with its efforts to negotiate contract differences with the contractor. It is too early to assess the impact of these initiatives. To improve USCIS's efforts for evaluating contractor performance and encourage quality services at call centers, we recommend that the Secretary of Homeland Security require the Director of USCIS take the following two actions: (1) finalize contract terms related to specific performance measurement requirements, before awarding new performance- based call center contracts; and (2) maintain readily available written records of performance assessments and performance evaluation meetings with the contractor. DHS and the contractor provided formal comments and technical comments on a draft of this report, which we have incorporated, as appropriate. DHS generally agreed with our recommendations. The contractor said the report accurately summarizes the complex nature of CIS's call center program and several challenges created by significant post-award changes to that program. Background: The Homeland Security Act of 2002 established USCIS within DHS.[Footnote 3] USCIS is responsible for several functions transferred on March 1, 2003, from the former Immigration Services Division of the Immigration and Naturalization Service (INS) under the Department of Justice.[Footnote 4] These functions include providing services or benefits to facilitate entry, residence, employment, and naturalization of legal immigrants; processing applications for U.S. citizenship/naturalization; and rendering decisions about immigration- related matters. Call Center Operations: The USCIS Information & Customer Service Division is responsible for operating the National Customer Service Center (NCSC), which was established in 1997 to provide nationwide assistance by telephone to customers calling about immigration services and benefits. When a customer calls the NCSC toll-free number (1-800-375-5283), the call is received by the interactive voice response system. The system features automated, self-service options 24-hours a day, 7 days a week. If the system cannot address a customer's concerns or needs or if a customer requests live assistance, then the call is generally routed to one of the four NCSC contract call centers, known as Tier 1.[Footnote 5] These four centers are operated by the contractor, Pearson. If a question posed by a customer is particularly complex or otherwise cannot be answered at the Tier 1 level, the call is transferred to one of the two USCIS-operated call centers, known as Tier 2.[Footnote 6] Figure 1 shows the organization of NCSC, including the call centers. Figure 1: Organization of USCIS's National Customer Service Center and Call Centers: [See PDF for image] [End of figure] In fiscal year 2004, almost half of 21.1 million calls made to NCSC were handled and completed by the interactive voice response system and the rest were generally routed to Tier 1. Customer service representatives (CSR) at Tier 1 respond to inquiries in English or Spanish. The CSRs focus primarily on providing administrative information to customers by using a series of scripts provided by USCIS. For example, if a customer needs what USCIS considers basic information, such as USCIS local offices' hours of operations, eligibility requirements, and procedures to follow, such questions are to be answered by CSRs at Tier 1 call centers using specific scripts. In addition, CSRs are to refer customers to USCIS service centers and local offices, for such things as changes of address and appointment scheduling at USCIS application support centers. (Some of these tasks may alternatively be performed by customers through the USCIS Web site- -www.uscis.gov.) As of April 2005, the four Tier 1 call centers employed over 450 CSRs.[Footnote 7] Figure 2 shows CSRs processing calls at a Tier 1 call center. Figure 2: Tier 1 Customer Service Representatives Processing Immigration-Related Calls at a Contractor-Operated Call Center: [See PDF for image] [End of figure] At the two USCIS-operated Tier 2 call centers, calls are handled by immigration information officers (IIO)--immigration specialists with in- depth knowledge of immigration laws, non-immigrant visas, naturalization, asylum and refugee status, and other related policies and procedures. As of April 2005, the Tier 2 call centers operated by USCIS had 111 IIOs. About 5 percent, or about 590,000, of the calls going to Tier 1 CSRs were rerouted to Tier 2 IIOs in fiscal year 2004. Figure 3 shows the call volume handled by the interactive voice response system, Tier 1 call centers, and Tier 2 call centers during fiscal year 2004. Figure 3: USCIS Call Flow and Call Volume for Fiscal Year 2004: [See PDF for image] [A] "Completed calls" refer to calls that have ended, not necessarily to calls where customers' questions or issues have been resolved. [End of figure] Performance-Based Service Contract: In January 2002, USCIS awarded a performance-based service contract for the management of four Tier 1 call centers.[Footnote 8] In making this award, USCIS obtained acquisition services from the Department of Veterans Affairs (VA), and the contracting officer who signed and was responsible for administering the contract was a VA employee working on behalf of USCIS.[Footnote 9] The contracting officer's technical representative (COTR), a USCIS employee, was also responsible for administering the contract. In commenting on a draft of this report, DHS noted that by agreement of both the VA and USCIS, on April 20, 2005, USCIS assumed responsibility for administering the contract. The contract was awarded for a base year, beginning on June 1, 2002, plus 4 option years (1-year renewable extensions of the contract, three of which have been exercised as of June 2005). Through calendar year 2004, USCIS paid $64.6 million to the contractor for the Tier 1 call center operations. According to the Office of Federal Procurement Policy (OFPP) in the Office of Management and Budget, performance-based service contracts are designed to focus on results.[Footnote 10] Their purpose is to ensure that contractors are given the freedom to determine how to meet the government's performance objectives, that appropriate quality levels are achieved, and that payment is made only for services that meet these levels. This type of contract is to emphasize standards for customer service and measurement of performance and may offer financial incentives, both positive and negative, to encourage quality performance. According to OFPP, call centers are suited to this type of contract because, among other things, they emphasize achieving results by meeting customer service standards. According to OFPP, with performance-based service contracts, incentive payments made to an independent contractor are to be contingent on the contractor's ability to meet the government's performance standards; the contract does not specify how those standards are to be met. Thus, the contractor retains discretion in determining how to meet performance standards specified in the contract, for example, how many CSRs to hire to ensure calls are answered within a contractually specified time. Other elements suggested for using a performance-based service contract include (1) identifying the agency's needs and addressing those needs with performance requirements that describe required service results; (2) establishing performance standards that describe the required performance level; and (3) establishing a quality assurance plan for assessing contractor performance in order to ensure that the contractor has performed in accordance with the standards. USCIS Established Performance Measures to Assess Contractor's Call Center Performance: USCIS used a multi-faceted approach to monitor and evaluate the quality of information and service provided by CSRs to customers calling contractor-operated Tier 1 call centers.[Footnote 11] This approach used seven performance measures. USCIS obtained performance data from the contractor's monitoring of selected telephone calls; customer satisfaction surveys; and a telecommunications vendor (telephone company).[Footnote 12] In addition, USCIS used an independent consulting firm to monitor CSRs' telephone calls and conduct a "mystery shopper" program assessing CSRs' responses to customers. USCIS Set Call Center Performance Measures for the Contractor to Meet: In order to monitor and evaluate the performance of the four contractor- operated Tier 1 call centers, USCIS planned to use seven performance measures. These measures were to evaluate the quality of customers' telephone interactions with CSRs; the accuracy of information provided to callers over the telephone; the accuracy of callers' information recorded by CSRs; callers' levels of satisfaction; how quickly CSRs handled calls (two measures); and the number of calls abandoned by customers put on hold. According to USCIS officials, USCIS established the performance measures based on a review of industry standards for both government and private-sector call center operations. The measures were described in a section of the contract called the Performance Requirements Summary (PRS). Under the PRS, these performance measures comprised one of three components upon which the contractor's performance score was based. The other two components were the standard, or goal, set for each measure, identifying the performance levels the contractor was expected to meet (e.g., callers will wait an average of 30 to 36 seconds before their calls are answered), and the performance calculation that USCIS would use to analyze performance data (e.g., total delay of all calls divided by the total number of calls).[Footnote 13] The PRS listing of the seven performance measures included a "sample calculation" for each of the measures, and stated that "actual calculations [are] to be determined during Contract negotiations."[Footnote 14] USCIS officials said they intended to negotiate and finalize the calculations after a 4- to 6-month phase-in period, and the contract was awarded with this provisional language. As to the performance measures and their related standards or goals, three of USCIS's performance measures are call quality monitoring, accuracy of information provided, and accuracy of capturing information. Data on these measures are to be collected by the contractor's quality assurance staff, who are to randomly monitor two calls per day for each CSR. (CSRs are not to know when they are being monitored.) The data collected are to be reported to USCIS on a monthly basis. Details on these three measures follow: * Call quality monitoring. Calls are to be monitored by the contractor's quality assurance staff to assess the CSRs' "soft skills," that is, their ability to interact with customers, establish customer rapport, maintain composure during a call, speak with clarity and professionalism, and other factors. Call quality monitoring data are to be captured on a standardized form. CSR responses for each of nine different "soft skills" are scored as percentages, with scores for the most highly valued skills, such as "active listening"--that is, whether the CSR was deemed to be attentive when listening to the customers-- given more weight than the scores for other skills. The nine scores (i.e., percentages) are combined for a total "soft skills" score, with 100 percent as the highest possible score. The performance standard stated in the PRS for this measure is that all calls monitored achieve an average score of 90 percent to 95 percent after the nine "soft skills" scores (i.e., percentages) for each call are combined. (See app. II for additional details on the criteria and methodology used to determine soft skills scores.) * Accuracy of information provided. Calls are to be monitored by the contractor's quality assurance staff to determine, among other things, whether CSRs provided accurate and complete responses. Using a standardized form, the staff score CSRs on five different efforts, such as whether the CSR used software tools appropriately and whether, when the callers were asked directly, they indicated that their needs had been satisfied. The five efforts are scored as percentages, with more weight given to the scores for certain efforts, such as "provides complete response." The scores are then combined for a total "accuracy of information provided" score, with 100 percent as the highest possible score. The performance standard stated in the PRS for this measure is that all calls monitored achieve an average score of 95 percent to 97 percent after the five accuracy scores (i.e., percentages) for each call are combined. (See app. II for additional details on the criteria and methodology used to determine accuracy of information provided.) * Accuracy of capturing information. Calls are to be monitored by the contractor's quality assurance staff to determine, among other things, whether CSRs accurately record and verify the callers' information. The staff assess this measure by scoring four efforts, including whether a referral to a local USCIS service center or local office was completed appropriately and correctly. The four efforts are scored as percentages, with more weight given to the scores for certain efforts, such as "verifies caller's information." The scores are then combined for a total "accuracy of information provided" score, with 100 percent as the highest possible score. The performance standard stated in the PRS for this measure is that all calls monitored achieve an average score of 95 percent to 97 percent after the four accuracy scores (i.e., percentages) for each call are combined. (See app. II for additional details on the criteria and methodology used to determine accuracy of capturing information.) A fourth performance measure of call quality--customer satisfaction-- was assessed by an independent consulting firm. Customer satisfaction surveys were conducted on a monthly basis to determine if customers were satisfied with the service that CSRs provided.[Footnote 15] At least 375 callers are to be randomly selected to be interviewed each month from a population of 10,000 randomly identified callers who called within the 30 days prior to the survey. To measure satisfaction with CSRs, customer responses to four interview questions about CSRs are compiled, and the overall percentage of respondents indicating satisfaction is calculated. The performance standard stated in the PRS for this measure is 80 percent to 85 percent of the customers surveyed indicating overall satisfaction with the CSRs' service. (See app. III for additional details on the criteria and methodology used to determine customer satisfaction.) Three other performance measures involve the collection of statistical data by the telecommunications vendor for determining how quickly calls are answered. The performance measures and standards in the contract for assessing how quickly CSRs answered customers' calls are as follows: * Service level. The telecommunications vendor under contract with USCIS is to collect information on the number of calls answered by CSRs in 20 seconds or less, that is, the number of callers who spoke to a CSR within 20 seconds after getting through the interactive voice response system. The performance standard stated in the PRS for this measure involves two factors: half-hour increments and the length of time it took CSRs to answer calls. The standard is that for 80 percent to 85 percent of the half-hour increments measured, 80 percent of the calls are to be answered in 20 seconds or less. * Average speed of answer. The telecommunications vendor under contract with USCIS is to collect information on the length of time it takes for CSRs to answer customers' calls after they are routed to Tier 1 by the interactive voice response system; that is, how long callers are on hold before a CSR answers their call. The performance standard stated in the PRS for this measure is that, for all calls routed to Tier 1, callers will wait an average of 30 seconds to 36 seconds. * Abandoned calls. The telecommunications vendor under contract with USCIS is to collect information on the number of calls abandoned by customers after getting through the interactive voice response system and waiting for a CSR to answer, that is, the number of times that customers hang up the telephone while waiting for a CSR. The performance standard stated in the PRS for this measure involves two factors: half-hour increments and how frequently callers abandon their calls. The standard is that for 85 percent to 95 percent of the half- hour increments measured, 1 percent to 2 percent of the calls are expected to be abandoned before a CSR answers. The contract stated that the contractor would be eligible to earn financial incentive awards if the average monthly performance met or exceeded the standards on a quarterly basis at each call center, and allowed USCIS to make deductions from payments to the contractor if the average monthly performance fell below the standards.[Footnote 16] According to the contract, the contractor is not eligible for an incentive award for a particular quarter if one of the performance standards is not met by one call center, and USCIS may make a deduction from payments to the contractor in that case. In addition, USCIS may, at its sole option, elect to include or waive financial incentives as it deems appropriate. USCIS Carried Out Additional Quality Assurance Efforts: In addition to the performance data collected by the contractor's own quality assurance staff, an independent consulting firm, and the telecommunications vendor, USCIS took two additional steps to measure call center performance for quality assurance purposes. First, to help ensure that the contractor's scoring of call-quality performance measures was reliable, USCIS used another independent consulting firm to validate the results of the contractor's efforts by monitoring two calls per month for each CSR. Data were gathered and provided to USCIS on a monthly basis. (See app. IV for additional details on the criteria and methodology used by the independent consulting firm to conduct call monitoring.) Second, in April 2003, USCIS engaged the same independent consulting firm to carry out a "mystery shopper" program to assess the completeness and accuracy of CSRs' answers to callers. Under this program, an independent consultant places random calls--1,200 each month--to Tier 1 call centers using various scripts provided by USCIS. As of April 2005, the scripts used in these calls covered 32 different scenarios, or types of calls, and 100 new scenarios were being developed. The calls are conducted in English and Spanish. (See app. V for an example of a mystery shopper scenario.) USCIS Evaluated Contractor's Performance but Suspended Use of Financial Incentives for More Than 2 Years Due to Performance Measurement Dispute: USCIS did not reach agreement with the contractor on how to apply the performance measurement requirements described in the PRS before awarding the performance-based service contract. USCIS suspended all financial incentives, positive or negative, while the parties negotiated this issue over a period of about 16 months without reaching agreement. After negotiations were abandoned, USCIS determined that, for the fourth quarter of 2004, the contractor had failed to meet four of seven performance measures and merited a payment deduction. The contractor disagreed on the grounds that the performance measurements had not been finalized and that changes in call center workloads affected the basis for applying financial incentives. In a separate matter, USCIS failed to ensure that all contractual, regulatory, and GAO standards pertaining to the documentation of the contractor's performance were fulfilled. USCIS Awarded the Call Center Contract without Finalizing Performance Measurement Requirements: The performance measurement requirements described in the PRS were not completely finalized before the contract was awarded. The language referring to "sample" calculations for determining how performance would be measured remained in the contract after it was in force. In commenting on a draft of this report, DHS said that at the time of the contract award, USCIS management believed it was appropriate to let the winning vendor have some input into the performance measurement methodology since this contract represented a transition to performance- based contracting for call center operations. The negotiations between USCIS and the contractor on this issue began in January 2003 (after a phase-in period) and continued intermittently until April 2004, when they were abandoned. While negotiations were taking place and after they were abandoned, USCIS obtained monthly data relating to the contractor's performance on the seven performance measures and compared those data to the standards.[Footnote 17] USCIS considered the measures and standards themselves to be nonnegotiable; the contractor, on the other hand, considered them as part of the "sample calculations" and, thus, negotiable. For over 2 years, USCIS did not use any of the resulting performance scores for the purpose of calculating financial incentive awards or payment deductions under the contract because the terms of the PRS remained unresolved between the parties. The contractor maintained that the performance scores were "potential scores" and were to be used by the parties in reaching an agreement on how to structure the PRS. On September 1, 2004, the contracting officer, representing USCIS, sent a letter to the contractor advising that USCIS would begin evaluating the contractor's performance and determining a financial incentive award or payment deduction for the fourth quarter of the calendar year (October 1 through December 31). USCIS officials told us they decided to take this action because they had concluded that negotiations with the contractor were unlikely to result in an agreement on the PRS. The contractor objected to USCIS's decision to carry out this evaluation. By letter dated November 29, 2004, the contractor stated that, under the terms of the contract, USCIS could not unilaterally determine the performance measurement requirements because all aspects of the requirements were negotiable, including the performance standards. The contractor further stated that an evaluation of its performance must take into account certain changes that took place to the work required under the contract. For example, the contractor stated that the number of USCIS-provided scripts, containing information for CSRs to address callers' inquiries, had grown to more than 2,300 pages from approximately 400 script pages in June 2002.[Footnote 18] According to the contractor, these changes significantly increased the average amount of time needed to handle a call and affected the contractor's ability to meet the performance standards imposed by USCIS. According to the contractor, USCIS's unilateral imposition of performance measurement requirements that did not account for the changed work requirements was inconsistent with Federal Acquisition Regulation (FAR) 16.402(g), which provides that "[i]t is essential that the Government and contractor agree explicitly on the effect that contract changes (e.g., pursuant to the Changes clause) will have on performance incentives." Nevertheless, by letter dated February 11, 2005, USCIS's contracting officer notified the contractor of the evaluation results for the period of October through December 2004. The results showed that the contractor met the standards for three of the seven performance measures and did not meet the standards for the other four measures. USCIS determined that, as a result of this performance, payments due to the contractor for services would be reduced. The letter noted that the contractor could submit its own data regarding performance during this period. Following the review of any data submitted, USCIS would take action to make the appropriate payment deduction, waive the payment deduction, or pay an appropriate incentive award. The contractor requested, by letter dated February 25, 2005, that USCIS waive implementation of the financial incentives, both positive and negative. The contractor reiterated its position that USCIS's unilateral implementation of the performance measurement requirements as currently written in the contract, without sufficient regard for substantial changes to the contract and the changing nature of the program, was not appropriate. The contractor stated that it was ready to resume negotiations on this subject so that fair and equitable financial incentives would be established. The contractor further stated that it had determined the payment deduction was incorrectly calculated by USCIS. USCIS's contracting officer responded, by letter dated April 15, 2005, that the government would not agree to waive implementation of the financial incentives and a deduction would be made from the next payment to the contractor. The letter stated that USCIS did not unilaterally create and impose the performance measurement requirements, which were included in the negotiated contract that USCIS and the contractor agreed to. Regarding the contractor's assertion that the average amount of time needed to handle calls had significantly increased, the letter noted that the performance measurement requirements would apply regardless of the average length of calls at any given time. According to FAR and OFPP guidance[Footnote 19] on performance-based service contracting, the precise method for measuring performance should have been agreed upon between USCIS and the contractor before the contract was signed and implemented. FAR § 16.401 states that performance-based service contracts should establish "reasonable and attainable [performance] targets that are clearly communicated to the contractor." According to OFPP, performance measurement techniques (i.e., how performance will be assessed to determine whether standards have been met) are essential elements of performance-based service contracting and should be clearly stated. In addition, according to OFPP, performance-based service contracts emphasize that all aspects of an acquisition be structured around the purpose of the work to be performed, that appropriate performance quality levels are achieved, that payment is made only for services that meet these levels, and that financial incentives are awarded to encourage quality performance. USCIS Plans to Solicit New Tier 1 Call Center Contracts with Changes to Improve Performance: Although the disagreement between the two parties had not been resolved, USCIS exercised its option to extend the current call center contract for another year through May 31, 2006, to allow time to solicit and award new call center contracts.[Footnote 20] The exercise of this option has no effect on the contract's performance measurement terms, which is the source of the parties' dispute. USCIS officials said they plan to award new performance-based service contracts for Tier 1 operations to two vendors, with the two vendors fully operational by June 2006, to improve the handling of customers' calls to Tier 1.[Footnote 21] USCIS officials told us they intend for the new contracts to include certain changes meant to improve Tier 1 call center operations and to incorporate OFPP guidance on performance-based contracting. USCIS officials told us that, unlike the current contract, the new PRS will clearly specify how contractor performance will be assessed and will not leave any terms open for post-award negotiation. In addition, USCIS officials said the new contracts will include independent call monitoring and the mystery shopper program as performance measurement tools to assess the quality of the Tier 1 CSRs' responses to customers, including the accuracy and reliability of the information provided. At the time of our review, USCIS officials said that the solicitation was going through DHS's contract review process and DHS had not issued the solicitation for a new contract containing these changes. DHS said in its comments on a draft of this report that the solicitation was with the DHS Procurement Office for review and issuance. USCIS and Contractor Did Not Fully Document Quality Assurance Activities: As part of its quality assurance responsibilities under the current contract, USCIS is to keep written records of observations about the contractor's performance based on periodic evaluations comparing performance data to standards in the PRS. USCIS's contracting officer's technical representative (COTR), who is responsible for administering the contract, is to use these written observations to notify the contractor if there are deficiencies--specifically, if the contractor does not meet the performance standards. The contractor is required to sign and date such observations to acknowledge that the COTR apprised it of any deficiencies. USCIS and contractor officials said they met at least quarterly (monthly, since October 2004) to discuss performance, performance data, and other items. USCIS officials said they provided the contractor with documentation containing performance and other data to discuss at these meetings. USCIS officials said some of this documentation identified performance deficiencies. However, contractor officials said they viewed the performance data as "potential scores" to be considered during negotiations. To the extent that USCIS considered the performance data as notification of deficiencies, it did not follow contractual procedures requiring the COTR to obtain the contractor's signature acknowledging notification of the deficiencies. In addition, neither USCIS nor the contractor kept minutes of these meetings. According to FAR § 46.104(c), the government should maintain, as part of the performance records of a contract, suitable records reflecting the nature of its contract quality assurance actions. With respect to any performance deficiencies, the government's records should include, among other things, the number and type of defects observed and any actions to correct deficiencies. Further, according to GAO's standards for internal control in the federal government, for an agency to run and control its operations, it must have relevant, reliable information relating to internal events.[Footnote 22] All transactions and other significant events need to be clearly documented, and the documentation should be readily available for examination. This information should be recorded and communicated to management and others within the agency who need it to carry out their responsibilities. GAO's standards provide a framework for establishing and maintaining internal control and for identifying and addressing major performance challenges. Appropriate and effective internal control is a key factor in helping agencies better achieve program results. The contract also requires the contractor to provide a quality assurance plan. The plan that was developed by the contractor describes the contractor's approach and strategy for ensuring the delivery of high-quality service. As part of the plan, the contractor is to conduct formal, biweekly internal performance review meetings to help with the identification and correction of performance deficiencies. These meetings are to be attended by contractor and USCIS officials, with contractor staff reporting on quality performance issues, and are to be in addition to the quarterly (now monthly) meetings discussed above. Minutes of the meetings are to identify action items, responsibilities, and solution time frames, and the minutes are to be published for USCIS review. However, a contractor official said that these meetings never took place. According to both contractor and USCIS officials, the quarterly meetings were used to discuss operations and performance and to focus senior management attention on any performance issues. Under FAR § 37.602-2, the government's quality assurance surveillance plans should include actions to help ensure that the contractor carries out its quality control obligations. By failing to ensure that the contractor held and documented performance review meetings as required by the contractor's quality assurance plan, USCIS did not meet its quality assurance obligations under FAR § 37.602-2 and GAO's internal control standards. In addition, USCIS's failure to obtain the contractor's written acknowledgment of USCIS-identified performance deficiencies did not meet the notification procedures established by the contract, documentation requirements of FAR § 46.104(c), and GAO's standards for internal control. In its comments on a draft of this report, DHS noted that the contract was administered by a component of the VA until April 20, 2005, and DHS said the VA was provided with documentation discussed at quarterly and monthly performance assessment meetings between USCIS and the contractor. According to DHS, the lack of a clear understanding between USCIS and the VA regarding their roles contributed to the fact that formal documentation and evaluations were not always properly maintained and formally transmitted to the contractor. DHS acknowledged that the agency procuring a service is ultimately responsible for the contract and, thus, USCIS should have clarified its and the VA's roles. USCIS Has Used Results of Monitoring Efforts to Identify Opportunities to Improve Customer Service and Call Flow at All Call Centers: USCIS used contractor performance data, including the results of surveys, call monitoring, and the mystery shopper program, to identify opportunities to improve customer service, including improving call- response times, help CSRs and IIOs better respond to customer inquiries, and manage the flow of calls into call centers. Following are examples of initiatives that USCIS recently implemented, or was planning to implement, as of April 2005. It is too early to assess the impact of these initiatives. * USCIS implemented "intelligent call routing" for Tier 1. With "intelligent call routing" in place since February 2004, telephone calls routed from the interactive voice response system to Tier 1 are now routed to the next available CSR, at any of the four Tier 1 contractor-operated call centers. Previously, telephone calls to Tier 1 were routed to the next available CSR in the call center that resided in the same geographic region of the country as the caller. USCIS officials said that by June 2005 they will also have implemented "intelligent call routing" for calls transferred from Tier 1 to the next available IIO at either of the two Tier 2 call centers. * USCIS implemented "overflow routing." USCIS started its "overflow routing" initiative in October 2004, enabling certain general call types, identified as "English Other" and "Spanish Other," to be routed directly from the interactive voice response system to Tier 2 USCIS- operated call centers, bypassing the Tier 1 contractor-operated call centers. Previously, all calls handled by IIOs at Tier 2 were first routed from the interactive voice response system to Tier 1, where CSRs then transferred the calls to Tier 2. USCIS officials said they expect the change will result in 1 to 5 percent of all calls being routed directly to Tier 2, which should help when Tier 1 CSRs cannot handle the call volume. * USCIS implemented interactive voice response system routing of certain telephone calls to USCIS service centers. USCIS changed its automated interactive voice response system in December 2004 so that certain types of customers' telephone calls--for example, certain issues concerning new permanent residents, cases already approved or denied, and pending cases--are now routed directly to USCIS service centers, bypassing CSRs at Tier 1. Previously, all customers' telephone calls that needed to be handled by USCIS service centers were routed by the interactive voice response system to Tier 1. Then, after talking with the customers, the CSRs referred the customers to the service centers (whose employees have access to case paperwork) via e-mail. CSRs were allowed to transfer customers' telephone calls to service center personnel only when customers requested emergency and expedited handling of applications. * USCIS implemented a portfolio management system. Private attorneys, paralegals, and other representatives can use the USCIS Internet Web site to check the status of their clients' immigration cases using a USCIS receipt number. Under the system, USCIS also notifies the representatives via e-mail when a case status changes; for example, when actions are taken, such as the approval or denial of an application. As of April 2005, over 300,000 customers, attorneys, and other representatives had used this system. * USCIS said it is planning to implement a referral management system. Currently, Tier 1 CSRs send, via e-mail, service request referrals to USCIS service centers and local offices for customers who call wanting to change addresses, schedule and reschedule appointments at application support centers, order forms, and resolve problems. After a referral is made, NCSC does not know whether the service center or local office responded to the customer in a timely manner or even responded at all. To better monitor this process, USCIS plans to implement a referral management system, with such service request referrals placed in a database and assigned a tracking number. The system is to (1) determine the proper service center or local office to process the referral, (2) assign the case to an adjudicator, (3) update the case on a daily basis, and (4) report once a month on case status. The referral management system is planned to be accessible to customers on USCIS's Internet Web site so they can make and track their own service request referrals. In addition, customers without Internet access are to be able to call on the telephone and CSR's will access the USCIS Web site and create referrals for them. USCIS plans that the referral management system will be fully operational during the summer of 2005. * USCIS is planning a customer service portal on USCIS's Web site. USCIS has a long-term goal of giving customers Internet access to information contained in the "scripts" used by Tier 1 CSRs to answer customers' questions. USCIS plans to establish a customer service portal on the USCIS Internet Web site, providing access to the information. The goal is to let customers with Internet access look up information themselves without having to call NCSC on the telephone, navigate the interactive voice response system, and wait for CSR's to answer. USCIS had not set a time frame for implementing this initiative. Conclusions: Immigration call centers are a vital information referral source used millions of times by immigrants and other interested parties seeking to obtain needed documents, regulatory information, up-to-date status information on immigration-related benefits and applications, and other information. To ensure that it serves its customers effectively and efficiently, USCIS appropriately used a performance-based contract, but its failure to finalize all aspects of the performance requirements before the contract was awarded hampered its ability to exercise performance incentives in the contract. As a result, USCIS lost the opportunity during the life of the contract to help ensure that it received the maximum level of service from the contractor. In addition, USCIS did not meet standards promulgated by federal acquisition regulations, GAO, and the contract itself pertaining to documenting the contractor's performance between 2002 and 2004, and adequately documenting notification of the contractor when the government perceived deficiencies in its performance. Failure to generate adequate documentation could impair USCIS's ability to conduct future contract negotiations and to preserve a complete and reliable record of contract performance needed to ensure accountability. Recommendation for Executive Action: To improve USCIS's efforts for evaluating contractor performance and encourage quality services at call centers, we recommend that the Secretary of Homeland Security require the Director of USCIS take the following two actions: (1) finalize contract terms related to specific performance measurement requirements before awarding new performance- based call center contracts; and (2) maintain readily available written records of performance assessments and performance evaluation meetings with the contractor. Agency Comments and Our Evaluation: DHS and the contractor provided formal comments and technical comments on a draft of this report, which we have incorporated, as appropriate. In its formal comments, DHS generally agreed with our recommendations. DHS said the draft solicitation for the new contracts specifically identifies performance requirements that are non-negotiable. DHS further stated that, as recommended by GAO, written records of performance assessments and performance evaluation meetings will be maintained and readily available for review by all interested parties. In its formal comments, the contractor provided additional language to further clarify this report. The contractor said the report accurately summarizes the complex nature of CIS's call center program and several challenges created by significant post-award changes to that program. DHS's and the contractor's formal comments are shown in appendixes VI and VII, respectively. As agreed, unless you publicly announce the contents of this report earlier, we plan no further distribution of it until 30 days after its issuance. At that time, we will send copies to the Director of USCIS and other interested parties. We will also make copies available to others upon request. In addition, the report will be available at no charge on GAO's Web site at http://www.gao.gov. If you or your staff have any questions regarding this report, please contact me at 202-512-8777 or jonesp@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix VIII. Paul L. Jones, Director Homeland Security and Justice Issues: List of Requesters: The Honorable John Conyers, Jr.: Ranking Minority Member: Committee on the Judiciary: House of Representatives: The Honorable Bennie G. Thompson: Ranking Minority Member: Committee on Homeland Security: House of Representatives: The Honorable Sheila Jackson Lee: Ranking Minority Member, Subcommittee on Immigration, Border Security, and Claims: Committee on the Judiciary: House of Representatives: The Honorable Loretta Sanchez: Ranking Minority Member, Subcommittee on Economic Security, Infrastructure Protection, and Cybersecurity: Committee on Homeland Security: House of Representatives: The Honorable Janice D. Schakowsky: House of Representatives: [End of section] Appendix I: Scope and Methodology: To determine what performance measures the U.S. Citizenship and Immigration Services (USCIS) established to monitor and evaluate the performance of contractor-operated call centers, we interviewed USCIS headquarters officials in Washington, D.C; Tier 1 contractor officials in Arlington, Virginia, and at a contractor-operated call center;[Footnote 23] and an official representing an independent consulting firm under contract to USCIS and located in Fairfax, Virginia. We also collected and analyzed pertinent USCIS and contractor documentation. We collected and analyzed information on the various types of monitoring and evaluation programs used by USCIS, including internal call monitoring, independent call monitoring, customer satisfaction surveys, the mystery shopper program, and telephone call data provided by a telecommunications vendor. To find out how USCIS used the performance measures to evaluate the contractor's performance, we interviewed USCIS headquarters officials in Washington, D.C., and Tier 1 contractor officials in Arlington, Virginia. We also collected and analyzed pertinent USCIS and contractor documentation. To determine what actions, if any, USCIS took or planned to take to strengthen call center operations, we interviewed USCIS headquarters officials in Washington, D.C., and Tier 1 contractor officials in Arlington, Virginia, and at a contractor-operated call center. We also collected and analyzed pertinent USCIS and contractor documentation. We assessed the reliability of telephone call volume data provided to USCIS by a telecommunications vendor, as well as USCIS and contractor staffing data. To carry out our data reliability assessments, we (1) reviewed information about the data, systems that produced the data, and data quality control procedures, and (2) interviewed USCIS and contractor officials knowledgeable about the data as necessary. We determined that the call volume and staffing data were sufficiently reliable for the purposes of this report. We conducted our work between May 2004 and May 2005 in accordance with generally accepted auditing standards. [End of section] Appendix II: Criteria and Methodology Used by Contractor to Measure Quality of Tier 1 Calls Monitored: Table 1: Call Quality Monitoring or "Soft Skills" Criteria: Establish rapport; Scoring range (0 - 3): 3; Section score (Percent): 11.5%. Criteria: Maintain composure; Scoring range (0 - 3): 3; Section score (Percent): 12.0%. Criteria: Conversational style; Scoring range (0 - 3): 3; Section score (Percent): 11.5%. Criteria: Active listening; Scoring range (0 - 3): 3; Section score (Percent): 12.5%. Criteria: Efficient call flow; Scoring range (0 - 3): 3; Section score (Percent): 12.5%. Criteria: Opening; Scoring range (0 - 3): 3; Section score (Percent): 10.0%. Criteria: Call on hold; Scoring range (0 - 3): 3; Section score (Percent): 10.0%. Criteria: Call transferred; Scoring range (0 - 3): 3; Section score (Percent): 10.0%. Criteria: Closing; Scoring range (0 - 3): 3; Section score (Percent): 10.0%. Criteria: Percent compliant; Section score (Percent): 100%. Source: USCIS contractor. [End of table] Table 2: Accuracy of Information Provided: Criteria: Uses software tools appropriately; Scoring range (0 - 3): 3; Section score (Percent): 9.0%. Criteria: Provides accurate response; Scoring range (0 - 3): 3; Section score (Percent): 28.0%. Criteria: Provides complete response; Scoring range (0 - 3): 3; Section score (Percent): 27.0%. Criteria: Provides USCIS content only; Scoring range (0 - 3): 3; Section score (Percent): 9.0%. Criteria: Satisfies caller's needs; Scoring range (0 - 3): 3; Section score (Percent): 27.0%. Criteria: Percent compliant; Section score (Percent): 100%. Source: USCIS contractor. [End of table] Table 3: Accuracy of Capturing Information: Criteria: Completes referral when appropriate; Scoring range (0 - 3): 3; Section score (Percent): 12.0%. Criteria: Completes referral record correctly; Scoring range (0 - 3): 3; Section score (Percent): 28.0%. Criteria: Captures caller's information; Scoring range (0 - 3): 3; Section score (Percent): 30.0%. Criteria: Verifies caller's information; Scoring range (0 - 3): 3; Section score (Percent): 30.0%. Criteria: Percent compliant; Section score (Percent): 100%. Source: USCIS contractor. [End of table] [End of section] Appendix III: Criteria and Methodology Used to Measure Customer Satisfaction with Customer Service Representatives: USCIS and an independent consulting firm jointly developed a telephone survey to measure customer satisfaction with the three levels of NCSC call center service--interactive voice response system, Tier 1 CSRs, and Tier 2 IIOs. To carry out the survey each month, representatives of the independent consulting firm call 375 randomly selected customers. To assess the customers' satisfaction with the CSRs, the representatives read several statements and ask questions for the customers to rate their experiences with CSRs. For the customer satisfaction performance measure required in the contract, USCIS collects and summarizes data on the customers' responses to the four statements below. The customers are asked to rate their agreement with each of the statements using a scale of 1 to 7 (1 is strongly agree and 7 is strongly disagree).[Footnote 24] 1. The representative seemed to fully understand my questions. 2. The representative was polite. 3. The representative did not rush me. 4. The representative answered my questions promptly. [End of section] Appendix IV: Criteria and Methodology Used by Independent Consulting Firm to Measure Quality of Calls Monitored: An independent consulting firm scored CSRs on 23 separate quality assurance factors as follows. Service orientation: Greeted customer; Verified customer; Established rapport; Used customer name; Expressed empathy; Maintained composure; Expressed commitment; Offered additional assistance; Terminated call appropriately. Issue identification: Obtained information; Listed actively. Issue resolution: Took responsibility; Provided thorough information; Summarized actions. Communications: Projected enthusiastic tone; Conveyed confidence; Spoke with clarity; Used appropriate language. Call management: Controlled call; Used time efficiently; Minimized "dead air"; Extended hold courtesies; Transferred call appropriately. [End of section] Appendix V: Mystery Shopper Scenario Example: [See PDF for image] [End of figure] [End of section] Appendix VI: Comments from the Department of Homeland Security: U.S. Department of Homeland Security: Washington, DC 20528: June 17, 2005: Mr. Paul L. Jones: Director, Homeland Security and Justice Issues: U.S. Government Accountability Office: 441 G Street, NW: Washington, DC 20548: Dear Mr. Jones: RE: Draft Report GAO-05-526, Immigration Services: Better Contracting Practices Needed at Call Centers (GAO Job Code 440318): The Department of Homeland Security (DHS) appreciates the opportunity to review and comrnerrt on the Government Accountability Office's draft report. As the auditors noted, United States Citizenship and Immigration Services (USCIS) call centers are a vital information referral source used millions of times by immigrants and other interested parries seeking to obtain needed documents, regulatory information, status on immigration-related benefits and applications, and other information. The Department and and USCIS appreciate that GAO recognized USCIS' use of contractor performance data, including the results of surveys, call monitoring, and the mystery shopper program to customer service. Management and staff at USCIS are proud of the success of centers to date. GAO concluded that to improve efforts for evaluating contractor performance and encourage quality services at call centers, USCIS officials should (1) finalize contract terms related to specific performance measurement requirements before awarding new performance- based call center contracts; and (2) maintain readily written records of performance assessments and performance evaluation meetings with the contractor. We generally agree with the GAO's recommendations and offer comments on each recommendation. In order to provide better context, we request that GAO mention early in the body of the report that the contract in question was awarded and administered by a component of the Veterans Administration for the Immigration and Naturalization Service (INS) which included what is now USCIS. The contract was awarded to Pearson Government Solutions (the vendor) in January 2002, fourteen months before the DHS was established This information is not found in the body of the draft report aside from a footnote. With respect to recommendation one, performance measures were, in fact, detailed in the call center contract. The final methodology by which performance against the group of stipulated critieria would be weighed remained to be finalized. At the time of the award, INS management belived it appropriate to let the winning vendor have some input into the mechanics of the methodology since this contract represented a stransition to performance based contracting for call center operations. However, this approach resulted in subsequent discussions with the vendor about performance that delayed application of the incentive and disincentive clauses. [See PDF for remainder of comment letter] [End of section] Appendix VII: Comments from Pearson Government Solutions: PEARSON GOVERNMENT SERVICES: June 15, 2005: Mr. Paul L. Jones: Director: Homeland Security and Justice: U.S. Government Accountability Office: 441 G Street NW: Washington, DC 20548: JOHN MCNAMARA CURTIS: PRESIDENT AND CHIEF EXECUTIVE OFFICER: PEARSON GOVERNMENT SOLUTIONS: 4250 N. FAIRFAX DRIVE: SUITE 1200: ARLINGTON,VA 22203: TEL (703) 284-5602: FAX (703) 284-5662: mac.curtis@pearson.com: www.pearsongov.com: Dear Mr. Jones: We appreciate the opportunity to comment on the GAO draft report (GAO- 05-526) regarding the USCIS management of call center contracts. Overall, the report accurately summarizes the complex nature of CIS' call center program, and several challenges created by significant post- award changes to that program. However, we believe the report would benefit from additional clarification on the following points: Contract Changes Resulted in Challenging Negotiations: The draft report explains that during a 16 month period following contract award, CIS and Pearson Government Solutions negotiated on how performance metrics would be scored. Pearson's objective during these negotiations was to define performance metrics that were reasonable and attainable given the significant and ongoing changes made to the contract following the award. In the end, although the negotiations were substantive they did not produce an agreement. CIS Metrics Do Not Reflect Reflect Changes: CIS unilaterally decided to apply performance measurements to the contract that were unacceptable to Pearson. We objected for two reasons: the contract requires mutually agreed upon performance measurements; and CIS' performance measurements did not take into account changes to the work required by the contract. For example, as the report states, the number of scripts CSRs are required to use in responding to callers increased from 400 to over 2,300. However, CIS's performance measurements did not reflect this or other changes. As a result CIS determined that PGS did not meet 4 of the 7 metrics for the fourth quarter of 2004, triggering a payment reduction. Although we object to this determination, we stand ready to work with CIS to establish fair and equitable financial incentives. Pearson Government Solutions manages citizen interaction programs for governments worldwide. We understand what it takes for contact center programs to be successful even with unexpected dynamic changes. As with all contracts, the contractor and government must work together to identify resources needed to respond to changes, agree on expected outcomes, and ensure that performance measurements and contract modifications reflect unexpected changes. The CIS program has had challenges in this regard. However, despite these challenges, we have worked closely with CIS to provide high-quality service and will continue to do so. In closing, I want to commend your team's efforts to thoroughly assess and accurately summarize these complex issues. We appreciate the opportunity to present these comments on the draft report. Please do not hesitate to contact me or my staff if you have any questions. Sincerely, Signed by: John McNamara Curtis: President and Chief Executive Officer: Pearson Government Solutions: [End of section] Appendix VIII: GAO Contact and Staff Acknowledgments: GAO Contacts: Paul L. Jones 202-512-8777: Staff Acknowledgments: In addition to the above, Darryl W. Dutton, Ronald G. Viereck, Brian J. Lipman, Christine F. Davis, Amy L. Bernstein, and Michele C. Fejfar made key contributions to this report. FOOTNOTES [1] An automated voice system with a touch-tone menu. [2] The contractor considers the locations of its call centers to be proprietary information. [3] Pub. L. No. 107-296, § 451 (Nov. 25, 2002). [4] Throughout this report, we use USCIS to refer to both the new USCIS and the former Immigration Services Division of INS. [5] If a customer attempts to obtain live assistance outside normal business hours, the customer is told when to call back to obtain the assistance. [6] In some cases, customers' calls are routed directly from the interactive voice response system to Tier 2, bypassing Tier 1. [7] The contractor considers the precise number of CSRs at its call centers and their locations to be proprietary information. [8] The term "contract" in this report refers specifically to "Delivery Order No. 591," which was awarded to the contractor following a competitive procedure through the General Services Administration's Federal Supply Schedule. [9] The contract was awarded before DHS was established while what is now USCIS was part of INS. [10] OFPP and the National Performance Review coordinated a federal interagency working group to develop generic performance-based concepts for government agencies to use in the procurement of call center services. The group developed a reference source to help program and acquisition officials draft performance requirements and standards related to call center performance-based contracts: Performance Based Concepts for Telephone Call Center Contracting, Office of Federal Procurement Policy (May 27, 1997). [11] USCIS monitors and evaluates operations at both Tier 1 and Tier 2 call centers. This report focuses on USCIS's evaluation of Tier 1 contractor-operated call centers. [12] This telecommunications vendor is also responsible for providing the interactive voice response system and routing customers' telephone calls from this system to Tier 1 and Tier 2 call centers. [13] In this report, we collectively refer to these three components as "performance measurement requirements." [14] The "sample" terms of the contract were never finalized, leading to a dispute between USCIS and the contractor that remains unresolved (see p. 15). Examples of "sample calculations" listed for different measures in the PRS were "total delay of all calls divided by total number of calls" for determining the average length of time it takes before calls are answered, and "total number of points achieved divided by total number of points available for all quality monitoring scores" for determining call quality. One sample calculation was listed for each measure. [15] The customer satisfaction surveys ask about customer satisfaction with the interactive voice response system and IIOs, as well as CSRs. [16] Up to 10 percent of payments can be added to or deducted from the contractor's billed amounts, depending upon whether or not the performance standards are met. [17] USCIS started collecting data on the contractor's performance in June 2002, the first month of the contractor's operations. We are not presenting the performance data used by USCIS because the performance measurement terms of the PRS were unresolved between USCIS and the contractor as of April 2005. [18] USCIS revised the scripts to provide new and updated immigration- related information on a variety of topics. In its technical comments on a draft of this report, DHS said that the contractor had been compensated for the increased script content and average call duration. [19] Performance Based Concepts for Telephone Call Center Contracting, Office of Federal Procurement Policy (May 27, 1997). [20] USCIS officials said that DHS will conduct the acquisition and USCIS will administer the contract, without seeking inter-agency assistance from the VA as it previously did with the current call center contract. [21] The current contractor may compete for the new contracts. [22] GAO, Standards for Internal Control in the Federal Government, GAO/AIMD-00-21.3.1 (Washington, D.C.: November 1999); Internal Control Management and Evaluation Tool, GAO-01-1008G (Washington, D.C.: August 2001). [23] The contractor considers the locations of its call centers to be proprietary information. [24] The complete scale for the four statements is as follows: 1- Strongly agree; 2-Mostly agree; 3-Somewhat agree, 4-Mixed, Neither agree nor disagree; 5-Somewhat disagree; 6-Mostly disagree; and 7- Strongly disagree. GAO's Mission: The Government Accountability Office, the investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through the Internet. GAO's Web site ( www.gao.gov ) contains abstracts and full-text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents using key words and phrases. You can print these documents in their entirety, including charts and other graphics. Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as "Today's Reports," on its Web site daily. The list contains links to the full-text document files. To have GAO e-mail this list to you every afternoon, go to www.gao.gov and select "Subscribe to e-mail alerts" under the "Order GAO Products" heading. Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. Government Accountability Office 441 G Street NW, Room LM Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000: TDD: (202) 512-2537: Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov Automated answering system: (800) 424-5454 or (202) 512-7470: Public Affairs: Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, D.C. 20548: