This is the accessible text file for GAO report number GAO-03-1068 entitled 'Contract Management: No Reliable Data to Measure Benefits of the Simplified Acquisition Test Program' which was released on September 30, 2003. This text file was formatted by the U.S. General Accounting Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Report to Congressional Committees: United States General Accounting Office: GAO: September 2003: CONTRACT MANAGEMENT: No Reliable Data to Measure Benefits of the Simplified Acquisition Test Program: Simplified Acquisition Test Procedures: GAO-03-1068: GAO Highlights: Highlights of GAO-03-1068, a report to the Senate and House Committees on Armed Services Why GAO Did This Study: In recent years, the federal government has introduced new ways to streamline the acquisition process. One of those vehicles is the simplified acquisition procedures test program, which removes some of the procedural requirements for buying commercial goods and services. Using the test program, federal procurement officials can make purchases faster than they have in the past for procurements not exceeding $5 million. Congress mandated that GAO determine the extent to which federal executive agencies—at a minimum, the Department of Defense (DOD)—have taken advantage of the test program and any benefits realized. One way to measure use is to examine test program data from the Federal Procurement Data System (FPDS). It is the central repository of contracting information. In addition to examining FPDS data, GAO looked at data from DOD’s data system. What GAO Found: Because the Federal Procurement Data System contains unreliable data about the simplified acquisition test program, GAO was unable to determine the extent to which federal executive agencies—including DOD— have used the test program and have realized any benefits. Specifically, the database indicated that the Departments of Treasury, Defense, and Justice were the three largest dollar-value users of the test program in fiscal year 2001 (the latest year with complete data available). But GAO found that FPDS either overstated or understated use of the test program by millions of dollars. The table below shows examples of discrepancies at different buying organizations within these three departments. Examples of Discrepancies with FPDS’s Data [See PDF for image] [End of table] GAO also found data reliability problems with contract data in DOD’s own data system—the Defense Contract Action Data System (DCADS)—which feeds into FPDS on an ongoing basis. For example, for fiscal year 2002, DCADS showed about $146 million in test program contract actions for two buying organizations for the Naval Air Systems Command and the Defense Intelligence Agency. After reviewing contract actions that had the highest dollar value, procurement officials at these two DOD buying organizations said that none of the entries were awarded through the test program. There were also reliability problems at other buying commands. The federal buying organizations we visited have not collected any other data that would allow us to assess whether the test program is helping to increase efficiency, improve contract prices, reduce administrative costs, or improve the delivery of goods and services. Anecdotal evidence indicates that the test program is getting favorable reviews. For example, nearly all procurement officials with whom GAO spoke at the buying organizations GAO visited indicated that the program’s primary benefit is the ability to process a contract more efficiently. What GAO Recommends: Before Congress decides whether to make the test program a permanent contracting vehicle, GAO recommends that DOD and other selected federal executive agencies ensure that reliable data are available to make program assessments. DOD agreed with GAO’s recommendation, while the other selected federal agencies had no comments on the recommendation. www.gao.gov/cgi-bin/getrpt?GAO-03-1068. To view the full product, including the scope and methodology, click on the link above. For more information, contact David Cooper at (202) 512-4125 or cooperd@gao.gov. [End of section] Contents: Letter: Results in Brief: Background: No Reliable Data Available to Assess Test Program: Conclusion: Recommendations for Executive Action: Agency Comments: Appendix I: Scope and Methodology: Appendix II: Comments from Department of Justice's Federal Bureau of Prisons: Table: Table 1: Examples of Discrepancies with FPDS's Data: Abbreviations: DCADS: Defense Contract Action Data System: DOD: Department of Defense: FAR: Federal Acquisition Regulation: FPDS: Federal Procurement Data System: GAO: General Accounting Office: OFPP: Office of Federal Procurement Policy: United States General Accounting Office: Washington, DC 20548: September 30, 2003: The Honorable John W. Warner Chairman The Honorable Carl Levin Ranking Minority Member Committee on Armed Services United States Senate: The Honorable Duncan Hunter Chairman The Honorable Ike Skelton Ranking Minority Member Committee on Armed Services House of Representatives: Acquisition reform has fundamentally changed the way the federal government procures billions of dollars worth of goods and services each year. The procurement process is more streamlined than ever before. Government buyers can make their purchases with less turnaround time, they have less paperwork, and they can rely on a variety of tools to help them expedite the process. One tool is the simplified acquisition procedures test program, which reduces the procedural requirements for buying commercial goods and services not exceeding $5 million. Congress mandated that we report on the test program and address the (1) extent to which federal executive agencies--at a minimum, the Department of Defense (DOD)--have used the test program, (2) benefits realized through its use, and (3) impact that the program has had on contract competition.[Footnote 1] To satisfy these objectives, we obtained and analyzed test program data from the Federal Procurement Data System (FPDS) and DOD's Defense Contract Action Data System (DCADS). We used these data to identify buying organizations that were among the largest total-dollar-value users of the test program and to review contract files at selected buying organizations to determine the accuracy of the data reported. We interviewed procurement officials to determine whether use of the test program has resulted in noticeable benefits or affected competition. Results in Brief: Because the FPDS and DCADS databases contain unreliable test program data, we were unable to determine the extent to which federal executive agencies, including DOD, have taken advantage of the program. Neither were we able to determine the test program's benefits and impact on competition. The federal buying organizations we visited have not collected any other data that would allow us to assess whether the test program is helping to increase efficiency, improve contract prices, reduce administrative costs, or improve the delivery of goods and services. Two years ago, we reported that data were not being collected to provide a basis for measuring whether the test program produced the desired results of maximizing efficiency and economy and minimizing administrative burden and cost. We recommended that data be collected to demonstrate the benefits of the test program. In response to our recommendation, DOD said it planned to convene an integrated process team to determine ways to measure the benefits of the test program. However, DOD has not taken action to measure the program's benefits. This report recommends that DOD and the Office of Federal Procurement Policy (OFPP) develop evaluation mechanisms for measuring test program benefits. It also recommends that the Departments of Treasury, Justice, and Defense improve the reliability of test program data to make program assessments. DOD partially concurred with the first recommendation and agreed with the second. The other two federal executive agencies had no comments on our recommendations. Background: To streamline the federal procurement process, Congress in 1994 authorized the use of simplified acquisition procedures for purchases not exceeding $100,000.[Footnote 2] Simplified procedures allow agency officials to expedite the evaluation and selection processes and keep documentation to a minimum. In 1996, Congress expanded the use of simplified acquisition procedures[Footnote 3] by authorizing a test program that allows government buyers to procure commercial items not exceeding $5 million[Footnote 4] in order to maximize efficiency and economy and minimize burden and administrative costs for both the government and industry.[Footnote 5] For example, government buyers: * may issue a combined synopsis and solicitation and may require proposal submission in fewer than 45 days, as would otherwise be required; * need not establish a formal evaluation plan or competitive range, conduct discussions with vendors, or score quotations or offers; and: * can minimize the documentation required to justify contract award decisions. Under simplified acquisition procedures, the Federal Acquisition Regulation (FAR) requires competition to the maximum extent practicable.[Footnote 6] The test program, which expires on January 1, 2004, is only one of a number of streamlined contracting vehicles that federal agencies use to procure goods and services. Other options include purchase cards, multiple award Federal Supply Schedule contracts, governmentwide acquisition contracts, and multiple award task and delivery order contracts. In one way or another, these options allow government buyers to simplify and expedite the procurement process. In 2001, to find out whether the test program was achieving desired results, we evaluated DOD's use of the program for commercial purchases.[Footnote 7] We reported that data had not been collected to provide a basis for measuring whether the test program was maximizing efficiency and economy and minimizing administrative burden and cost. However, our report summarized a 1999 OFPP survey showing that procurement executives believed the program improved the federal procurement process and that the test program should be made permanent. We asked Congress to consider requiring the OFPP to develop a method for demonstrating that the test program was producing desired results. FPDS is the central repository of federal contracting information, and it contains detailed data on contract actions exceeding $25,000. Although federal agencies collect contract data using their own data systems, their data must be transmitted to and consolidated in FPDS on an ongoing basis. FPDS can assist procurement managers in making such decisions as understanding the consequences of their purchasing decisions, projecting future needs, or leveraging overall buying power. FPDS was designed to provide basic contracting information, such as whether the simplified acquisition test program was used. FPDS also was designed to provide insight on small business participation and competition, among other things. Federal officials can use the data to perform oversight responsibilities. The General Services Administration, through the Federal Procurement Data Center, operates and maintains FPDS. DOD accumulates similar data on contract actions of over $25,000 in the DCADS database, and, like other federal agencies, transmits contract information to FPDS. No Reliable Data Available to Assess Test Program: We found significant data-reporting errors related to the test program in both FPDS's and DCADS's databases. Because of unreliable data, we were unable to determine the extent to which federal executive agencies have used the simplified acquisition test program and what benefits they may have realized from its use. We also could not determine the impact that the test program has had on contract competition. To verify FPDS's data, we visited the Departments of Treasury, Defense, and Justice--the three largest dollar-value users of the simplified acquisition test program in fiscal year 2001, as reported in FPDS. FPDS's data showed that the Departments of Treasury, Defense, and Justice had test program contract actions worth about $303 million, $209 million, and $157 million, respectively, in fiscal year 2001, the latest year with the most complete data available. We found that these figures were either overstated or understated by millions of dollars. For example, as shown in table 1, FPDS's data showed that the Department of the Treasury's U.S. Mint had about $242 million in test program contract actions, making it the largest user of the test program in fiscal year 2001. However, the U.S. Mint officials told us that they did not use the program at all. We also found reporting errors with FPDS's data for DOD. An official at one of DOD's buying commands--the Defense Logistics Agency--said it had about $146 million in test program contract actions, but FPDS's data showed only $4 million. In the case of the Department of Justice, FPDS's data showed that the Federal Prison Industries, which is a part of the Federal Bureau of Prisons, had about $118 million in test program contract actions. Federal Prison Industries officials told us they used the test program extensively, but when we asked them to review a portion of FPDS's data, about $31 million worth of contract actions, they said that none of the contract actions listed were part of the test program. Table 1: Examples of Discrepancies with FPDS's Data: Department's buying organization: Department of the Treasury's U.S. Mint; Value of test program contracts, according to FPDS (fiscal year 2001): $242 million; What procurement officials said about FPDS's data: U.S. Mint said it did not use the test program at all. Department's buying organization: DOD's Defense Logistics Agency; Value of test program contracts, according to FPDS (fiscal year 2001): $4 million; What procurement officials said about FPDS's data: Defense Logistics Agency said it obligated $146 million in test program contracts. Department's buying organization: Department of Justice's Federal Prison Industries; Value of test program contracts, according to FPDS (fiscal year 2001): $118 million; What procurement officials said about FPDS's data: After reviewing portions of FPDS's data, about $31 million in contract actions, Federal Prison Industries said none of those items were purchased under the test program. Sources: FPDS (data); GAO (analysis). [End of table] As with other federal agencies, DOD has its own database system to collect contract data, and it transmits those data to FPDS on an ongoing basis. We decided to take a closer look at DOD's database-- DCADS--for fiscal year 2002, the latest year with the most complete data. As we found with FPDS, there were reporting errors in DCADS. According to DCADS, DOD had a total of $1.9 billion in test program contract actions. For verification, we reviewed selected test program contract actions for DOD's buying organizations that were major dollar- value users of the test program, according to DCADS. While we did find that one Air Force buying organization correctly reported its test program contract actions, other buying commands reported them incorrectly in DCADS. For example, DCADS's data showed that an organization within the Naval Air Systems Command had about $122 million in test program contract actions and that the Defense Intelligence Agency's Virginia Contracting Activity had $24 million. We asked procurement officials at DOD's buying organizations to review DCADS's data for contract actions that had the highest dollar value. They said that none of those listed were awarded under the test program. Three other DOD buying organizations also had reporting errors. In addition to the data reliability problems we found with FPDS's and DCADS's test program data, we also found that federal buying organizations had not collected any other data to document whether the test program is helping to increase efficiency, improve contract prices, reduce administrative costs, or improve the delivery of goods and services. However, indications are that the test program is well received. Nearly all procurement officials with whom we spoke at selected buying organizations view the test program favorably. They cite as the program's primary benefit the ability to process a contract more efficiently, and the majority advocates making the test authority permanent. In commenting on our 2001 report, DOD stated its intention to convene an integrated process team to consider ways for measuring the benefits of the test program. However, DOD has not acted on this initiative. In discussing the results of this effort, DOD officials stated that they are willing to assess the benefits of the test program. Justice and Treasury Department officials stated that they would work to improve the reliability of test program's data. Conclusion: The simplified acquisition test program, which streamlines the process for buying commercial items that do not exceed $5 million, expires on January 1, 2004. Most procurement officials with whom we spoke at selected buying organizations said that they would like the test program to be made permanent and that there are benefits associated with buying commercial items using simplified procedures. However, anecdotal information is not enough to determine this program's overall impact and benefits. Inherent in any test program is the expectation that federal agencies establish evaluation mechanisms for assessing program results, which includes ensuring that reliable data are collected and used for the assessments. Our observations are that there is no reliable information for measuring the test program's benefits. Recommendations for Executive Action: Before Congress provides permanent authority for using simplified procedures to acquire commercial items costing up to $5 million, we recommend that DOD work with the Administrator of OFPP to develop evaluation mechanisms for measuring test program benefits. In addition, the Departments of Treasury, Justice, and Defense should independently take appropriate actions to ensure that reliable FPDS test program data are available to make program assessments. Agency Comments: We asked OFPP and the Departments of Treasury, Justice, and Defense for written comments on the draft report. OMB, which oversees OFPP, and DOD provided oral comments. OMB did not comment on the specifics of the report but made a general observation that the test program provides a benefit if used correctly. It said the test program enables agencies to gain timely access to the marketplace while still reaping the benefits of open market competition. OMB further noted that the test program may be especially beneficial as an alternative to the Federal Supply Schedule and multiple or single award task and delivery order contracts, when prenegotiated terms and conditions of these vehicles are not suitable to meet an agency's needs. DOD partially concurred with our recommendation that it work with OFPP to develop evaluation mechanisms for measuring test program benefits. DOD stated that it would develop a methodology to evaluate the benefits of the test program, on the basis of such metrics as procurement lead- time, and share the results with OFPP. Using this methodology, the benefits of the test program would be measured by sampling test program contracts from the contract reporting system. DOD concurred with our recommendation that it take appropriate actions to ensure that reliable FPDS test program data are available to make program assessments. DOD is planning to issue a memorandum to the military departments and defense agencies emphasizing the need for all test program data to be entered into the contract-reporting system accurately, so reliable data are available to demonstrate the continuing need for this program. The Department of Justice's Federal Bureau of Prisons concurred with the information reflected in the report; the bureau's comments appear in appendix II. The Department of the Treasury's U.S. Mint agreed with our finding that it did not use the test program. We are sending copies of this report to other interested congressional committees and the Secretary of Defense; Secretary of the Air Force; Secretary of the Army; Secretary of the Navy; Director, Defense Logistics Agency; Director, Office of Management and Budget; Administrator, Office of Federal Procurement Policy; Administrator of General Services; Assistant Attorney General for Administration, Department of Justice; Secretary of the Treasury; and Director, U.S. Mint. We will also make copies available to others on request. In addition, this report will be available at no cost on the GAO Web site at http://www.gao.gov. If you have questions about this report, please contact me at (617) 788-0500 or Ralph Dawn at (202) 512-4544. Key contributors to this assignment were Jeffrey Rose, Ralph Roffo, Marie Ahearn, Lily Chin, and Julia Kennon. Sincerely yours, David E. Cooper Director, Acquisition and Sourcing Management: Signed by David E. Cooper: [End of section] Appendix I: Scope and Methodology: Our objectives were to determine the (1) extent to which federal executive agencies used the simplified acquisition test authority, (2) benefits they realized through its use, and (3) impact that the program has had on contract competition. To satisfy these objectives, we obtained and analyzed test program data from the Federal Procurement Data System (FPDS) and the Department of Defense's (DOD) Defense Contract Action Data System (DCADS). Using the data, we identified buying organizations that were among the largest dollar-value users of the test program, and we reviewed contract files at selected buying organizations to determine the accuracy of the data reported. We interviewed procurement officials from the Office of Federal Procurement Policy and from the Departments of Treasury, Defense, and Justice to obtain their views of the benefits from using the test authority. In addition, we reviewed federal regulations and available test program guidance. We initially identified, using FPDS's data, the major dollar-value users of the test program during fiscal year 2001, the year of the most recent and complete data we had available at the time of our review. According to the data, the Departments of Treasury, Defense, and Justice were the largest dollar-value users of the test program. We then used the data to identify the buying organizations within each of these departments that were the largest dollar-value users. We met with procurement officials at selected buying organizations to verify the reliability of FPDS's test program data and to discuss the benefits realized. We also reviewed the DCADS database to respond to our congressional mandate's minimum requirements. Using DCADS's complete fiscal year 2002 data, we selected the buying organizations within DOD commands that were major dollar-value users of the test authority. The selected organizations were the (1) Air Force Air Mobility Command, (2) Air Force 21st Contracting Squadron, (3) Army Communications-Electronics Command, (4) Naval Air Systems Command's Naval Air Warfare Center, (5) Defense Intelligence Agency's Virginia Contracting Activity and (6) Defense Logistics Agency's Defense Supply Center Philadelphia. We discussed with procurement officials at each of these buying organizations the reliability of DCADS's test program data and the program benefits realized. We conducted our work from March through August 2003 in accordance with generally accepted government auditing standards. [End of section] Appendix II: Comments from Department of Justice's Federal Bureau of Prisons: U.S. Department of Justice Federal Bureau of Prisons: Washington, DC 20534: September 9, 2003: Ralph Dawn, Assistant Director Acquisition and Sourcing Management General Accounting Office Washington, DC 20548: Dear Mr. Dawn: The Bureau of Prisons appreciates the opportunity to formally respond to the General Accounting Office's draft report entitled Contract Management: No Reliable Data to Measure: Benefits of the Simplified Acquisition Test Program. We have completed our review and concur with the information reflected in the report. The report does not contain any recommendations for the BOP. Thank you for the opportunity to review the draft copy of this report. If you have any questions regarding this response, please contact Jessie Hobbs, External Liaison, at (202) 305-7401. Sincerely, Michael W. Garrett: Senior Deputy Assistant Director: Signed by Michael W. Garrett: FOOTNOTES [1] Our reporting mandate is found in the Bob Stump National Defense Authorization Act for Fiscal Year 2003, Pub. L. No. 107-314, sec. 812, Dec. 2, 2002. [2] Federal Acquisition Streamlining Act of 1994, Pub. L. No. 103-355, Oct. 13, 1994. [3] Clinger-Cohen Act of 1996, Pub. L. No. 104-106, Div. D, Feb. 10, 1996 (short title changed from Federal Acquisition Reform Act of 1996 to Clinger-Cohen Act of 1996, Pub. L. No. 104-208, sec. 808, Sept. 30, 1996). [4] Federal Acquisition Regulation, Subpart 13.5. [5] Since then, there has been another effort to simplify procurement in special situations. The Homeland Security Act of 2002 temporarily expands the use of simplified acquisition procedures (Pub. L. No. 107- 296, sec. 855(b), Nov. 25, 2002). The act authorizes executive agencies to use simplified procedures in any procurement of property or services acquired as part of the fight against terrorism and related threats. [6] FAR sec. 13.104. [7] U.S. General Accounting Office, Contract Management: Benefits of Simplified Acquisition Test Procedures Not Clearly Demonstrated, GAO-01-517 (Washington, D.C.: Apr. 20, 2001). GAO's Mission: The General Accounting Office, the investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through the Internet. GAO's Web site ( www.gao.gov ) contains abstracts and full-text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents using key words and phrases. You can print these documents in their entirety, including charts and other graphics. Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as "Today's Reports," on its Web site daily. The list contains links to the full-text document files. To have GAO e-mail this list to you every afternoon, go to www.gao.gov and select "Subscribe to e-mail alerts" under the "Order GAO Products" heading. Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. General Accounting Office 441 G Street NW, Room LM Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000: TDD: (202) 512-2537: Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov Automated answering system: (800) 424-5454 or (202) 512-7470: Public Affairs: Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S. General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C. 20548: