This is the accessible text file for GAO report number GAO-14-54 entitled 'VA Health Care: Additional Guidance, Training, and Oversight Needed to Improve Clinical Contract Monitoring' which was released on October 31, 2013. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States Government Accountability Office: GAO: Report to Congressional Requesters: October 2013: VA Health Care: Additional Guidance, Training, and Oversight Needed to Improve Clinical Contract Monitoring: GAO-14-54: GAO Highlights: Highlights of GAO-14-54, a report to congressional requesters. Why GAO Did This Study: VA must frequently contract with non-VA health care providers so that clinical providers are available to meet veterans’ health care needs. While recent studies have disclosed problems with VA’s development of contracts for clinical services, there has been little scrutiny of how VA monitors and evaluates the care contract providers give to veterans. GAO was asked to review VA’s efforts to monitor clinical contractors working in VA facilities. This report examines the extent to which VA establishes complete performance requirements for contract providers, challenges VA staff encounter in monitoring contract providers’ performance, and the extent to which VA oversees VAMC staff responsible for monitoring contract providers. GAO reviewed VA acquisition regulations and other guidance. In addition, GAO visited four VAMCs that varied in geographic location and selected a nongeneralizable sample of three types of clinical contracts from each of the four VAMCs to review. GAO discussed how VAMC and VISN staff monitor and oversee these contracts and reviewed contract monitoring documentation. What GAO Found: All 12 contracts GAO reviewed from the four Department of Veterans Affairs’ (VA) medical centers (VAMC) visited contained performance requirements consistent with VA acquisition policy. However, the performance requirements lacked detail in six categories: type of provider or care; credentialing and privileging; clinical practice standards; medical record documentation; business processes; and access to care. GAO identified these categories from reviews of VA acquisition regulations, VA policies, and hospital accreditation standards; and VA officials verified that these six categories were an accurate reflection of performance requirements that should be in VA clinical contracts. GAO found, for example, one VAMC cardiothoracic contract that had detailed performance requirements while another VAMC’ s cardiothoracic contract did not contain a statement describing the contract provider’s responsibilities for reporting and responding to adverse events and patient complaints. GAO also found that contracting officials lack tools, such as standard templates, that provide examples of the performance requirements that should be included in common types of clinical contracts. Such tools would help ensure consistency in requirements across contracts. Contracting officer’s representatives (COR) cited two main challenges in monitoring contract providers’ performance-—too little time to monitor clinical contractors’ performance effectively and inadequate training. Most of the 40 CORs at the four VAMCs in GAO’s review said that their clinical contract monitoring duties were a collateral duty and that they had other primary responsibilities, such as serving as a business manager or administrative officer for a specialty clinic within the VAMC. GAO found that, on average, each of these 40 CORs spent about 25 percent of their time monitoring an average of 12 contracts. CORs said the demands of their primary positions at times prevented them from fully monitoring contract providers’ performance. Further, VA’s current guidance related to COR responsibilities does not include any information on how VAMCs are to determine the feasibility of whether a COR’s workload—-including both COR and primary position responsibilities-—will allow them to carry out their tasks as CORs for monitoring contract provider performance. GAO also found that current VA COR training programs focus on contracts that buy goods, not clinical services, and include little information on monitoring responsibilities. CORs questioned the usefulness of the COR training VA uses to prepare them for monitoring clinical contracts. VA Central Office conducts limited oversight of COR and contract monitoring activities. VA Central Office reviews of COR clinical contractor monitoring activities are limited to a small number of annual file reviews that focus on verifying the presence of required documentation only and do not assess the quality of CORs’ monitoring activities. Since implementing the program in March 2013 these reviews have been conducted in 4 of 21 network contracting offices and as of August 2013 none of the 4 offices has received feedback on these reviews. Without a robust monitoring system, VA cannot ensure that all CORs in its VAMCs are properly monitoring, evaluating, and documenting the performance of contract providers caring for veterans. What GAO Recommends: GAO recommends that VA develop and disseminate standard templates that provide examples of performance requirements for clinical contracts, revise guidance for CORs to include workload information, modify COR training, and improve the monitoring and oversight of clinical contracts. VA concurred with GAO’s recommendations. View [hyperlink, http://www.gao.gov/products/GAO-14-54]. For more information, contact Randall Williamson at (202) 512-7114 or williamsonr@gao.gov. [End of section] Contents: Letter: Background: Reviewed Contracts Included Performance Requirements, but Lacked Complete Detail Needed to Hold Contractors Accountable in Some Cases: Selected VA Clinical Contracts Did Not Always Include Clear and Measurable Performance Standards: CORs Reported that Heavy Workloads and Inadequate Training Make It Difficult to Effectively Monitor Contract Providers' Performance: VA Central Office Oversight of VAMC Clinical Contract Monitoring is Limited: Conclusions: Recommendations for Executive Action: Agency Comments: Appendix I: Objectives, Scope and Methodology: Appendix II: Comments from the Department of Veterans Affairs: Appendix III: GAO Contact and Staff Acknowledgments: Tables: Table 1: Level of Detail in Performance Requirements for 12 Selected Contracts at Four VA Medical Centers (VAMC), by Contract Type and Performance Requirement Category: Table 2: Extent to Which Selected Contracts from Four VA Medical Centers (VAMC) Included Clearly Stated Performance Standards, by Contract Type and Performance Requirement Category: Table 3: Extent to Which Selected Contracts from Four VA Medical Centers (VAMC) Included Measurable Targets for Performance Standards, by Contract Type and Performance Requirement Category: Table 4: VA Requirements for Contracting Officer's Representative (COR) Monitoring of Clinical Contractor Performance: Table 5: A Sample of Position Descriptions of Contracting Officer's Representatives' (COR) Primary Positions at the Four VA Medical Centers (VAMC) We Visited: Table 6: Contract Types Reviewed at Each Selected VA Medical Center (VAMC): Table 7: Performance Requirement Categories and Required Components: Figures: Figure 1: Organization of the Veterans Health Administration (VHA) Procurement and Logistics Office: Figure 2: Performance Requirement Categories for VA Clinical Contracts: Abbreviations: CBOC: community-based outpatient clinic: CO: contracting officer: COR: contracting officer's representative: FAC: Federal Acquisition Certification: FAR: Federal Acquisition Regulation: OIG: Office of the Inspector General: SAO: Service Area Office: VA: Department of Veterans Affairs: VAAR: VA Acquisition Regulation: VAMC: VA medical center: VHA: Veterans Health Administration: VISN: Veterans Integrated Service Network: [End of section] GAO: United States Government Accountability Office: 441 G St. N.W. Washington, DC 20548: October 31, 2013: Congressional Requesters: The Department of Veterans Affairs (VA) employs the majority of the clinical providers that care for veterans in VA-operated facilities, such as VA medical centers (VAMC) and community-based outpatient clinics (CBOC).[Footnote 1] However, in order to meet the needs of the veterans it serves and ensure that clinical providers are available to treat veterans, VA frequently contracts with non-VA health care providers to provide services in VA facilities. According to VA, every VAMC has at least one contract in place with a non-VA health care provider to help supplement the number of providers employed by VA facilities. These non-VA health care providers either provide services that the VAMC or CBOC do not currently offer or supplement the capacity of the VA facility by providing additional clinicians to treat veterans. Previous studies have highlighted challenges VA has faced developing and administering its clinical contracts. In recent years, for example, the VA Office of the Inspector General (OIG) highlighted challenges VA faces in developing its clinical contracts and found systemic weaknesses in the process VA uses to award contracts. [Footnote 2] These weaknesses were attributed to VA's decentralized oversight of the initial stages of the contracting process before a contract is awarded to a contractor. However, there have been no comprehensive reviews of VA's efforts to monitor clinical contractors once a contract is awarded and contract providers begin caring for veterans in VA facilities. You asked us to review VA's efforts to monitor clinical contractors working in VA facilities. In this report we examine: (1) the extent to which VA establishes complete performance requirements for clinical contractors; (2) the extent to which VA clinical contracts include clear and measurable performance standards for assessing whether or not clinical contractors met the acceptable quality levels defined in selected contracts; (3) challenges VA staff encounter in monitoring clinical contractors' performance; and (4) the extent to which VA Central Office provides oversight of VA staff responsible for monitoring clinical contracts. To examine the extent to which VA establishes complete performance requirements for clinical contractors, we reviewed applicable VA Acquisition Regulations (VAAR), policies, and handbooks to determine what should be included in clinical contracts' performance requirement statements.[Footnote 3] We also interviewed officials from VA Central Office involved in the oversight of acquisitions at VAMCs to discuss their roles in managing the staff responsible for the day-to-day oversight of clinical contractors--including the Medical Sharing Office.[Footnote 4] In addition, we conducted site visits to four VAMCs and reviewed how they monitor and oversee clinical contractors. Each of the four VAMCs was located in a different Veterans Integrated Service Network (VISN) and had several types of clinical contracts in place at their facility.[Footnote 5] These four VAMCs were located in Lebanon, Pennsylvania; Nashville, Tennessee; Minneapolis, Minnesota; and Seattle, Washington. We also spoke with officials from the four network contracting offices responsible for administering and executing all contracts for these four VAMCs about how their staff manage these clinical contracts and how they interact with VAMC staff responsible for the daily monitoring of clinical contractors.[Footnote 6] Information obtained from these VAMCs and network contracting offices cannot be generalized to all VAMCs and network contracting offices. Finally, we reviewed three contracts from each of the four VAMCs we visited--including one contract for a CBOC, one contract for the services of a specialist, and one contract for the services of a temporary clinical provider.[Footnote 7] To analyze the performance requirements included in the 12 selected clinical contracts, we identified six performance requirement categories by reviewing: (1) the VAAR; (2) VA policy, guidance, and training documents;[Footnote 8] and (3) The Joint Commission's hospital accreditation standards.[Footnote 9] We verified these categories with officials from the Veterans Health Administration (VHA) Procurement and Logistics Office to ensure they were an accurate reflection of performance requirements that should be included in VA clinical contracts. The six categories included in our analysis are: (1) type of provider or care; (2) credentialing and privileging; (3) clinical practice standards; (4) medical record documentation; (5) business processes; and (6) access to care.[Footnote 10] We analyzed the 12 selected clinical contracts to determine if they included all, at least one, or none of the required components of these categories. To examine the extent to which VA clinical contracts include clear and measurable performance standards for assessing whether or not clinical contractors met the acceptable quality levels defined in selected contracts, we reviewed the performance standards from the 12 selected contracts. To complete this review, we identified whether the contracts contained performance monitoring plans that included performance standards in five of the same six categories vital to VA operations we previously identified.[Footnote 11] We analyzed performance standards in each category to determine if they included all, some, or none of the required components of the category. We also interviewed contracting officers (CO) and contract supervisors from the four network contracting offices that are responsible for administering and executing the contracts from the four VAMCs we visited regarding the use and development of performance standards within clinical contracts.[Footnote 12] Finally, we spoke with contracting officers' representatives (COR) and medical directors responsible for overseeing the clinical contractors for the12 selected contracts about their experiences using performance standards within clinical contracts to assess contractor performance.[Footnote 13] To examine challenges VA staff encounter in monitoring clinical contractors' performance, we held structured interviews with and administered a data collection instrument to the 40 CORs who oversaw clinical contracts in the four VAMCs we visited. These structured interviews and the data collection instrument were designed to capture information about various aspects of CORs' tasks and responsibilities-- including how many contracts CORs managed, whether or not serving as a COR was the individual's primary position or a collateral duty, and experiences with COR training required by VA. Results from this data collection instrument cannot be generalized to all CORs throughout VA's health care system, but provide important insights. In addition, we interviewed the CORs and medical directors responsible for monitoring the 12 contracts we reviewed about their experience monitoring these contracts. Finally, we reviewed VA's COR training courses to assess their content and structure. To examine the extent to which VA Central Office provides oversight of VA staff responsible for monitoring clinical contracts, we interviewed VA Central Office officials responsible for overseeing the acquisition process, including officials from the three regional Service Area Offices (SAO) and the VHA Procurement and Logistics Office, about their role in overseeing COs and VAMC-based CORs.[Footnote 14] We also reviewed tools used by these offices to conduct oversight of COs and CORs, such as checklists and procedure guides. In addition, we spoke with COs and CORs responsible for managing and monitoring the 12 contracts we reviewed about their experience working with these VA Central Office officials. For additional details about the scope and methodology used in this report, see appendix I. We conducted this performance audit from October 2012 to October 2013 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Background: Clinical contracts at VA are used to acquire the services of clinical personnel, such as physicians, pharmacists, and nurses. These contracts can be used to fill vacancies for clinicians in specialties that are difficult to recruit, supplement existing VAMC capacity by providing additional clinicians in high-volume areas where VA also manages a staff of its own employees, or fill critical staffing vacancies on a long-or short-term basis. Clinical contracting at VA is governed by two sets of regulations--the FAR and VAAR--that contain general requirements for all VA acquisitions. The FAR contains government-wide regulations and establishes uniform policies and procedures used by all executive branch federal agencies for their acquisitions. VA supplements the FAR with the VAAR to establish uniform policies and procedures for all VA acquisitions of supplies and services. In addition to the VAAR, VA provides guidance to its acquisition workforce in the form of policy directives and handbooks. Acquisition Workforce at VA: Both acquisition and clinical staff at VA work together to plan, execute, and monitor clinical contracts at VA. On the acquisition side, COs are responsible for planning, awarding, and administering contracts on behalf of the federal government. Each CO is able to obligate federal funds up to a specified limit and a CO must formally approve all clinical contracts at VA. Common tasks of a CO include developing acquisition planning documents used to begin a clinical contract, conducting market research to determine pricing and availability for a clinical contract, and completing the formal competitive or non-competitive solicitation process for contracts. Each CO works within a network contracting office and is overseen by managers within that office who report directly to VA Central Office. There are 21 network contracting offices throughout VA's health care system that manage all the contracting activities of a single VISN. [Footnote 15] Two types of VAMC staff have monitoring responsibilities for clinical contracts--CORs and medical directors. For each VA clinical contract, the CO responsible for the contract designates a COR at the VAMC to help develop the clinical contract and monitor the contract provider's performance once the provider begins work. Common tasks delegated to the COR include providing input on the performance requirements for the clinical contract, determining how the contract provider's performance will be measured and monitoring performance once work has begun, validating the contract provider's invoices to ensure their accuracy, managing contract modifications, and assisting the CO in resolving any issues that may arise with the contract provider. At VA, CORs are commonly administrative personnel responsible for managing the operations of a specialty care line at a VAMC--such as primary care, surgery, etc.--where the contractor will be working. Medical directors in various specialty care lines often assist CORs in monitoring contract provider performance because CORs lack the expertise to evaluate a contract provider's clinical abilities. [Footnote 16] Medical directors are responsible for overseeing the clinical care provided by all providers within a specialty care line, including VA-employed and contract clinicians. Medical directors use existing VA processes, including the credentialing and privileging process, to monitor contract providers.[Footnote 17] This clinical monitoring includes an initial assessment of the clinician's competency during the first 90 days after the contract provider begins working at a VA facility and typically includes evaluations in several areas--such as patient care, clinical knowledge, and interpersonal communication skills.[Footnote 18] Upon successful completion of the initial competency evaluation, medical directors monitor contract providers through ongoing evaluations of their performance according to the specific performance standards used to evaluate all clinicians working at VA within their specialty.[Footnote 19] Medical directors provide the results of these initial and ongoing evaluations to the COR and work with the COR to resolve any issues that may arise as a result of a contract provider's clinical performance. CORs are responsible for maintaining the official record of the contract provider's performance and providing official performance assessments to the CO. VA Acquisition Oversight Structure: VA Central Office has primary responsibility for overseeing network contracting offices and manages clinical contracting activities through the VHA Procurement and Logistics Office. There are five primary offices within the VHA Procurement and Logistics Office that are responsible for overseeing various aspects of clinical contracting activities and report to VHA's Deputy Chief Procurement Officer. (See figure 1.) Figure 1: Organization of the Veterans Health Administration (VHA) Procurement and Logistics Office: [Refer to PDF for image: organizational chart] Top level: * VHA Chief Procurement and Logistics Officer. Second level, reporting to VHA Chief Procurement and Logistics Officer: * VHA Deputy Chief Procurement Officer. Third level, reporting to VHA Deputy Chief Procurement Officer: * Medical Sharing; * Procurement Operations: - Procurement Audit; * Procurement Policy; * Service Area Offices. Source: GAO analysis of VA information. [End of figure] Medical Sharing Office. The Medical Sharing Office is responsible for providing guidance to network contracting offices regarding the content and structure of solicitations for clinical contracts and for reviewing several types of clinical contracts. The Medical Sharing Office reviews solicitations of all competitive clinical contracts valued at over $1.5 million, all non-competitive clinical contracts valued at over $500,000, and all organ transplant contracts.[Footnote 20] All Medical Sharing Office reviews are conducted before a solicitation is issued to ensure that all the necessary provisions are in place prior to any competition or award. Procurement Operations Office. The Procurement Operations Office is responsible for providing ongoing guidance and monitoring of the COR population at VA. The Procurement Operations Office conducts reviews of COR files and publishes a COR newsletter. Procurement Audit Office. The Procurement Audit Office is responsible for ensuring compliance with VA policies and procedures related to contracting. This office conducts internal compliance audits of contracts, including clinical contracts, once they are executed to ensure that all required documentation was included in the final contract and audits the activities of network contracting offices and SAOs to ensure their compliance with VA policies and regulations. Procurement Policy Office. The Procurement Policy Office is responsible for providing guidance to VA's acquisition workforce in network contracting offices and SAOs. This office produces and updates standard operating procedures for CORs and COs. Service Area Offices. SAOs are the regional contract management entities created to oversee the activities of the 21 network contracting offices and the COs and supervisors that work within them. VHA created three SAOs--East, West, and Central--to manage the contracting activities of six to eight VISNs each. SAOs review solicitations for most clinical contracts during their initial stages to ensure that all necessary provisions are in place prior to any competition or award. Reviewed Contracts Included Performance Requirements, but Lacked Complete Detail Needed to Hold Contractors Accountable in Some Cases: We found that all 12 contracts we reviewed from the four VAMCs we visited contained performance requirements consistent with VA acquisition policy. However, we found that 10 of the 12 contracts we reviewed lacked detailed descriptions of contractors' performance requirements in one or more of the six categories we assessed. [Footnote 21] We analyzed the content of these contracts using six performance requirement categories we established through reviews of the VAAR, VA policies, and the standards of the leading hospital accreditation organization. (See figure 2) Figure 2: Performance Requirement Categories for VA Clinical Contracts: [Refer to PDF for image: illustration] Type of provider or care: * Licensure requirements; * Description of specialty area; * List of personnel and rules for substitution. Credentialing and privileging: * Use of VA's electronic system; * Requirements for initial reviews of providers; * Renewal of privileges after the initial contract term. Clinical practice standards: * Compliance with medical staff bylaws; * Adverse event and patient complaint procedures; * Compliance with accreditation entity standards. Medical record documentation: * Maintaining electronic medical records; * Response to requests for consultations, etc. * Timely entry of information. Business processes: * Invoicing procedures; * Work hours and timecard procedures; * Personnel security requirements. Access to care[A]: * Compliance with scheduling policies; * Hours of operation; * Providers required to be present for operation. Source: GAO (analysis); VA (information); The Joint Commission (information). Note: We identified these performance requirement categories by reviewing the VA Acquisition Regulation (VAAR); VA policy, guidance, and training documents; and The Joint Commission's hospital accreditation standards. Officials from the VHA Office of Procurement and Logistics verified these six performance requirement categories. [A] We added access to care as a performance requirement category for community-based outpatient clinic (CBOC) contracts because VA policy requires CBOCs to comply with the same timeliness goals required of VAMCs. [End of figure] We also found that the level of detail in contract performance requirements contained in the 12 contracts we reviewed varied both by the type of contract--CBOC contract, specialty care contract, or temporary clinical provider contract--and by the six performance requirement categories we assessed. (See table 1.) Table 1: Level of Detail in Performance Requirements for 12 Selected Contracts at Four VA Medical Centers (VAMC), by Contract Type and Performance Requirement Category: Community-based outpatient clinic (CBOC): Contract type and location: Lebanon VAMC; Performance requirement category[A]: Type of provider or care[B]: complete; Credentialing and privileging[C]: complete; Clinical practice standards[D]: complete; Medical record documentation[E]: complete; Business processes[F]: complete; Access to care: complete. Contract type and location: Minneapolis VAMC; Performance requirement category[A]: Type of provider or care[B]: complete; Credentialing and privileging[C]: partial; Clinical practice standards[D]: complete; Medical record documentation[E]: complete; Business processes[F]: complete; Access to care: complete. Contract type and location: Nashville VAMC; Performance requirement category[A]: Type of provider or care[B]: complete; Credentialing and privileging[C]: complete; Clinical practice standards[D]: complete; Medical record documentation[E]: complete; Business processes[F]: complete; Access to care: complete. Contract type and location: Seattle VAMC; Performance requirement category[A]: Type of provider or care[B]: complete; Credentialing and privileging[C]: partial; Clinical practice standards[D]: complete; Medical record documentation[E]: complete; Business processes[F]: partial; Access to care: partial. Specialty care: Contract type and location: Lebanon VAMC; Performance requirement category[A]: Type of provider or care[B]: complete; Credentialing and privileging[C]: partial; Clinical practice standards[D]: complete; Medical record documentation[E]: complete; Business processes[F]: complete; Access to care: N/A. Contract type and location: Minneapolis VAMC; Performance requirement category[A]: Type of provider or care[B]: complete; Credentialing and privileging[C]: partial; Clinical practice standards[D]: partial; Medical record documentation[E]: complete; Business processes[F]: complete; Access to care: N/A. Contract type and location: Nashville VAMC; Performance requirement category[A]: Type of provider or care[B]: complete; Credentialing and privileging[C]: partial; Clinical practice standards[D]: partial; Medical record documentation[E]: partial; Business processes[F]: complete; Access to care: N/A. Contract type and location: Seattle VAMC; Performance requirement category[A]: Type of provider or care[B]: complete; Credentialing and privileging[C]: partial; Clinical practice standards[D]: complete; Medical record documentation[E]: complete; Business processes[F]: complete; Access to care: N/A. Temporary clinical provider: Contract type and location: Lebanon VAMC; Performance requirement category[A]: Type of provider or care[B]: partial; Credentialing and privileging[C]: incomplete; Clinical practice standards[D]: partial; Medical record documentation[E]: partial; Business processes[F]: complete; Access to care: N/A. Contract type and location: Minneapolis VAMC; Performance requirement category[A]: Type of provider or care[B]: complete; Credentialing and privileging[C]: partial; Clinical practice standards[D]: partial; Medical record documentation[E]: partial; Business processes[F]: complete; Access to care: N/A. Contract type and location: Nashville VAMC; Performance requirement category[A]: Type of provider or care[B]: complete; Credentialing and privileging[C]: partial; Clinical practice standards[D]: partial; Medical record documentation[E]: partial; Business processes[F]: complete; Access to care: N/A. Contract type and location: Seattle VAMC; Performance requirement category[A]: Type of provider or care[B]: complete; Credentialing and privileging[C]: partial; Clinical practice standards[D]: partial; Medical record documentation[E]: incomplete; Business processes[F]: complete; Access to care: N/A. Legend: N/A = not applicable: Source: GAO analysis of VA information. [A] Each category was recorded as complete, partial, or incomplete, based on whether the contract contained performance requirement statements for all, at least one, or none of the components identified for each category. [B] Type of provider or care includes requirements for contractor personnel and qualifications, such as licensing, and a description of the specialty area in which the contractor will provide care. [C] Credentialing and privileging includes requirements for providers to submit required information for the credentialing and privileging process applicable to their specialty before providing care and renewal or review requirements. [D] Clinical practice standards includes requirements regarding compliance with VAMC medical staff bylaws and the provider's response to adverse events or patient complaints. [E] Medical record documentation includes requirements for entering information into the veteran's electronic medical record, such as referrals, consults, and a plan of care. [F] Business processes includes administrative requirements, such as invoicing procedures and time card submission. [G] Access to care applies only to CBOC contracts and includes requirements regarding compliance with VA's timeliness and scheduling standards, CBOC hours of operation, and types of contract providers required to be on site. We added access to care as a performance requirement category for CBOC contracts because VA policy requires CBOCs to comply with the same timeliness goals required of VAMCs. [End of table] The CBOC contracts we reviewed were the most detailed, followed by specialty care contracts and temporary clinical provider contracts. CBOC contracts. Contracts for CBOCs generally contained the most detailed performance requirements. For example, all four of the CBOC contracts we reviewed had complete performance requirements for three of the six applicable categories--the type of provider or care, clinical practice standards, and medical record documentation. However, two of the four CBOC contracts--at the Minneapolis and Seattle VAMCs--lacked detail in one or more of the performance requirement categories we reviewed. For example, the Seattle VAMC CBOC contract was the only CBOC contract we reviewed that did not include a required set of operating hours for the contracted facility, which was a component we assessed within the access to care category. Specialty care contracts. The amount of detail in performance requirements within specialty care contracts varied across our four selected VAMCs. Overall, performance requirements in these contracts lacked specificity in key categories. For example, two of our selected specialty care contracts--for the Seattle and Minneapolis VAMCs--were for the services of cardiothoracic surgeons and had very similar results in our review, except for their descriptions of performance requirements within the clinical practice standards category. For this category, the Seattle VAMC cardiothoracic contract had detailed performance requirements, but the Minneapolis VAMC cardiothoracic contract did not contain a statement describing the provider's responsibilities for the reporting of and response to adverse events and patient complaints, a component we assessed within the clinical practice standards category.[Footnote 22] In addition, the Nashville VAMC specialty care contract for the services of a psychiatrist had limited detail because it did not contain requirements for the clinical contractor's privileges to be renewed at the beginning of each new contract term, and received a partial rating in the medical record documentation category because the contract did not contain performance requirements for the timely entry of information into VA's electronic medical record. Temporary clinical provider contracts. We found that temporary clinical provider contracts included the least detailed performance requirements among the three contract types we reviewed. For example, the Lebanon VAMC temporary clinical provider contract for a gynecologist was missing descriptions of requirements for providers to undergo VA's credentialing and privileging process, comply with The Joint Commission requirements that apply to gynecology, and enter relevant data into veterans' electronic medical records. These specific requirements are components of three separate categories we reviewed--credentialing and privileging, clinical practice standards, and medical record documentation. In addition, the Seattle VAMC temporary clinical provider contract for a pharmacist was missing information on contractor requirements related to both the clinical practice standards and medical record documentation categories. According to VA officials, temporary clinical provider contracts are often written using a format that is designed to allow for shorter and faster contract award timeframes; however, this shorter format likely contributed to less complete performance requirement statements in the contracts we reviewed. We also found the level of detail in contract performance requirements varied significantly by performance requirement category, with a majority of reviewed contracts missing some component of the credentialing and privileging performance requirement category. Credentialing and privileging. This category included a review of three specific components of performance requirements: (1) requirements for all contract providers to initially undergo VA's credentialing and privileging process; (2) the use of VA's electronic credentialing system for that process; and (3) requirements for the renewal of privileges.[Footnote 23] In our review, we found that 10 of the 12 contracts were missing performance requirements from one or more of these three components. Six of the 10 contracts did not contain statements describing the rules for renewal of privileges, which is part of the credentialing and privileging process at VA. Privileges are required to be renewed every 2 years by VA policy. The term of a contract provider's privileges may not extend beyond the term of the contract regardless of whether the VAMC intends to renew or extend the contract; this means that privileges need to be renewed more often than every 2 years for contract providers depending on the length of the contract.[Footnote 24] In addition, 5 of the 10 contracts did not contain statements requiring the use of VA's electronic credentialing system for the credentialing process. Type of provider or care. This performance requirement category had three components: (1) provider licensure requirements, (2) a description of the specialty area care was contracted for, and (3) a listing of key personnel providing services under the contract. [Footnote 25] Eleven of the 12 contracts we reviewed included detailed performance requirements in this category, with the exception of the Lebanon VAMC temporary clinical provider contract, which did not include the name of the gynecologist selected for the contract or rules for the substitution of another provider if needed. Business processes. This performance requirement category contained three components: (1) invoicing policies; (2) on-and off-duty hours and time card submission; and (3) personnel security responsibilities, such as background checks. Eleven of the 12 contracts included detailed performance requirements in this category, with the exception of the Seattle VAMC CBOC contract, which did not list on-and off-duty hours for staff, but included all other business process components. COs and CORs we interviewed told us that they lack available tools, such as standard templates, that provide examples for the types of performance requirements that should be included in the types of clinical contracts we selected for our review. Several COs and CORs we spoke with told us that they refer to previously awarded contracts as their source for developing performance requirements for future contracts. However, the previous contracts used as sources may not contain categories of performance requirements that adhere to VA policies and allow CORs to hold contractors accountable for their performance. In addition, several COs and CORs we interviewed said they would find it useful to have standard templates for clinical contracts. VA Central Office officials with whom we spoke said that there is a template for CBOC contracts that has been distributed among contracting officials and that templates for six common specialty care contracts, including cardiology and anesthesiology, are currently in development as of August 2013, and that VA's progress in developing and implementing these templates is dependent upon VA Central Office and the VA OIG agreeing on the six specialties that will have templates developed. Absent available tools, such as standard templates for common types of contracts--including those for CBOCs, specialty care, and temporary clinical providers--VA cannot reasonably ensure that critical requirements for contract providers' performance are consistently included in VA clinical contracts and are standardized across VAMCs. Without assurance that these critical performance requirements are included in clinical contracts, VA may not be able to hold clinical contractors accountable for providing the high-quality services VAMCs need to serve veterans. Selected VA Clinical Contracts Did Not Always Include Clear and Measurable Performance Standards: Two of the 12 clinical contracts we reviewed did not include any performance standards and 7 contracts did not include performance standards in key categories. Additionally, the performance standards included within the 12 clinical contracts we reviewed were not always stated clearly and did not always include measurable targets. We found that VA did not provide guidance to COs and CORs on how to develop performance standards that are clear and measurable for determining whether or not clinical contractors met the acceptable quality levels defined in selected contracts. The FAR requires agencies to ensure that requirements for services are clearly defined and appropriate performance standards are developed so that the agency's requirements can be understood by potential offerors and that performance in accordance with contract terms and conditions will meet the agency's requirements.[Footnote 26] In addition, agencies, in order to successfully measure the performance of their operations, use performance standards that demonstrate results, are limited to a vital few activities, and provide useful information for decision-making.[Footnote 27] To determine whether performance standards used by agencies meet these qualities, we previously found that performance standards should have several attributes--including being clearly stated and using measurable targets.[Footnote 28] We assessed each of the 12 selected contracts from the four VAMCs we visited to determine if the performance standards for five of the six categories we previously identified included these two attributes of successful performance standards.[Footnote 29] Performance standards were not clearly stated. Two of the 12 clinical contracts we reviewed--the Lebanon VAMC CBOC and temporary clinical provider contracts--did not include any monitoring plan provisions defining the performance standards against which the VAMC would assess the contractor's performance. Performance standards included in the monitoring plans of the remaining 10 clinical contracts we reviewed lacked clarity in what would be used to determine a clinical contractor's performance. For example, the Nashville VAMC CBOC contract includes a performance standard in the clinical practice standards category that requires the contractor to meet a specific target for three of four outpatient satisfaction survey scores. However, VA's outpatient satisfaction survey includes significantly more than four elements and the contract's performance standard does not clearly specify which four survey elements will be used for this analysis. In another instance, the Seattle VAMC temporary clinical provider contract includes a performance standard for prescription dispensing that includes multiple standards that cover numerous tasks related to prescription processing and validation in inpatient and outpatient settings. By including multiple parts in a single performance standard, it is unclear what the COR will actually be measuring to determine the clinical contractor's performance in this area. As table 2 shows, none of the 12 contracts we assessed included clearly stated performance standards in all five categories. Table 2: Extent to Which Selected Contracts from Four VA Medical Centers (VAMC) Included Clearly Stated Performance Standards, by Contract Type and Performance Requirement Category: Community-based outpatient clinic (CBOC): Contract type and VAMC location: Lebanon VAMC; Performance requirement category[A]: Type of provider or care[B]: no performance standards were included in the contract; Credentialing and privileging[C]: no performance standards were included in the contract; Clinical practice standards[D]: no performance standards were included in the contract; Medical record documentation[E]: no performance standards were included in the contract; Access to care[F]: no performance standards were included in the contract. Contract type and VAMC location: Minneapolis VAMC; Performance requirement category[A]: Type of provider or care[B]: all performance standards were clearly stated; Credentialing and privileging[C]: no performance standards were included in the contract; Clinical practice standards[D]: all performance standards were clearly stated; Medical record documentation[E]: all performance standards were clearly stated; Access to care[F]: all performance standards were clearly stated. Contract type and VAMC location: Nashville VAMC; Performance requirement category[A]: Type of provider or care[B]: all performance standards were clearly stated; Credentialing and privileging[C]: all performance standards were clearly stated; Clinical practice standards[D]: some performance standards were clearly stated; Medical record documentation[E]: all performance standards were clearly stated; Access to care[F]: all performance standards were clearly stated. Contract type and VAMC location: Seattle VAMC; Performance requirement category[A]: Type of provider or care[B]: all performance standards were clearly stated; Credentialing and privileging[C]: no performance standards were included in the contract; Clinical practice standards[D]: all performance standards were clearly stated; Medical record documentation[E]: no performance standards were included in the contract; Access to care[F]: all performance standards were clearly stated. Specialty care: Contract type and VAMC location: Lebanon VAMC; Performance requirement category[A]: Type of provider or care[B]: some performance standards were clearly stated; Credentialing and privileging[C]: all performance standards were clearly stated; Clinical practice standards[D]: some performance standards were clearly stated; Medical record documentation[E]: all performance standards were clearly stated; Access to care[F]: N/A. Contract type and VAMC location: Minneapolis VAMC; Performance requirement category[A]: Type of provider or care[B]: all performance standards were clearly stated; Credentialing and privileging[C]: some performance standards were clearly stated; Clinical practice standards[D]: some performance standards were clearly stated; Medical record documentation[E]: no performance standards were included in the contract; Access to care[F]: N/A. Contract type and VAMC location: Nashville VAMC; Performance requirement category[A]: Type of provider or care[B]: some performance standards were clearly stated; Credentialing and privileging[C]: all performance standards were clearly stated; Clinical practice standards[D]: some performance standards were clearly stated; Medical record documentation[E]: some performance standards were clearly stated; Access to care[F]: N/A. Contract type and VAMC location: Seattle VAMC; Performance requirement category[A]: Type of provider or care[B]: all performance standards were clearly stated; Credentialing and privileging[C]: no performance standards were included in the contract; Clinical practice standards[D]: no performance standards were included in the contract; Medical record documentation[E]: all performance standards were clearly stated; Access to care[F]: N/A. Temporary clinical provider: Contract type and VAMC location: Lebanon VAMC; Performance requirement category[A]: Type of provider or care[B]: no performance standards were included in the contract; Credentialing and privileging[C]: no performance standards were included in the contract; Clinical practice standards[D]: no performance standards were included in the contract; Medical record documentation[E]: no performance standards were included in the contract; Access to care[F]: N/A. Contract type and VAMC location: Minneapolis VAMC; Performance requirement category[A]: Type of provider or care[B]: no performance standards were included in the contract; Credentialing and privileging[C]: some performance standards were clearly stated; Clinical practice standards[D]: all performance standards were clearly stated; Medical record documentation[E]: no performance standards were included in the contract; Access to care[F]: N/A. Contract type and VAMC location: Nashville VAMC; Performance requirement category[A]: Type of provider or care[B]: some performance standards were clearly stated; Credentialing and privileging[C]: no performance standards were included in the contract; Clinical practice standards[D]: all performance standards were clearly stated; Medical record documentation[E]: no performance standards were included in the contract; Access to care[F]: N/A. Contract type and VAMC location: Seattle VAMC; Performance requirement category[A]: Type of provider or care[B]: some performance standards were clearly stated; Credentialing and privileging[C]: no performance standards were included in the contract; Clinical practice standards[D]: some performance standards were clearly stated; Medical record documentation[E]: Access to care[F]: N/A. Legend: N/A = not applicable: Source: GAO analysis of VA information. Note: The business processes category includes only administrative requirements, such as invoicing procedures and time card submission. We did not assess performance standards in this category because these requirements do not require performance standards for their validation. [A] To assess if a performance standard was clear, we reviewed each performance standards included in a selected contract to determine if each performance standards clearly stated what the contractor was responsible for and included specific information about how the contractor would be assessed. Each category was scored on a scale of complete, partial, or incomplete, based on whether the performance standards included in the category met these criteria. [B] Type of provider or care includes performance standards that define specific assessments of clinical outcomes for the type of provider covered by the contract. [C] Credentialing and privileging includes performance standards related to the credentialing and privileging process. [D] Clinical practice standards includes performance standards related to general clinical practice at VA, including compliance with VAMC policies and response to adverse events and patient complaints. [E] Medical record documentation includes performance standards related to contract providers' use of VA's electronic medical record. [F] Access to care applies only to CBOC contracts and includes performance standards related to contract providers' compliance with VA's national timeliness goals, CBOC hours of operation, and types of clinical providers required to be onsite. We added access to care as a performance requirement category for CBOC contracts only because VA policy requires CBOCs to comply with the same timeliness goals required of VAMCs. [End of table] It is important to ensure that performance standards included in clinical contracts are clearly stated to minimize confusion during the monitoring of clinical contractors' performance. Without clearly articulated performance standards, COs and CORs may not be able to effectively assess the performance of clinical contractors and VA may not be able to hold contractors accountable for poor performance. Performance standards did not include measurable targets. We found that there were no categories that included fully measurable performance standards in the 12 contracts we assessed. As previously noted, 2 of the 12 contracts--the Lebanon VAMC CBOC and temporary clinical provider contracts--did not include any monitoring plan provisions defining the performance standards against which the VAMC would assess the contractor's performance. Of the remaining 10 contracts with monitoring plans, only 1 contract--the Lebanon VAMC specialty contract--included fully measurable performance standards in all categories, and the other 9 contracts either did not include performance standards for a critical category or had performance standards that were only partially measurable in at least one category. (See table 3.) For example, the Nashville VAMC temporary clinical provider contract included a performance standard in the clinical practice standards category that stated that the contractor should have no more than two patient or staff complaints. However, this contract did not define how frequently this target would be measured and therefore it is not possible to determine whether this threshold applies to complaints on a monthly, quarterly, or annual basis. This limited information on how often the complaints would be reviewed and measured for the clinical contractor makes it difficult to determine what a successful performance outcome would be. In another instance, the Minneapolis VAMC specialty contract includes several performance standards in the credentialing and privileging category that indicate the clinical contractor is responsible for maintaining medical and clinical knowledge and engaging in practiced- based learning. However, this contract does not include information on how the clinical contractor's performance in these areas would be assessed and does not specify a clear numerical target for determining whether or not these standards have been fulfilled. Finally, the Nashville VAMC specialty contract includes a performance standard in the medical record documentation category that states the clinical contractor should maintain documentation in VA's electronic medical record system of 95 percent, but does not include a description of how the COR would assess compliance with this performance standard. Table 3: Extent to Which Selected Contracts from Four VA Medical Centers (VAMC) Included Measurable Targets for Performance Standards, by Contract Type and Performance Requirement Category: Contract type and VAMC location: Community-based outpatient clinic (CBOC). Contract type and VAMC location: Lebanon VAMC; Performance requirement category[A]: Type of provider or care[B]: no performance standards were included in the contract; Credentialing and privileging[C]: no performance standards were included in the contract; Clinical practice standards[D]: no performance standards were included in the contract; Medical record documentation[E]: no performance standards were included in the contract; Access to care[F]: no performance standards were included in the contract; Contract type and VAMC location: Minneapolis VAMC; Performance requirement category[A]: Type of provider or care[B]: all performance standards were measurable; Credentialing and privileging[C]: no performance standards were included in the contract; Clinical practice standards[D]: some performance standards were measurable; Medical record documentation[E]: some performance standards were measurable; Access to care[F]: some performance standards were measurable. Contract type and VAMC location: Nashville VAMC; Performance requirement category[A]: Type of provider or care[B]: all performance standards were measurable; Credentialing and privileging[C]: Clinical practice standards[D]: Medical record documentation[E]: all performance standards were measurable; Access to care[F]: all performance standards were measurable. Contract type and VAMC location: Seattle VAMC; Performance requirement category[A]: Type of provider or care[B]: some performance standards were measurable; Credentialing and privileging[C]: no performance standards were included in the contract; Clinical practice standards[D]: all performance standards were measurable; Medical record documentation[E]: no performance standards were included in the contract; Access to care[F]: all performance standards were measurable. Specialty care: Contract type and VAMC location: Lebanon VAMC; Performance requirement category[A]: Type of provider or care[B]: all performance standards were measurable; Credentialing and privileging[C]: all performance standards were measurable; Clinical practice standards[D]: all performance standards were measurable; Medical record documentation[E]: all performance standards were measurable; Access to care[F]: N/A. Contract type and VAMC location: Minneapolis VAMC; Performance requirement category[A]: Type of provider or care[B]: Credentialing and privileging[C]: Clinical practice standards[D]: all performance standards were measurable; Medical record documentation[E]: no performance standards were included in the contract; Access to care[F]: N/A. Contract type and VAMC location: Nashville VAMC; Performance requirement category[A]: Type of provider or care[B]: all performance standards were measurable; Credentialing and privileging[C]: some performance standards were measurable; Clinical practice standards[D]: some performance standards were measurable; Medical record documentation[E]: some performance standards were measurable; Access to care[F]: N/A. Contract type and VAMC location: Seattle VAMC; Performance requirement category[A]: Type of provider or care[B]: all performance standards were measurable; Credentialing and privileging[C]: no performance standards were included in the contract; Clinical practice standards[D]: no performance standards were included in the contract; Medical record documentation[E]: all performance standards were measurable; Access to care[F]: N/A. Temporary clinical provider: Contract type and VAMC location: Lebanon VAMC; Performance requirement category[A]: Type of provider or care[B]: no performance standards were included in the contract; Credentialing and privileging[C]: no performance standards were included in the contract; Clinical practice standards[D]: no performance standards were included in the contract; Medical record documentation[E]: no performance standards were included in the contract; Access to care[F]: N/A. Contract type and VAMC location: Minneapolis VAMC; Performance requirement category[A]: Type of provider or care[B]: no performance standards were included in the contract; Credentialing and privileging[C]: some performance standards were measurable; Clinical practice standards[D]: all performance standards were measurable; Medical record documentation[E]: no performance standards were included in the contract; Access to care[F]: N/A. Contract type and VAMC location: Nashville VAMC; Performance requirement category[A]: Type of provider or care[B]: some performance standards were measurable; Credentialing and privileging[C]: no performance standards were included in the contract; Clinical practice standards[D]: some performance standards were measurable; Medical record documentation[E]: no performance standards were included in the contract; Access to care[F]: N/A. Contract type and VAMC location: Seattle VAMC; Performance requirement category[A]: Type of provider or care[B]: some performance standards were measurable; Credentialing and privileging[C]: no performance standards were included in the contract; Clinical practice standards[D]: some performance standards were measurable; Medical record documentation[E]: some performance standards were measurable; Access to care[F]: N/A. Legend: N/A = not applicable: Source: GAO analysis of VA data. Note: The business processes category includes only administrative requirements, such as invoicing procedures and time card submission. We did not assess performance standards in this category because these requirements do not require performance standards for their validation. [A] To assess if a performance standards was measurable, we reviewed each performance standard included in a selected contract to determine if each performance standard included a measurable target for the contractor, including a numerical target. Each category was scored on a scale of complete, partial, or incomplete, based on whether the performance standards included in the category met these criteria. [B] Type of provider or care includes performance standards that define specific assessments of clinical outcomes for the type of provider covered by the contract. [C] Credentialing and privileging includes performance standards related to the credentialing and privileging process. [D] Clinical practice standards includes performance standards related to general clinical practice at VA, including compliance with VAMC policies and response to adverse events and patient complaints. [E] Medical record documentation includes performance standards related to contract providers' use of VA's electronic medical record. [F] Access to care applies only to CBOC contracts and includes performance standards related to contract providers' compliance with VA's national timeliness goals, CBOC hours of operation, and types of clinical providers required to be onsite. We added access to care as a performance requirement category for CBOC contracts only because VA policy requires CBOCs to comply with the same timeliness goals required of VAMCs. [End of table] It is important to ensure that the performance standards included in clinical contracts be measurable in order to allow COs and CORs to confidently and effectively measure contractors' performance. VA has not provided detailed guidance to COs and CORs on how to develop performance standards that allow CORs to conduct meaningful performance monitoring. VA requires that each contract contain appropriate quality assurance standards--including a detailed description of the monitoring procedures that the CO and COR will use to evaluate the performance of clinical contractors and data collection that will be performed.[Footnote 30] However, this guidance does not include any information on what type of performance standards should be included in a clinical contract, does not include examples of effective performance standards, and does not define the elements of successful performance standards. Without ensuring that COs and CORs have access to detailed guidance on how to construct meaningful performance standards that are both clearly stated and measurable, VA cannot ensure that all clinical contractors will be monitored appropriately or that these performance assessments will be based on the most appropriate measurement of the care they provided to veterans in VA facilities. CORs Reported that Heavy Workloads and Inadequate Training Make It Difficult to Effectively Monitor Contract Providers' Performance: During our review of 12 clinical contracts at four VAMCs, CORs reported two challenges that may compromise VA's monitoring of contractors' performance--the heavy workload associated with the COR position and the lack of adequate training for CORs. CORs Reported Facing Heavy Workloads that May Compromise VAMC Monitoring of Clinical Contractors: CORs at the four VAMCs we visited consistently reported facing significant challenges in effectively carrying out their COR responsibilities for monitoring clinical contractors. Most CORs at the four VAMCs we visited reported that they had other primary duties-- including managing staff--that required them to approach their COR responsibilities as a collateral duty. This workload challenge may have led to CORs being unable to effectively monitor clinical contractors. Current VA guidance requires VAMCs to provide CORs with the time to complete their responsibilities and ensure that contract compliance is managed by a knowledgeable COR. Specifically, VA's standard operating procedure for CORs requires VAMCs to provide CORs with the time and resources necessary to complete required training and fulfill their duties as a COR. Further, the policy governing clinical contracting at VA states that the COR is responsible for ensuring contract compliance and specifies that the COR must have sufficient knowledge of the operation of the facility and the specific specialty requesting the contract.[Footnote 31] In addition, to monitor clinical contracts effectively CORs are required to perform a number of key functions according to VA's standard operating procedure for CORs. This guidance requires CORs to submit documentation to the CO and carry out a number of responsibilities as part of their monitoring efforts. (See table 4.) Table 4: VA Requirements for Contracting Officer's Representative (COR) Monitoring of Clinical Contractor Performance: Monitoring task: Quarterly contract progress reports; Description: These reports contain the COR's assessment of the contract provider's performance related to patient satisfaction, access to care, timeliness of services, compliance with contract requirements and service quality, and compliance with reporting requirements. Monitoring task: Quarterly quality assurance reports; Description: These reports contain the COR's documentation of their monitoring of performance standards included in each contract and any clinical quality assurance reviews that were conducted by VA clinical staff, such as medical directors. Monitoring task: Quarterly invoice audit reports; Description: These reports contain the COR's certification that all of a contract provider's invoices were verified against records of their actual work hours and any improper invoices were returned to the contract provider for correction. Monitoring task: Quarterly contract provider training reports; Description: These reports include copies of all required training records certifications, licenses, and records of changes in contractor personnel. Monitoring task: Annual past performance evaluations; Description: These evaluations are completed by the COR at the end of each contract term in order to record the COR's evaluation of the contractor in several categories, including the quality of service provided, contract costs, timeliness of performance, and business relations. Monitoring task: Regularly assisting the contracting officer (CO); Description: CORs are required to assist COs in identifying and resolving issues with the contract provider--including identifying delays in contract delivery, analyzing contract provider claims and recommending settlement options to the CO, and determining when the CO should intervene and begin formal procedures for terminating a contract. Source: GAO analysis of VA information. [End of table] The CORs responsible for monitoring clinical contracts at the VAMCs we visited were often serving in this role as a collateral duty and most had other primary duties that limited their ability to monitor clinical contracts. Thirty-seven of the 40 CORs (93 percent) that completed our data collection instrument, administered to all CORs with responsibility for clinical contracts at the four VAMCs we visited, stated that the COR position is a collateral duty to their primary position. Many of these CORs' primary positions require them to manage staff, maintain budgets, and oversee other clinical providers. (See table 5.) Table 5: A Sample of Position Descriptions of Contracting Officer's Representatives' (COR) Primary Positions at the Four VA Medical Centers (VAMC) We Visited: Primary position job title: Business manager for a clinical specialty; Position description: The business manager serves as the principle administrative advisor to the medical director of the specialty. This role requires the incumbent to develop and implement policies within the specialty and serve as the key staff member for planning, budgeting, and developing proposals for the utilization of resources and operations of the specialty. This individual's responsibilities also include supervising staff and conducting workforce planning for the specialty. Primary position job title: Administrative officer for a clinical specialty; Position description: As an administrative officer, the individual is responsible for the management of all administrative staff and activities within a specialty. This role requires the incumbent to develop and implement policies for the specialty, advise the medical director and VAMC management on administrative issues, and develop the budget for the specialty. The administrative officer also serves as the primary financial analyst for the specialty, supervises the administrative staff working within the specialty, and assists in the management of clinical staff working in the specialty. Primary position job title: Medical director for a specialty; Position description: The medical director is responsible for the management of all clinical and administrative staff working within the specialty, implementing policies for the specialty--including clinical procedures and requirements mandated by federal and state law, and advising VAMC leadership on issues related to the specialty. The medical director is also responsible for the specialty's budget and monitoring expenses and for maintaining the specialty's teaching program for students in applicable clinical training programs. Source: GAO analysis of VA information. [End of table] Based on the results from our data collection instrument administered to 40 CORs at the four VAMCs we visited, we found that the average COR spends about one-quarter of his or her time monitoring approximately 12 contracts, according to estimates provided by CORs; however, some of these CORs were responsible for overseeing significantly more contracts. For example, we found that 6 of these 40 CORs managed nearly 190 of the 452 (41 percent) contracts in place at the four VAMCs we visited and told us they estimated spending at most 30 percent of their work time to their COR duties. These 6 CORs were each managing about 20 more contracts than the average, in our group of 40 CORs completing the data collection instrument, in less time and all 6 of these CORs had other more time-consuming primary duties, such as serving as an administrative officer for a clinical specialty or as a program manager. In addition to CORs who responded to our data collection instrument, during our review of the 12 contracts for various clinical services in the four VAMCs we visited, we found that the CORs responsible for managing these contracts frequently did not have the time to effectively monitor the performance of contract providers. The majority of these CORs said that one of their greatest challenges is not having adequate time to fulfill their COR duties and responsibilities and that the demands of their primary positions prevented them from fully monitoring clinical contractors' performance. Specifically, CORs for 8 of the 12 contracts reported that the demands of their primary positions have at times prevented them from fully monitoring contract providers' performance. In addition, CORs for 6 of these 12 contracts stated that they could not complete certain elements of their COR responsibilities due to limited time and resources. For example, one COR stated that she did not adequately monitor costs for the 50 contracts she managed because the duties of her primary position demanded significant portions of her work day and caused frequent interruptions that did not allow her to focus on her duties as a COR. Another COR noted that she only had time to monitor contract issues that were a cause for concern or needed to be addressed immediately. As a result of this time pressure and the demands of their primary positions, CORs reported they did not complete several key monitoring functions, including: (1) submitting required reports, such as quarterly quality assurance reports; (2) documenting contract provider performance issues for annual evaluations; and (3) monitoring contract expenditures. CORs managing 5 of our 12 selected contracts reported that to help address challenges presented by time constraints and the demands of their primary positions, they often worked extended hours in order to complete at least some of their COR duties and manage their day-to-day responsibilities. For example, one COR responsible for managing 1 of our selected contracts, along with 49 other contracts, said that she worked after hours to devote a block of time to her COR work that she could not devote during the normal work day due to the responsibilities of her primary position. In addition, a few CORs also stated that they believe COR duties should be a separate, stand-alone, full-time position rather than a collateral duty to ensure that the monitoring of clinical contractors and other COR duties receives full attention. COs responsible for managing the 12 selected contracts echoed what CORs told us about the heavy workloads and monitoring challenges CORs face. However, none of the COs responsible for the 12 contracts we reviewed directly participate in the monitoring of contract providers' performance, and all have delegated this responsibility directly to the COR for each clinical contract. Most of the 10 COs responsible for our 12 selected clinical contracts told us that they also rely on the CORs to identify performance issues with contract providers. The majority of these COs recognized that CORs face competing demands and heavy workloads that may prevent them from effectively monitoring contract providers' performance. Specifically, 7 of the 10 COs responsible for the 12 selected contracts in our review reported that primary position duties and COR duties combined resulted in heavy workloads for CORs. As a result, these COs noted that CORs often do not have enough time to devote to their COR duties. Half of the COs responsible for the 12 selected contracts expressed that the COR responsibility should be a full-time, stand-alone position. We found that VA's current guidance related to COR responsibilities does not include any information on how VAMCs are to determine the feasibility of whether a COR's workload--including both COR and primary position responsibilities--will allow them to carry out their tasks as CORs for monitoring contract provider performance. The COR standard operating procedure also does not provide any guidance for determining when COR duties should be assigned as a collateral duty or a full-time responsibility. Without clear guidance on how to determine a COR's workload, VAMCs can unintentionally assign COR duties to a staff member who does not have the time available to properly monitor clinical contractors. If CORs' workloads prevent proper monitoring of clinical contracts, VA risks missing the opportunity to proactively identify and correct performance issues with contract providers and to recognize patient safety concerns potentially resulting from contract providers' actions. By failing to identify performance concerns with contract providers, VA could unknowingly be receiving sub-standard service from these contractors, continue to receive services from these contract providers that do not meet the needs of the VAMCs, and risk patient safety problems when these contracts are extended for additional years. VA Training Does Not Adequately Prepare CORs to Monitor Contract Providers' Performance: CORs from the four VAMCs we visited noted weaknesses in VA's COR training courses and our own analysis of these courses confirmed these limitations. Specifically, over half of the 40 CORs that completed our data collection instrument at the four VAMCs we visited responded that either their COR training did not prepare them for their role as a COR or were neutral on whether or not this training was helpful preparation. In addition, CORs for 8 of the 12 contracts we reviewed did not find the required COR training helpful or applicable to VA clinical contracting. Several of the CORs monitoring the 12 selected contracts and CORs managing other contracts at the four VAMCs we visited stated that VA's required COR training is focused on Department of Defense contracting and is not tailored to CORs managing clinical contracts at VA.[Footnote 32] For example, one COR stated that the training covered very broad areas of contracts and did not include specific information on which kinds of contracts need detailed quality assurance plans or information on how to manage a clinical contract rather than a supply contract. In addition, a few CORs stated that the instructors for their training courses had limited knowledge of clinical contracting. VA requires CORs to complete training courses to obtain the Federal Acquisition Certification (FAC) for CORs or FAC-COR.[Footnote 33] These courses are developed and offered by VA's Acquisition Academy and can be taken both in person and through instructor-led, web-based courses. These courses generally aim to provide individuals with the knowledge and tools to carry out the COR responsibilities, which include the monitoring of clinical contractors' performance. Previously we found that well-designed training and development programs are linked to both agency goals and to the organizational, occupational, and individual skills and competencies needed for the agency to perform effectively.[Footnote 34] We reviewed the content of the 32-hour FAC-COR Level II course administered by the VA Acquisition Academy and found that this course has several limitations in preparing CORs to manage clinical contracts in VAMCs. * Focused on contracts that buy goods, not services. The primary examples used in the course do not include a discussion of clinical contracts at VA and instead walk students through the contracting process using examples such as replacing carpet, a large computer equipment purchase, and soup contents and production. There were no examples focused on how to evaluate or measure the quality of services provided by a contract provider in a VAMC clinical setting. By focusing on the types of examples in the course content, CORs did not have the opportunity to learn and practice the skills necessary for developing and managing service contracts that do not focus on easily quantifiable elements and involve complex assessments of the performance of clinicians. * Included little information on monitoring responsibilities. The course content includes limited information for CORs on post-award monitoring responsibilities for clinical contracts and instead is heavily weighted to discussing the pre-award development of a contract. By failing to incorporate a robust discussion of how CORs should be monitoring contract providers in the course, VA has not provided these critical personnel with a solid foundation for how to conduct their responsibilities for monitoring the performance of contract providers serving in clinical roles at VAMCs. To supplement the required FAC-COR Level II course, VA's Medical Sharing Office recently developed and implemented an 8-hour training course for CORs managing clinical contracts. However, VA does not currently require this course be completed by all CORs managing clinical contracts.[Footnote 35] This course covers primarily pre- award contract development responsibilities of CORs and does not include any significant information on the post-award monitoring responsibilities of CORs managing clinical contracts. Without developing required training on clinical contract monitoring, VA cannot ensure that all CORs are receiving adequate information on how to carry out this critical contract management responsibility and cannot ensure that CORs are well trained in how to monitor contract providers. This training places VA at risk of having CORs miss performance weaknesses by clinical contractors that could affect patient safety, and extending clinical contracts for contract providers who do not help VAMCs effectively meet their clinical staffing needs or provide high-quality care to veterans. VA Central Office Oversight of VAMC Clinical Contract Monitoring is Limited: VA has not established a robust method for overseeing the monitoring of clinical contractors by COs and CORs throughout VA's health care system. Standards for internal control in the federal government state that agencies should design internal controls that assure ongoing monitoring occurs in the course of normal operations, is continually performed, and is ingrained in agency operations.[Footnote 36] VA's primary oversight entity for health care contracting activities, the VHA Procurement and Logistics Office, has a limited role in overseeing the monitoring actions of COs and CORs once a contract has been approved and initiated at a VAMC. The VHA Procurement and Logistics Office conducts limited oversight of contracting activities throughout the VA health care system through its SAOs and Procurement Operations Office.[Footnote 37] * Service Area Offices. According to officials from the three SAOs, the role of the three SAOs in clinical contract monitoring is limited to an audit of the records COs maintain in VA's electronic Contract Management System. These reviews focus only on the completeness of COs' electronic contracting files--including documentation that a COR with current training records was assigned to the contract. SAO electronic Contract Management System audits do not include any reviews of CORs' monitoring of clinical contractors. * Procurement Operations Office. The Procurement Operations Office is the only VHA Procurement and Logistics Office entity responsible for overseeing the monitoring activities of CORs; however, the reviews conducted by this office are limited to a remote electronic documentation review of a small sample of COR files.[Footnote 38] Officials from the Procurement Operations Office told us that to select COR files for these reviews, a Procurement Operations Office staff member aims to select 25 COR files for active contracts per network contracting office--at most 2.1 percent of clinical contracts in an average VISN if all 25 selected COR files are for clinical contracts.[Footnote 39] VA officials told us that, while the Procurement Operations Office sets a goal to review COR files from two network contracting offices each month, since implementing the program in March 2013 these reviews have been completed in only four network contracting offices and none of these four offices have received feedback on the outcomes of these reviews as of August 2013.[Footnote 40] Officials added that this limited review schedule is due to only one Procurement Operations Office staff member being assigned to complete these reviews. These reviews also have a narrow focus on the completeness of COR files because the Procurement Operations Office staff member reviewing the files relies on a checklist to verify the presence or absence of required documentation of COR monitoring activities and does not review the quality of information contained within CORs records.[Footnote 41] The limited review schedule and narrow focus on file completeness do not allow the Procurement Operations Office to comprehensively assess the monitoring activities of COs and CORs throughout VA's health care system, as recommended by federal internal control standards. Without a robust monitoring system in place, VA cannot reasonably assure that all CORs in all VAMCs are maintaining the proper records of their efforts to monitor the activities of clinical contractors caring for veterans. Conclusions: Clinical contracts serve an important role in helping to ensure that VA can provide health care to our nation's veterans. Contract providers allow VAMCs and CBOCs to supplement the capacity of VA- employed providers and the ability to provide additional services at these VA facilities. However, VA must maintain robust monitoring of these contract providers to ensure they provide high quality care to veterans and fulfill the responsibilities of their contracts. We identified weaknesses in four areas that limit VA's ability to effectively monitor clinical contractors and provide support to VAMC- based CORs and contracting officials responsible for conducting these critical reviews. First, the lack of available tools such as templates for common types of clinical contracts requires COs and CORs to rely on less consistent sources for constructing critical performance requirements in clinical contracts. As a result, most clinical contracts we reviewed were insufficiently detailed in their descriptions of contract providers' performance requirements and these performance requirements were not consistently applied throughout our sample of clinical contracts. Without making tools such as templates available to COs and CORs, VA cannot ensure that all clinical contracts include performance requirements in key categories necessary to VA's operations and patient care, such as credentialing and privileging and medical record documentation. Second, VA lacks guidance for COs and CORs that defines the types of performance standards and key elements of performance expected to be included in clinical contracts. Without access to this type of guidance, VA cannot ensure that contracting officials have the ability to establish clear and measurable performance standards. Consistency in performance measurement and attention to key elements--including clearly stating performance standards, and ensuring these performance standards can be accurately and consistently measured--is critical to VA's efforts to ensure that all contract providers are monitored appropriately and can be held accountable for providing quality patient care. Third, two significant challenges CORs face in the monitoring of clinical contracts--COR workload and training--if left unresolved, will likely have continuing negative impacts on the quality of the monitoring that CORs are able to perform. Better guidance to VAMCs on how to determine an appropriate COR workload would allow CORs to fulfill key functions required to effectively monitor clinical contractors' performance. Currently, VA guidance is silent on how to determine whether or not a potential COR has the time to take on the critical responsibilities of monitoring contract providers. With the current practice of assigning COR duties to VAMC staff as a collateral duty, it is critical that VA define how VAMC staff should allocate COR duties to ensure that contracts are monitored effectively. In addition, CORs are receiving insufficient training to prepare them for monitoring clinical contracts. Currently, CORs for clinical contracts cannot apply the current training to their duties because it focuses on purchasing supplies rather than on acquiring clinical care services. While VA has undertaken efforts to provide supplemental training to CORs on clinical contracts, this additional training fails to address how CORs should monitor clinical contracts and instead focuses on the development of contracts. Without ensuring that COR training addresses all of the critical monitoring functions CORs perform, VA cannot ensure that all CORs are approaching this role with adequate preparation and skills and VA risks failing to identify weaknesses in contract providers' performance that could affect patient safety. Finally, VA Central Office currently provides limited oversight of clinical contract monitoring activities. Without a robust monitoring program that includes involvement from VA Central Office, VA does not have reasonable assurance that all CORs in VAMCs throughout VA's health care system are properly monitoring clinical contractors and is missing the opportunity to identify system-wide limitations in the monitoring of these critical contract providers. Recommendations for Executive Action: To improve the monitoring and oversight of clinical contracts, we recommend that the Secretary of Veterans Affairs direct the Under Secretary for Health to take the following five actions: * Develop and disseminate tools, such as standard templates, for the most common types of clinical contracts in VA's health care system. Such tools should include performance requirement statements covering key categories of VA health care policy and guidance--such as credentialing and privileging, provider qualifications, and expectations for compliance with critical VA policies and medical record documentation requirements. * Develop and issue guidance on the performance standards that could be included in common types of clinical contracts--including CBOC, specialty, and temporary clinical provider contracts--to ensure that these performance standards are clearly stated in the contracts and have measurable targets for assessing contract provider performance. * Revise current standard operating procedures for CORs to provide guidance on the number of contracts, based on size and complexity, each COR should manage to ensure that all CORs maintain a workload that allows them to fulfill their duties as a COR and their primary position responsibilities. * Modify existing COR training to ensure it includes examples and discussion of how to develop and monitor service contracts--including contracts for the provision of clinical care in VAMCs. * Increase SAO and VA Central Office oversight of COs and CORs by ensuring that post-award contracting files are regularly reviewed for all network contracting offices. Agency Comments: VA provided written comments on a draft of this report, which we have reprinted in appendix II. In its comments, VA generally agreed with our conclusions, concurred with our five recommendations, and described the agency's plans to implement each of our recommendations. In its comments, VA stated that to address our first and second recommendations, VHA's Medical Sharing Office will work with VHA's Patient Care Services to update and complete templates for clinical contracts so that they include performance requirement statements covering key categories included in our review. To address our third recommendation, VA noted that VHA's Procurement Policy and Operations Offices will collaborate to revise existing COR standard operating procedures to include guidance on the number of contracts, based on size and complexity, that each COR should manage to ensure that all CORs maintain a workload that allows them to fulfill their duties as a COR and their primary position responsibilities. To address our fourth recommendation, VA stated that VHA's Medical Sharing Office will provide training to CORs that covers post-award COR monitoring responsibilities. We support VA's efforts to provide this training content to CORs, especially those that monitor clinical contracts, but encourage the agency to also modify the content of its existing required COR training to ensure that CORs have the adequate preparation and skills to effectively monitor contract providers. Finally, to address our fifth recommendation, VA noted that VHA's Procurement Policy and Operations offices will collaborate to revise existing COR standard operating procedures to include Service Area Office and CO oversight to ensure that regular reviews take place for post-award contract files from all network contracting offices. We are sending copies of this report to the Secretary of Veterans Affairs, appropriate congressional committees, and other interested parties. In addition, the report is available at no charge on the GAO website at [hyperlink, http://www.gao.gov]. If you or your staffs have any questions about this report, please contact me at (202) 512-7114 or at williamsonr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III. Signed by: Randall B. Williamson: Director, Health Care: List of Requesters: The Honorable Patty Murray: Chairman: Committee on the Budget: United States Senate: The Honorable Bernard Sanders: Chairman: The Honorable Richard Burr: Ranking Member: Committee on Veterans' Affairs: United States Senate: The Honorable Jeff Miller: Chairman: The Honorable Michael Michaud: Ranking Member: Committee on Veterans' Affairs: House of Representatives: The Honorable Mike Coffman: Chairman: Subcommittee on Oversight and Investigations: Committee on Veterans' Affairs: House of Representatives: The Honorable Bill Johnson: House of Representatives: [End of section] Appendix I: Objectives, Scope and Methodology: This appendix describes the information and methods we used to examine: (1) the extent to which the Department of Veterans Affairs (VA) establishes complete performance requirements for clinical contractors; (2) the extent to which VA clinical contracts include clear and measurable performance standards for assessing whether or not clinical contractors met the acceptable quality levels defined in selected contracts; (3) challenges VA staff encounter in monitoring clinical contractors' performance; and (4) the extent to which VA Central Office provides oversight of VA staff responsible for monitoring clinical contracts. Specifically, we discuss our methods for selecting VA medical centers (VAMC) and network contracting offices to review; identifying appropriate VA Central Office Officials to interview; selecting clinical contracts for review and assessment;[Footnote 42] assessing the performance requirements included in our selected contracts; assessing the performance standards included in our selected contracts; administering a data collection instrument to contracting officer's representatives (COR) at our selected VAMCs; and evaluating VA's COR training. Site Selection Methodology and Interviews with VAMC and Network Contracting Office Officials: We conducted four site visits to VAMCs to obtain the perspectives of VAMC officials responsible for monitoring the day-to-day activities of clinical contractors. To identify VAMCs for our site visits, we: * Selected VAMCs that were located in different Veterans Integrated Service Networks (VISN) to ensure that our selected VAMCs varied in their geographic locations and reported to different VISN management officials; * Ensured that our selected VAMCs had a variety of active clinical contracts in place to allow for a variety of COR perspectives on clinical contractor monitoring; and: * Selected at least one VAMC from each of VA's three Service Area Offices (SAO)--the regional contract management entities created to oversee the activities of the 21 network contracting offices and the contracting officers (CO) and supervisors that work within them. [Footnote 43] Using these criteria, we selected four VAMCs to visit during our field work located in Lebanon, Pennsylvania; Minneapolis, Minnesota; Nashville, Tennessee; and Seattle, Washington. During our site visits to these locations, we interviewed each VAMC's leadership team; CORs responsible for managing all active clinical contracts; quality management staff; and the CORs and medical directors responsible for overseeing the clinical contractors for a select sample of clinical contracts. We spoke with these officials about a variety of topics-- including COR training and workload, monitoring procedures for clinical contractors, and their role in developing performance requirements and performance standards for clinical contracts. In addition, we spoke with officials from the four network contracting offices responsible for managing the contracting activities of our selected VAMCs to discuss how COs and management staff manage the monitoring of clinical contractors and interact with CORs at VAMCs. Information obtained from our visits to selected VAMCs and interviews with selected network contracting offices cannot be generalized to all VAMCs and network contracting offices throughout VA's health care system, but provide important insights. Interviews with VA Central Office Officials: We also interviewed VA Central Office officials responsible for developing policies and procedures for VA clinical contracting activities and overseeing the actions of CORs and COs throughout VA's health care system. We spoke with the following offices within the Veterans Health Administration (VHA) Procurement and Logistics Office: (1) the Medical Sharing Office; (2) the Procurement Audit Office; (3) the Procurement Operations Office; and (4) all three SAOs. Clinical Contract Selection and Analyses: To assess the monitoring of clinical contracts at VA, we reviewed the contracts and accompanying COR documentation for a sample of clinical contracts from the four VAMCs we visited during our field work. Contract Selection: To select our sample of clinical contracts, we reviewed a list of all active clinical contracts for our selected VAMCs provided by the network contracting offices responsible for overseeing the contracting actions of the selected VAMCs. For each VAMC we visited, we selected three contracts to review in detail--one community-based outpatient clinic (CBOC) contract, one specialty care contract, and one temporary clinical provider contract.[Footnote 44] (See table 6.) Table 6: Contract Types Reviewed at Each Selected VA Medical Center (VAMC): Contract type: Community-based outpatient clinic (CBOC) contracts; Description: These contracts can be used to acquire all the services of a CBOC located in a community that is geographically separate from a VAMC--including the physical location of the clinic and all required clinical personnel, such as primary care and mental health providers. CBOC contracts are typically executed for a 1-year base term with four 1-year options to extend the contract. Contract type: Specialty contracts; Description: These contracts are used to acquire the services of clinical specialists that work within an existing VAMC, such as cardiologists and anesthesiologists. Specialty contracts are typically executed for a 1-year base term with four 1-year options to extend the contract. Contract type: Temporary clinical provider contracts; Description: These contracts are used to acquire the services of clinical providers on a temporary basis and their terms are typically for 1 year or less. Source: GAO. [End of table] Analyses of Performance Requirements: To analyze the performance requirements included in our 12 selected clinical contracts, we identified six performance requirement categories for this analysis by reviewing: (1) VA Acquisition Regulation (VAAR);[Footnote 45] (2) VA policy, guidance, and training documents;[Footnote 46] and (3) The Joint Commission's hospital accreditation standards.[Footnote 47] We also verified these categories with officials from the Medical Sharing Office, Procurement Operations Office, and Procurement Audit Office to ensure they were an accurate reflection of performance requirements that should be included in VA clinical contracts. During our assessment of the 12 selected contracts, we analyzed the performance requirements included in each clinical contract's statement of work against these six performance requirement categories and determined whether or not performance requirements existed that matched each categories' required components. (See table 7.) Results of our reviews were recorded as complete, partial, or incomplete based on whether the contract's performance requirements covered all, some, or none of the specific components reviewed within each category. Table 7: Performance Requirement Categories and Required Components: Type of provider or care; Required components of performance requirements: * Licensure requirements for all clinicians serving under the contract; * Description of the specialty area covered under the contract; * A listing of key personnel serving under the contract and rules for substituting personnel if necessary. Credentialing and privileging; Required components of performance requirements: * Use of VA's electronic credentialing and privileging system; * Requirement for all personnel serving under the contract to be credentialed and privileged by VA staff at the initiation of the contract; * Requirement for all personnel serving under the contract to have their privileges renewed at the start of each new contract period. Clinical practice standards; Required components of performance requirements: * Compliance with VA medical staff bylaws and VAMC policies; * Adverse event and patient complaint procedures; * Compliance with applicable accreditation entity standards. Medical record documentation; Required components of performance requirements: * Entry of pertinent information into VA's electronic medical record system; * Response to consult requests, referrals, and other communication in VA's electronic medical record system; * Rules for the timely entry of information into VA's electronic medical record system. Business processes; Required components of performance requirements: * Invoicing policies and procedures; * Specification of work hours and time card procedures; * Personnel security requirements each clinical contractor must uphold. Access to care[B]; Required components of performance requirements: * Adherence to VA's scheduling policies and timeliness goals; * Specification of clinic hours of operation; * Requirements for the types of providers that must be on site for the operation of the clinic. Source: GAO (analysis); VA (information); The Joint Commission (information). [A] We identified these performance requirement categories by reviewing the VA Acquisition Regulation (VAAR); VA policy, guidance, and training documents; and The Joint Commission's hospital accreditation standards. Officials from the VHA Office of Procurement and Logistics verified these six performance requirement categories. [B] We added access to care as a performance requirement category for CBOC contracts because VA policy requires CBOCs to comply with the same timeliness goals required of VAMCs. [End of table] Analyses of Performance Standards: To analyze the performance standards included in our 12 selected clinical contracts, we reviewed whether each contract contained monitoring plan provisions that included performance standards in five of the same six categories we identified were vital to VA operations during our review of performance requirements. We did not include the business processes category in our analysis of performance standards because these requirements do not require performance standards for their validation and assessment. For the 12 selected clinical contracts, we grouped performance standards contained in each contract's monitoring plan under the five performance requirement categories used for this analysis--type of provider or care, credentialing and privileging, clinical practice standards, medical record documentation, and access to care (for CBOC contracts only). We analyzed these performance standards using criteria previously identified by GAO. Specifically, that performance standards should have several attributes--including being clearly stated and using measurable targets.[Footnote 48] We assessed each of the 12 selected contracts from the four VAMCs we visited to determine if the performance standards included in their monitoring plans met these two attributes of successful performance standards. To determine whether performance standards were clearly stated, two analysts independently determined whether or not each performance standard had specified a desired outcome that was clearly stated and understandable. To determine whether performance standards used measurable targets, two analysts independently determined whether or not each performance standard had a numerical target and included information on the frequency and calculation method for this target. Each analyst entered results as complete, partial, or incomplete for each standard and entered a reason for their determination if the result was partial or incomplete. A third analyst reviewed these assessments and reconciled any differences that arose. Data Collection Instrument Administered to CORs: We administered a data collection instrument to the 40 CORs responsible for managing a clinical contract at the four VAMCs we visited to determine their perspectives on COR training and workload, and to gather information on whether or not they were serving as a COR as a collateral duty. The data collection instrument included questions on each CORs highest training level, assessment of VA COR training, amount of time devoted to COR duties, and primary position title and description. Each COR completed this data collection instrument in the presence of a GAO analyst during our field work at each of our four selected VAMCs. Evaluation of VA COR Training: To evaluate VA's required COR training, three GAO analysts attended an offering of VA's COR training in September 2012. These analysts reviewed the content of this training to determine if it met previously identified criteria for successful training--that well- designed training and development programs are linked to both agency goals and to the organizational, occupational, and individual skills and competencies needed for the agency to perform effectively. [Footnote 49] To determine if these criteria were met, we reviewed the content of this course to determine if it included information that was linked to VA's agency goals and the organization, occupational, and individual skills needed for a COR monitoring clinical contractors in a VAMC. [End of section] Appendix II: Comments from the Department of Veterans Affairs: Department Of Veterans Affairs: Washington, DC 24420: October 17, 2013: Mr. Randall B. Williamson: Director, Health Care: U.S. Government Accountability Office: 441 G Street, NW: Washington, DC 20548: Dear Mr. Williamson: The Department of Veterans Affairs (VA) has reviewed the Government Accountability Office's (GAO) draft report, "VA Health Care: Additional Guidances Training, and Oversight Needed to improve Clinical Contract Monitoring (GAO-14-54). VA generally agrees with GAO's conclusions and concurs with GAO's five recommendations to the Department. The enclosure specifically addresses GAO's five recommendations and provides an action plan for each. VA appreciates the opportunity to comment on your draft report. Sincerely, Signed by: Jose D. Riojas: Chief of Staff: Enclosure: Department of Veterans Affairs (VA) Response to Government Accountability Office (GAO) Draft Report WA Health Care: Additional Guidance, Training, and Oversight Needed to improve Clinical Contract Monitoring" (GAO-14-54): GAO Recommendation 1: To improve the monitoring and oversight of clinical contracts, GAO recommends that the Secretary of Veterans Affairs direct the Under Secretary for Health to take the following five actions: Recommendation 1; Develop and disseminate tools, such as standard templates, for the most common types of clinical contracts in VA'S health care system. Such tools should include performance requirement statements covering key categories of VA health care policy and guidance—-such as of credentialing and privileging, provider qualifications, and expectations for compliance with critical VA policies and medical record documentation requirements. VA Comment; Concur. The Veterans Health Administration's (VHA) Medical Sharing Office (MSO) will work with VHA's Patient Care Services (PCS) to update existing clinical performance work statement (PWS) templates and complete the remaining clinical PWS templates. The template will include performance requirement statements covering key categories including credentialing and privileging, provider qualifications, and expectations for compliance with critical VA policies and medical record documentation requirements. Anticipated completion date is July 31, 2014. Recommendation 2: Develop and issue guidance on the performance standards that could be included in common types of clinical contracts—-including CBOC, specialty, and temporary Clinical provider contracts—-to ensure that these performance standards are clearly stated and have measurable targets For assessing contract provider performance. VA Comment: Concur. VHA'S MSO will work with PCS la update existing Clinical PINS templates and complete the remaining clinical PWS templates. Anticipated completion date is July 31, 2014. Recommendation 3: Revise current standard operating procedures for CORs to provide guidance on the number of contracts, based on size and complexity, each COR should manage to ensure that all CORs maintain a workload that allows them to fulfill their duties as a COR and their primary position responsibilities. VA Comment: Concur. VHA's Procurement Policy Office in collaboration with VHA's Procurement Operations Office, will revise existing VHA Contracting Officer Representative (COR) standard operating procedures (SOP) to include guidance on the number of contracts, based on size and complexity, that each COR should manage to ensure that all CORs maintain a workload that allows them to fulfill their duties as COR and their primary position responsibilities. Anticipated completion date is July 31, 2014. Recommendation 4: Modify existing COR training to ensure it includes examples and discussion of how to develop and monitor service contracts—-including contracts for the provision of clinical care in VAMCs. VA Comment; Concur. VHA's MSO will deploy a new Tier IV Target Customer Training. This training will cover the fundamentals for contract performance surveillance; invoice processing, documentation; and working with contracting officers {CO} in the post administration for each service. Anticipated completion date is July 31, 2014. Recommendation 5. Increase SAO and VA Central Office oversight of COs and CORs by ensuring that post-award contracting files are regularly reviewed for all network contracting offices. VA Comment: Concur. VHA's Procurement Policy Office in collaboration with VHA's Procurement Operations Office will revise existing VHA COR SOP to include Service Area Office and CO oversight to ensure post- award contracting files are regularly reviewed for all network contracting offices. Anticipated completion dale is July 31. 2014. [End of section] Appendix III: GAO Contact and Staff Acknowledgments: GAO Contact: Randall B. Williamson, (202) 512-7114 or williamsonr@gao.gov: Staff Acknowledgments: In addition to the contact named above, Marcia A. Mann, Assistant Director; A. Elizabeth Dobrenz; Pamela Dooley; Carolyn Garvey; Katherine Nicole Laubacher; Laurie Pachter; Said Sariolghalam; Jennifer Whitworth; and Malissa G. Winograd made key contributions to this report. [End of section] Footnotes: [1] VA's health care system includes 152 VAMCs. In addition to VAMCs, VA operates over 800 CBOCs that are located in areas surrounding VAMCs and that provide primary care and some specialty care services that do not require a hospital stay. CBOCs help reduce the travel time of veterans seeking treatment for outpatient care. [2] See Department of Veterans Affairs Office of Inspector General, Audit of Veterans Integrated Service Network Contracts, 10-01767-27 (Washington, D.C.: Dec. 1, 2011) and Department of Veterans Affairs Office of Inspector General, Audit of VA Electronic Contract Management System, 08-00921-181 (Washington, D.C.: July 30, 2009). In the 2011 report the VA Office of the Inspector General (OIG) found problems with the oversight of Veterans Integrated Service Network (VISN) contracts and a lack of tools to effectively manage VISN contracting activities. In the 2009 report the VA OIG found that VA's electronic Contract Management System is not used effectively and procurement information in the system is incomplete. In addition, the VA OIG found that VA cannot rely on the electronic Contract Management System to determine the total number of procurements accurately or the total estimated value of these procurements. As a result of the VA OIG's findings, we did not include data from VA's electronic Contract Management System, for example the number of clinical contracts across VA's health care system, in this report's findings. [3] VA established the VAAR to codify and publish uniform policies and procedures for VA's acquisition of supplies and services, which supplement the Federal Acquisition Regulations (FAR) applicable to all executive branch agencies. The FAR and agency supplements are codified in title 48 of the Code of Federal Regulation. [4] The Medical Sharing Office is a part of the Veterans Health Administration's (VHA) Procurement and Logistics Office. [5] VISNs oversee the day-to-day functions of VAMCs that are within their network. Each VAMC is assigned to a single VISN. [6] There are 21 network contracting offices within VHA that report directly to VA Central Office and manage all the contracting activities of a single VISN and all the VAMCs assigned to that VISN. [7] CBOC contracts can be used to acquire all the services of a CBOC located in a community that is geographically separate from a VAMC-- including the physical location of the clinic and all required clinical personnel, such as primary care and mental health providers. Specialty care contracts are used to acquire the services of clinical specialists that work within an existing VAMC, such as cardiologists and anesthesiologists. Both CBOC and specialty care contracts are typically executed for a 1-year base term with four 1-year options. Temporary clinical provider contracts are used to acquire the services of clinical providers on a temporary basis and the terms of these contracts are typically 1 year or less. VA refers to these temporary clinical providers as locum tenens. [8] See VA Acquisition Academy, Acquisition Planning Guide, Version 2, release 1 (Washington, D.C.: undated); VA Veterans Health Administration, Quality of Medical Services Performed Within VA Facilities by Academic Affiliates Under Contract, Directive 2009-040 (August 31, 2009); Veterans Health Administration, Veterans Health Care Service Standards, Directive 2006-041 (June 27, 2006); and Veterans Health Administration, VA Outpatient Scheduling Processes and Procedures, Directive 2010-027 (June 9, 2010). [9] The Joint Commission, Hospital Accreditation Standards 2012: Standards, Elements of Performance, Scoring, Accreditation Policies (Oakbrook Terrace, IL: 2012); and VAAR § 873.109(b). The Joint Commission is an independent organization that accredits and certifies health care organizations and programs in the United States. [10] The access to care performance requirement category was used for CBOC contracts because VA policy requires CBOCs to comply with the same timeliness goals required of VAMCs. [11] VA Directive 1663 requires all clinical contracts to have monitoring plans that ensure the appropriate quality assurance standards and data methods are in place, collection of these data is performed, and the performance of the clinical contractor is monitored. We did not include the business processes category in our assessment of performance standards because the administrative requirements covered by this category, such as invoicing procedures and time care submission, do not require performance standards for their validation. [12] COs are authorized to enter into, administer, or terminate contracts and make related determinations and findings. 48 C.F.R. §§ 1.602-1(a), 801.602. [13] CORs are appointed by the CO responsible for a clinical contract and are responsible for monitoring clinical contractor performance. [14] Within VA, VHA is the organization responsible for providing health care to veterans at medical facilities across the country. SAOs are VHA's regional contract oversight entities organized into three regions--East, West, and Central--that manage the contracting activities and acquisition personnel of six to eight VISNs each. SAOs report directly to the VHA Procurement and Logistics Office and do not report to VISN or VAMC directors. [15] While network contracting offices manage the contracting activities of a single VISN, they are managed by VA Central Office regional contracting management entities, SAOs, and have no managerial link to VISN leadership. [16] For the purposes of our report, we use the term medical directors to mean the same as individuals commonly known as service chiefs, service line directors, or section chiefs. [17] During the credentialing process, VAMC staff collect and review information such as a provider's professional training, malpractice history, peer references, and other components of professional background to determine whether providers have suitable abilities and experience for appointment to a VAMC's medical staff. During the privileging process, VAMCs determine which health care services--known as clinical privileges--the provider should be allowed to provide. After a provider is hired, the credentialing and privileging processes are repeated at least every 2 years. [18] These initial assessments of clinician competency are referred to as focused professional practice evaluations and typically occur for all VA clinical personnel who require clinical privileges at the time of their initial appointment to the medical staff. The process may include periodic chart review, direct observation, monitoring of diagnostic and treatment techniques, or discussion with other individuals involved in the care of patients. [19] These ongoing evaluations of clinicians are referred to as ongoing professional practice evaluations and are regularly conducted at least every 6 months on all VA clinical personnel who require clinical privileges. These ongoing professional practice evaluations include a number of reviews of professional practice trends that impact the quality of care provided by a clinician and patient safety. [20] The Medical Sharing Office does not review any contracts for nursing services. Nursing contracts are processed and reviewed by SAOs. [21] Results of our reviews were recorded as complete, partial, or incomplete based on whether the contract's performance requirements covered all, at least one, or none of the specific components reviewed within each category. [22] See Veterans Health Administration, Disclosure of Adverse Events to Patients, Handbook 1004.08 (Oct. 2, 2012). According to VHA, adverse events are untoward incidents, diagnostic or therapeutic misadventures, iatrogenic injuries, or other occurrences of harm or potential harm directly associated with care or services provided within the jurisdiction of VA's health care system. [23] See VHA Handbook 1100.19. VHA's electronic credentialing system must be used for credentialing all providers who are granted clinical privileges or credentialed for other reasons. According to VHA policy, the use of this system is necessary to reduce the potential for human error in the credentialing process. [24] See VHA Handbook 1100.19. [25] See Department of Veterans Affairs, Health Care Resources Contracting--Buying, Title 38 U.S.C. 8153, VA Directive 1663 (Aug. 10, 2006). Directive 1663 states that contracts for full-time providers must identify the key personnel proposed to provide the service and their qualifications. [26] 48 C.F.R. § 37.503(a). [27] See GAO, Executive Guide: Effectively Implementing the Government Performance and Results Act, [hyperlink, http://www.gao.gov/products/GAO/GGD-96-118] (Washington, D.C.: June 1996). Prior GAO work had determined that these performance standards: (1) demonstrate results by showing an organization's progress toward achieving an intended level of performance, (2) should be limited to core program activities to allow managers to make decisions without excess of data, and (3) should provide timely information in a format that helps managers make decisions to improve program performance. [28] See: GAO, Tax Administration: IRS Needs to Further Refine Its Tax Filing Season Performance Measures, [hyperlink, http://www.gao.gov/products/GAO-03-143] (Washington, D.C.: November 22, 2002). For the purposes of this report, we selected two attributes--clarity and measurability--to evaluate VA clinical contracts' performance standards due to their applicability to performance standards for smaller scale projects. We previously found several other attributes of successful performance standards, including: (1) objectivity, (2) reliability, (3) core program activities, (4) limited overlap, (5) balance, (6) government-wide priorities, and (7) linked to agency priorities. We did not include these additional seven attributes in our assessment of each selected contract's performance standards because they were designed to measure the success of broader agency-wide performance standards rather than those included in a single contract. [29] We did not include assessments of the business processes category in our final results because these requirements are administrative and performance standards are not necessary to determine success in this area. [30] See VA Directive 1663. [31] See Veterans Health Administration, Standard Operating Procedure: Contracting Officer Technical Representative, (May 20, 2011). See also Department of Veterans Affairs Directive 1663, Health Care Resources Contracting - Buying Title 38 U.S.C. 8153, (August 10, 2006). [32] The Defense Acquisition University develops most acquisition training courses, including those for CORs, and federal civilian agencies, such as VA, modify this training to meet their agency's needs. [33] There are three levels of FAC-COR certifications, which directly correlate with the years of COR's contracting experience. Specifically, the FAC-COR Level I certification is an 8-hour training and does not require previous experience as a COR, the FAC-COR Level II certification is 40 hours of training (Level I combined with an additional 32 hours of training) and requires 1 year of previous experience serving as a COR, and the FAC-COR Level III certification is 60 hours of training and requires 2 years of previous experience serving as a COR. [34] GAO, Human Capital: A Guide for Assessing Strategic Training and Development Efforts in the Federal Government, [hyperlink, http://www.gao.gov/products/GAO-04-546G] (Washington, D.C.: March 2004). [35] In June 2013, the Chief of the Medical Sharing Office reported that VA had developed a proposal that makes this training course a requirement for all CORs of clinical contracts and submitted it to the agency's labor relations partners for approval. However, there is no target date for completing this review and instituting this requirement. [36] See GAO, Internal Control: Standards for Internal Control in the Federal Government, [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1] (Washington, D.C.: November 1999). [37] In June 2013, officials from the Medical Sharing Office reported that they are beginning to assess whether they can provide oversight to the post-award monitoring of COs and CORs; however, these officials noted that they do not currently have the necessary staff support to conduct post-award oversight. [38] Because COR files are not maintained in VA's electronic Contract Management System, the CORs for the contracts selected to be part of these reviews must send copies of their files by email to the Procurement Operations Office staff member conducting the review. [39] Officials from the Procurement Operations Office told us that the actual number of files being reviewed has been typically around 21. COR files selected for these electronic documentation reviews may be for any active contract over $250,000 that originates in the network contracting office subject to the review. These contracts can include clinical contracts, supply contracts, construction contracts, and any other type of active contract. [40] An official from the Procurement Operations Office told us the office has set a goal to complete a total of 8 to 10 of these electronic documentation reviews of COR files by the end of fiscal year 2013. [41] The file reviews assess the presence of documentation in seven key areas: (1) COR training and delegation; (2) the contract and any modifications made to the contract; (3) records of inspections they have completed and any actions taken as a result of these inspections; (4) records of technical and financial reports--including copies of invoices and purchase orders; (5) copies of all required annual contractor performance reviews and security documents; (6) copies of all communications with the contractor and CO; and (7) verification that all contract providers have completed required VHA training. [42] Clinical contracts at VA are used to acquire the services of clinical personnel, such as physicians, pharmacists, and nurses. These contracts can be used to fill vacancies for clinicians in specialties that are difficult to recruit, supplement existing VAMC capacity by providing additional clinicians in high-volume areas where VA also manages a staff of its own employees, or fill critical staffing vacancies on a long-or short-term basis. [43] VHA created three SAOs--East, West, and Central--to manage the contracting activities of six to eight VISNs each. SAOs report directly to the VHA Procurement and Logistics Office in VA Central Office. [44] VA refers to these temporary clinical providers as locum tenens. These temporary clinical providers work in a VA facility on a temporary basis in order to fill critical staffing vacancies. [45] See 48 C.F.R. § 873.109(b), which provides that contracts for health care resources should, in most instances, specify qualifications or limitations, such as time limits for service delivery, medical certification or credentialing requirements, and small business or other socio-economic preference. [46] See VA Acquisition Academy, Acquisition Planning Guide, Version 2, release 1 (Washington, D.C.: undated); VA Veterans Health Administration, Quality of Medical Services Performed Within VA Facilities by Academic Affiliates Under Contract, Directive 2009-040 (August 31, 2009); Veterans Health Administration, Veterans Health Care Service Standards, Directive 2006-041 (June 27, 2006); and Veterans Health Administration, VA Outpatient Scheduling Processes and Procedures, Directive 2010-027 (June 9, 2010). [47] The Joint Commission, Hospital Accreditation Standards 2012: Standards, Elements of Performance, Scoring, Accreditation Policies (Oakbrook Terrace, IL: 2012. The Joint Commission is an independent organization that accredits and certifies health care organizations and programs in the United States. [48] See: GAO, Tax Administration: IRS Needs to Further Refine Its Tax Filing Season Performance Measures, [hyperlink, http://www.gao.gov/products/GAO-03-143] (Washington, D.C.: November 22, 2002). For the purposes of this report, we selected two attributes--clarity and measurability--to evaluate VA clinical contracts' performance standards due to their applicability to performance standards for smaller-scale projects. We previously found several other attributes of successful performance standards, including: (1) objectivity, (2) reliability, (3) core program activities, (4) limited overlap, (5) balance, (6) government-wide priorities, and (7) linkage. We did not include these additional seven attributes in our assessment of each selected contract's performance standards because they were designed to measure the success of broader agency-wide performance standards rather than those included in a single contract. [49] GAO, Human Capital: A Guide for Assessing Strategic Training and Development Efforts in the Federal Government, [hyperlink, http://www.gao.gov/products/GAO-04-546G] (Washington, D.C.: March 2004). [End of section] GAO’s Mission: The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO’s website [hyperlink, http://www.gao.gov]. Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to [hyperlink, http://www.gao.gov] and select “E-mail Updates.” Order by Phone: The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO: Connect with GAO on facebook, flickr, twitter, and YouTube. Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts. Visit GAO on the web at [hyperlink, http://www.gao.gov]. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; E-mail: fraudnet@gao.gov; Automated answering system: (800) 424-5454 or (202) 512-7470. Congressional Relations: Katherine Siggerud, Managing Director, siggerudk@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, DC 20548. Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, DC 20548. [End of document]