KPMG Consulting LLP, B-290716; B-290716.2, September 23, 2002 * REDACTED DECISION

KPMG Consulting LLP, B-290716; B-290716.2, September 23, 2002 * REDACTED DECISION

DIGEST

Attorneys

DECISION

KPMG Consulting LLP protests an award to International Business Machines Corporation (IBM) under request for quotations (RFQ) No. MDA210-02-T-0003, issued by the Department of Defense (DOD), Defense Finance and Accounting Service (DFAS), for the development and maintenance of a DOD-wide Financial Management Enterprise Architecture (FMEA) for the Office of the Under Secretary of Defense (Comptroller).

We deny the protest.

The RFQ, issued November 29, 2001, contemplated a single-award blanket purchase agreement (BPA) to cover orders under General Services Administration (GSA) Federal Supply Schedule (FSS) contracts for 5 years, or until expiration of the GSA schedule, whichever comes first.

The RFQ set forth a "best value" evaluation plan considering the following evaluation factors listed in descending order of importance: technical, management, past performance, and price. The RFQ stated that the technical factor "is more important than the other factors combined." RFQ at 9. Under the technical factor, the RFQ identified four equally weighted subfactors: (1) understanding the problem, (2) solution approach, (3) integrated solution, and (4) risk mitigation. Under the management factor, the RFQ identified three equally weighted subfactors: (1) management approach, (2) delivery schedule, and (3) personnel. /1/ RFQ at 5-6, 9. Under the price factor, the RFQ indicated that the cost of the initial task order, which constituted the largest portion of the FMEA, would be evaluated, although this was "not susceptible to a high degree of estimation, but primarily will consist of labor." A government estimate of labor hours for various categories was stated in the RFQ, and vendors were requested to submit "applicable proposed discounted labor rates" to be applied to these estimates to evaluate the vendors' prices. RFQ at 7, 12.

The RFQ, as amended, stated that the agency might require oral presentations, and that such presentations and corresponding materials would be considered by the agency in making a final award selection. However, oral presentations, if held, would not be considered discussions, and the agency reserved the right to make award without discussions. RFQ at 13 (added by amend. 0001).

The RFQ stated that "[r]esponses shall arrive by January 25, 2002." RFP amend. 0003 at 1. No late submission clause was included in the RFQ. Several vendors, including KPMG and IBM, submitted quotations by that date.

Also on January 25, a source selection plan (SSP) for this acquisition was approved identifying the source selection authority (SSA), the source selection evaluation board (SSEB) chairperson, and the other SSEB members. Agency Report, Tab M-2, Initial SSP (Jan. 25, 2002), at 2, app. A. On January 28, the SSEB convened to evaluate quotations. The SSEB evaluated each submission using evaluation worksheets provided in the SSP identifying the evaluation factors and subfactors stated in the RFQ, and providing evaluation criteria for each subfactor under the technical and management factors and for the past performance factor. The worksheets provided for assigning a numerical score for each criterion using a three-point scale with two points assigned if the quotation was "responsive," one point for quotations needing "clarification," and zero points for "nonresponsive" quotations. The worksheets also provided space for describing strengths and weaknesses. The worksheets provided for totaling the numerical scores under each subfactor for the technical and management factors and for the past performance factor, and assigning color/adjectival ratings-blue/exceptional, green/acceptable, yellow/marginal, and red/unacceptable--based on various point ranges stated for each subfactor or factor. Agency Report, Tab M-2, Initial SSP (Jan. 25, 2002), apps. C, D.

On February 5, the SSEB chairperson reported the evaluation results to the contracting officer. The SSEB had assigned numerical scores and color/adjectival ratings for each subfactor and factor as provided for in the SSP. KPMG's and IBM's quotations were both rated blue/exceptional overall with IBM's numerical score exceeding KPMG's by a very narrow margin. Agency Report, Tab J-2, Initial SSEB Report (Feb. 5, 2002) at 2.

Also on February 5, the Director of DFAS appointed as SSA a person different from the one identified in the initial SSP. /2/ However, the Director states that there was miscommunication regarding who should be appointed the SSA and that the individual appointed, though discussed as a potential SSA, had not been recommended for appointment. Agency Submission (Aug. 7, 2002), encl. 2, Affidavit of the DFAS Director. On February 22, the Director rescinded his initial appointment and appointed as SSA the Deputy Under Secretary of Defense (Financial Management), the same person identified as the SSA in the initial SSP. Id.; Agency Report, Tab T-3, SSA Appointment Memo (Feb. 22, 2002).

Subsequently, the SSA established a source selection advisory board (SSAB). Agency Submission (Aug. 7, 2002), encl. 1, Affidavit of SSA. On March 1, the SSEB chairperson and the contracting officer briefed the SSAB on the SSEB's evaluation. The contracting officer also identified the evaluated prices of the quotations--approximately $33 million for IBM's quote and approximately $20 million for KPMG's--and recommended award to KPMG. The SSAB reviewed the quotations and determined that material differences existed between them, and that the SSEB evaluation lacked sufficient detail to permit the SSAB to identify significant discriminators among the quotations or to assess their relative merit considering price and technical factors. On March 11, the SSAB recommended that the SSEB reevaluate quotations using evaluation worksheets with a larger numerical rating scale (i.e., 0-unacceptable, 1-marginal, 3-acceptable, and 5-exceptional), and provide additional narrative descriptions of strengths and weaknesses at the subfactor level. The contracting officer agreed with the SSAB and revised the SSP accordingly. Contracting Officer's Statement at 6; Agency Report, Tab I-3, Memo from SSAB to the Contracting Officer (Mar. 11, 2002); Tab I-4, Briefing Slides (Mar. 1, 2002).

On March 18, the contracting officer received an unsolicited notice from IBM that the vendor was removing a team member from its quotation. Contracting Officer's Statement at 6. The team member, Arthur Andersen LLP, had notified IBM that GSA had suspended Arthur Andersen from all contracting except ongoing subcontract work, and that, since an agreement for Arthur Andersen to be a subcontractor under IBM's quotation had not been executed prior to the suspension, Arthur Andersen was withdrawing from IBM's quotation. Agency Report, Tab O-8, Letter from Arthur Andersen to IBM (Mar. 18, 2002). IBM's notice to the agency included the following statements:

Please note there will be no impact to IBM's ability to perform in accordance with our previously submitted proposal. In particular, there will be no impact to our technical solution, price or schedule. The sole Arthur Andersen individual identified in the key personnel section of our proposal will be replaced with an individual of equal or superior qualifications.

In assembling our team, we deliberately sought a breadth and depth of talent including two "Big Five" [public accounting] firms, Arthur Andersen and KPMG LLP. /3/ As a result, the void created by the withdrawal of Arthur Andersen will be completely addressed by IBM and its other team members. In fact, we have conferred with our teammates and confirmed that the scope previously allotted to Arthur Andersen can be absorbed by IBM and its team members. Agency Report, Tab O-7, Letter from IBM to Agency (Mar. 18, 2002). The contracting officer forwarded IBM's notice to the SSEB without comment. Contracting Officer's Statement at 6.

On March 25, the SSEB completed the reevaluation requested by the SSAB and reported the results to the contracting officer. The SSEB assigned overall ratings of green/acceptable to both IBM's and KPMG's quotations. With one exception, each of these quotes received ratings of green/acceptable under every factor and subfactor.

The lone exception was that the SSEB rated KPMG's quotation yellow/marginal under the first technical subfactor--understanding the problem. Agency Report, Tab J-1, SSEB Memo to the Contracting Officer (Mar. 25, 2002), at 10. The SSEB stated that, although KPMG addressed certain aspects of that subfactor "very well," the quotation "did not evidence a thorough understanding of the complexity and scope of the problem . . . lacked information recognizing data standardization efforts and the relationship of the effort to the joint Operational Architecture, [and] was somewhat weak on describing the organizational complexities associated with the FMEA." Id. at 11. In contrast, the SSEB stated that IBM's quotation under this same subfactor demonstrated "an excellent understanding of the scope and complexities, both organizationally and functionally, associated with the effort," and addressed the "historical challenges of DOD in pursuing financial reform." Id. at 7.

The SSEB's general summary of the quotations reflects that both vendors were found to have relevant experience and "very acceptable" quotations. The SSEB did identify a significant difference between the two quotations. It found that KPMG's quote emphasized a "partnership" with the agency with several actions contingent upon agency approval, and focused on "system compliance and identification of systems that will be part of the FMEA solution." In contrast, the SSEB found that IBM's quote was "built on a clear understanding of the magnitude and complexity of the FMEA" and its impact on DOD, that IBM viewed the project as a "transformation" within the agency that was larger than merely redesigning current business processes, and that IBM expressed an intent to assume a "very strong leadership/'take charge' role" applying experience and knowledge from its own successful corporate transformation that IBM stated rivaled the scope and complexity of the FMEA. Id. at 7, 11.

On March 28, the SSAB began review of the SSEB reevaluation and the underlying quotations. That same day, the contracting officer briefed the SSA (with the SSAB in attendance) on the SSEB reevaluation. The contracting officer again recommended award to KPMG. Contracting Officer's Statement at 9-10.

On April 4, KPMG and IBM made oral presentations to the SSA and the SSAB. Following the oral presentations, the SSAB issued a memo to the SSA that summarized the SSEB evaluation, and identified areas where the SSAB's review showed significant differences between the two quotations. Contracting Officer's Statement at 10. The memo also presented the SSA with both the color/adjectival ratings and numerical scores from the SSEB's reevaluation, which indicated that the quotations had the same overall color/adjectival rating, green/acceptable, with similar overall numerical scores--KPMG's quotation received 3.54 on a scale of 5 and IBM's quotation received 3.48. Agency Report, Tab I-1, Memo from the SSAB to the SSA (Apr. 4, 2002), attach. 1, at 1-2. However, as detailed below, the SSAB determined that there were significant discriminators between the quotations in four areas.

The first discriminator, "understanding the problem," was evaluated as arising under the first technical subfactor of the same name. The SSAB considered this issue key to the successful development of the FMEA, since a complete understanding of the problem is critical to performing the subsequent tasks. Of the shortcomings apparent in KPMG's quotation, as identified in the SSEB's evaluation, the SSAB considered most noteworthy KPMG's shortcomings in understanding DOD's previous attempts at financial reform. Although KPMG had been assisting the agency for some time on related components of this problem, and thus should have the capability to demonstrate an understanding of the agency's financial management problems, its quotation did not identify all relevant components of the problem, and when it did generally identify a problem area, KPMG failed to elaborate on it. The SSAB considered this weakness to be exacerbated by KPMG's claim that it would improve upon the RFQ's required milestone/timeframe requirements for completing the FMEA. The SSAB considered the RFQ timeframes to be aggressive and that, considering the complexity of the problem, even the most capable team may not be able to meet those timeframes. The SSAB considered the disjoint--the incongruity between KPMG's shortcomings in identifying the problem and its claim that it would reduce timeframes--as evidencing that KPMG's quotation was unduly optimistic and did not evidence an understanding of the enormity and complexity of the agency's problems. Agency Report, Tab I-1, Memo from the SSAB to the SSA (Apr. 4, 2002), at 2-3.

In contrast, the SSAB found that IBM's quotation not only elaborated on issues that KPMG addressed only minimally, but also identified and discussed key points that KPMG missed; for example, IBM detailed the flaws in the current financial management processes, as well as the forces, both internal and external, that affect those processes and will affect attempts to change those processes. Id. at 3.

With regard to this discriminator, the SSAB determined that IBM demonstrated a firm understanding of the scope and difficulty of the problems, and KPMG did not. Given KPMG's apparent lesser understanding of the problem, the SSAB concluded that, compared to IBM, KPMG likely will "[r]equire more work on the part of the government to overcome the KPMG shortcomings, [r]equire more time 'getting up to speed' before KPMG develops concrete quotations for developing architectures, and [i]ncrease the risk KPMG will not develop a workable solution in the required timeframe." Id.

The second discriminator, "financial management transformation," was evaluated as arising under the first three technical subfactors--understanding the problem, solution approach, and integrated solution. The SSAB identified this difference between the vendors' plans for business process reengineering as being closely related to their identification of the problem. /4/ The SSAB recognized that DOD's current procedures, processes and systems are ineffective. The essential difference between the vendors' solutions is that KPMG's solution focuses on modernizing the agency's existing systems to achieve compliance with applicable standards, whereas IBM goes beyond that, addressing the extensive transformation that will be needed within DOD to make successful implementation of the system possible. The SSAB concluded that "KPMG['s] approach has less chance of success . . . produc[ing] the substantive reengineering [that DOD] needs in order to resolve the root causes of DOD's financial management problems." Id. at 3-4.

The third discriminator, "leading vice partnering," was evaluated as arising under the first management subfactor, management approach, and the second technical subfactor, solution approach. The SSAB stated that the FMEA effort is a "momentous undertaking, requiring an unprecedented effort by the contractor" to assess and recommend solutions to the agency's business and financial management processes. This represents a "financial management transformation" for which solution approaches that "rely heavily on involvement by government personnel are unlikely to be successful, because [DOD] lacks sufficient experience in tackling such problems," and because the "disparate DOD communities are typically resistant to working together and reluctant to compromise their requirements for the requirements of others."

The SSAB determined that KPMG's quotation relies on forming a "partnership" with DOD to develop the FMEA, which essentially relies on DOD to lead the transformation, rather than on outside expertise/leadership to overcome the entrenched communities within the agency. The SSAB found that this did not represent a dynamic approach that will push DOD to achieve goals that it has been unable to do in previous attempts.

On the other hand, the SSAB found that IBM's quotation was built upon the vendor's experience and lessons learned from that firm's earlier development of an enterprise architecture and transformation of its own and its customers' financial management systems, and providing the leadership for DOD to develop the FMEA. The SSAB characterized IBM's approach as one of "bringing in an outside consultant to tear apart an organization and rebuild it"--a "take charge" management approach, which the SSAB stated was "the approach needed here." In making this assessment, the SSAB disagreed with an SSEB assessment of risk that IBM's "take charge" approach so limited interaction with the agency as to be detrimental to DOD's interests, finding that the SSEB concern arose from the frequency of meetings contemplated in the quotation, rather than any actual exclusion of agency consultation. Indeed, consistent with the SSEB evaluation, the SSAB found that IBM proposed necessary "collaborative processes with DOD to include full-time representation from DOD's functional experts [and other] designated DOD points of contact." The SSAB believed that the SSEB's caution arising from its risk assessment kept the SSEB from rating IBM higher than it did and properly identifying a meaningful advantage of IBM's quotation. Id. at 4-5.

The fourth discriminator, personnel, was evaluated as arising under the third management subfactor of the same name, as well as under the price factor. The SSAB first noted a limitation in the price evaluation that, being based on the government's estimate of labor categories and hours, did not account for differences in the vendors' approaches. Nevertheless, the SSAB stated that analysis of the labor rates in each quotation provided significant information for evaluation. On average, KPMG's labor rates were substantially lower than competing quotes and the government estimate, whereas IBM's quote was in line with both. The SSAB considered two possible explanations for KPMG's substantially lower rate--"reduced overhead or less-qualified personnel." The SSAB determined that KPMG's facilities could not provide the level of savings in overhead costs to account for the large margin in labor rates, and therefore, KPMG's lower price indicated that the quotation must be based on providing "less-qualified personnel to perform the work." The SSAB believed that IBM's higher labor rates reflected IBM's experience with a similar type of efforts that recognized the need for more experienced, better qualified personnel. As part of this analysis, the SSAB also prepared a side-by-side comparison of the job description information in IBM's and KPMG's quotations, from which the SSAB determined that, in many areas, the differences in experience are "sizeable," and concluded that IBM will devote more experienced, and better qualified personnel "in virtually all job categories," thus increasing the likelihood of success at the quotation price. Id. at 5-6, attach. 2.

Here, too, the SSAB referred to the SSEB evaluation, which found that both vendors would provide adequate key management personnel and rated the quotations almost identically. While the SSAB did not disagree with the SSEB's description of key personnel qualifications, it determined that, aside from the fact that the SSEB did not have access to the personnel cost information in the vendors' quotations, the SSEB evaluation did not consider the overall level of experience and qualifications of offered personnel generally. The SSAB determined that the difference in labor costs and experience/qualification levels was consistent with the differences in the vendors' solution approaches, and that KPMG's greater reliance on DOD personnel for performance "shifts cost and risk for accomplishing the task to the government." Id. at 5-6.

In conclusion, the SSAB recommended award to IBM because, even at a significantly higher price, IBM's quotation represented the best value because, based on the SSAB's comparison of the significant discriminators between the quotations, the SSAB found that IBM's quote offers the team most capable of accomplishing the requirements and meeting the timeframes of the RFQ with the least risk of failure. Id. at 6-7.

On April 5, the SSA issued her source selection decision. The decision stated that it was based on the SSA's review of the vendors' quotations and oral presentations, the evaluation of the SSEB, and the SSAB Memo of April 4. The SSA also stated that she met with the members of the SSAB and discussed their views. Her decision analyzed the quotations of KPMG and IBM as follows:

I find the quotations of KPMG and IBM to be acceptable but fundamentally different. The KPMG proposal is built around systems compliance and bringing the department's financial management systems into compliance with law. The proposal describes substantial collaboration with the government to execute this approach. It is rooted in efforts to review individual systems and to conform them to federal accounting standards. It does not, in my judgment, adequately address the business process reengineering that is fundamental to this effort. The IBM proposal, on the other hand, recognizes that business process reengineering is fundamental to the development of successful enterprise architectures, addressing compliance as a necessary component. IBM's proposal utilizes proven enterprise architectures to enable the design of a more comprehensive DoD enterprise architecture solution in the available time. It does not rely on government leadership or collaboration but on its own experience gained from developing its own financial management architecture.

The quotations also reflect important differences in personnel, an element that was noted in the RFQ as being material to successful execution of program objectives. Although both quotations describe acceptable management personnel, the skill and experience levels of KPMG's personnel are generally less, often significantly so, than those of the proposed IBM personnel. This difference is consistent with the differences in the proposed labor rates for the two teams.

I find that IBM's direct experience in the development of enterprise architectures means less learning will be required by IBM to execute this requirement. I consider KPMG's collaborative approach to involve more risk, because the government does not have personnel with the expertise necessary to participate effectively in a collaborative approach. Because the department seeks improved efficiency and not just compliance with existing standards and laws, I find KPMG's approach less effective than IBM's and more likely to require additional effort and expense in order to obtain improved efficiency. I find that KPMG's less skilled and less experienced personnel will also likely result in additional cost by necessitating additional work.

While KPMG's proposed rates are initially attractive, I associate substantial risk with them. The government cannot uphold its end of a collaborative arrangement with KPMG, and this approach, coupled with less skilled and less experienced personnel, exposes the government to significant cost risk and unacceptable performance risk. IBM's proposal is significantly less risky. It offers direct and relevant experience in the development of large-scale enterprise architectures. It does not rely excessively on government experience and participation. It offers higher skilled and more experienced personnel at rates consistent with the government's independent estimate.

The lower risk of the IBM proposal comes at a substantially higher cost to the government. However, as noted in the evaluation factors, cost in this procurement is by far the least important of the evaluation factors, while the technical factor is more important than all of the other factors combined, including cost. The reason for this order of importance is the importance of this requirement to the department. The success of this undertaking is essential to the mission of the entire department and ultimately to the availability of resources to Warfighters. It is one of the Secretary's top priorities and is responsive to Congress' growing concern about how the department uses and controls its resources.

In view of this importance, I find that the higher cost of the IBM proposal is more than offset by the greater value of significantly reduced risk and greater chance for success offered by its Technical and Management proposal. I conclude that the proposal of the IBM team represents the best value for the government and that the BPA be awarded to IBM.

Agency Report, Tab H, Source Selection Decision, at 1-2.

On April 9, the agency offered the BPA to IBM, which IBM accepted. Following a debriefing, KPMG filed an agency-level protest. The agency denied the protest on June 10. This protest followed.

Under the FSS program, an agency is not required to issue a solicitation to request quotations, but rather may simply review vendors' schedules and, using business judgment to determine which vendor's goods or services represent the best value and meet the agency's needs at the lowest overall cost, may directly place an order under the corresponding vendor's FSS contract. 10 U.S.C. Sec. 2302(2)(C) (2000); OSI Collection Servs., Inc.; C.B. Accounts, Inc., B-286597.3 et al., June 12, 2001, 2001 CPD Para. 103 at 4. If, however, the agency issues an RFQ and thus shifts the burden to the vendors for selecting the items from their schedules, the agency must provide guidance about its needs and selection criteria sufficient to allow the vendors to compete intelligently. Where, as here, the agency intends to use the vendors' responses as the basis of a detailed technical evaluation and cost/technical trade-off, the agency has elected to use an approach that is more like a competition in a negotiated procurement than a simple FSS buy, and the RFQ is therefore required to provide for a fair and equitable competition. COMARK Fed. Sys., B-278343, B-278343.2, Jan. 20, 1998, 98-1 CPD Para. 34 at 4-5. While we recognize that the procedures of Federal Acquisition Regulation (FAR) Part 15, governing contracting by negotiation, do not govern competitive procurements under the FSS program, Computer Prods., Inc., B-284702, May 24, 2000, 2000 CPD Para. 95 at 4, where the agency has conducted such a competition and a protest is filed, we will review the record to ensure that the evaluation is reasonable and consistent with the terms of the solicitation and with standards generally applicable to negotiated procurements. OSI Collection Servs., Inc.; C.B. Accounts, Inc., supra at 4-5; Amdahl Corp., B-281255, Dec. 28, 1998, 98-2 CPD Para. 161 at 3.

KPMG alleges that IBM's removal of Arthur Andersen from its quotation cannot be accepted because it is a late proposal modification. The protester asserts that because this acquisition more closely resembles a competition in a negotiated procurement than a simple FSS buy, FAR Part 15 should be applied, in particular FAR Sec. 15.208, which permits consideration of quotations, revisions and modification received after the date for submission stated in the solicitation only in limited circumstances not present here. KPMG therefore asserts that IBM's late quotation modification should have been rejected, and thus IBM's quotation, based on teaming with Arthur Andersen, a suspended contractor, should be rejected.

It is well established that the standard for late quotations does not generally apply to requests for quotations. An RFQ, unlike a request for proposals (or an invitation for bids), does not seek offers that can be accepted by the government to form a contract. Rather, the government's purchase order represents the offer that the vendor may accept through performance or by a formal acceptance document. DataVault Corp., B- 248664, Sept. 10, 1992, 92-2 CPD Para. 166 at 2. It follows that language in an RFQ requesting quotations by a certain date cannot be construed as establishing a firm closing date for receipt of quotations, absent a late quotation provision expressly providing that quotations must be received by that date to be considered. Instruments & Controls Serv. Co., B-222122, June 30, 1986, 86-2 CPD Para. 16 at 3. An agency may consider "late" quotations or quotation modifications, so long as the award process has not begun and other offerors would not be prejudiced. Id.; ATF Constr. Co., Inc., B-260829, July 18, 1995, 95-2 CPD Para. 29 at 2.

Here, the RFQ did not contain a late quotation provision. Therefore, the agency was not required to reject a modification received after the date stated in the RFQ for submission of quotations. In the case of IBM's modification of its quotation, the agency received this modification several days after the SSAB had requested a reevaluation and prior to the SSEB's completion of that reevaluation, and prior to the oral presentations and the SSA's award decision. No competitor was prejudiced by the agency's acceptance of IBM's modification. Therefore, the agency could accept IBM's removal of Arthur Andersen from its quotation. /5/

KPMG also protests that the agency failed to evaluate the effect that removing Arthur Andersen had on IBM's quotation. The contributions of Arthur Andersen to IBM's quotation that the protester alleges are significant are a key personnel position, a steering committee member, compliance expertise, and experience with DFAS. The agency responds that the modification was considered in the reevaluation, but the changes were not considered material.

The agency's evaluation of IBM's quotation does not mention the Arthur Andersen individual's steering committee membership, that firm's compliance expertise, or its experience with DFAS, either before or after the reevaluation and IBM's modification. This reasonably supports the agency's position that the modification was not material in the overall evaluation of quotations.

With regard to the key personnel position, IBM initially identified, and submitted a resume for, a person from Arthur Andersen for the key personnel position of data repository manager. Agency Report, Tab O-1, IBM Quotation Executive Summary, at 9; Tab O-3, IBM Management Proposal, at A-23. Upon removing Arthur Andersen from its quotation, IBM stated that this would have no impact on its quotation and that the Arthur Andersen key personnel position would be replaced "with an individual of equal or superior qualifications." Agency Report, Tab O-7, Letter from IBM to Agency (Mar. 18, 2002). During IBM's oral presentation, IBM specifically identified the replacement, an assistant vice president with another member of IBM's team, and described his experience, which included significant data repository, DFAS and other DOD experience. Agency Report, Tab O-6, IBM's Oral Presentation Slides, at 3. Therefore, though not in the traditional format of a resume, IBM did provide a resume for this individual. Though the agency did not solicit this information from IBM, the RFQ stated that the agency would consider the information presented at oral presentations in the evaluation. Where the solicitation states, as here, that oral presentations will be considered in the evaluation of quotations, the content of the oral presentation is part of the proposal and should be considered in the agency's selection process. Compare Kathpal Tech., Inc.; Computer & Hi-Tech Mgmt., Inc., B-283137.3 et al., Dec. 30, 1999, 2000 CPD Para. 6 at 12-13, aff'd, Department of Commerce-Request for Modification of Recommendation, B-283137.7, Feb. 14, 2000, 2000 CPD Para. 27, (where solicitation states that oral presentations will be considered part of the quotations, agency cannot eliminate quotations from consideration for award without considering the corresponding oral presentations), with S. C. Myers & Assocs., Inc., B-286297, Dec. 20, 2000, 2001 CPD Para. 16 at 5 (agency properly did not consider proposal revision stated in the oral presentation where solicitation stated that oral presentations would not be considered part of the quotations). Thus, we find no basis to question the award selection based on IBM's change in this key personnel position.

KPMG next alleges that the agency's evaluation and cost/technical tradeoff are defective because they are not consistent with the evaluation factors and subfactors stated in the RFP. KPMG argues that since the SSEB rated IBM's and KPMG's quotations the same overall on a color/adjectival basis, and rated KPMG's quote slightly higher numerically, the quotations must be considered at least equivalent as to the non-price factors, and since KPMG's evaluated price is substantially lower than IBM's, the agency has no reasonable basis to make a cost/technical tradeoff in favor of the higher-priced quotation.

Point scores and adjectival ratings are only guides to assist source selection officials in evaluating quotations; they do not mandate automatic selection of a particular proposal. Grey Advertising, Inc., B-184825, May 14, 1976, 76-1 CPD Para. 325 at 9; PRC, Inc., B-274698.2, B-274698.3, Jan. 23, 1997, 97-1 CPD Para. 115 at 12. Those officials have broad discretion in determining the manner and extent to which they will make use of the technical and cost evaluation results, subject only to the tests of rationality and consistency with the evaluation criteria. Grey Advertising, Inc., supra; A & W Maint. Servs., Inc.--Recon., B-255711.2, Jan. 17, 1995, 95-1 CPD Para. 24 at 4. Where, as here, higher-level officials determine that the lower-level evaluators' ratings do not reflect the actual technical differences in quotations and the award is protested, the agency must show that its ultimate determination is reasonable, with sufficient detail to permit our Office to review the determination for reasonableness. Chemical Demilitarization Assocs., B-277700, Nov. 13, 1997, 98-1 CPD Para. 171 at 6.

Here, as indicated above, the SSAB provided a detailed written analysis of the differences in the quotations that, though generally identified by the SSEB, were not reasonably reflected in the SSEB evaluation ratings and scores. The SSAB found these differences to be significant and of value to the government, and the bases for its conclusions are well documented in the SSAB memo to the SSA summarized above. While KPMG argues that this analysis ignored certain strengths in KPMG's quotation that were found by the SSEB, KPMG only points to certain areas where KPMG's quotation received slightly higher point scores, without persuasively elaborating on why these differences represent actual technical superiority; this amounts to mere disagreement with the agency's evaluation which does not render it unreasonable. Dual, Inc., B-279295, June 1, 1998, 98-1 CPD Para. 146 at 3.

KPMG argues, however, that there was no reasonable basis for the evaluation of the four discriminators identified by the SSAB and SSA and, in any event, those discriminators represent unstated evaluation factors that the agency could not properly consider.

In reviewing an agency's technical evaluation under an FSS competitive acquisition, we will not reevaluate the quotations, but (as with protests of negotiated procurements) will examine the record of the evaluation to ensure that it was reasonable and in accordance with the stated evaluation criteria. Digital Sys. Group, Inc., B-286931, B-286931.2, Mar. 7, 2001, 2001 CPD Para. 50 at 7. It is fundamental, however, that offerors must be advised of the bases upon which their proposal will be evaluated. H.J. Group Ventures, Inc., B-246139, Feb. 19, 1992, 92-1 CPD Para. 203 at 4. This means that the agency may not consider unstated evaluation criteria that are not reasonably related to the stated evaluation factors. Compare MCA Research Corp., B-278268.2, Apr. 10, 1998, 98-1 CPD Para. 129 at 8-9 (agency's identification of relative differences in technical quotations that are reasonably related to the stated evaluation factors are not unstated evaluation criteria) and Computer Sys. Dev. Corp., B-275356, Feb. 11, 1997, 97-1 CPD Para. 91 at 7 (same) with Lloyd H. Kessler, Inc., B-284693, May 24, 2000, 2000 CPD Para. 96 at 4-5 (where agency evaluates each quotation for a specific type of experience and accords that evaluation significant weight, it is a significant evaluation subfactor that must be disclosed in the solicitation). Based on our review, we find the discriminators identified by the agency were reasonable and did not represent unstated evaluation factors not reasonably related to those stated in the RFQ.

As noted, the first of the four discriminators was understanding the problem. This was a stated subfactor under the technical factor. The RFQ states under this subfactor that each quotation shall demonstrate an understanding of DOD's financial management environment "including issues, context, and scope." RFQ at 5. Here, the SSEB rated KPMG's quote yellow/marginal under this subfactor because it failed to evidence a thorough understanding of the scope and complexity of the problem. Agency Report, Tab J-1, SSEB Final Evaluation Report, at 10-11. The SSEB also determined that IBM's quote demonstrated an excellent understanding of the problem and rated IBM's quote higher under the subfactor. Id. at 7. The SSAB concurred in this distinction. Moreover, the SSAB considered the performance timeframe required by the RFQ to be difficult to achieve, yet KPMG claimed that it would perform in a shorter timeframe, which the SSAB considered unreasonable and supported its conclusion that KPMG did not demonstrate a complete understanding of the problem. This determination is consistent with the stated subfactor and is reasonable.

The next two discriminators were financial management transformation and leading (IBM) versus partnering (KPMG). The SSAB identified the former discriminator as arising under the first three stated technical subfactors, and the latter as arising under the first management subfactor and the second technical subfactor. These discriminators stem from the difference in the way these vendors identified the problem. IBM, identifying the problems as being of broader scope and complexity than KPMG, had a more expansive solution and management approach to address the problems.

The protester alleges that terms used by the SSAB and SSA to refer to this evaluated difference--financial management "transformation," a "take charge" management approach, and "collaboration" as it relates to leading versus partnering--are unstated evaluation criteria not reasonably related to those stated in the RFQ. The first of these terms was used by the agency to describe the difference between the vendors in the scope of business process reengineering. As noted above, business process reengineering was expressly called for under the RFQ and both vendors addressed it in their quotations. As characterized by the agency, KPMG's approach was to modernize the existing processes to achieve compliance, whereas IBM's approach was to go beyond achieving compliance and provide for extensive transformation, recognizing the difficulty that the agency has had in previous attempts to improve its financial management systems.

The "take charge" management approach was a description of IBM's leadership approach, and "collaboration" was a term the SSA used to further summarize the SSAB's comparison of the two vendors' approaches. These terms essentially characterize the greater extent to which IBM's quotation provided for the contractor to assume responsibility for interim decision-making and performance of other tasks. We do not think that this terminology in any way created new evaluation criteria; it merely described differences between quotations that spanned many of the stated subfactors.

The protester contends that the repeated use of the term "collaboration" in the source selection decision shows that the SSA favored non-collaboration with the government, which is contrary to the terms of the RFQ, which clearly called for a collaborative effort. See, e.g., RFQ, Performance Work Statement MDA210-02-T-0003, at 4-7, 10-13. A cursory reading of the SSA's decision (quoted above) could give support to the protester's concerns, since the source selection decision indeed said that KPMG's quotation contemplated "substantial collaboration" and that IBM's does "not rely on government leadership or collaboration"; that KPMG's "collaborative approach [involves] more risk"; and that the agency "cannot uphold its end of a collaborative arrangement with KPMG." Looking at these statements in isolation, however, distorts the SSA's summary of the SSAB's detailed analysis of the real differences between the quotations. Contrary to the protester's contention, IBM's quotation does not contemplate a non-collaborative approach with the government. In fact, the SSAB noted, in defusing SSEB concerns about IBM contemplating less frequent meetings than might be desirable, that IBM's quotation provides for extensive interaction with relevant government personnel. /6/ See Agency Report at 12-13; Tab O-2, IBM Quotation's Technical Proposal at 2, 36, 38-39, 47, 78-79, 101, 103, 105, 109. Reading the SSA's selection decision in its entirety, together with the more detailed SSAB memo to the SSA, it is apparent that the point that was being made in the source selection decision with its references to "collaboration" was that the nature of KPMG's collaboration was more dependent on agency personnel and direction, whereas the nature of IBM's collaboration was a consultative relationship with fewer demands on government resources. We think this is consistent with the stated RFQ subfactors that address the vendors' solutions and management plans.

The final discriminator concerned personnel, which the SSAB identified as arising under both the personnel subfactor and the price factor. The source selection decision stated that a risk of increased costs was associated with KPMG's less skilled and experienced personnel. This risk was identified by the SSAB as originating with its questions concerning why KPMG's labor rates and overall price were so much lower than the government estimate and the other vendors' prices. The SSAB memo to the SSA stated that the structure of the RFQ and quotations did not facilitate a detailed analysis of prices, so the SSAB identified factors that could have a substantial impact on price. The only factor that the SSAB thought could conceivably account for the significant labor rate differences was KPMG relying on less experienced personnel in the labor categories. Agency Report, Tab I-1, SSAB Memo to the SSA (Apr. 4, 2002), at 5.

After reaching this conclusion, the SSAB turned to the labor category descriptions included in each quotation and concluded that IBM's descriptions contemplated significantly higher experience levels on average than did KPMG's. Id. at 6, attach. 2. KPMG objects to this latter comparison because the schedules address experience levels differently in that KPMG's schedules generally identify a minimum years of experience figure for each labor category whereas IBM's generally identify a maximum figure. While we agree that this comparison was not a definitive indicator of relative experience because of the different nature of the vendors' schedules, this comparison does support the agency's otherwise reasonable assumption to explain KPMG's price variance from the agency's estimate and other competitors' prices. Even now, we find no more reasonable explanation than KPMG proposing less experienced and less qualified personnel to account for KPMG's substantial price advantage. We therefore find the agency's assessment of the resulting risk on performance is reasonable.

In conclusion, we find, based on our review, that the SSAB's and the SSA's analysis of the quotations and the lower-level evaluation results, and their identification of significant differences that were not reflected in the lower-level evaluation results, were reasonable. As a result of their review, the SSAB and SSA essentially determined that the SSEB's evaluation ratings and scores did not reasonably reflect the relative merits of the quotations of KPMG and IBM. The resulting record of the relative merits prepared by the SSAB for the SSA, and on which the SSA's source selection decision rests, is very detailed and sufficient to support the source selection decision.

Contrary to the protester's allegations, the cost/technical tradeoff presented in the source selection decision was reasonable and consistent with the terms of the solicitation, and with the standards of making such decisions in negotiated procurements in general. The SSAB and SSA were well aware of the substantial difference in the evaluated prices of these two quotations, but also reasonably concluded there was a significant advantage in merit offered by IBM's identification of the problem, its solution approach, and its management plan for performance. The agency reasonably determined that this advantage rendered IBM's quotation more likely than KPMG's to result in success on a challenging project that is of great importance to DOD. This was consistent with the RFQ's evaluation scheme, which provided that the technical factor was significantly more important than all of the other factors combined, the management factor was the second-most-important factor, and price was of least importance. The agency also reasonably determined that there was a risk that KPMG's apparent price advantage would decrease in the face of performance. In sum, we find the agency's selection of IBM's quotation at a significantly higher price to be reasonable.

The protest is denied.

Anthony H. Gamboa General Counsel

1. The management factor also contained a discrete fourth subfactor, security, which was to be evaluated only as acceptable or unacceptable.

2. Although an SSA was identified in the initial SSP, the record does not show that one was appointed until February 5.

3. KPMG LLP is a corporate entity different from the protester.

4. The RFQ does not state or define the term "business process reengineering." However, the performance work statement states the following as a key objective of the development of the FMEA: Reengineer the Department's financially related business processes to ensure routine availability of reliable, accurate and timely financial management information upon which to make the most effective business decisions. RFQ, Performance Work Statement MDA210-02-T-0003, at 4.

5. Since this properly accepted quotation modification from IBM was not solicited by the agency, it did not constitute improper or unequal discussions with only one vendor. Also, we need not decide whether IBM's proposal would have been unacceptable if it had maintained Arthur Andersen as a subcontractor.

6. For this reason, we reject KPMG's argument that IBM's quotation is technically unacceptable for failing to meet the RFQ's colloboration requirements.

* DOCUMENT FOR PUBLIC RELEASE

The decision issued on the date below was subject to a GAO Protective Order. This redacted version has been approved for public release.