Matter of: Keane Federal Systems, Inc. File: B-280595 Date: October 23, 1998 * Redacted Decision

Matter of: Keane Federal Systems, Inc. File: B-280595 Date: October 23, 1998 * Redacted Decision

DIGEST

Attorneys

DECISION

Keane Federal Systems, Inc. protests the award of contracts to Computer Sciences Corporation (CSC), Electronic Data Systems Corporation (EDS), and Lockheed Martin Technical Services, Inc. (LM) under request for proposals (RFP) No. HQ-97-13, issued by the Department of Justice, Immigration and Naturalization Service (INS), in support of its Service Technology Alliance Resources (STARS) program, under which INS intends to meet all of its information technology needs into the next century. Keane primarily contends that INS improperly evaluated its proposal with respect to management and cost and conducted an irrational and arbitrary best value analysis that improperly distorted the solicitation's evaluation scheme.

We deny the protest.

BACKGROUND

The STARS program is the result of INS's desire to form an alliance with information technology contractors to obtain a full range of technological solutions to support the agency in achieving its mission objectives and information technology needs into the 21st century. RFP Sec. C.1.1. This solicitation, issued on September 16, 1997, is the vehicle by which INS planned to obtain the services of its STARS contractors.

To fulfill the STARS program strategy INS intended to award one contract for system management and integration services; three hybrid, indefinite-delivery performance contracts /1/ to three other firms for systems development, implementation, operations, and maintenance services; and one contract to a fifth firm for independent verification and validation services. RFP Sec. C.1.2. All work to be performed under the STARS contracts will be assigned by task orders issued over 1 base year, with up to 4 option years available. RFP Sec. B, C.1.2. This protest concerns the award of the performance contracts.

The performance contractors will be responsible for, among other things, ongoing systems engineering and integration to support implementation of a comprehensive information systems architecture; systems maintenance and sustaining engineering; systems management; designing, developing, installing, operating, and maintaining existing and planned communications and electronic systems; and assisting INS with system definition and providing development, installation, operation, maintenance of existing and planned end-user automated data processing support systems or services, user training, and video services. RFP Sec. C.4.

Since software development is the heart of the STARS program, the RFP required offerors to possess a documented Software Engineering Institute (SEI) Software Capability Maturity Model (CMM) at level 2 or higher. /2/ Offerors not meeting this mandatory requirement were not eligible for evaluation or award--their proposals were to be rejected. Offerors with documented Software CMM levels higher than 2 were to be given credit in the overall best value analysis. /3/ RFP Sec. L.7.2.1., M.2.

Awards were to be made to the offerors whose proposals represented the greatest overall value to the government, cost and other factors considered. RFP Sec. M.3.2. The solicitation set forth five evaluation factors: management, experience and past performance, technical, cost, and other business factors. RFP Sec. M.4.

Within the management factor, the contract management subfactor was more important than the corporate management subfactor; the RFP also set forth numerous elements to be considered under each management subfactor. RFP Sec. M.4.1(a), M.4.3.1. The experience and past performance factor consisted of two equally important subfactors, experience and past performance. RFP Sec. M.4.1(a). Within the technical factor, each sample task response was considered to be equally important. RFP Sec. M.4.1(b). All of these factors and their components were to be adjectivally rated. RFP Sec. M.4.2.

The management and experience and past performance factors were considered to be equally important, and these two factors were significantly more important than the technical factor. RFP Sec. M.4.1(a). Collectively, the management, experience and past performance, and technical factors represented the performance capability factors, and the performance capability factors were significantly more important than the cost factor. RFP Sec. M.4, M.4.1(c).

Costs and other business factors were not to be adjectivally rated. Information in the cost volumes was to be evaluated to determine the total proposed contract cost, cost realism, reasonableness, and cost risk. RFP Sec. M.4.4. The cost factor was significantly more important than the other business factors. RFP Sec. M.4.1(d). The other business factors were to be evaluated and considered and were also critical to INS's responsibility determination. RFP Sec. M.4.4.2.

Offerors' management and experience and past performance volumes were to be evaluated by the management, experience & past performance evaluation committee (MEPPEC). Offerors' technical volumes were to be evaluated by the technical evaluation committee (TEC). Finally, offerors' cost and other business factors volumes were to be evaluated by the business evaluation committee (BEC). At the conclusion of the evaluation process, each committee was to submit its findings to the source selection advisory council (SSAC), which was to review the committees' findings; conduct comparative analyses of the proposals; and present its recommendations to the source selection authority (SSA). The SSA was to make the final source selection decision. RFP Sec. M.3.1.

The determination of greatest value was to be made by comparing the differences in the value of the management, experience and past performance, and technical factors with differences in the costs proposed. INS would not make an award at a significantly higher overall cost to achieve only slightly superior performance capability. INS planned to make its greatest value assessment through the development of trade-off analyses and other analytic studies that involve the assessment of benefits of superior performance capability features versus the added cost. RFP Sec. M.3.3.

INS received proposals from five offerors, each consisting of a prime contractor and numerous subcontractors. The contracting officer included all five proposals in the competitive range based upon the initial evaluation. INS conducted extensive written and oral discussions with each offeror and received best and final offers (BAFO) in April 1998. Each evaluation committee reviewed the BAFOs and revised their final reports as appropriate. The BEC listed both the offerors' proposed costs, corrected for any errors, and their most probable costs, inclusive of the BEC's cost realism adjustments.

The SSAC was given the committees' final reports, each of which consisted of detailed documentation of the strengths and weaknesses of each proposal in connection with each evaluation factor, subfactor, and element. The SSAC voted unanimously to accept the findings of the committees. SSAC Recommendation at 27. The SSAC decided to consider offerors' proposed corrected costs as their evaluated costs since it believed the BEC's cost realism adjustments reflected upper

bounds on quantifiable risks rather than corrections for unrealistically low cost proposals. /4/ Id. The final relevant evaluation results were as follows: /5/

CSC EDS LM Keane

Management Outstanding Outstanding Outstanding Good

Contract Management Outstanding Outstanding Outstanding Good

Corporate Management Outstanding Outstanding Outstanding Outstanding

Experience and Past Performance Good Good Good Outstanding

Experience Outstanding Outstanding Outstanding Outstanding

Past Performance Good Good Good Outstanding

Technical Outstanding Good Acceptable Outstanding

Proposed Costs (Millions) $272 $296 $244 $283

The SSAC used various methods to rank the offerors but achieved different results depending on the method used. In any event, the SSAC decided that a further ranking of proposals was unnecessary since it required a detailed best value analysis. Id. at 29. The best value working group (BVWG) was appointed by the SSAC Chair to conduct this analysis. As discussed in detail below, the BVWG began its task by identifying 53 potential discriminators based upon a review of the evaluation committee reports. BVWG Report, Exhibit 1. The BVWG next reviewed each proposal to ascertain which of the potential discriminators were actual discriminators--superior features and risks--and arrived at a list of 12 superior features and 5 risks. BVWG Report, Exhibit 2.

After undertaking a detailed review of each proposal, the BVWG determined that CSC's proposal offered 11 superior features and 1 risk; EDS's proposal offered 9 superior features and 1 risk; LM's proposal offered 6 superior features and 1 risk; and Keane's proposal offered 5 superior features and 2 risks. SSAC Recommendation at 31. The BVWG compared these results with the offerors' proposed costs and reached some conclusions.

First, Keane's proposal did not present a sufficient number of features to warrant paying a $39 million premium (the difference between Keane and LM's proposed costs) for slightly fewer features and added cost risk. Second, LM's proposal offered a significant number of superior features, was the lowest cost offer, and had very little cost and performance risk. Third, CSC was the highest-ranked offer based on the performance capability factors and its proposal, ranked third based on cost, offered a significantly greater number of superior features than any other offeror with little cost or performance risks. Fourth, EDS's proposal offered superior features that were virtually identical to those proposed by CSC, its proposed costs were $24 million higher than CSC's, and the superior features found in EDS's proposal significantly exceeded those found in Keane's proposal with a cost that was slightly higher. Id. at 32.

The BVWG recommended that awards be made to CSC, EDS, and LM, and the SSAC adopted this recommendation. Id. at 33-35. The SSA decided to award the contracts to CSC at a maximum value of $817.9 million; to EDS at a maximum value of $890.2 million; and to LM at a maximum value of $734.1 million. /6/ In his Source Selection Decision, the SSA stated:

The CSC proposal offers the most performance capability, third lowest cost, and represents the greatest value to the Government of all offerors.

The EDS proposal offers high performance capability and a significant number of superior features. Although the EDS proposal carries a cost premium over the unsuccessful offeror's proposals, it represents the second greatest value to the Government.

The [LM] proposal offers high performance capability, a number of superior features, and is the lowest cost proposal. As such it represents the third greatest value to the Government.

In making this determination, I have considered the rankings, ratings, and recommendations of the [SSAC] and evaluation committees and have used them as the basis to arrive at my own independent decision.

Keane filed this protest after it received its debriefing. Keane primarily argues that INS improperly evaluated its management volume with respect to the contract management subfactor; improperly evaluated its cost information in assessing it several cost risks; and conducted an irrational and arbitrary best value analysis that improperly distorted the evaluation scheme.

EVALUATION OF MANAGEMENT VOLUME

Evaluating the relative merits of competing proposals is a matter within the discretion of the contracting agency since the agency is responsible for defining its needs and the best method of accommodating them, and must bear the burden resulting from a defective evaluation. Advanced Tech. and Research Corp., B-257451.2, Dec. 9, 1994, 94-2 CPD Para. 230 at 3. Where an evaluation is challenged, we will examine the agency's evaluation to ensure that it was reasonable and consistent with the evaluation criteria. Professional Software Eng'g, Inc., B-272820, Oct. 30, 1996, 96-2 CPD Para. 193 at 4. A protester's disagreement with the agency's conclusion does not itself establish that the agency acted unreasonably. Id.

Keane argues that if EDS's proposal was rated outstanding under the management factor its proposal should have received the same rating. Keane points to the fact that EDS's proposal was rated "outstanding" under the management factor even though it had two fewer major strengths and four fewer minor strengths than did Keane's proposal under the contract management subfactor, and the fact that Keane's proposal was rated "outstanding" under the corporate management subfactor.

As an initial matter, as discussed below, the source selection here did not depend upon these adjectival ratings or the mere number of strengths or weaknesses found in given proposals. Keane's narrow focus on these matters is meaningless because it fails to consider the underlying nature of the strengths found in each proposal and their relative importance to the evaluation as a whole. See Innovative Logistics Techniques, Inc., B-275786.2, Apr. 2, 1997, 97-1 CPD Para. 144 at 9. Keane's only substantive challenge to the evaluation is its assertion that its proposal and EDS's proposal were treated unequally, citing several examples. We address three of these.

Keane complains that INS evaluated EDS's proposal as having a major strength for "High Visibility of STARS Program within EDS" because its proposed STARS program is four organizational levels from the EDS chairman, while Keane's proposed STARS program, three organizational levels from its chairman, was not evaluated as a strength. The record shows that EDS's evaluated strength was not attributable to the mere placement of the STARS program within its organization, but to many additional factors as well. Final MEPPEC Report at II-41-42 and Appendix 3 at D-2. As a result, we have no basis to question INS's evaluation.

Keane incorrectly contends that the MEPPEC's final report erroneously indicated that EDS's proposal was rated as having two major strengths for staffing when only one major strength appears in the backup documentation. The record shows that, from the beginning, EDS's proposal was considered to have two major strengths for staffing. Final MEPPEC Report, Appendix 2 at D-18 and Appendix 3 at D-12.

Keane contends that the MEPPEC improperly rated EDS's proposal as having a minor strength for its goal of awarding 16 percent of its annual revenue to small businesses and its commitment to a minimum set-aside of 10 percent, whereas Keane's proposal was rated as having the same minor strength for its proposal to allocate 28.1 percent of its work to small businesses and to a minimum set-aside of 12.5 percent. While there are differences between the two proposals, Keane has provided no reason for us to conclude that these differences merited a materially different assessment.

EVALUATION OF COST VOLUME

The BEC's evaluation of Keane's proposal disclosed three significant cost risks associated with the status of its purchasing system, its proposed indirect rates, and its cost accounting standards (CAS) status. Keane argues that each of these cost risks is unreasonable. We address each in turn.

One of the other business factors was "approval of business systems." Offerors were required to state whether all of their business systems which required government approval, such as their purchasing systems, were approved or, if not, to explain the status of their approval. RFP Sec. L.7.7.6.6. In this regard, contractors with approved purchasing systems generally need not obtain administrative contracting officer (ACO) consent for cost-reimbursement subcontracts, while contractors without approved purchasing systems generally do need to obtain such consent. Federal Acquisition Regulation (FAR) Sec. 44.201-2. In considering whether to grant consent, the ACO is to consider numerous factors associated with the way the prime contractor selected the subcontractor; "particularly careful and thorough consideration" is necessary when the prime contractor's purchasing system is inadequate or when subcontracts are proposed on a cost-reimbursement basis. FAR Sec. 44.202-2.

To assist the BEC with its cost proposal evaluation, offerors were required to furnish a copy of each cost proposal and the analysis made thereof for each subcontract valued at $100,000 or more. Offerors were required to have each of these subcontractors submit a full set of cost forms documenting various elements of their proposed costs. RFP Sec. L.7.7.5.1.

Keane's initial proposal did not advise INS whether its purchasing system was approved and, in addition, the BEC found that it had not been given sufficient cost information to evaluate Keane's subcontractors. During discussions, Keane was asked to address the status of its purchasing system and to provide sufficient information on its proposed subcontractors to allow for their evaluation. Keane's answers did not specifically address the status of its purchasing system and provided only general information concerning its subcontractors. Keane's Response to Deficiency Reports BEC-DR-346, BEC-DR-410.

Amendment No. 0005 revised RFP Sec. L.7.7.5.3. to specifically instruct offerors that:

All prime contractors must show, or as a minimum explain what analysis (cost analysis, price analysis, etc.) was performed on subcontractor proposals to determine the reasonableness of the subcontractor pricing. If the prime contractor has an approved purchasing system, the prime must affirmatively state whether they did or did not adhere to their own purchasing system requirements when evaluating the subcontractors proposals.

Keane's BAFO provided some additional explanation of its subcontractor agreements and arrangements, BAFO Cost Volume at V-B-14 through V-B-22, but the BEC concluded that these general and unsupported statements failed to meet the requirements of RFP Sec. L.7.7.5.3. Keane also advised that it did not have an approved purchasing system. See BAFO Other Business Factors Volume at VI-17.

The BEC found that Keane was the only offeror that did not provide required documentation of the process by which it selected subcontractors in accordance with FAR Part 44, and that its analysis was not sufficient to permit the contracting officer to authorize approval of subcontracting at the time of award. Final BEC Report at 2. This lack of information, which might not have been necessary had Keane possessed an approved purchasing system, represented significant risk, particularly since Keane planned to subcontract out 40 percent of the contract effort. The contracting officer would have to approve each subcontractor until Keane's purchasing system was reviewed and approved. Id. at 18-19. Keane's assertion that it identified each subcontractor for the government to evaluate, making its lack of an approved purchasing system irrelevant, is wholly unresponsive to INS's concerns, and we conclude that the BEC's assessment was reasonable.

Next, in evaluating Keane's initial proposal the BEC was concerned that the firm's proposed indirect rates declined over the period of performance and were both significantly lower than historical rates and unsupported. During discussions, Keane was asked to explain and support its proposed declining rates. Keane generally asserted that it had experienced declining rates due to increased business volume and that these rates would continue to decline with the addition of STARS business. Keane's Responses to Clarification Requests BEC-CR-623, BEC-CR-620. One of Keane's subcontractors was also advised, during discussions, that its proposed indirect rates were unrealistically low and unsupported, and its response was similarly vague and unsupported. Id., BEC-CR-167, BEC-CR-445, BEC-CR-166, BEC-CR-628, BEC-CR-146. These explanations did not persuade INS that either firm's proposed indirect rates were reliable, and the BEC repeated its concerns during oral discussions.

Keane's BAFO provided no substantive support for its proposed indirect rates but only generalized statements about its projected growth which Keane itself termed "optimistic." BAFO Cost Volume at V-B-5 through V-B-7. Accordingly, the BEC concluded that Keane's indirect rates, and those of its subcontractor, represented a significant cost risk. The issuance of any competitive task order on a cost-reimbursement basis would require that these indirect rates be routinely checked with the Defense Contract Audit Agency to enable the contracting specialist to monitor trends. Final BEC Report at 18-20.

Keane's argument that the BEC mechanically adjusted the firm's--and its subcontractor's--indirect rates without considering the positive impact that receiving the STARS contract would have on lowering their historical indirect rates is simply untrue and fails to address INS's concerns regarding the lack of support for these proposed indirect rates. Given the unsupported discrepancy between Keane's actual indirect rates and its proposed indirect rates, we see nothing unreasonable about INS's evaluation. See Booz-Allen & Hamilton, Inc., B-275934.2, May 29, 1997, 97-1 CPD Para. 222 at 5.

Finally, since the performance contracts are subject to CAS coverage, FAR Appendix Sec. 9901.306, the RFP asked offerors to submit their CAS Disclosure Statements--written descriptions of their cost accounting practices and procedures--as part of their cost volumes. RFP Sec. L.7.7.5.3., K.3.21. In addition, approval of an offeror's accounting system was one of the business systems specified under the "approval of business systems" factor discussed above. RFP Sec. L.7.7.6.6., M.4.4.2.6. In its initial proposal Keane advised that it had not submitted a CAS Disclosure Statement because it had not previously been required to do so, and that it would submit the required statement in 1998. Initial Proposal Volume VI at VI-17.

Business units selected to receive a CAS-covered contract of $25 million or more are required to submit a CAS Disclosure Statement before award. FAR Appendix Sec. 9903.202-1(b)(1). As a general matter, contracting officers are not to award a CAS-covered contract until the ACO has made a written determination that a required Disclosure Statement is adequate. FAR Sec. 30.202-6(b). Accordingly, during discussions, the BEC asked Keane about its CAS compliance status and was advised that the firm would submit its disclosure statement concurrent with award of this contract. Keane's Response to Clarification Request BEC-CR-619. The BEC repeated its inquiry during oral discussions, and expressed concern about the time it takes to review such statements.

The BEC gave Keane's proposal a passing rating under the approval of business systems factor since it had not previously been required to submit a CAS Disclosure Statement. /7/ However, since the contracting officer may generally not award a CAS-covered contract until the ACO has determined that the required Disclosure Statement is adequate, which it could not do here, and since the performance contracts contemplated the issuance of high-dollar value cost-reimbursement task orders, the BEC found the fact that Keane did not have approval of its cost accounting standards to be a significant risk because the government did not know if its cost accounting practices would meet government requirements. Final BEC Report at 18-19. Keane's argument that it is CAS-compliant does not minimize the agency's concerns about whether it actually is CAS-compliant, or its concern regarding the firm's ability to obtain a review of its CAS Disclosure Statement in a timely fashion. Under the circumstances, we cannot conclude that INS's concerns here were unreasonable.

While Keane is correct that the awardees' proposals evidence various CAS non-compliance issues, its implication that their proposals should have been found to present cost risks as well is unsupported. Keane is incorrect when it argues that INS ignored these matters with respect to the awardees. The record shows that the agency reviewed the issues, conducted discussions to gain more information, and ultimately concluded that the risks, if any, were not significant. Keane has given us no reason to think that the risks, if any, presented by these proposals were equivalent to the risk presented by its own.

As for Keane's assertion that these areas of risk were undisclosed evaluation criteria, the consideration of risk involved in an offeror's proposed approach is inherent in the evaluation of proposals. DIGICON Corp., B-275060, B-275060.2, Jan. 21, 1997, 97-1 CPD Para. 64 at 4. Moreover, the RFP here specifically informed offerors that the cost risks associated with their proposals would be evaluated, and that their CAS status, methods of selecting subcontractors, purchasing systems, and indirect rates would be reviewed. Thus, the agency's consideration of risk here was not improper. Id. at 5. Finally, Keane's argument that INS failed to conduct meaningful discussions on these issues is without support. Aside from the fact that applicable regulations and RFP provisions put Keane on notice of most of these matters, as noted above, the record is replete with evidence that INS conducted meaningful discussions in each of these areas. /8/

BEST VALUE ANALYSIS AND SOURCE SELECTION DECISION

INS took particular care to inform offerors how their proposals were to be evaluated against the RFP's terms and how their proposals were to be selected for award. The RFP put offerors on notice that a set of evaluation committees was to review, analyze, evaluate, and rate the various volumes of their proposals, gave them the evaluation roadmap that each committee planned to follow, and instructed them that the committees' findings were to be given to the SSAC. The RFP put offerors on notice that the SSAC's role was to review the findings of the evaluation committees, conduct comparative analyses of the proposals, and present its recommendations to the SSA, who would make the final source selection decision.

In this regard, offerors were told that awards were to be made to those offerors whose proposals represented the greatest overall value to the government, cost and other factors considered. Section M.3.3. of the RFP told offerors precisely how the "greatest overall value" was to be determined:

The determination of greatest value will be made by comparing the differences in the value of the Management, Experience & Past Performance, and Technical Factors with differences in the costs proposed. These evaluation factors, when combined, are significantly more important than cost. However, the Government will not make an award at a significantly higher overall cost to the Government to achieve only slightly superior performance capability. The Government will make this assessment through the development of trade-off analyses and other analytic studies that involve the assessment of benefits of superior performance capability features (e.g., economic benefits clearly attributable to increased INS productivity, probability of successful contract performance, and/or unique and innovative approaches or capabilities) versus the added cost. Overall cost to the Government may become the ultimate determining factor for award of the contracts as proposals become more equal based on the other factors. The degree of equality between the Offerors' proposals will be measured by the quantity, significance, and applicability of the superior features proposed.

The record shows that INS followed the RFP's instructions.

Each evaluation committee reviewed, analyzed, evaluated, and rated the proposals of each offeror. The evidence of their labor is found in hundreds of pages of detailed documentation which contains factor-by-factor, subfactor-by-subfactor, and element-by-element analysis of how each proposal measured up to the solicitation's requirements. Each evaluation report culminated in a set of adjectival ratings or adjusted costs, as applicable. The SSAC reviewed these findings and convened the BVWG to assist with the comparative analysis of proposals. The process followed for the best value analysis and source selection is set forth in the BVWG's Report and the SSAC's Recommendation, and was further illuminated by the MEPPEC Chair (a BVWG member), the SSAC Chair, and the SSA at a hearing.

The BVWG's goal was to identify the differences--discriminators--betwe en proposals that would be important to the INS mission and/or program objectives, and that would be expected to affect INS's operation under the contract. SSAC Final Recommendation at 30. To ascertain the universe of potential discriminators, the BVWG prepared a matrix which listed each evaluation factor, subfactor, and element as set forth in section M of the RFP, and contained a column for each offeror. The BVWG reviewed each committee's final evaluation report, factor-by-factor, to identify superior features and risks--potential discriminators--found in the offerors' proposals. The BVWG inserted each potential discriminator into the matrix under the appropriate evaluation factor, and inserted a plus (for a superior feature) or a minus (for a risk) in the column of each firm whose proposal contained the potential discriminator. Id.; BVWG Exhibit 1; Hearing Transcript (Tr.) at 26-34, 70-71. The BVWG identified 53 potential discriminators.

Next, the BVWG set out to determine which of these potential discriminators were actual discriminators--features that would have a significant impact on INS's mission productivity, effectiveness, and cost. The BVWG reviewed each proposal, factor-by-factor, to ensure that it fully understood what was offered and why it did or did not add value. The evidence of their labor is found in more than 100 pages of documented analysis which links each discriminator to an RFP requirement, discusses the relevant proposal feature, and analyzes the value of that feature, if any. BVWG Exhibit 2 Backup Documentation; Tr. at 34-35, 37-39. To summarize its findings the BVWG prepared the following matrix which lists each actual discriminator and inserts pluses or minuses, as applicable, with respect to each proposal:

Qualitative Discriminators CSCEDS LM Keane

CMM Level +

Understanding/Embracing of STARS Program + + + +

Location/Facilities + + + +

Program Visibility + +

Corporate Support for the STARS Performance Team + + +

Merit of Proposed Subcontractors + +

Senior Program Manager + + +

Management Team Composition + +

Quality Assurance Role and Responsibility + + + +

Corporate Resources + + +

Degree of Corporate Commitment +

Sample Tasks + +

Performance Risk

Direct Labor Costs-Risk

Indirect Rates (Prime)-Risk -

Subcontractor Costs-Risk - - -

Distribution of Work Effort-Risk -

Net Totals: 10 8 5 3

Each SSAC member was given the BVWG's full report, and the SSAC received an extensive debriefing on the reasons various proposals did or did not offer the various discriminators. SSAC Recommendation at 30-33; Tr. at 42-43, 89, 125-30. The SSAC concurred with the BVWG's recommendation and provided an extensive debriefing to the SSA. As part of his debriefing the SSA was given and reviewed the SSAC's Final Recommendation, which contained the BVWG's report, as well as the BVWG's detailed analysis of proposals. Tr. at 136, 143-46, 175-78.

Keane argues that this best value analysis improperly disregarded and distorted the evaluation scheme set forth in section M of the RFP and improperly ignored many of Keane's major strengths as found by the lower-level evaluation committees. According to Keane, since the SSAC unanimously voted to accept the findings of the committees, it should have made award based on their adjectival ratings.

In a negotiated procurement with a best value evaluation plan, adjectival ratings are only guides to assist contracting agencies in evaluating proposals; they do not mandate automatic selection of particular proposals. Chemical Demilitarization Assocs., B-277700, Nov. 13, 1997, 98-1 CPD Para. 171 at 6. Source selection officials, which includes officials at an intermediate level, are not bound by the recommendations or evaluation judgments of lower-level evaluators, even though the working level evaluators may normally be expected to have the technical expertise required for such evaluations. PRC, Inc., B-274698.2, B-274698.3, Jan. 23, 1997, 97-1 CPD Para. 115 at 7. Source selection officials have broad discretion in determining the manner and extent to which they will make use of the technical and cost evaluation results, and their judgments are governed only by the tests of rationality and consistency with the stated evaluation criteria. Id.; Chemical Demilitarization Assocs., supra.

INS did not discard the lower-level evaluation results and, along with them, the RFP's evaluation scheme. The record shows that INS analyzed those results and drew from them a list of discriminators with which to make its best value analysis, in accordance with the solicitation. /9/

The solicitation explicitly advised offerors that INS planned to assess the benefits of superior features versus added cost. RFP Sec. M.3.3. Moreover, where detailed technical proposals are sought and detailed technical evaluation criteria are used to enable the agency to make comparative judgments about the relative merits of competing proposals, offerors are on notice that qualitative distinctions will be made. Computer Sys. Dev. Corp., B-275356, Feb. 11, 1997, 97-1 CPD Para. 91 at 6.

Information regarding specific proposal advantages or disadvantages is the type of information that agencies should make available to source selection officials to enable them to reasonably determine whether and to what extent adjectival evaluation ratings indicate meaningful differences in proposals and the resulting value of such differences. Israel Aircraft Indus., Ltd., MATA Helicopters Div., B-274389 et al., Dec. 6, 1996, 97-1 CPD Para. 41 at 7. Such considerations are the essence of any best value source selection decision, F2M-WSCI, B-278281, Jan. 14, 1998, 98-1 CPD Para. 16 at 8, and the BVWG's designation of discriminators was simply and properly a tool to help assess which proposals represented the best value. See Engineering and Professional Servs., Inc., B-262179, Dec. 6, 1995, 95-2 CPD Para. 266 at 5.

Keane complains that the BVWG's selection of discriminators improperly distorted the evaluation scheme by "abandoning" the experience and past performance evaluation factor. Keane points out that the management factor is represented by numerous discriminators. In contrast, the experience and past performance factor, which was just as important as the management factor, is represented by only one discriminator, and that a negative one--performance risk. According to Keane, the selection of discriminators gave disproportionate weight to the management factor and other, less important factors, and failed to give Keane credit for its superior past performance rating.

The record shows that INS did not "abandon" the experience and past performance factor, but that the BVWG simply did not consider favorable past performance to be a discriminator. As a BVWG member stated at the hearing, past performance instills confidence that a company will probably do well but does not in and of itself guarantee that a company will do well, whereas poor past performance is a good indicator of performance risk. Tr. at 92-96, 102-106.

There is no requirement to have a discriminator for each evaluation factor, or to have an equivalent number of discriminators for equally important evaluation factors. See Computer Sys. Dev. Corp., supra. Moreover, whenever equal factors are considered, the fact that one is chosen as more valuable does not mean that the relative weights of the evaluation factors have been changed or that one has been abandoned. It simply means that one has become the discriminator between competing proposals. Calspan Corp., B-258441, Jan. 19, 1995, 95-1 CPD Para. 28 at 14. There is also no requirement that award discriminators be the most heavily weighted factors. Research for Better Schools, Inc., B-270774.3, June 17, 1996, 96-2 CPD Para. 41 at 8. So long as the less heavily weighted criteria have been disclosed to the offerors in the RFP, as they were here, there is nothing improper in their becoming the discriminator where competing proposals are evaluated as equal in the more heavily weighted ones. Duke/Jones Hanford, Inc., B-249367.10, July 13, 1993, 93-2 CPD Para. 26 at 10 n.8.

While Keane's proposal was rated "outstanding" under the past performance subfactor, to the other offerors' "good" ratings, the BVWG considered the offerors' past performance to be approximately the same. Tr. at 100. Indeed, our review of the MEPPEC's findings show virtually identical strengths for all offerors, and there is no basis for us to conclude that the minor concerns with respect to the awardees' past performance amounted to a material distinction. As a result, we cannot conclude that the agency improperly omitted a positive discriminator for past performance here.

Keane also asserts that INS improperly failed to credit the firm for offering two discriminators, one for a superior senior program manager, and one for possessing a CMM level higher than 2. With respect to the first, Keane merely argues that it received a major strength for its senior program manager and, thus, should have been credited with this discriminator. Since Keane does not address the BVWG's detailed rationale for crediting other offerors with this discriminator, and not itself, we have no basis to question the agency's assessment. See BVWG Exhibit 2 Backup Documentation; Final MEPPEC Report, Appendix 3 at C-11; Appendix 2 at D-18; Appendix 3 at F-9. With respect to the second, Keane's allegation is untimely.

On the day of its debriefing Keane was given redacted copies of BVWG Exhibits 1 and 2. The first potential discriminator listed on exhibit 1 was "CMM Level . . . Higher than Level 2, per RFP." As noted above, the RFP specifically provided that offerors with documented Software CMM levels higher than 2 were to be given credit in the overall best value analysis. RFP Sec. M.2. There was no plus in Keane's column for this discriminator. This discriminator was abbreviated and carried over onto exhibit 1; again, there was no plus in Keane's column for this discriminator.

Despite Keane's knowledge on that day that offerors with CMM levels higher than 2 were to be given credit in the best value analysis; its belief that it offered a CMM level 3; and its knowledge that it was not given credit for having a CMM level higher than 2 in the best value analysis as set forth in BVWG exhibits 1 and 2, Keane failed to raise this issue until its comments on the agency report. In view of the detail available to Keane when it filed its protest, it was required to raise the issue then and its failure to do so renders the issue untimely. Global Eng'g & Constr. Joint Venture, B-275999.4, B-275999.5, Oct. 6, 1997, 97-2 CPD Para. 125 at 4.

Finally, Keane argues that the award decision was irrational and arbitrary because it reduced the entire source selection decision to a tradeoff between the net totals of superior features versus cost risks. As discussed above, while the BVWG exhibits list net totals, and the SSAC's recommendation and source selection decision speak in terms of the numbers of features, the record is abundantly clear that the award decision was not based on a simple "scorecard" as Keane suggests. Instead, the SSAC and SSA relied on these numbers as a shorthand for the well-documented, reasonable findings of the BVWG; the source selection decision was clearly based on the BVWG's findings as a whole. See Israel Aircraft Indus., Ltd., MATA Helicopters Div., supra, at 7; Tr. at 111-17, 147-48, 178-80.

The rationale for the source selection decision is clearly outlined in the full BVWG Report, the SSAC Recommendation, and the source selection decision. As discussed above, the rationale is rational, supported by the evaluation record, and consistent with the evaluation criteria. As a result, there is no basis to object to INS's selection of CSC, EDS, and LM for contract award, and no basis to object to INS's decision not to award a contract to Keane. Calspan Corp., supra, at 14.

The protest is denied.

Comptroller General of the United States

1. INS intended to award three performance contracts but reserved the right to award more or fewer such contracts based on the evaluation results. RFP Sec. C.1.2.

2. The CMM is a tool for assessing and evaluating the maturity of an organization's software processes. The model, which identifies five levels of maturity, provides for software process assessments--or self-assessments--that allow organizations to implement improvement programs and for software capability evaluations that allow evaluators to identify the risks of selecting among different contractors for award. Research Analysis and Maintenance, Inc., B-272261, B-272261.2, Sept. 18, 1996, 96-2 CPD Para. 131 at 4 n.1.

3. Since INS evaluated Keane's proposal and considered it for award, and since we conclude that the evaluation and source selection decision were reasonable, we need not resolve the question, raised by INS during the pendency of the protest, whether Keane's proposal actually met this mandatory requirement. As discussed further below, Keane's allegation that INS improperly failed to credit it with possessing a documented CMM level 3 in the best value analysis is untimely.

4. As a result, our consideration of INS's cost realism analysis need only address those areas of cost risk that became important in the best value analysis.

5. The ratings assigned to the adjectivally rated other business factors are not at issue here, and all offerors were rated as passing the remaining other business factors.

6. These values are based upon the RFP's overall contract maximum value, which was computed by multiplying each offeror's total evaluated costs by a factor of three. This formula assumes the possibility that each performance contractor might perform all of the work ordered under the performance contracts. RFP Sec. H.10.2.2.

7. As revised by amendment No. 0005, offerors were to be given passing ratings if none of their business systems that required approval had significant outstanding citations of non-approval that might affect their eligibility for a cost-type contract, and if the offeror had no significant CAS violations that might affect their eligibility for a cost-type contract. RFP Sec. M.4.4.2.6.

8. In its comments Keane pointed out that the BEC found six significant cost risks in EDS's proposal but the BVWG only considered one of these to be a discriminator. In response to our request, INS provided a detailed explanation for the BVWG's decision, which Keane failed to address in its subsequent response. As a result, we consider the issue to have been abandoned and will not consider it. IT Corp., B-258636 et al., Feb. 10, 1995, 95-1 CPD Para. 78 at 8 n.11.

9. The 17 discriminators selected by the BVWG are not, as Keane insists, undisclosed evaluation criteria. All are drawn directly, and many verbatim, from section M of the solicitation.

* DOCUMENT FOR PUBLIC RELEASE

The decision issued on the date below was subject to a GAO Protective Order. This redacted version has been approved for public release.