B-234558, Jun 21, 1989, 68 Comp.Gen. 511

B-234558, Jun 21, 1989, 68 Comp.Gen. 511

Source selection decision to award to the lowest cost, but lowest technically evaluated offeror, is not supported by the record, where the solicitation provided that technical merit was the most important evaluation factor, and the agency source selection justification does not explain why the offeror's lowest cost offsets its relatively low technical and high risk rating, considering the protester's technical rating was significantly higher.

TRW, Inc.:

TRW, Inc., protests the award of a contract to GTE, Government Systems Corporation under request for proposals (RFP) No. F01620-88-R 0002, issued by Gunter Air Force Base, Alabama, for a combination cost plus and firm- fixed price contract for modernization of software for the Air Force Command and Control Systems (AFC2S). TRW contends that the Air Force used cost as the predominant source selection factor, even though cost was secondary to technical considerations under the RFP. TRW also contends that GTE's subcontractor, Evaluation Research Corporation International (ERCI), had an organizational conflict of interest which should disqualify GTE from award.

We sustain the protest on the first ground.

The RFP called for nonpersonal technical services to modernize software and for providing hardware necessary to support modernization of the current AFC2S. GTE, TRW, and two other concerns submitted proposals in response to the RFP.

The RFP advised offerors that this was "a technical competition with cost considered subordinate to other factors." Offerors were also advised that "although price/cost is listed last, cost reasonableness, cost realism, and cost risk will be significant considerations for award as part of an integrated assessment with other evaluation areas." The RFP explained that cost realism would be evaluated by comparing proposed costs with the government's independent cost estimate including an evaluation of the extent to which proposed costs "indicate a clear understanding of, and sound approach to, the requirements of the program." Realism was also evaluated by development of most probable cost estimates.

The Air Force used a color rating and risk assessment scheme for evaluating technical proposals. "Blue" was considered "exceptional" and was defined as exceeding specified performance or capability with high success probability and no significant weakness. "Green" was considered "acceptable" and was defined as meeting standards with good probability of success and weaknesses which could be readily corrected. "Yellow" was considered "marginal" and was defined as failing to meet standards, with low probability of success and significant but correctable deficiencies. "Red" was considered "unacceptable." Risk assessments were defined according to the potential a proposal had for causing risk of disruption of schedule, increase in cost, or degradation of performance. "High" risk was defined as being "likely" to cause "significant serious risk," "moderate" risk as "potentially" causing "some" risk, and "low" risk having "little potential" for causing risk.

The eight technical subfactors were listed in descending order of importance: (1) "Management Approach," most important; (2) "Requirements Analysis and Integration," (3) "Software Design, Development, and Integration," and (4) "Implementation," of equal value and next in importance; (5) "AFC2S Modernization Support Environment" and (6) "Data Base Management Environment," of equal value; and next in importance; and (7) "Logistics" and (8) "Corporate and Personnel Qualifications," of equal value and last in importance.

The Air Force evaluated each offeror's technical and cost proposals and conducted three rounds of discussions, including clarification requests, deficiency reports, and face to face discussions with each offeror. All four offerors were included in the competitive range and best and final offers (BAFOs) were obtained from each. TRW was rated by the evaluation team as the highest technically of the offerors, receiving "blue" ratings for subfactors 3, 4, 5, and 6 and "green" ratings for the other 4 subfactors. The risk assessment of TRW's proposal was "low" on five evaluation factors, moderate on two factors and high on one factor.

GTE was the lowest technically rated of the four offerors with "yellow" ratings in subfactors 4, 5, and 6 and an overall "very high" cost risk with "high" risk assessments for half of the subfactors. The evaluation team found that GTE had "failed to meet standards in 3 technical items" and was the only offeror whose costs were "judged to be unrealistic." The team concluded:

"This offeror has significant technical deficiencies and is considered high risk for several items. The deficiencies of this offeror are considered potentially correctable. ... However, attempting to correct this offeror's deficiencies would immediately delay the AFC2S program. Further, attempted correction of this offeror is likely to significantly increase GTE's actual costs, making them at least as expensive as the other offerors."

The government's independent cost estimate was $498,815.50. As evaluated for cost realism, TRW's evaluated most probable cost was $444,043,200, while GTE's evaluated most probable cost was $259,168,400. The other two offerors' evaluated costs each exceeded $400 million.

The source selection authority was briefed on the relative strengths, weaknesses, and risks of the various offerors. In part because he was concerned with the significant cost disparity between GTE and the other offerors, the selection authority requested that the various cost risk assessments be quantified and added to the most probable cost figures for each offeror.

This analysis was accomplished in a three-step process in which the offerors' costs were first normalized with regard to expected needs. Second, the relative technical ratings were normalized by calculating and adding the expected cost impact of correcting "yellow" ratings to "green." The third step involved a so-called "Monte Carlo" risk model /1/ to determine the relative confidence in each offeror's cost estimate that the contract could be successfully performed at that cost. The "Monte Carlo" analysis showed a confidence of 92 percent for TRW's evaluated most probable cost of $444 million and 90 percent confidence in an estimate of $439.1 million. For GTE, a confidence level of 0 percent was estimated for its $259,168,400 evaluated most probable cost and a 90 percent confidence level for a figure of $413.9 million.

In his source selection decision selecting GTE for award, the selection authority listed various GTE strengths identified by the evaluation team in technical factors 1, 2, 3, and 8, and concluded that GTE offered "the best overall value" based upon an "integrated assessment," but provided no reasons for this conclusion.

In a negotiated procurement, the contracting agency's selection officials have broad discretion in determining the manner and extent to which they will make use of the technical and cost evaluation results. DLI Engineering Corp., B-218335, June 28, 1985, 85-1 CPD Para. 742, aff'd, DLI Engineering Corp.-- Reconsideration, 65 Comp.Gen. 34, 85-2 CPD Para. 468. Cost/technical tradeoffs may be made, and the extent to which one may be sacrificed for the other is governed only by the tests of rationality and consistency with the established evaluation criteria. Id. An agency may award to a lower priced, lower scored offeror if it determines that the cost premium involved in awarding to a higher rated, higher priced offeror is justified given the acceptable level of technical competence at the lower cost. Dayton T. Brown, B-229664, Mar. 30, 1988, 88-1 CPD Para. 321. However, award to a lower priced, technically "average" proposal over technically superior proposals, where cost is secondary to technical considerations, requires an adequate justification. See DLI Engineering Corp., B-218335, supra, at 6; see also DynCorp, B-232999, Feb. 14, 1989, 89-1 CPD Para. 152.

Here, the source selection statement only listed the various GTE strengths identified by the evaluation team in technical factors 1, 2, 3, and 8, but did not mention the more numerous corresponding identified weaknesses, nor any strengths or weaknesses for factors 4, 5, 6, and 7, despite the fact of GTE's substandard "marginal" (yellow) ratings for factors 4, 5, and 6. While he stated that GTE's cost proposal was reasonable and fair, and "provided the lowest evaluated cost," no mention was made of cost realism or risk, despite GTE's evaluated high risk. Finally, the source selection authority did not discuss the technical ratings of the other proposals, including TRW's significantly superior blue/green proposal. The report indicates that the source selection statement also relied upon the "Monte Carlo" risk model which indicated that GTE's proposal could be upgraded to "green" and still be the low cost offeror.

While the selection authority states that he made an "integrated assessment" and that his decision considered technical risks and costs, these statements fall far short of the requirement to justify cost/technical tradeoff decisions. See DLI Engineering Corp., B-218335, supra. In view of the significant evaluated technical superiority of TRW and specific evaluated weaknesses (yellow ratings and overall high risk assessments in GTE's proposal), the absence of a justification indicates that no tradeoff analysis was made and that the Air Force effectively made cost more important than technical considerations, notwithstanding the RFP statement that technical factors were most important.

The "Monte Carlo" risk analysis cannot here be relied upon to justify the selection. Although we do not question that risk analysis can be a useful evaluation tool, it does not take the place of a cost/technical tradeoff analysis. As is recognized in the Air Force's own instructions for use of this model, the results must be carefully interpreted, since the input does not account for all factors involved and thus the results "demand subjective evaluation." Here, for example, the Air Force's risk analysis of GTE's marginal, high risk proposal only indicates what the eventual cost of GTE's proposal might be in the event of an award, if corrections are necessary. The risk analysis does not reveal whether GTE's lower- priced, technically inferior proposal is more advantageous to the government than TRW's technically superior proposal. Moreover, the assumptions underlying risk analysis are extremely speculative and may be unreliable. For example, although the "Monte Carlo" model was designed to provide costs necessary to bring technical deficiencies (yellows) to acceptable levels (greens), the fact remains that GTE's proposal did not change. Its approach still warranted the yellow-high risk ratings and there is no indication in the record that GTE was required to adopt the Air Force solutions necessary to achieve acceptability.

Moreover, even if we accept the assumptions and results of the "Monte Carlo" analysis, it only indicates a 6 percent cost difference between GTE's hypothetically upgraded proposal and TRW's technically superior proposal at the point where the analysis indicated a 90 percent confidence level that the firms would be able to successfully perform the contracts. Given the admonition concerning the use of the "Monte Carlo" results, this could well be within the range of statistical error.

Although TRW's cost is more than 50 percent higher than GTE's evaluated most probable cost, the source evaluation team documentation reveals that it had significant concerns with GTE's "high risk," lower rated proposal, even after conducting three rounds of discussions. Even though these deficiencies were considered "potentially correctable," the evaluation team believed that GTE's unrealistically low costs would ultimately approximate those of the other offerors. Where, as here, the awardee's low cost is far below both the government's own estimate and the other offers submitted, the great difference may be less indicative of a cost premium for technical superiority and more indicative of the low offeror's lack of understanding of the government's needs. DLI Engineering Corp., B-218335, supra.

It is true that the source selection authority is not bound by the ratings and recommendations of the source evaluation team. See Associations for the Education of the Deaf, Inc., B-220868, Mar. 5, 1986, 86-1 CPD Para. 220. Nevertheless, he is required to have a reasonable basis for the award selection which is not apparent here. See DLI Engineering Corp., B-218335, supra. In particular, the source selection decision does not explain how GTE's low cost offsets its relatively low technical and high risk ratings. Consequently, in the absence of any comments in the source selection statement concerning relative technical merit, it appears that the source selection decision was based on the lowest proposed cost. That technical factors had predominant weight under the RFP evaluation scheme was repeatedly emphasized to TRW during discussions. /2/In short, the record provides no assurance that the award to GTE was consistent with the terms of the solicitation, which assigned primary importance to technical considerations and we sustain the protest on this basis. DynCorp, B-232999, supra; DLI Engineering Corp., B-218335, supra. Accordingly, we recommend that the agency conduct and document a cost/technical tradeoff to determine whether TRW's technical superiority is insufficient to justify the cost premium of accepting its offer and whether an award to GTE in the face of its low technical scores and high risk is justified. If the Air Force determines that TRW's (or another offeror's) evaluated cost premium is justified, then the contract with GTE should be terminated for the convenience of the government.

TRW also had contended that GTE should be excluded from the competition due to an alleged conflict of interest on the part of GTE's subcontractor, ERCI. As a subcontractor on a Department of Energy contract, ERCI produced a draft statement of work for the software modernization program which the Air Force revised and rewrote before including it in the RFP. Prior to the issuance of the RFP, the contracting officer determined that ERCI could nevertheless compete for the AFC2S contract. TRW maintains that not all aspects of ERCI's prior performance were considered in that determination. In particular, TRW claims that ERCI had access to inside information which assisted GTE in preparing its proposal.

We believe it is clear that TRW suffered no prejudice from ERCI's alleged conflict of interest, inasmuch as GTE apparently was given no great insight since it was ranked last in technical merit. Moreover, we note that TRW was aware of the potential for ERCI's conflict of interest and the Air Force decision clearing its participation for nearly a year before filing this protest, which it only did after being apprised it did not receive the award. Under the circumstances, and in the absence of credible evidence showing an actual conflict (see Petro Engineering, Inc., B-218255.2, June 12, 1985, 85-1 CPD Para. 677), we find no merit to this protest basis.

In any event, since we have found that the Air Force selection decision was flawed, we find that TRW is entitled to the reasonable costs of filing and pursuing the protest. Bid Protest Regulations, 4 C.F.R. Sec. 21.6(d)(1).

The protest is sustained.

/1/ This cost risk analysis model is generally used for quantifying the lowest and highest possible costs of weapons systems, based upon estimated costs of various components.

/2/ We do not agree with TRW that the discussions conducted with it were less than meaningful or misleading, inasmuch as (1) the RFP also clearly indicated that technical merit was the predominant evaluation factor; (2) TRW's costs were considered reasonable for its technical approach; and (3) deficiencies and corrections were extensively pointed out in the three rounds of discussions.