Skip to main content

Matter of: Chemical Demilitarization Associates File: B-277700 Date: November 13, 1997 * Redacted Decision

B-277700 Nov 13, 1997
Jump To:
Skip to Highlights

Highlights

Cannot be determined reasonable where the evaluation record reflects differences in the merit of competing technical proposals and the agency's rationale for finding the proposals equivalent under every criterion is not supported by the record. CDA alleges that the source selection decision is unreasonable. Demilitarization facilities for destroying chemical weapons have been or will be constructed and operated at nine sites where the weapons are located. A contract for the prototype chemical demilitarization facility was awarded in 1986 to Raytheon for the Johnston Atoll Chemical Agent Disposal System (JACADS). A contract for the second facility was awarded in 1989 to EG&G. A third contract was awarded in 1996 to Westinghouse Electric Corporation for the Anniston Chemical Agent Disposal Facility in Alabama.

View Decision

Matter of: Chemical Demilitarization Associates File: B-277700 Date: November 13, 1997 * Redacted Decision

DIGEST

Attorneys

DECISION

Chemical Demilitarization Associates (CDA), a joint venture of EG&G, Inc. and Morrison Knudsen, protests an award to Raytheon Demilitarization Company under request for proposals (RFP) No. DAAA09-92-R-0351, issued by the Department of the Army, Armament, Munitions and Chemical Command (AMCCOM), Rock Island, Illinois, for construction, equipment installation, systemization, operation and closure of the Pine Bluff Chemical Agent Disposal Facility (PBCDF), Arkansas. CDA alleges that the source selection decision is unreasonable.

We sustain the protest.

Demilitarization facilities for destroying chemical weapons have been or will be constructed and operated at nine sites where the weapons are located. A contract for the prototype chemical demilitarization facility was awarded in 1986 to Raytheon for the Johnston Atoll Chemical Agent Disposal System (JACADS). A contract for the second facility was awarded in 1989 to EG&G, Inc. for the Tooele Chemical Agent Disposal Facility in Utah. A third contract was awarded in 1996 to Westinghouse Electric Corporation for the Anniston Chemical Agent Disposal Facility in Alabama. Most recently, a fourth contract was awarded in February 1997 to Raytheon for the Umatilla Chemical Agent Disposal Facility in Oregon.

The RFP, issued on July 8, 1994, contemplated the award of a combination fixed-price and cost-plus-award-fee contract. The RFP stated that award would be made on a best value basis considering the areas of technical, management, past performance, and cost/price. The RFP, as amended, stated the relative importance of each of these areas as follows:

Technical approach is more important than management. Management is somewhat more important than either of past performance or cost/price. Past performance and cost/price are of equal importance.

The technical and management areas were divided into elements, which in turn were divided into factors and, in the case of one factor, subfactors--this amounted to 27 constituent evaluation criteria in these two areas whose relative importance was stated in the RFP. [1] As stated in the RFP, the technical and management areas would be numerically scored and assessed for proposal risk.

Past performance had no stated subordinate elements or factors (although the RFP stated in narrative what would be considered under this area). A performance risk assessment group (PRAG) was to evaluate the quality of an offeror's past performance and assign a risk rating of low, moderate, high, or unknown (i.e., no relevant experience).

Cost/price was to be evaluated, but not numerically scored. The cost/price evaluation would also include an integrated assessment of proposal risk.

CDA, Raytheon, and Westinghouse responded to the RFP. /2/ The agency conducted multiple rounds of discussions and ultimately requested two rounds of best and final offers (BAFO). Offerors submitted their second round of BAFOs by April 28, 1997.

The source selection evaluation board (SSEB) evaluated the proposals under the technical and management areas by assigning adjectival ratings and corresponding scores on a 100-point scale. /3/

The evaluation resulted in CDA's proposal receiving a technical score of 90.6 compared to 84 for Raytheon. Within the technical area, CDA's proposal received a higher score than Raytheon's under 12 of the 16 constituent criteria, of which the difference between the two proposals' scores for about half of these was 15 or more points. Raytheon's proposal did not receive a higher score than CDA's under any of the technical criteria. This difference in scores resulted in CDA's proposal receiving a higher score than Raytheon's under every evaluation element of the technical area.

Under the management area, Raytheon's proposal received a score of 89.8 compared to 84.6 for CDA. In contrast to the technical area, the margin here resulted solely from Raytheon's proposal receiving a higher score for one of the constituent criteria, corporate baseline management, which was the most important element in the management area and had no subordinate factors. This was the only criterion in the management area for which Raytheon's proposal received a higher score than CDA's proposal.

In addition to the scores and ratings, the SSEB identified and described with narrative comments the advantages and disadvantages associated with each proposal under each constituent criterion. A panel then determined which of these advantages and disadvantages were considered "programmatic" (i.e., whether an advantage or disadvantage enhanced or degraded performance, or reduced or increased performance risk). The programmatic advantages and disadvantages were those presented to the source selection authority (SSA) as those that were important. Hearing Transcript (Tr.) at 28, 482-83. The following table shows the number of advantages and disadvantages, both overall and programmatic (prog.), found for each proposal:

Tech- Adv- Tech- Disadv- Manage- Adv- Manage- Disadv- nical antages nical antages ment antages ment antages Offeror Total Prog. Total Prog. Total Prog. Total Prog.

CDA 22 7 /4/ 0 0 3 2 2 1

Raytheon 4 3 3 0 4 1 2 1

The overall combined weighted score under the technical and management areas was 88.1 for CDA's proposal and 86.4 for Raytheon's proposal. Both were rated "outstanding" overall with low proposal risk. Under past performance, CDA was rated a moderate performance risk and Raytheon a low performance risk. The cost/price evaluation assessed a moderate risk to both cost proposals. The evaluated prices and costs follow (in millions of dollars):

Offeror Fixed Price Proposed Cost Upward Cost Adjustment Total

CDA $217.6 $276.5 $5.8 $499.9

Raytheon $206.5 $305.1 $3.0 $514.6

The SSEB reported its findings to the source selection advisory council (SSAC). The SSAC analyzed the SSEB's findings and reported to the SSA. The SSAC report stated the following conclusions:

All offerors determined to be outstanding, based on evaluation of their proposals against [technical/management] criteria.

CDA and [Raytheon] determined to represent equivalent, lowest proposal risk, based on [Technical/Management] Tradeoff Analysis.

[Raytheon] determined to represent lowest performance risk, based on Past Performance Assessment.

The SSAC report then analyzed the cost/price evaluations showing that CDA was lowest overall and stated the following conclusion:

[Raytheon] represents best value to the Government: -Compared to CDA: [Raytheon's] lower performance risk offsets 2.9 [percent] cost advantage of CDA . . .

The SSA determined that there were no meaningful differences under any of the 27 technical and management evaluation criteria between Raytheon's and CDA's proposals. Tr. at 608. The SSA's source selection decision document stated:

a. All offerors' Technical and Management proposals are outstanding. First, I reviewed Technical and Management areas, which had been individually evaluated and scored. In order to determine the overall integrated benefits offered by the proposals, I also assessed Technical and Management as a whole, considering the relative weights of the two areas, as well as each offeror's unique set of programmatic advantages and disadvantages, and the relative importance of those programmatic advantages and disadvantages. As a result of this review, I find that CDA and [Raytheon] are equivalent in Technical/Management approaches, and also provide the lowest overall integrated Technical/Management proposal risk. . . . CDA represents a higher performance risk based on EG&G's past performance at Tooele . . . .

5. This contract award is one of a succession of contract awards for the [Chemical Stockpile Demilitarization Program]. . . . Because of the successive solicitations including discussions and debriefings, all the offerors have been able to improve their technical and management proposals and produce technical/management plans which are overall considered outstanding. Similarly, the offerors' cost proposals have also become extremely competitive. . . . CDA and [Raytheon] each have one of [the two currently] operating plants. . . . At this point in time, the Government has more confidence in the performance potential for [Raytheon] than it does for CDA, based on current activities and experiences at these two facilities.

6. Award Decision: After carefully reviewing and assessing the results of the Technical, Management, Past Performance, and Cost/Price evaluations of the three offerors, I find that [Raytheon] represents the best value to the Government for the PBCDF project. When compared to CDA, both have outstanding and low risk technical/management approaches. However, [Raytheon's] lower performance risk clearly justifies the cost premium. . . .

The agency awarded the contract to Raytheon on July 25. After requesting and receiving a debriefing, CDA filed this protest. The protest challenges the source selection decision with respect to the determination of technical/management equivalence and the relative assessments of performance risk. The agency has stayed performance pending resolution of this protest.

TECHNICAL/MANAGEMENT EQUIVALENCE

CDA first alleges that the determination that proposals were equivalent under the technical and management areas is unreasonable. CDA explains that despite the clear evaluated differences between CDA's and Raytheon's proposals, the agency unreasonably determined that the proposals were equivalent under every individual technical and management evaluation criterion. Thus, CDA alleges that the source selection decision improperly failed to give any weight to the technical and management evaluation areas, even though they were supposed to account for 60 percent of the evaluation weight.

In response, the agency contends that the disparity in scores in and of itself does not evidence an evaluated superiority of either proposal under any criterion. Rather, the agency asserts that the design and operation restrictions of the RFP, combined with both offerors' having competed in several similar procurements, allowed little margin for differentiation under any evaluation area other than past performance.

In a negotiated procurement with a best value evaluation plan, point scores and adjectival ratings are only guides to assist contracting agencies in evaluating proposals; they do not mandate automatic selection of a particular proposal. Grey Advertising, Inc., 55 Comp. Gen. 1111, 1118 (1976), 76-1 CPD Para. 325 at 9; PRC, Inc., B-274698.2, B-274698.3, Jan. 23, 1997, 97-1 CPD Para. 115 at 12. Source selection officials have broad discretion in determining the manner and extent to which they will make use of the technical and cost evaluation results, subject only to the tests of rationality and consistency with the evaluation criteria. Grey Advertising, Inc., supra; A & W Maintenance Servs., Inc.--Recon., B-255711.2, Jan. 17, 1995, 95-1 CPD Para. 24 at 4.

Where, as here, the agency determines that a higher technical score does not reflect actual technical superiority, it must show that the agency reasonably concluded that no such technical superiority exists. DynCorp, 71 Comp. Gen. 129, 133-34 (1991), 91-2 CPD Para. 575 at 6-7. Such a showing must be sufficiently detailed to permit our Office to review the determination for reasonableness. Id.; compare Dayton T. Brown, Inc., B-229664, Mar. 30, 1988, 88-1 CPD Para. 321 at 5-7 (reasonable justification for determining that evaluated merit did not reflect significant actual difference) with DLI Eng'g Corp., B-218335, June 28, 1985, 85-1 CPD Para. 742 at 6-8 (unreasonable justification).

In the course of a hearing conducted by our Office, the evaluation and source selection officials provided testimony on the determination of equivalence under the technical and management areas. All agreed that the proposals were found equivalent under the technical area as well as under the management area. Tr. at 355, 484, 575, 614-15. The SSA testified that the two proposals had no meaningful differences under any of the 27 technical and management evaluation criteria. Tr. at 608. He confirmed this judgment with regard to his assessment of various individual elements and factors. Tr. at 511-12, 540-50, 575, 596-98. In our view, the evaluation and hearing records do not support this determination.

The most marked difference between the proposals is under technical, the most important area, where CDA's proposal received a higher raw score under three quarters of the criteria, a large number of which reflected a significant difference in score, and Raytheon's proposal did not receive the higher score on any of the criteria. /5/ Although these scores are only guides, our review of the detailed evaluation narratives describing the advantages and disadvantages showed that the higher scores appear to reflect actual qualitative differences between the two proposals.

The evaluation and source selection personnel recognize that differences exist, but state that such differences are not meaningful or significant. Tr. at 344, 488, 542, 546-48, 655. However, when explaining why the proposals are equivalent with no meaningful or significant differences under any given criterion, the witnesses responded in very general terms, essentially stating without further elaboration that the agency considered the differences and did not find them meaningful; that equivalence was to be expected because of the procurement history, detailed RFP requirements, and successive rounds of discussions; or that the similarity of adjectival ratings showed equivalence. Tr. at 339-41, 356-57, 420-22, 486-88, 540-43, 545-50, 576-81, 588-98, 653-60.

Where, as here, the evaluation record evidences relative differences in proposal merit, such general statements are inadequate to show equivalency; the agency must compare the relative merits of the proposals in a manner that reasonably supports a determination of equivalency. See Matrix Int'l Logistics, Inc., B-272388.2, Dec. 9, 1996, 97-2 CPD Para. 89 at 9-10.

Although the agency's position is that the procurement history, the detailed requirements of the RFP, and the successive BAFOs support a conclusion that proposals are equivalent in merit, /6/ these events themselves do not show that the proposals actually are equivalent. Nor is similarity in the adjectival ratings proof of technical equality because the pre-established large range of scores within each adjectival rating allowed for large variations in evaluated merit between proposals receiving the same rating. Since the evaluation record shows documented differences in merit between these two proposals, the similarity of adjectival ratings here does not establish that the proposals are equivalent. See Matrix Int'l Logistics, Inc., supra, at 8-9.

To the extent the SSA could be said to have substituted his judgment for that documented by the evaluators, the record shows that he may not have reasonably considered the relevant information. In this regard, although the SSA's testimony was for the most part not specific enough for us to assess the basis of his judgment. In the one instance where he did specifically explain relative proposal merits, his explanation does not establish that the proposals are equivalent. In that specific instance, i.e., Quality Assurance/Quality Control (QA/QC), a factor under the most important technical element, CDA's proposal was scored at 92 and Raytheon's proposal was scored at 60. In terms of adjectival ratings, these scores placed CDA's proposal above the middle of the "Outstanding" range, while Raytheon's was at the bottom of the "Excellent" range (bordering on "Satisfactory"). CDA's identified programmatic advantage was that "[DELETED]." CDA was also found to have a [DELETED], which was likewise determined to be a programmatic advantage, although this was inadvertently omitted from the final list of programmatic advantages presented to the SSA. Tr. at 126.

On the other hand, Raytheon's proposal had one nonprogrammatic advantage attributed to [DELETED] when that change occurs. Raytheon's proposal also had the following nonprogrammatic disadvantage:

[DELETED]

[DELETED]

[DELETED]

[DELETED]

During the hearing conducted by our Office, the SSEB chairperson testified that the SSEB never characterized the two proposals as equivalent under this factor. Tr. at 134. The stated reason that Raytheon's disadvantage was not designated as programmatic (and therefore not included in the SSAC report to the SSA) was that the cognizant evaluators decided to designate such advantages or disadvantages as either a programmatic advantage for one proposal, or a programmatic disadvantage for another, but not as programmatic for both, so as to avoid "overly penaliz[ing] for a particular finding if it applied to a single attribute like quality." Tr. at 131. Thus, the SSAC report to the SSA showed only a programmatic advantage for CDA under this criterion and no programmatic disadvantage for Raytheon.

Although we think that this evaluation documentation evidences a significant difference between the proposals on QA/QC, the SSA stated that this was not a meaningful difference. He understood the evaluated difference to be that [DELETED], and he thought the differences "could be easily - not easily but certainly handled administratively once you assign the contract." Tr. at 511. This explanation does not address the totality of the problems found in the evaluation of Raytheon's QA/QC approach, in that it does not account for [DELETED]. We also note that, but for CDA's programmatic advantage under this criterion, the concerns with Raytheon's QA/QC plan should have been designated a programmatic disadvantage for Raytheon. Tr. at 131-32. Thus, the SSA's specific testimony suggests that he was not cognizant of, or did not understand, the evaluated differences between the proposals with regard to QA/QC, and that he may not have fully understood how programmatic differences were designated. In any event, the determination that the proposals are equivalent under QA/QC was not supported by the evidence. /7/

Similarly, the agency has not explained why the documented differences in the evaluation of the proposals for the other criteria, particularly those where CDA's proposal received a notably higher (15 points) score (general approach, laboratory operations, environmental compliance, and utilization of core employees) were not meaningful. /8/

The agency and intervenor state that the differences on any given criterion are not significant because the weights of the individual criteria are low overall, considering the presence of so many other criteria. This argument fails to account for the underlying rationale of the equivalence determination and the ultimate selection decision--that is, that the two proposals were equivalent on every technical and every management criterion--because the designated weight of any given criterion is irrelevant in determining whether two proposals are or are not equivalent under that criterion.

Here, once the proposals were found equivalent under the technical and management areas, the source selection decision was based on a trade-off between past performance risk and cost/price. Tr. at 503-05, 565, 614-18. In other words, the two most heavily weighted evaluation areas, technical and management, were ultimately discarded from the source selection decision. Since the agency has not reasonably determined that the proposals are equivalent under the technical and management areas, the agency has not meaningfully assessed the relative merits of the proposals under the stated evaluation plan. The trade-off analysis was thus unreasonable and inconsistent with the stated selection plan and cannot be the basis for awarding the contract. See Matrix Int'l Logistics, Inc., supra, at 10.

Counsel for the agency and the intervenor contend, in the alternative, that even if CDA's proposal is technically superior to Raytheon's proposal, Raytheon's proposal is superior to CDA's under the management area and, applying the established factor weights, the overall weighted technical/management scores are so close as to indicate overall equivalence attributable to offsetting proposal strengths and weaknesses. Although we will consider information submitted during a protest, including the parties' arguments, explanations, and hearing testimony, in considering the entire record we accord greater weight to contemporaneous evaluation and source selection material. See Matrix Int'l Logistics, Inc., supra, at 5-6. A finding of equivalence based on offsetting merit may be reasonable where the source selection officials have made such a determination. See, e.g., Employee Assistance Serv., B-207057, July 19, 1982, 82-2 CPD Para. 56 at 3. A properly documented, considered judgment of the source selection authority is critical in this case, however, since the evaluation results, when correctly considered under the stated evaluation criteria, are so close as to make selection of either offeror a reasonable possibility.

The SSA testified that he did not perform a trade-off analysis along the lines of the alternative defense suggested by counsel. Tr. at 596-98. Moreover, although the SSEB chairperson initially testified that meaningful evaluated differences between CDA's and Raytheon's proposals existed, Tr. at 117-18, 145-48, 152-53, the agency recalled the witness on a later date to deny that any meaningful differences existed. Tr. at 342-44. This alternate technical/management trade-off rationale thus does not appear to be the view of the evaluation and source selection officials, but rather that of counsel seeking to defend the agency's source selection. Because it has been offered in the heat of an adversarial process and may not represent the fair and considered judgment of the agency, which is a prerequisite of a rational evaluation and source selection process, we accord this alternate rationale little weight. Boeing Sikorsky Aircraft Support, B-277263.2, B-277263.3, Sept. 29, 1997, 97-2 CPD Para. 91 at 15.

PAST PERFORMANCE RISK

CDA also protests that the past performance risk evaluations and the assessment of those evaluations by the SSA are unreasonable. CDA essentially contends that not enough significance was accorded in Raytheon's risk assessment to the release of a chemical agent at JACADS in 1994, and that too much significance was accorded in CDA's risk assessment to recent unfavorable customer satisfaction assessments of EG&G's performance at Tooele. We believe that the record does not demonstrate that the past performance risk ratings are unreasonable.

Raytheon's actions were apparently not the sole cause for the chemical agent release at JACADS. The agency identified a government design defect as playing a significant role in the release, and that defect has been corrected at JACADS as well as in the design of other facilities. Although Raytheon's staff reportedly did not properly execute relevant procedures at the time, such lapses have not occurred since. The SSA was aware of the design defect and the lapse in following procedure. Ultimately, he determined that the correction of the design defect will prevent the same type of release from happening at any facility. This, in his view, together with the quality performance of Raytheon at JACADS since 1994, provides a sufficient level of confidence to reduce the level of risk that would usually attach to a contractor's release of a chemical agent. Tr. at 495-97, 520-24.

In contrast with this problem from 1994, the customer satisfaction issues for EG&G at Tooele have occurred in the last year. The concerns were associated with cost overruns, a failure to follow procedures resulting in a plant shutdown, the failure to implement required systems such as a cost control system, and a general lack of corporate support. Tr. at 269-70. Although some of the agency's concerns have been corrected, and EG&G has discussed with the agency the contractor's plans for addressing the other concerns, the agency has determined that an increased level of risk will remain until all of the proposed action is successfully implemented. Tr. at 498-502, 557-61.

Neither of these risk assessments is unreasonable. An agency may reasonably give less weight to older performance problems where the contractor's subsequent performance has been good. E. Huttenbauer & Son, Inc., B-257778, B-257779, Nov. 8, 1994, 94-2 CPD Para. 206 at 7. An agency may also place more significance on recent performance problems. See Federal Envtl. Servs., Inc., B-250135.4, May 24, 1993, 93-1 CPD Para. 398 at 9. To the extent the agency's concerns here became the significant discriminator in the past performance area, the RFP specifically advised offerors that such a finding in any element of an offeror's performance history could be "an important consideration in the source selection process."

CONCLUSION

Since the agency did not reasonably consider the relative merits of the proposals under the technical and management areas, we cannot find that the source selection decision is reasonable and consistent with the terms of the RFP. We recommend that the agency make and document a new source selection decision consistent with the stated evaluation criteria. If an offeror other than Raytheon is selected, the agency should terminate Raytheon's contract and make award accordingly. We also recommend that the protester be reimbursed the reasonable costs of filing and pursuing its protest, including attorneys' fees. 4 C.F.R. Sec. 21.8(d)(1) (1997). The protesters certified claim for costs must be submitted to the agency within 60 days of receiving this decision. 4 C.F.R. Sec. 21.8(f)(1).

The protest is sustained.

Comptroller General of the United States

1. The record refers to 28 criteria. One of these criteria covering facility closure was evaluated under another criterion (i.e., general approach) and was not separately scored.

2. Because Westinghouse's proposal is not at issue in this protest, it is not discussed further.

3. The source selection plan, which was not released to the offerors, identified the following rating and point scale:

Score Rating

80 - 100 Outstanding

60 -

40 -

20 -

0 -

4. The report carried forward to the SSA identifies only six technical programmatic advantages for CDA. The SSEB chairperson stated that the seventh programmatic advantage was inadvertently omitted. Tr. at 126.

5. The reasonableness of the underlying evaluations is not challenged by any party.

6. The SSA described his "mind-set" when he reviewed the technical and management evaluations as expecting the proposals from these offerors to be outstanding. Tr. at 548-49.

7. Although an SSAC member testified about another aspect of Raytheon's proposal that was rated as a programmatic advantage under another criterion that had some application to QA/QC, he stated that this crossover was "very small." Tr. At 667.

8. For example, protester's counsel questioned the SSA about the most important evaluation element, i.e., operations conceptual approach (under which 4 of the constituent criteria CDA had a higher score of at least 15 points), going through numerous examples of evaluated differences, and the SSA only acknowledged advantages in CDA's proposal not present in Raytheon's (e.g., [DELETED]). In response to this questioning, the SSA did not provide a specific explanation as to why those differences are not significant or meaningful. See, e.g.,Tr. at 576-81. While the SSEB chairperson later denied that there were meaningful differences between the proposals under any of the criteria, Tr. at 342-344, he initially testified that the proposals were not equivalent with regard to QA/QC, Tr. at 134, and that there were meaningful differences favoring CDA under laboratory operations, Tr. at 117-18, and the utilization of core employees, Tr. at 145, and that, considering the entire basis for the evaluation, CDA's and Raytheon's proposals were not equivalent under either the technical or management area, Tr. 152-53. Finally, when questioned about specific evaluated differences between the two proposals, an SSAC member testified that there was a difference under laboratory operations, but he could not say whether it was significant, Tr. at 653-54, and that the proposals were not equivalent under QA/QC, Tr. at 655.

* DOCUMENT FOR PUBLIC RELEASE

The decision issued on the date below was subject to a GAO Protective Order. This redacted version has been approved for public release.

GAO Contacts

Office of Public Affairs