B-227839.2, Nov 9, 1987, 67 Comp.Gen. 58

B-227839.2, Nov 9, 1987, 67 Comp.Gen. 58

PROCUREMENT - Bid Protests - GAO Procedures - Protest Timeliness - 10 Day Rule DIGEST: 1. Where the protester does not learn of the weight the agency gave to certain technical/performance evaluation factors until the debriefing conference, a protest that the agency gave too much weight to those technical/performance factors and too little weight to price is timely when filed within 10 working days after the debriefing conference. 2. Protest that the Army's testing of protective masks and analysis of those test results bear no relation to real battle situations and therefore should not have been used to predict casualties is dismissed as untimely where the protester was aware of the test methods, witnessed the tests, and apparently was satisfied with the testing during the 2 1/2 year period during which tests were conducted. It was only after the protester's mask was shown to be rated lower than the awardee's mask that the protester voiced complaints about testing and analysis-- about 8 months after the completion of testing. PROCUREMENT - Competitive Negotiation - Requests for Proposals - Evaluation Criteria - Cost/Technical Tradeoffs - Weighting 3. Where the request for proposals (RFP) indicates that technical/performance, cost, and production capability will be considered in the evaluation of proposals, without any indication of each factor's relative weight, each factor is assumed to be accorded substantially equal weight in the evaluation; protest of the evaluation is sustained where the agency considered the technical/performance factor to be significantly more important than the other factors set forth in the RFP.

ILC Dover, Inc.:

ILC Dover, Inc., protests the award of a fixed-price, multi-year contract to Scott Aviation Company by the Department of the Army under request for proposals (RFP) No. DAAA15-87-R-0035. Under the contract, Scott is to produce and supply the Army with 300,000 chemical/biological protective masks and related items. The contract also contains an option for an additional 150,000 masks. Scott is to supply 87,900 units of the basic mask (designated the XM40) for use by infantrymen and 212,100 units of a variant of that mask (designated the XM42) for use by combat vehicle crewmen. ILC Dover contends that the award to Scott was improper because the Army did not evaluate proposals in accord with the evaluation criteria set forth in the RFP.

We sustain the protest.

BACKGROUND

In 1982, the Army canceled a program to develop a new mask, designated the XM30, because the mask proved unsatisfactory during testing (in large part due to limitations in its polyurethane flexible lens). At that time, the Army began a two-phase program to develop the XM40 series mask (the subject of the present RFP). The XM40 mask and its variants will replace the Army's current family of protective masks, and the mask design chosen by the Army under the present RFP will be the Armed Forces' chemical/biological mask through the end of this century.

Late in 1982, the Army solicited proposals from several firms for Phase I of the XM40 series mask development program. Contracts were awarded to three firms: ILC Dover, Scott, and Mine Safety Appliances Company. The contractors were to conduct design studies using "minimum change/minimum risk" approaches to develop a new protective mask by retaining the positive features of the XM30 mask and incorporating the rigid lens system of a previously proven mask (the M17A1). The contractors also were to fabricate a small number of prototypes for testing, issue a design report, and submit proposals for Phase II of the program. Based on the design reports, evaluation of the test prototypes, and the fabrication proposals, the Army approved mask designs and awarded Phase II contracts to ILC Dover and Scott only.

Under the Phase II contracts, ILC Dover and Scott each produced over 1,000 masks of their own design for extensive evaluation and testing by the Army. In addition, the two contractors provided engineering support to the Army during product testing, fabricated the necessary tooling and molds, and updated the XM30 mask technical data package to document the approved mask design and incorporate changes made during testing. Testing was completed in December of 1986. Based upon these tests, the Army concluded that both the ILC Dover and mask candidates fulfilled the Joint Services Operational Requirements (JSOR), which set forth the essential characteristics and levels of protection required for protective masks.

The Army determined that it was in the public interest to limit the competition for the initial production contract to Scott and ILC Dover, the only two firms that had participated in the second phase of the development program. Accordingly, the Army issued the present RFP to Scott and ILC Dover on February 6, 1986. Both firms submitted timely proposals, and after evaluation of proposals and the Phase II test results, the Army awarded a contract to Scott on June 24, 1987. The reason for selecting Scott, even though ILC Dover's offer was considerably lower in price, was the Army's conclusion that Scott's proposal was better than ILC Dover's in the two most important (of the 11) technical/performance subfactors: "protection" and "reliability, availability and maintainability" (RAM). ILC Dover filed its protest in our Office on July 2.

PROTEST ISSUES

ILC Dover contends that the Army's decision to award the contract to Scott is contrary to the evaluation scheme set forth in the RFP. ILC Dover believes that its masks meet or exceed all requirements of the JSOR and are substantially as good as the Scott masks. ILC Dover also believes that its production capabilities are superior to Scott's. ILC Dover concludes that it therefore should have been awarded the contract because its price for the basic quantity was only $35,019,750, while Scott's price was $51,823,181, or $16,803,431 higher.

ILC Dover also complains that the Army's evaluation was unreasonable because the Army tried to predict the number of battlefield casualties that would occur using each mask from "quantitative fit" laboratory tests the Army conducted; ILC Dover contends that there is no direct relationship between the laboratory tests and the protection afforded soldiers when the masks are used on the battlefield. ILC Dover alleges that the Army's statistical analysis of the protection factor data collected in laboratory tests was flawed in a number of respects so that while it appeared that the Scott mask might have a significant protection factor advantage over the ILC Dover mask, the data collected in fact do not reveal a statistically significant difference between the masks.

TIMELINESS

The Army and Scott argue that it was clear from the RFP's evaluation scheme that the Army intended to give significant weight in its evaluation to the protection and RAM subfactors. Therefore, they contend that ILC Dover's basic argument-- which they construe as being that the Army had to give the same score to both masks on technical/performance factors because both met all JSOR requirements-- is untimely under section 21.2(a)(1) of our Bid Protest Regulations. 4 C.F.R. Part 21 (1987).

Scott and the Army misconstrue ILC Dover's protest. The thrust of ILC Dover's protest is that the Army did not actually evaluate proposals in accord with the criteria set forth in the RFP. ILC Dover acknowledges that the Army properly could evaluate proposals under each of the 11 separate technical/performance subfactors set out in the RFP, but argues strenuously that the Army gave too much weight to the protection and RAM subfactors and too little weight to price. The Army told ILC Dover at a June 30th debriefing that its award decision was based upon the Source Selection Authority's determination that Scott's significant superiority in the protection and RAM areas outweighed the cost savings of ILC Dover's proposal. Since ILC Dover filed its protest on July 2, within 10 working days after the debriefing conference at which it learned the specific basis for the award, this portion of the protest is timely. Intelcom Educational Services, Inc., B-220192.2, Jan. 24, 1986, 86-1 CPD Para. 83; 4 C.F.R. Sec. 21.2(a)(2).

ILC Dover also argues that during development of the MX40 mask design, the Army's testing of mask prototypes was flawed in a number of ways. Further, ILC Dover believes the Army's analysis of the test results was invalid and the Army improperly tried to predict actual battlefield casualties from tests that bear little or no relation to real battle conditions.

The Army admits that to some degree its testing methods have weak points with respect to predicting what might occur on the battlefield. The Army is adamant, however, that it did the best job possible and that its test methods and analysis were reasonable. In this respect, the Army reports that it built its mask testing facility and designed its testing procedures only after extensive consultation with government and private industrial hygiene experts throughout the world.

This basis of protest is untimely. The Army reports that ILC Dover representatives were brought into the test facility to assist the Army in conducting some of the earliest tests. The Army also reports that ILC Dover has used the Army's test facility on more than one occasion to evaluate the protective capabilities of various prototypes it was developing for the Army. The contracting officer states that ILC Dover was intimately familiar with the test procedures used by the Army and witnessed the testing of its mask. In this regard, we note that the Army consulted with ILC Dover when its mask was performing poorly due to a leakage problem, that ILC Dover was allowed to fix the leakage by partially redesigning its mask, and that the Army used only test results for ILC Dover's mask obtained after the mask was fixed. The Army also states that ILC Dover was on notice of the statistical methods the Army intended to use to evaluate the test results as early as May 16, 1986, when the Army provided ILC Dover with some of the test results.

ILC Dover apparently was satisfied with the test methods throughout the 2 -1/2 year period of testing, and only complained of the alleged deficiencies when the tests showed its mask to be less technically proficient than the Scott mask. In our opinion, it is unreasonable for ILC Dover to lodge its first complaints about testing and analysis 8 months after the tests were completed. See, for example, Cadillac Gage Co., B-209102, July 15, 1983, 83-2 CPD Para. 96. In fact, we note that in its initial protest letter the protester made only a general statement that the difference between its design and Scott's design in protection scores was statistically meaningless. ILC Dover waited almost 8 week longer-- until it filed its comments on the Army report and a conference on its initial protest-- to provide our Office and the Army any substantive statement on just why it believed the Army's analysis to be flawed. We dismiss this issue as untimely. 4 C.F.R. Sec. 21.2.

In any event, the protester's arguments concerning the validity of the agency's testing and analysis boil down, in our view, to a disagreement over the concept of using laboratory data to predict battlefield results. While we would agree with Dover that there may not be a perfect correlation, we have no basis here to conclude that the agency acted unreasonably in using what it determined were the best testing and analysis methods available.

WERE THE EVALUATION AND THE AWARD DECISION IN ACCORD WITH THE RFP's STATED SCHEME?

The evaluation of proposals is the function of the procuring agency, requiring the exercise of informed judgment and discretion. Our review is limited to examining whether the agency's evaluation was fair and reasonable and consistent with the stated evaluation criteria. We will question contracting officials' determinations concerning the technical merits of proposals only upon a clear showing of unreasonableness, abuse of discretion, or violation of procurement statutes or regulations. KET, Inc., B-190983, Dec. 21, 1979, 79-2 CPD Para. 429.

As the basis for award, the RFP stated: "The Government will select that mask which represents the best overall value to the Government, performance, cost and other factors considered." The RFP stated that proposals would be evaluated under three factors: (1) performance requirements-- based upon the JSOR requirements and the assessment of Phase II and follow-on tests; (2) cost, including proposed price, maintenance and repair parts costs, warranty costs, and the costs of integrating certain planned product improvements; and (3) the offeror's production capability. The RFP listed eleven performance subfactors as follows: A. Protection B.Reliability, Availability and Maintainability (RAM) C. Vision/Optical Coupling D. Speech/Communication E. Filter Change F. Wearability/Comfort G. Drinking H. Compatibility I. Logistical Supportability J. NBC Survivability K. Climatic Considerations

The RFP stated: "Protection is significantly more important than any other of the above factors. All other factors are listed in descending order of importance."

The Army admits that its extensive testing revealed that ILC Dover's mask meets or exceeds the protection standards set out in the JSOR, but the Army stresses that it was not its intent to purchase masks that merely meet the JSOR minimum requirements. The Army argues that while ILC Dover's test results surpassed the JSOR standards, Scott's test results were superior to ILC Dover's in two critical areas: protection and RAM. The Army points out that the RFP reserved to the Army the right to award a contract to other than the lowest-priced offeror.

In selecting the Scott proposal, the Source Selection Authority offered the following rationale:

"With regard to performance, the Scott Aviation candidate offers enhanced protection with resulting casualty reduction over the ILC Dover candidate. In the area of production both prospective contractors were assessed as having the capability to produce their respective designs. Finally, with regard to cost, the Scott Aviation mask was found to have a higher proposal price than the ILC Dover mask; however, the enhanced protection offered by the Scott Aviation candidate, with the resultant reduced chemical casualties, outweighs the proposal price advantage offered by the ILC Dover candidate."

The Army argues that the primary mission of the mask selected is protection of soldiers and that the superiority of the Scott mask in protection and RAM will allow it to fulfill that mission much better than the ILC Dover mask. Specifically, the Army determined that the probability that a Scott mask would successfully complete a 72-hour mission was 98.5 percent compared to ILC Dover's 98.0 percent probability of a successful mission. The Army also predicted that ILC Dover's mask would suffer 33.3 percent more failures than Scott's mask (20 failures per 1,000 masks for ILC Dover compared to 15 failures per 1,000 masks for Scott) and that Scott's mean time to repair was just 1.8 minutes compared to ILC Dover's 4.2 minutes. Based upon the RAM data and the protection factor results, the Army concluded that there would be approximately twice as many casualties incurred if the ILC Dover mask was purchased instead of the Scott mask. For the 300,000 masks purchased under the basic contract, the Army calculates that ILC Dover's mask will have 1500 more life threatening failures than the Scott mask. Thus, the Army believes its decision to pay more for the Scott design is justified.

As indicated above, the RFP stated that proposals would be evaluated in three areas: (1) technical/performance against the JSOR requirements; (2) offeror's production capability and (3) cost. The RFP gave no indication that any one of these factors would be considered more important than any other. Our Office has held that where the solicitation informs offerors that the evaluation will consider certain factors for award purposes, absent any indication in the RFP that the factors will be given other than equal consideration, the factors are to be considered substantially equal in weight. University Research Corp., B-196246, Jan. 28, 1981, 81-1 CPD Para. 50.

We do not believe the Army considered all three evaluation factors equally; therefore, the Army's evaluation was not in accord with the RFP's stated scheme. The Army's decision to award to Scott clearly gave paramount weight to the protection and RAM subfactors of the technical/performance factor at the expense of the cost and production capability factors.

In nine of the eleven technical/performance subfactors, the Scott and ILC Dover proposals were rated exactly even. That is, each proposal received 45 evaluation points (out of a possible 90) and was rated "satisfactory." Thus, the only differences in the proposals under the technical/performance factor were in the protection and RAM subfactors. Admittedly, these two subfactors were identified by the RFP as the two most important technical/performance subfactors, with protection identified as significantly more important than any other subfactor. The evaluation record shows that Scott's mask was rated at 8 on a 10-point scale for protection (clearly surpassing the JSOR requirements), while ILC Dover's mask was rated at 6 (also exceeding the JSOR requirements). the RAM evaluation, Scott's mask was rated at 7 (again, clearly surpassing the JSOR requirements), while ILC Dover's mask was rated at 6 (exceeding the JSOR requirements). We note that one of the evaluators would have given ILC Dover's mask a RAM score of 7; he wrote a minority narrative on RAM in which he stated, "The operational reliability of the ILC mask ... is so high that it must be pointed out as providing a significant operational advantage." Thus, Scott's overall rating in all technical/performance subfactors was 60 points, to 57 points for ILC Dover. It is clear that both mask designs surpassed the JSOR standards by wide margins under all significant protection criteria. Even the Army's own independent evaluation of the test results states that neither the Scott nor the ILC Dover design has any major deficiencies.

With respect to the evaluation factor for production capability, the Source Selection Authority concluded that both Scott and ILC Dover would be able to produce the required number of protective masks within the proposed contract schedule. However, the evaluators expressed concern about Scott's manufacturing plan and its facilities. Basically, the evaluators were concerned because Scott proposed to manufacture its masks at a plant that had been empty for 2 years, had no equipment in place, and needed major repairs. The evaluators also noted that Scott's manufacturing plan lacked sufficient detail. After allowing Scott to provide supplemental information about these perceived deficiencies, the Army's evaluators were satisfied that Scott probably would be able to meet the production schedule. The evaluators concluded, however, that Scott's proposal contained "medium" risk, while ILC Dover's proposal was rated "low" in risk assessment. Thus, the ILC Dover proposal was rated as slightly superior to the Scott proposal under this evaluation factor.

Finally, in the cost area, ILC Dover's proposal was much lower than Scott's proposal. ILC Dover proposed a fixed price of $35,019,750 for the contract quantity, while Scott proposed a fixed price of $51,823,181; thus Scott's proposed price was $16,803,431, or about 48 percent, more than ILC Dover's proposed price. Even after the Army added the costs of maintenance, spare parts, and product improvements over the 10-year life of the masks, ILC Dover's total cost was calculated to be only $51,354,660, while Scott's total cost was $68,470,980. Accordingly, using the Army's own figures, Scott's masks will still cost about $17,116,320, or approximately 33 percent, more than ILC Dover's masks. Thus, ILC Dover's proposal was significantly superior in this evaluation area.

CONCLUSION

Our review of all the evaluation materials shows ILC Dover's proposal to be much lower in the cost area, slightly better under the production capability factor, and only slightly inferior under the technical/performance factor. In accord with the RFP's evaluation scheme, all three of these factors were to be considered equal in selecting a proposal for award. Because the Army's selection decision was not based on the RFP evaluation scheme, we sustain the protest.

Clearly, the Army has determined that RAM and protection are critical to selection of the appropriate mask design and it wants to purchase a mask that exceeds the JSOR requirements by as much as possible. In fact, the selection of the Scott design effectively was preordained by the results of the Phase II testing; it would seem that ILC Dover never really had a chance to win the competition given its design. /1/ A reopening of the competition based on the ILC Dover design therefore would serve no useful purpose. In these circumstances, ILC Dover is entitled to recover its proposal preparation expenses as well as the reasonable costs of filing and pursuing its protest. See 4 C.F.R. Sec. 21.6(e).

The record is clear that the Army always has intended to obtain competition based on the winning design. Therefore, and in view of the statutory mandate for full and open competition, we are recommending by letter of today to the Secretary of the Army that the Army terminate Scott's contract for the convenience of the government and procure its requirements on a competitive basis using the Scott design and the revised technical data package. /2/

The protest is sustained.

/1/ We recognize in this regard that the Army did not complete its analysis of the testing until well into the competition.

/2/ In the determination and findings issued to support a limited competition (between Scott and ILC Dover) for this contract, the Army stated that the firm which designed the selected mask must produce an additional quantity of the masks in order to validate the technical data package and to verify production procedures, processes and techniques. However, we know of no reason why production by Scott, as opposed to any other competent contractor, is needed to provide it. This matter currently is the subject of a seperate review by our Office.