Matter of: J&J Maintenance, Inc. File: B-284708.2; B-284708.3 Date: June 5, 2000

Matter of: J&J Maintenance, Inc. File: B-284708.2; B-284708.3 Date: June 5, 2000

DIGEST

Attorneys

DECISION

J&J Maintenance, Inc. protests the Army's award of a contract to Day & Zimmermann Services (D&Z) pursuant to request for proposals (RFP) No. DAKF11-98-R-0011. /1/ The protester contends that the decision to award to D&Z on the basis of its higher-priced proposal was flawed because the agency unreasonably downgraded the protester's proposal and evaluated proposals unequally, giving D&Z's proposal higher ratings in a number of areas even though J&J's proposal contained similar or better features.

We sustain the protest.

Issued on June 8, 1998, the RFP solicited proposals for maintenance and repair of family housing and for operation of a "self-help" center at Fort Polk, Louisiana. RFP amend. 7, Sec. C.1.1. The RFP contemplated award of a fixed-price requirements contract to include a 1-month phase-in period, a 1-year basic contract period, and two optional 1-year periods. Id., Sec. B.

The contract was to be awarded on the basis of best value. The evaluation factors (and their relative weights) were quality ([deleted] percent), price ([deleted] percent), and past performance ([deleted] percent). RFP amend. 0009, Secs. M.2, M.5; Agency Report, Tab Q, Source Selection Decision, at 1. Within the quality factor, management and technical were equally important subfactors. Management approach, staffing and qualifications, subcontracting plan, and phase in/phase out were listed as elements of the management subfactor. Resources, quality control/corrective action program, and methodology were listed as elements of the technical subfactor. RFP amend. 9, Sec. M.6. Only quality and past performance were given point scores; price was evaluated for reasonableness and balance. Id., Sec. M.7. The RFP required the quality portion of each offeror's proposal to be presented orally, with slides, and indicated that the agency intended to award the contract without discussions. RFP amend. 0007, Sec. L.13.1.

Six firms, including J&J, the incumbent contractor, submitted proposals by the February 10, 1999 closing. Each offeror made an oral presentation in April, and revised price proposals were received in August. /2/ After evaluation, two proposals were rated unsatisfactory, while the four acceptable offers were ranked as follows:

Offeror Quality/Past Adjectival Total Price Performance Rating Score

D&Z [deleted] [deleted] [deleted]

Offeror B [deleted] [deleted] [deleted]

Offeror C [deleted] [deleted] [deleted]

J&J [deleted] [deleted] [deleted]

Agency Report, Tab P, Combined Pre-Negotiation Objective/Price Negotiation Memorandum, at 10-14. After reviewing the evaluations, the source selection official (SSO) decided to award the contract on the basis of initial proposals without discussions. Noting that D&Z's proposal had the highest score under the quality factor, the SSO determined that D&Z's proposal represented the best overall value to the government and that it would be worth the additional expenditure to have D&Z, rather than J&J, perform the work. Agency Report, Tab Q, Source Selection Decision Document, at 1, 3-4. Accordingly, the contract was awarded to D&Z on February 14. After a debriefing, J&J filed this protest.

Our Office will question an agency's evaluation and selection decisions only where they violate a procurement statute or regulation, lack a reasonable basis, or are inconsistent with the stated evaluation and selection scheme. See B. Diaz Sanitation, Inc., B-283827, B-283828, Dec. 27, 1999, 2000 CPD Para. 4 at 6. We cannot perform a meaningful review of an agency's evaluation and source selection if the agency record lacks adequate documentation to support the evaluation of proposals and the selection decision. Biospherics Inc., B-278508.4 et al., Oct. 6, 1998, 98-2 CPD Para. 96 at 4.

In this regard, we rely primarily on the documentation required by the Federal Acquisition Regulation (FAR). Specifically, FAR Sec. 15.305(a) requires agencies to document in the contract file the relative strengths, deficiencies, significant weaknesses, and risks supporting proposal evaluation. Of particular relevance to this case, FAR Sec. 15.102(e) requires the contracting officer to maintain a record of oral presentations to document what the agency relied upon in making the source selection decision. The source selection authority selects the method of recording the oral presentations, and FAR Sec. 15.102(e) gives the following examples: videotaping, audio tape recording, written record, Government notes, copies of offeror briefing slides or presentation notes. In addition, FAR Sec. 15.308 requires that a source selection decision be documented, and that the documentation include the rationale for any business judgments or tradeoffs made or relied on.

Even where an agency has acted improperly (such as by failing to maintain adequate documentation of its decisions), we will not sustain the protest unless the improper action prejudiced the protester, that is, there is a reasonable possibility that, but for the agency's actions, the protester would have had a substantial chance of receiving the award. McDonald-Bradley, B-270126, Feb. 8, 1996, 96-1 CPD Para. 54 at 3; see Statistica, Inc. v. Christopher, 102 F.3d 1577, 1581 (Fed. Cir. 1996). In this protest, we are faced with both an evaluation that is unsupported as to many elements and an undocumented cost/technical tradeoff, in which the protester's and awardee's proposals were quite close in terms of their technical ratings (D&Z's [deleted] points versus J&J's [deleted]) and their price (approximately [deleted] for D&Z versus [deleted] for J&J). Because the protester's price was lower, any but a de minimis change to the technical rating of the protester's proposal creates the reasonable possibility that the proposal would have been selected for award. Accordingly, even though the evaluation defects discussed below relate to only a small number of points (or even fractions of points), we view them, collectively, as establishing that the agency's actions prejudiced the protester.

In this procurement, the proposals addressing the quality factor consisted of oral presentations, briefing slides, and resumes. No written technical proposals were allowed. /3/ While the RFP stated that the agency could videotape and/or record the oral presentations, the agency did not do so. RFP amend. 9, at L-6. In this case, the offerors' slides and the evaluators' notes are the only record of what was included in the oral presentations. We recognize the SSO's discretion to decide the method and level of detail of the record of an oral presentation and that the use of offerors' slides and government notes are two of several possible methods of documenting an oral presentation under FAR Sec. 15.102(e). Here, however, the slides and notes do not present sufficient information to determine if the evaluations of J&J's and D&Z's proposals were conducted in a reasonable and equal manner. The slides provide only an outline--in most cases, the slides include general headings only--and do not describe what was included in the approximately 2-hour oral presentations and question-and-answer sessions. /4/ The evaluators' notes are not summaries of the presentation but, for the most part, are selective comments which are sketchy and provide almost no elaboration, or even a description, of what was contained in the portion of the oral presentation being commented upon by the evaluator.

In addition, the consensus evaluation ratings consist only of numerical scores for each offer in each evaluation element, without any accompanying narratives to show how the panel reconciled the divergent opinions of the individual evaluators on any particular facet of an oral presentation or how the panel achieved a consensus rating for any element of the evaluation. There is no evidence in the consensus ratings sheets (or elsewhere in the record) that the evaluators discussed the strengths and weaknesses of the proposals or used any other method to reconcile their differences to arrive at the consensus scores. Agency Report, Tabs I, J, Consensus Ratings for Quality Proposals (D&Z and J&J, respectively).

We understand that oral presentations are an effective way to streamline the source selection process and to enhance the agency's understanding of an offeror's approach, and we do not believe that the FAR requirement for documentation described above necessarily limits this flexibility. FAR Secs. 15.102(e) and 15.308 do not require a particular method of establishing a record of what was said by offerors during oral presentations or place an excessive burden on contracting agencies in recording oral presentations. Whatever method is chosen, FAR Secs. 15.102(e), 15-305(a) and 15-308 do establish an obligation to provide a reasonably adequate record of such presentations and the evaluation thereof. Such a record permits our Office to perform a meaningful review of the agency's selection decision. Here, the oral presentations constituted the offerors' entire technical proposals (the only written portions of proposals related to past performance). The record of the oral presentations and the evaluation is so sketchy, that we have no means to determine, based upon the record before us, the reasonableness of the agency's selection.

In discussing the following five examples, we necessarily examine the notes and comments made by individual evaluators, because those notes (and the offerors' slides) are the only record available to us.

First, regarding evaluation of the quality control/corrective action program, D&Z's proposal received a [deleted] consensus evaluation score ([deleted] points) than J&J's proposal ([deleted] points) from the evaluation panel. Agency Report, Tab I, Consensus Rating for Quality Proposals (D&Z), at 1-2; Tab J, Consensus Rating for Quality Proposals (J&J), at 1-2. J&J contends that the agency unfairly and inexplicably rated D&Z's proposal higher than J&J's proposal in this area. Supplemental Protest at 12, 23; Protester's Supplemental Comments at 10-12.

J&J points out that at least one evaluator made a very favorable comment regarding J&J's quality control program, stating,"[deleted]" Id. at 10; Agency Report, Tab H, Individual Evaluator Worksheets--J&J Quality Proposal, at 18. J&J also points out that at least one evaluator made what J&J interprets as less positive comments regarding D&Z's quality control program, "[deleted]" and "[deleted]." Protester's Supplemental Comments at 10; Agency Report, Tab G, Individual Evaluator Worksheets--D&Z Quality Proposal, at 8. J&J believes that its quality control program was superior to D&Z's and, therefore, should have been given at least as high a rating. Protester's Supplemental Comments at 11.

The Army argues that the protester is merely disagreeing with the evaluation and has not shown that the evaluation and selection should be overturned. Agency Supplemental Report at 6. Neither the record nor the Army's post-protest filings explain why J&J's proposal was rated lower than D&Z's regarding the quality control programs, and the agency offers no way to reconcile those ratings with the evaluators' comments noted by the protester.

Second, regarding evaluation of management approach, the record shows that D&Z's proposal received a [deleted] consensus evaluation score ([deleted] points) than J&J's proposal ([deleted] points) from the evaluation panel. Agency Report, Tab I, Consensus Rating for Quality Proposals (D&Z), at 1-2; Tab J, Consensus Rating for Quality Proposals (J&J), at 1-2. J&J challenges the agency's position that the protester's proposal contained a weakness because J&J "[deleted]." /5/ Agency Report, Tab T, Debriefing Slides, at 14. J&J asserts that its oral presentation emphasized the use of [deleted]. /6/ Among other things, J&J points out that its [deleted] J&J states that it demonstrated the [deleted] during its oral presentation, explained how the [deleted], and answered questions about [deleted] at the end of its oral presentation. Supplemental Protest at 10-11.

The record shows that one evaluator apparently downgraded J&J's proposal in evaluating management approach, commenting that J&J "[deleted]," and that the same evaluator, when evaluating the methodology for accomplishing tasks element commented that J&J's "[deleted]." Agency Report, Tab H, Individual Evaluator Worksheets--J&J Quality Proposal, at 30, 35. Another evaluator appeared to be favorably impressed with J&J's [deleted], commenting, "[deleted]." That same evaluator gave J&J's proposal a [deleted] score when evaluating cost control measures; however, due to the change from a hybrid type to a fixed-price type of contract, cost control was not considered in the final evaluation. Id. at 22.

The Army contends that its evaluators were aware that the [deleted], and points to several notes in the individual evaluators' scoresheets as evidence of this fact. Agency Supplemental Report at 5-6. /7/ The agency does not otherwise justify the ratings assigned to the two competing proposals, and the agency offers no way to reconcile those ratings with the evaluators' comments noted by the protester.

A third example concerns J&J's approach to [deleted]." The agency perceived this approach as a weakness of the proposal, stating: "[deleted]." Agency Report, Tab T, Debriefing Slides, at 15.

J&J states that it explained during its oral presentation that it would [deleted] J&J also states that it is currently successfully using [deleted]. The protester alleges that the Army ignored this information in the evaluation. Supplemental Protest at 16.

The record shows that one evaluator, while recognizing that the idea had merit, questioned whether J&J's [deleted] concept could be applied at Fort Polk, because "[deleted]." That evaluator downgraded the proposal on the evaluation of resources. Agency Report, Tab H, Individual Evaluator Worksheets--J&J Quality Proposal, at 34. However, at least two other evaluators made favorable comments regarding J&J's approach. One specifically noted the concept as an advantage of the proposal, giving it a [deleted] score in the methodology for accomplishing tasks element, stating: "[deleted]." Id. at 27. Another also noted that, using this method, J&J would be "[deleted]" and gave J&J [deleted] rating on methodology for accomplishing tasks. Id. at 11, 18.

The Army responds only that its evaluators noted in several places that J&J would use [deleted], so they clearly did not ignore J&J's oral presentation on this point, but the Army states that, in the opinion of at least one evaluator, the concept might not be as advantageous as J&J thinks. The Army again contends that J&J is merely disagreeing with the Army's judgment and has provided no basis for finding that judgment unreasonable. Agency Supplemental Report at 8. The agency offers no other support for the ratings given or explanation for the contrast between the evaluators' positive comments and the consensus ratings.

The fourth instance concerns the agency perception that J&J's quality proposal had a weakness because it was "[deleted]." Agency Report, Tab T, Debriefing Slides, at 14. J&J states that it explained during its oral presentation that it was [deleted]. J&J alleges that the Army ignored this portion of its oral presentation and downgraded its proposal on the resources element. Supplemental Protest at 15-16. J&J also asserts that the Army has failed to explain why J&J's approach provides any greater risk than D&Z's approach, since J&J contends that [deleted] , and therefore it should have been rated at least equal to D&Z on this element. Protester's Supplemental Comments at 13.

The record includes a reference to one evaluator criticizing J&J's having its [deleted], and indicates that this criticism affected the evaluator's evaluation of both the resources and methodology for accomplishing tasks elements. Agency Report, Tab H, Individual Evaluator Worksheets--J&J Quality Proposal, at 34, 35. The only other evaluator that specifically noted this approach gave J&J much higher ratings on the same two elements. Id. at 19, 26, 27. The Army response is simply that its evaluators took note of the protester's approach and determined it risky, and that the protester merely disagrees with the evaluators. Agency Supplemental Report at 7-8.

As with the other examples, we cannot tell if the evaluation of this aspect of J&J's oral presentation was reasonable, or if the agency treated J&J and D&Z equally, because there is no adequate record of either J&J's or D&Z's oral explanation of their [deleted], and because the evaluation record provides no explanation of how the evaluators reconciled their different opinions on this point.

A final example concerns D&Z's proposal receiving a higher rating than J&J's on the phase in/phase out element of the management subfactor; D&Z's proposal received a consensus score of [deleted] points, while J&J's proposal received a consensus score of only [deleted] points. Agency Report, Tab I, Consensus Rating for Quality Proposals (D&Z), at 1-2; Tab J, Consensus Rating for Quality Proposals (J&J), at 1-2.

The protester asserts that, since it is the incumbent contractor, it has a significant advantage over D&Z because J&J will not have to phase in. The protester alleges that J&J and D&Z were not treated equally in the evaluation and that its proposal should have been rated at least equal to D&Z's. Second Supplemental Protest at 7; Protester's Supplemental Comments at 22-23. In this regard, the offerors' slides show that both J&J and D&Z addressed the phase in/phase out element. While D&Z's [deleted] on this element includes [deleted], J&J's [deleted] also indicate that it [deleted]. Agency Report, Tab E, D&Z's Quality Proposal Slide No. 14; Tab F, J&J's Quality Proposal Slides Nos. 13, 14.

[Deleted] evaluators made favorable comments about J&J's oral presentation on this point and gave it relatively high scores. One evaluator wrote: "[deleted]." Another commented: "[deleted]." [Deleted] evaluator stated: "[deleted]." [Deleted] evaluator gave J&J [deleted] score and wrote: "[deleted]." Agency Report, Tab H, Individual Evaluator Worksheets--J&J Quality Proposal, at 5, 15, 24, 32.

The Army responds that, even though J&J was the incumbent and would not have to go through a phase-in period, it was still required to address the phase-out period. The Army states that D&Z addressed both phases of the contract in sufficient detail to impress the evaluators and argues that J&J merely disagrees with the evaluation. Army Second Supplemental Report at 4.

Again, because there is no adequate evaluation record, we have no way to determine whether the evaluation was fair and reasonable. Further, since the consensus evaluation worksheets include no narrative comments or explanation, we cannot determine how the evaluators reconciled their different opinions and evaluation comments on this element of the evaluation.

As illustrated by these examples, we conclude that the Army failed to maintain an adequate record of the oral presentations, as required by FAR Sec. 15.102(e), or adequate documentation of the relative strengths, deficiencies, significant weaknesses, and risks supporting proposal evaluation, as required by FAR Sec. 15.305(a). In addition, there is no meaningful evaluation record that supports, explains and reconciles the individual evaluator scores and comments.

We also conclude that the source selection document does not meet the requirements of FAR Sec. 15.308. That provision requires that the selection decision be documented and that the documentation include the rationale for any tradeoffs made or relied on. Here, while the source selection document lists several of the perceived advantages of D&Z's oral presentation, it does not address any of the perceived advantages or disadvantages of J&J's presentation. Instead, the SSO merely compares the overall scores of the two offers in the quality and past performance evaluations and their total prices. /8/ Agency Report, Tab Q, Source Selection Decision, at 1-4. Because of the SSO's apparently exclusive reliance on point differentials in the selection decision, we conclude that the source selection decision, which required a tradeoff between J&J's lower-rated, but lower-priced proposal and D&Z's, failed to include documentation showing the rationale for the tradeoff made between those two proposals. See Teltara Inc., B-280922, Dec. 4, 1998, 98-2 CPD Para. 124 at 3-4.

For the reasons set out above, we sustain the protest. We recommend that the Army (1) allow offerors to make new oral presentations and to submit revised written portions of their proposals; (2) make an adequate record of the revised oral presentations and proposal evaluations; (3) conduct discussions, if deemed necessary; and (4) make a new source selection decision. If the agency determines that the contract should be awarded to an offeror other than D&Z, the contract with D&Z should be terminated for the convenience of the government and the contract awarded to the offeror selected. /9/ We also recommend that the protester be reimbursed the reasonable costs of filing and pursuing the protest, including attorneys' fees. 4 C.F.R. Sec. 21.8(d)(1) (2000). The protester should submit its certified claim for costs, detailing the time expended and the costs incurred, directly to the contracting agency within 60 days after receipt of this decision. 4 C.F.R. Sec. 21.8(f)(1).

The protest is sustained.

Comptroller General of the United States

1. Previously, J&J had protested the Army's failure to solicit and evaluate revised technical proposals after amending the RFP's evaluation scheme. Initial Protest at 6-9. We dismissed that earlier protest (reference No. B-284708) as untimely on March 10, 2000.

2. Initially, the RFP contemplated award of a hybrid cost plus award fee/fixed-price contract. RFP Sec. L.9. After proposals were submitted and oral presentations made for the quality proposal, the RFP was amended to state that a fixed-price contract would be awarded; offerors were requested to submit revised price proposals, but no revisions to technical proposals or additional oral presentations were allowed. RFP amend. 7, at 2, 3.

3. J&J's also submitted an index and a 19-page executive summary as parts of its proposal, but these items were not evaluated because they were neither solicited nor authorized. The agency only evaluated each offeror's oral presentation, slides, and resumes. Agency Report, Tab F, Quality Proposal (Slides)--J&J; Memorandum from the Contracting Officer to Army Contract Appeals at 1 (May 9, 2000).

4. Each offeror was allowed to present up to 30 slides during the oral presentation. RFP amend. 7, Sec. L.13.2(g)(3)

5. The RFP required the contractor to have a computerized system for processing and recording work-related data and for generating contract reports. RFP Sec. C.4.6.1.

6. [deleted].

7. The Army fails to explain why, during the debriefing, J&J was told that one of the perceived weaknesses of its proposal was its [deleted].

8. The SSO simply states that "J&J Maintenance, although offering the lowest total price, does not represent the degree of expected performance specified by the solicitation's weighting of [deleted] for the non-price factors. The numerical differences in the Quality evaluations between Day & Zimmermann and J&J Maintenance are significant, and not to be dismissed as trivial or otherwise virtually the same.' Day and Zimmermann presents an offer rated as [deleted],' while the lower-priced J&J, based on the point differential, rates [deleted]." Agency Report, Tab QQ, Source Selection Decision, at 3-4.

9. J&J contends that D&Z should be precluded from any further participation in this procurement because the Army improperly released J&J's protected initial protest letter, which contained some proprietary information, to D&Z directly. Supplemental Protest at 7-9; Protester's Supplemental Comments at 29-30. The contracting officer, who admits that the improper release occurred, reports that the Army promptly retrieved the protected materials from D&Z. Contracting Officer's Statement at 3. The Army also reports that it gave J&J very similar information about D&Z's proposal (including D&Z's price, relative quality score, and several of its perceived technical advantages) during the debriefing, and concludes that J&J has therefore not been prejudiced. Agency Supplemental Report at 3-4. While the release was clearly improper, we see no basis to punish D&Z because of the Army's error, particularly in light of the limited evidence of prejudice to J&J.