Matter of: Olympus Building Services, Inc. File: B-285351; B-285351.2 Date: August 17, 2000

Matter of: Olympus Building Services, Inc. File: B-285351; B-285351.2 Date: August 17, 2000

DIGEST

Attorneys

DECISION

Olympus Building Services, Inc. protests the exclusion of its proposal from the competitive range under request for proposals (RFP) No. GS06P-99-GXC-0021, issued by the General Services Administration (GSA) for janitorial and related services at the Thomas F. Eagleton Courthouse, St. Louis, Missouri. Olympus contends that the exclusion was the result of GSA's improperly downgrading Olympus's proposal on the basis of the mechanical application of arbitrary evaluation criteria that were not disclosed in the RFP.

We sustain the protest.

The solicitation, issued September 16, 1999, provides for the best value award of a fixed-price contract for a base year with four 1-year options. RFP Secs. B.2, M.2.a.

The RFP identifies the following evaluation factors, listed in descending order of importance (with the two subfactors of equal value):

1. Experience 2. Past Performance 3. Staffing and Authority Standards 4. Management Approach

a. Quality Control b. Resources

RFP Sec. M.2.b.

Under GSA's source selection plan (which was not disclosed in the RFP), each factor and subfactor was assigned a possible point score and a weight, for a possible total weighted score of 1,000, as indicated below:

Points Weight Total Possible Score

Experience 50 10 500

Past Performance 50 6 300

Staffing and Authority 60 2 120 Standards

Management Approach

Quality Control 20 2 40

Resources 20 2 40

Total 1,000

Agency Report, Tab 3, Source Selection Plan, at 4; Contracting Officer's Statement at 4.

The RFP advised offerors to address each of the above-listed evaluation factors and subfactors in their proposals, and provided general guidelines, discussed below, regarding the information that offerors were expected to include under each factor. RFP Sec. L.2.

[Deleted] proposals, including Olympus's, were received by the November 15 closing date. Contracting Officer's Statement at 2. The proposals were evaluated and scored by the source selection evaluation board (SSEB), which applied specific criteria contained in the source selection plan. Id. at 4-5. The five highest technically rated offers included two proposals rated "excellent," with weighted point scores of [deleted], and three proposals rated "poor," one with a weighted point score of [deleted] and two with weighted point scores of [deleted]. /1/ Id. at 5. Olympus's proposal was assigned an overall "poor" rating based on its weighted point score of [deleted]. Olympus's evaluated price was the second low of these five, and was substantially lower than the prices of the two highest-scored proposals. Because there was a "natural break" in technical scores between the two highest-rated proposals and the next three proposals, the agency determined to include only the two highest technically rated proposals in the competitive range. Supplemental Agency Report at 4. By letter dated April 19, GSA notified Olympus that its proposal was eliminated from the competitive range and after a May 8 debriefing, Olympus filed this protest with our Office. /2/

The protester argues that GSA improperly applied undisclosed standards in evaluating the experience, staffing and authority standards, and management approach factors which resulted in the downgrading and exclusion from the competitive range of Olympus's proposal. Protester's Comments at 2-10. Olympus points out that while the solicitation "generically" described what GSA wished to see in a technical proposal, the source selection plan, which was not disclosed to offerors, contained a specific evaluation methodology that included "undisclosed benchmarks" that offerors had to include in proposals in order to achieve high scores. Id. at 4-5. Olympus contends that the agency was required to disclose these criteria because an offeror whose proposal did not meet these undisclosed benchmarks had little, if any, chance of being considered for award and, therefore, these benchmarks effectively served as "pass/fail" or "accept/reject" criteria. Id. at 2.

TECHNICAL EVALUATION

Experience

The solicitation provided no specific explanation of how the technical evaluation factors -- including experience -- would be evaluated. The sole reference to experience in the solicitation (other than the above-cited listing among the evaluation criteria) consists of the instructions in section L of the RFP advising offerors of the requirement to submit a minimum of three but not more than five references "of comparable type facility at least 600,000 SF [square feet] of one (1) year duration within the last five (5) years." RFP Sec. L.2.A. Olympus listed references for five facilities in its proposal. The SSEB evaluated each reference in accordance with the source selection plan, which provided that each reference would be rated on a scale of 0 to 10 points, with an "unacceptable" rating earning 0 to 4 points; a "poor" rating earning 5 to 6 points; a "good" rating earning 7 to 8 points; and an "excellent" rating earning 9 to 10 points. The plan further provided that to be rated "good," a reference had to be judged comparable and be 600,000 to 719,000 square feet. To be rated "excellent," a reference had to be judged comparable and be 720,000 or more square feet. Agency Report, Tab 3, Source Selection Plan, at 6. The scores for each reference were totaled, and the total score was divided by the number of references provided to determine the average score. The average score was then multiplied by 5 (the maximum number of references) to determine the raw score. Id. This raw score was multiplied by 10 to arrive at the weighted score.

The agency determined that four of Olympus's five references were for comparable facilities. Two of these references were for space that met or exceeded the 720,000 square foot minimum for an "excellent" rating. These two references were evaluated as "excellent" and each received 10 points. Agency Report, Tab 5, Olympus's Scores, at 11-12, 19-20. A third comparable reference was for a facility of approximately 700,000 square feet which, because it was in the high end of the 600,000 to 719,000 square foot range for a "good" rating, was evaluated as "good" and received 8 points. Id. at 23-24. The fourth reference was for a facility of 610,000 square feet. It was evaluated as comparable space and, because it was in the low end of the "good" range, was awarded 7 points. Id. at 15-16. Olympus's fifth reference was evaluated as "not comparable" space because it was not a high rise building but consisted of several buildings, the largest of which was approximately 300,000 square feet. This reference was rated "unacceptable" and awarded 1 point. Id. at 7-8. Based on these five scores and using the above-explained formula that was detailed in the source selection plan, Olympus's proposal received a total of 360 out of 500 possible points under the experience factor. /3/ Id. at 4.

Olympus argues that its proposal was unfairly downgraded, which essentially eliminated it from consideration, as a result of the application of the undisclosed source selection plan formula. Protester's Comments at 5-7. The protester argues that offerors should have been advised that scores would be averaged and that the agency was awarding its top rating and score only for a building with 720,000 or more square feet, rather than merely indicating the 600,000 square feet minimum requirement. Id. at 5. Olympus contends that had it been informed of the evaluation scheme it would not have provided all the references it included in its proposal. Id. at 6.

As a general rule, a contracting agency need not specifically identify the subfactors comprising the evaluation criteria if the subfactors are reasonably related to the stated criteria. FMS Corp., B-255191, Feb. 8, 1994, 94-1 CPD Para. 182 at 6; KMS Fusion, Inc., B-242529, May 8, 1991, 91-1 CPD Para. 447 at 9. Agencies also need not disclose evaluation guidelines for rating proposal features as more desirable or less desirable since agencies are not required to inform offerors of their specific rating methodology. ABB Power Generation, Inc., B-272681, B-272681.2, Oct. 25, 1996, 96-2 CPD Para. 183 at 4. However, the particular method of proposal evaluation utilized must provide a rational basis for source selection and be consistent with the evaluation criteria set forth in the solicitation. Brown & Root, Inc. and Perini Corp. -- A Joint Venture, B-270505.2, B-270505.3, Sept. 12, 1996, 96-2 CPD Para. 143 at 9. Here, the undisclosed source selection plan methodology used by GSA in computing scores for experience is unreasonable and does not provide a rational basis for the resulting evaluations.

Specifically, under the solicitation, while an offeror's experience was to be evaluated on the basis of at least three facility references, offerors were invited to submit up to five references. Olympus submitted five references; three of these -- satisfying the minimum number required by the solicitation -- were scored as "excellent" or in the high "good" range because they were deemed comparable and ranged from approximately 700,000 to 1.4 million square feet. However, the agency went on to essentially deduct points for the two references that were for facilities substantially less than 720,000 square feet or were judged "not comparable." Thus, Olympus lost 3 points for one of these references (which met the stated 600,000 square feet criterion) and 9 points for the other. All of the scores were added and averaged.

The application of this formula resulted in Olympus's proposal being evaluated as of a lesser quality simply because, in addition to its experience on three fully comparable buildings, Olympus also indicated experience on two unnecessary, smaller, "non-comparable" buildings. In other words, while Olympus's score for the five references was 360, had the protester's score been based only on the three most comparable references, it would have received an additional 105 points for a total score of 465. /4/ Crediting only these additional points would increase the protester's total point score to 849, for an overall "good" rating. In our view, GSA's mechanical formula for scoring the experience factor inappropriately penalized Olympus for including essentially extra references, which resulted in an arbitrary score that did not bear a meaningful relation to the offeror's actual experience. SDA Inc., B-248528.2, Apr. 14, 1993, 93-1 CPD Para. 320 at 8-12.

GSA argues that the fact that Olympus included one reference the agency judged as "not comparable" warranted the reduced score because it demonstrated that the protester "did not have the requisite knowledge to discern comparable facilities . . . ." Supplemental Agency Report at 6. GSA's position is that by submitting this one non-comparable reference, Olympus "failed to demonstrate its understanding of the types of requirements (e.g., staffing, supplies, materials and level of effort) needed to clean the St. Louis Federal Courthouse." Id.

This argument is without merit. Although GSA could conceivably have included a criterion that was tailored to evaluate offerors' understanding or ability to discern the importance of experience with projects of size and requirements similar to the current procurement, such a criterion was not present here under the experience factor. The RFP identified this evaluation criterion as an assessment of the offerors' experience, not as an ability to identify comparable projects. Moreover, Olympus did provide four directly comparable facility references -- one more than the minimum required by the solicitation. Nothing in GSA's contemporaneous record of its evaluation of Olympus's proposal suggests a concern that Olympus "failed to demonstrate its understanding of the types of requirements" needed under this solicitation. Indeed, the only weaknesses noted in the rating sheets concerning the reference GSA judged to be non-comparable were that the reference was "not comparable type space," "not comparable to Class A [office space]" and "multi type space, mixed use -- not comparable to the courthouse." Agency Report, Tab 5, Olympus's Scores, at 7-8. /5/ Further, under the past performance factor, which is actually relevant to understanding the requirements in that it evaluates actual performance, Olympus's proposal received a score of 282 out of a possible 300, warranting an assessment of "excellent." Id. at 4. The agency's argument in its protest report that Olympus's listing of non-comparable experience reflected, or was reasonably evaluated as, a lack of understanding of the requirements is thus not supported by the record.

The agency also argues that it averaged the scores for the references submitted because, without averaging, an offeror submitting five minimal or marginal references would be rated higher than an offeror submitting three superior comparable references. Supplemental Agency Report at 7. This argument ignores the fact that simply averaging the three highest scored references would accomplish the same result without penalizing an offeror for including additional experience beyond the minimum requirement. In this regard, GSA admits that its "formula may, under limited circumstances, yield what appear to be inequitable results . . . ." Id. at 6. In sum, the undisclosed evaluation methodology for scoring and rating experience was irrational and resulted in the Olympus proposal being inappropriately downgraded.

Staffing and Authority Standards

Olympus next challenges the agency's evaluation of its proposal under the staffing and authority standards factor under which Olympus's proposal received a weighted score of 70 (35 x 2) out of a possible total of 120 points. Again, the solicitation provided no explanation of how this factor would be evaluated. Rather, the solicitation instructed each offeror to include the project manager's resume and to define the "Project Manager's level of authority in terms of monetary limits." RFP Sec. L.2.C. The source selection plan provided that a "good" rating and a score of 31 to 45 points would be assigned for a project manager who, among other things, has a level of authority from $2,500 to $24,999; an "excellent" rating and 46 to 60 points would be assigned a level of authority over $25,000. /6/ Agency Report, Tab 3, Source Selection Plan, at 8.

Olympus included a resume for its proposed project manager, listing education, skills, knowledge and experience. The proposed project manager had experience managing an 800,000 square foot building. Olympus also listed the project manager's monetary level of authority as $2,500. Agency Report, Tab 4, Initial Source Selection Report, at 6. In its evaluation of this factor, GSA found that the level of authority was at the lower end of the range and, deducting points primarily on this basis, assigned the protester 35 points and a "good" rating under this criterion. Id.

Olympus argues that the solicitation should have disclosed the required or desired monetary levels of authority for the proposed project manager. The protester contends that the solicitation merely advised offerors to submit the monetary level of authority, but failed to explain that this subfactor would "dramatically affect the points allocated to an offeror" on this criterion or that fewer points will be awarded if the manager's monetary level of authority was not equal to GSA's arbitrary "secret standard." Protester's Comments at 7-8, 9.

GSA states that it required the project manager's level of monetary authority to evaluate the manager's ability to quickly address and resolve on-site problems and to respond to emergency situations. Supplemental Agency Report at 8. The agency contends that the $2,500 to $24,999 standard "was neither arbitrary nor capricious," but was "deemed appropriate for a project of this size." Id. GSA argues that it did not impose mandatory monetary limits because "it recognized offerors would `parrot' the solicitation rather than submit an independently and intelligently arrived at offer." Id. at 9. The agency argues that this standard allowed it "to rationally distinguish and discriminate among competing proposals." Id.

In our view, GSA's application of the source selection plan used an undisclosed criterion, the predetermined monetary level of authority for the project manager, to downgrade proposals whose proposed monetary levels of authority were less than that level. Contrary to the agency's claim, the record shows that GSA mechanically applied this standard and did not, as it asserts in its report, "rationally distinguish and discriminate among competing proposals." The evaluation documents do not contain a single instance where the agency analyzed an offeror's proposed monetary level of authority to examine strengths and weaknesses of the proposed level within the context of the proposed management approach, staffing, resources, or experience of the offeror. The initial source selection report for all proposals contains only brief factual statements, such as: "Contained spending authority of $5,000," "Manager's spending authority was $2,500.00," and "Dollar authority is limited to $2,500.00," in the evaluation summaries of this criterion. Agency Report, Tab 4, Initial Source Selection Report, at 2-4. While we do not question the agency's need to evaluate an offeror's ability to quickly respond to problems and emergencies, which GSA explains that this criterion was intended to measure, the solicitation failed to indicate the agency's intention. Rather, GSA arbitrarily and mechanically applied an undisclosed standard or estimate, which, in conjunction with the other evaluation improprieties discussed in this decision, effectively denied Olympus a fair opportunity to compete.

Management Approach

Finally, Olympus complains that the agency did not disclose the specific estimate it used to evaluate the resources subfactor under management approach. Olympus's proposal received 6 of 20 possible points, for a weighted score of 12 of 40 possible points under this subfactor. As with the other factors, the RFP here advised offerors as to the information they should submit in their proposals without providing any explanation of the evaluation methodology. Specifically, the RFP advised offerors that they must "submit manhour requirements, with assignment of square footage per hour to be cleaned by productive workers. Offeror submits a listing of quantities and types of equipment, supplies and materials to be used in performance of the contract." RFP Sec. L.2.D.2. The source selection plan specified that an offeror that listed quantities and types of equipment, supplies and materials and proposes staff-hour requirements and square footage assignments of 2,700 to 3,100 "are within industry standards" and would be rated "good" and awarded 11 to 15 points. An offeror proposing staff-hours requirements or square footage assignments in excess of 3,100 and with no listing of proposed quantities and types of equipment, supplies, and materials would be rated "poor" and awarded 6 to 10 points. /7/ Agency Report, Tab 3, Source Selection Plan, at 9.

GSA found that the protester's proposal "[c]ontained listing of supplies and equipment along with quantities. Square foot assignments proposed were [deleted] per hour vs. industry standards of 2700-3100 per hour." /8/ Agency Report, Tab 4, Initial Source Selection Report, at 6. The agency also noted that Olympus proposed fewer productive hours than the government estimate. Id. The Olympus proposal was rated "poor" on this subfactor and awarded 6 points.

Olympus objects that offerors were not advised to support their staffing levels and that the agency did not consider any reasons or circumstances of a particular offeror "which may have justified a higher or lower per man hour estimate of productivity." Protester's Comments at 11.

In response, GSA explains that this factor was utilized to determine if "offerors understood the amount of resources and manpower needed to successfully perform the contract requirements." Supplemental Agency Report at 9. GSA explains that its estimate was based on its own analysis of the cleaning requirements, results from the Cleaning Maintenance and Management Survey, the Sanitary Supply Association Cleaning Time Estimator and the GSA Custodial Management Desk Guide. Agency Report at 7-8. GSA argues that agencies may rely on their own estimates of required staffing levels and contends that if it advised offerors of the level of staffing needed, the agency "would have no reasonable or rational way of determining whether the offeror clearly had the knowledge, skills and ability to perform the requisite services." Supplemental Agency Report at 10.

While an agency may rely on its own estimates of the staffing levels necessary for satisfactory performance when negotiating a fixed-price contract, we have found that it is improper for a contracting agency to reject fixed-price proposals simply because the offerors' estimated staff-hours differ significantly from the government's estimate, where the government's estimate was not disclosed to the offerors and the agency failed to conduct discussions with the offerors concerning the discrepancy. Allied Cleaning Servs., Inc., B-237295, Feb. 14, 1990, 90-1 CPD Para. 275 at 3-4. Because estimates are no more than informed guesses which may have little or no applicability to other companies, an agency's absolute reliance on estimates could have the effect of arbitrarily and unfairly penalizing an innovative or unusually efficient firm. Id. at 4.

Here, GSA mechanically downgraded the protester's proposal because its proposed staff-hours reflected an hourly cleaning rate that was not within the range of the government's undisclosed estimate. In fact, the record shows, in this regard, that several of the offerors' estimated staff hour levels differed substantially from the government's estimate. Agency Report, Tab 4, Initial Source Selection Report, at 2-15.

In our view, a reasonable evaluation of this factor would have taken into consideration each offeror's individualized approach and would have recognized the possibility that the hourly cleaning rate and the resulting proposed staff-hours reasonably could vary. Here, as noted above, Olympus asserts that its innovative approach using [deleted] provided efficiencies which GSA did not consider, because its scoring methodology did not take into account any differences in technical approach that might have offered efficiencies. We offer no opinion on whether or not the protester's contention about its innovative approach is correct; nor on this record could we do so. However, in these circumstances, the agency's failure to consider each offeror's approach by mechanical application of an undisclosed hourly square footage rate was unreasonable.

In sum, the arbitrary and inflexible approach reflected by GSA's mechanical evaluation method does not represent a rational evaluation of the offerors' experience, staffing and authority standards, and management approach/resources. Had the criteria been disclosed and applied in a rational manner, Olympus's proposal, which was substantially lower-priced than the two competitive range proposals, might well have received a technical score above the 900 point "natural break" which the agency used to determine which proposals to include in the competitive range.

Accordingly, we sustain the protest. We recommend that the agency amend the solicitation to provide a rational and reasonably disclosed evaluation scheme, request and evaluate revised proposals and make a new source selection decision. If a proposal other than MMMM's is selected for award, the agency should terminate the contract awarded to that firm. We also recommend that the protester be reimbursed the reasonable costs of filing and pursuing its protest, including attorneys' fees. 4 C.F.R. Sec. 21.8(d) (2000). The protester should submit its claim for costs, detailing and certifying the time expended and cost incurred with the contracting agency within 60 days after receipt of this decision. 4 C.F.R. Sec. 21.8(f)(1).

The protest is sustained.

Robert P. Murphy General Counsel

1. The source selection plan provided that proposals with total scores of 900 or more would be rated "excellent," 800 to 899 would be rated "good," 500 to 799 would be rated "poor," and equal to or below 499 would be rated "unacceptable." Agency Report, Tab 3, Source Selection Plan, at 18.

2. The agency held discussions with the two competitive range offerors on April 24, and requested best and final offers (BAFO) by May 4. Contracting Officer's Statement at 4 n.1. After evaluation of BAFOs, the decision to award to Mitch Murch's Maintenance Management Company (MMMM) was made on May 16. Agency Report, Tab 30, Source Selection Final Report, at 4. By letter dated June 30, GSA notified our Office that the agency had determined that urgent and compelling circumstances required contract award; MMMM began to perform in July.

3. Specifically, Olympus received scores of 10, 10, 8, 7 and 1, for a total of 36 points. Using GSA's formula, the 36 points were divided by 5 (the number of references) to compute Olympus's average score of 7.2. The average was multiplied by 5 (the maximum number of references) to determine the raw score of 36. This score was multiplied by 10 (the weight for the experience factor) for a total of 360.

4. Using only Olympus's three highest rated references, 10, 10 and 8, Olympus's total score would have been 28. The protester's average score would then equal 9.3 (28 divided by 3) and its raw score would be 46.5 (9.3 times 5). The raw score times 10 (the weight of the experience factor) equals 465.

5. Olympus argues that the agency improperly downgraded one reference because it was for a facility comprised of many buildings rather than a single building. The protester points to RFP language requiring references "of comparable type facility" to support its position that references did not have to be single buildings. RFP Sec. L.2.A. The protester's interpretation of the RFP is not reasonable because it renders meaningless the RFP's numerous references to "building," for example, in its request for the building name and the building gross square footage. RFP Sec. L.2.B. Read as a whole, we agree with the agency that the RFP did call for references of comparable single building facilities.

6. The agency also evaluated whether the project manager's resume included information regarding the manager's education, knowledge, experience and skills and if the experience was within the past 5 years in a 600,000 or larger square foot building. Agency Report, Tab 3, Source Selection Plan, at 7-8.

7. An offeror would be awarded an "excellent" rating and 16 to 20 points if it listed quantities and types of equipment, supplies and materials, proposed square footage assignment within the 2,700 to 3,100 industry standard and submitted proposed staff-hour requirements and hours broken out by daily and periodic requirements. Agency Report, Tab 3, Source Selection Plan, at 9.

8. Olympus's proposal did not state a figure for the square footage to be cleaned in one hour, so the agency used the [deleted] square foot rate submitted for the protester's [deleted] as the average. Contracting Officer's Statement at 7. In its protest, Olympus argues that it had proposed to clean [deleted] square feet per hour, which was feasible because of its innovative use of [deleted], whose work was augmented [deleted], and that this figure could be calculated from its proposal documents. In responding to the protest, the agency reevaluated the protester's proposal and discovered that the [deleted] square foot figure could be derived from the proposal, but determined that under the source selection plan criteria, this would not change Olympus's proposal's score on this criterion. Id.