Matter of: ENMAX Corporation File: B-281965 Date: May 12, 1999

Matter of: ENMAX Corporation File: B-281965 Date: May 12, 1999

DIGEST

Attorneys

DECISION

ENMAX Corporation protests the award of a contract to Carlisle Research, Inc. pursuant to request for proposals (RFP) No. F19628-98-R-0445, issued by the Department of the Air Force for engineering services in support of the software development activities of the Materiel Systems Group at Hill Air Force Base, Utah. ENMAX argues that the Air Force evaluation of Carlisle's proposal was improper in three areas--the conclusion that the proposal was technically acceptable, the decision that the proposal presented low performance risk, and the review of Carlisle's proposed compensation rates.

We sustain the protest.

BACKGROUND

The RFP was issued on September 21, 1998, and limited the competition to participants in the Small Business Administration's (SBA) section 8(a) small disadvantaged business program, pursuant to 15 U.S.C. Sec. 637(a) (1994). The RFP contemplated award of an indefinite-delivery, indefinite-quantity, fixed-price labor-hour contract /1/ with a term of 5 years. RFP Secs. B.A., F.A. The work was priced using on-site and off-site labor rates. The RFP's evaluation scheme, described in greater detail below, advised that award would be made using a performance/price tradeoff, which "permits tradeoffs between price/cost and the past performance evaluation for technically acceptable proposals." RFP amend. 0002, Sec. M.A.8.

Under the RFP's evaluation scheme, proposals were first reviewed for technical acceptability under three evaluation factors: (1) management of multi-disciplinary teams in an information technology environment; (2) sustainment/maintenance in an SEI/CMM /2/ environment; and (3) enhancement/modernization in an SEI/CMM environment. RFP amend. 0002, Sec. M.B.2. Each of these factors was followed by a narrative description of the scope of the factor, which the evaluators transformed into a [deleted] checklist; each item on the checklist was graded on a pass/fail basis. Evaluation Plan, Oct. 15, 1998, at 11. Proposals found to be technically acceptable were next evaluated for performance risk, using ratings of low, moderate, or high. RFP amend. 0002, Sec. M.B.3.b. In addition, the Air Force evaluated each offeror's price. The RFP advised that past performance and price would be equally important, with award to the offeror whose proposal presented the best combination of performance risk and evaluated price. Id. Sec. M.A.8.

After receiving and reviewing eight proposals, the Air Force concluded that [deleted] of them, including those submitted by ENMAX and Carlisle, were technically acceptable. The agency next concluded that [deleted] of the acceptable proposals (again including those submitted by ENMAX and Carlisle), presented low performance risk based on the past performance review. Of the [deleted] technically acceptable, low-risk proposals, Carlisle had the lowest evaluated price [deleted] followed by ENMAX [deleted]. Based on this information, the agency selected Carlisle for award. Memo for Record, Feb. 10, 1999, at 3.

DISCUSSION

In its challenges to the Air Force's evaluation of Carlisle's proposal, ENMAX focuses on whether Carlisle has gained sufficient technical expertise and relevant past performance from its prior contract experience. Carlisle's predominant experience has been in providing support for the [deleted]. Carlisle Technical Proposal, Oct. 29, 1998, at 2. Among other things, Carlisle maintained and administered the [deleted] Wide Area Network (WAN), and the Air Force explains that Carlisle "has successfully supported this system at all engineering levels and project management levels." Air Force Reply to ENMAX Protest, Agency Report, Tab 2, attach. 11, at 5.

Technical Acceptability of Carlisle's Proposal

ENMAX contends that Carlisle's proposal did not meet the minimum technical requirements set forth in the RFP, and that the Air Force improperly concluded that the proposal was technically acceptable. Specifically, ENMAX claims that Carlisle has failed to demonstrate 2 or more years of experience in the ORACLE, UNIX, NT, and Powerbuilder computing environments. /3/

In considering a protest challenging an agency's evaluation of proposals, we will examine the record to determine whether the agency's judgment was reasonable and consistent with stated evaluation criteria and applicable statutes and regulations. ESCO, Inc., B-225565, Apr. 29, 1987, 87-1 CPD Para. 450 at 7. We have reviewed the pleadings, the solicitation, the proposals, and the evaluation materials, and, based on the record before us, we are unable to conclude that the evaluation was reasonable.

To adequately discuss ENMAX's challenge in this area, we begin with a more detailed review of the solicitation's instructions to potential offerors. As stated above, the RFP identified three evaluation factors, and in the narrative immediately preceding the recitation of these factors, the RFP explained, in relevant part, that:

Each factor, identified below, will be considered equally important and be rated as either pass or fail. The technical evaluation team will use relevant experience to correlate with specific factors below. Only those offerors which receive a "pass" evaluation on all three factors will be eligible for award. The offeror shall ensure that each and every area of each factor has been addressed. Since the technical area is rated as pass/fail, failure to do so may result in failure of the technical proposal.

RFP amend. 0002, Sec. M.B.2. The evaluation factors followed the above-quoted narrative, and each factor was followed by its own narrative expanding on the scope of the factor.

The requirement that the agency review an offeror's experience in ORACLE, UNIX, NT, and Powerbuilder, arises from the third of the three technical evaluation factors, which is set forth below:

FACTOR 3: ENHANCEMENT/MODERNIZATION IN A SEI/CMM ENVIRONMENT

The Government will evaluate the ability to direct the overall system(s) migration from legacy mainframe architecture to an open architecture, client/server environment. The Government will evaluate experience of offeror (2 plus years) in client/server development in ORACLE, UNIX, NT, and Powerbuilder environments and emerging technologies such as web enabled applications development and large scale (over 1 terabyte) data depot/data warehouse projects. The Government will evaluate offeror's experience in transitioning a system(s) to a shared data environment, containing standard data elements and facilitating global access via a distributed computing environment (a form of data warehousing) and experience in data management to include data standardization, modeling, integration, architecture, design, and data extraction/loading. The Government will evaluate the ability to transition a system to the Integration and Runtime Specification (I&RTS) in accordance with Defense Information Infrastructure (DII) Common Operating Environment (COE) compliance, and successful DII certification. The Government will evaluate offeror experience in successfully reusing software code components, modules, objects and/or segments when transitioning or enhancing a system to include maintenance of software code libraries and the offeror's ability to recognize and apply proven state-of-the-industry technologies to evolve existing system(s).

Id. (emphasis added).

As described above, the agency evaluators generated a [deleted] checklist from the requirements identified in the RFP. Of relevance here is the [deleted] item on the checklist:

[deleted].

1. Evaluation Plan for Engineering Services Support, Oct. 15, 1998, attach. 1. The checklist items were graded on a pass/fail basis. Since Carlisle's proposal does not identify experience with [deleted], ENMAX argues it was unreasonable to conclude that Carlisle should receive a grade of "pass" under this checklist item. Thus, ENMAX argues it was unreasonable to conclude that the Carlisle proposal was technically acceptable.

As a preliminary matter, we note that it is the solicitation, not the internal evaluation plan, that is the touchstone for whether offerors have been treated fairly in an evaluation. Requirements stated in evaluation and source selection plans are not disclosed to offerors, and thus, do not give outside parties any rights. Mandex, Inc.; Tero Tek Int'l, Inc., B-241759 et al., Mar. 5, 1991, 91-1 CPD Para. 244 at 7. Our review here is limited to the RFP language, not the evaluation checklist items that are the focus of the protester's arguments.

Carlisle's technical proposal shows that, at best, Carlisle has experience in the [deleted]. /4/ Carlisle Technical Proposal, supra, at 13-14. We see no evidence to support a conclusion that Carlisle has any experience in [deleted], other than the proposal's assertion that the [deleted] WAN "incorporate[s] the same principles" as these systems. Id. at 14. Nonetheless, the evaluators gave Carlisle a rating of "pass" under the "[d]emonstrated experience (2 plus years) in client/server development in ORACLE, UNIX, NT, and Powerbuilder environments" [deleted]. /5/

The next issue is whether Carlisle's failure to show the required experience in all four computing environments should have led the agency to conclude that the proposal was technically unacceptable under the third evaluation factor (a finding which, under the terms of the RFP, would have resulted in an overall rating of technically unacceptable). The RFP does not specify that failure to meet any one of the technical areas under an evaluation factor would result in a finding of technical unacceptability under the factor overall. In this regard, while the RFP does require offerors to address each and every area within each factor, it says only that failure to do so "may result in failure of the technical proposal." RFP amend. 0002, Sec. M.B.2. While, given this language, we cannot conclude that failure to address any one area must result in a finding of technical unacceptability, it clearly could have affected the evaluators' conclusion that Carlisle was technically acceptable. Accordingly, we conclude that the evaluators had to consider whether Carlisle's failure to demonstrate the required experience made it technically unacceptable under the third evaluation factor, and we sustain the challenge to the evaluation in this area.

Past Performance

ENMAX argues that it was unreasonable for the Air Force to award Carlisle a low performance risk rating because Carlisle's past performance has limited relevance to the effort covered by the solicitation here. According to ENMAX, the heart of the current effort is writing code for a large scale management information system (MIS) designed using ORACLE, UNIX, NT, and Powerbuilder, while Carlisle's past experience is less broad and has been largely limited to maintaining the [deleted] WAN. In addition, ENMAX argues that the survey questionnaire used by the Air Force to collect past performance information was inadequate.

With respect to ENMAX's challenge to the agency's conclusions about Carlisle's past performance, our review consists of examining the record to determine whether the agency's judgment was reasonable and consistent with stated evaluation criteria and applicable statutes and regulations. Aerospace Design & Fabrication, Inc., B-278896.2 et al., May 4, 1998, 98-1 CPD Para. 139 at 12. For the reasons set forth below, we conclude that the evaluation was reasonable.

The record here shows that Carlisle provided [deleted] past performance references it deemed relevant to this effort. All of these references were sent surveys, and all of them completed and returned the surveys. Each survey consisted of 17 questions for which the reference rated the offeror as "above average," "average," "below average," or "not observed." Surveys, Agency Report, Tab 6, attach. 3. A response of "above average" was assigned 3 points; a response of "average" was assigned 2 points; a response of "below average" was assigned 1 point. Memo to the File, Evaluation of Past Performance, Dec. 17, 1998, at 1. Responses of "not observed" were not counted further. These responses were then averaged across the survey, and translated to adjectival ratings as follows: 2.5 to 3.0, low risk; 1.5 to 2.4, moderate risk; and, 0 to 1.4, high risk. All [deleted] of Carlisle's references rated Carlisle above 2.5 and all [deleted] indicated they would hire Carlisle in the future. Surveys, supra.

In evaluating Carlisle's past performance information, the Air Force noted that some of the work cited by Carlisle was not precisely the type of work covered by the solicitation. In this regard, the agency stated:

[deleted].

Air Force Reply to ENMAX Protest, Agency Report, Tab 2, attach. 11, at 14.

We start our analysis by noting that there is a difference between ENMAX's claims about the skills that lie at the heart of this effort, and the RFP's guidance regarding the evaluation of past performance. Specifically, while ENMAX correctly asserts that the solicitation emphasized the agency's desire for proposals showing past performance of comparable complexity, and with similar technical requirements, RFP amend. 0002, Sec. M.B.3.c, the solicitation also stated that it would consider the offeror's past experience with:

Information technologies management, systems modernization, operations and maintenance in a software development environment to include the ability to provide the necessary organization, experience, accounting and operational controls, technical skills, and equipment.

Id. Sec. M.B.3.c.(1). Nothing in this language limits offerors to precisely the same effort as covered by the solicitation here; rather, the solicitation anticipates review of a fairly broad range of experience in developing software. /6/

In addition, we see nothing about Carlisle's references, their responses, or the Air Force's assessment of the relevancy of the references, which precludes a reasonable determination that Carlisle's proposal presents low performance risk. First, there is nothing in the record to support a conclusion that Carlisle does not have experience that falls within the broader range of software development experience highlighted in the past performance portion of section M of the RFP, and quoted above. In addition, to the extent that the referenced experience is somewhat different from the scope of the solicited effort, the Air Force recognized the differences and concluded that Carlisle's experience was sufficiently close to be relevant. Finally, the actual responses received were overwhelmingly favorable. Accordingly, we conclude that the agency reasonably assigned Carlisle's proposal a low performance risk rating based on these references. See Apache Enters., Inc., B-278855.2, July 30, 1998, 98-2 CPD Para. 53 at 3.

With respect to ENMAX's argument that the survey questions used here were inadequate to assess the relevance of Carlisle's experience, ENMAX claims that the survey questions should have been more tailored to uncover information about the similarity between the effort solicited here, and the effort covered by the survey. Generally, past performance surveys do not request information about the relationship between the solicited effort and the prior contract; in fact, to do so would substantially increase the burden on individuals asked to complete such surveys, as they would need information about the solicited effort. Instead, each offeror's past performance proposal was required to include an explanation of how the identified contracts are relevant to the solicited effort. RFP amend. 0002, Sec. L.F.3.d. This information was then reviewed by the agency, which was in the best position to make judgments about the relevance of earlier projects.

In addition, our review of the past performance survey provides no basis to conclude that it was unreasonable, improper, or failed to discern the information which it sought. As explained above, the survey consisted of a total of 17 questions for rating the offerors (and a small number of additional questions). Also, the survey was broken into areas designed to correspond with the three technical evaluation factors in the RFP. Surveys, supra. The questionnaire covered the kind of information routinely sought by such questionnaires, including information about whether the agency would like to do business with the company in the future. Given that the record shows that the questionnaires were adequate to survey prior customers of Carlisle, and that the ratings reflected the judgment of cognizant contracting officials with specific knowledge of the contracts in question, we find that the questionnaires used here were reasonably designed to generate information of sufficient reliability to support the Air Force's assessment of past performance. See Pacific Ship Repair and Fabrication, Inc., B-279793, July 23, 1998, 98-2 CPD Para. 29 at 4.

Price Evaluation

ENMAX argues that the Air Force failed to perform an adequate price realism evaluation because the agency simply compared each offeror's proposed hourly labor rates with standard rates compiled by the government. According to ENMAX, this approach, without more, was insufficient to determine the realism of proposed prices, and violates the scheme set forth in Federal Acquisition Regulation (FAR) section 52.222-46, "Evaluation of Compensation for Professional Employees."

As an initial matter, price realism ordinarily is not considered in the evaluation of proposals for the award of a fixed-price contract because the government's liability is fixed and the risk of cost escalation is borne by the contractor. The Centech Group, Inc., B-278715, B-278715.2, Mar. 5, 1998, 98-1 CPD Para. 108 at 5. However, because there is a risk of poor performance in certain circumstances--such as when a contractor attempts to provide products or services at little or no profit, id. at 5-6, or when a contractor fails to offer sufficient compensation levels to obtain and keep qualified professionals, see Research Management Corp., B-237865, Apr. 3, 1990, 90-1 CPD Para. 352 at 8--an agency may in its discretion provide for a price realism analysis in the solicitation of fixed-price proposals. The Centech Group, Inc., supra, at 6. The nature and extent of an agency's price realism analysis is a matter within the sound exercise of the agency's discretion. Cardinal Scientific, Inc., B-270309, Feb. 12, 1996, 96-1 CPD Para. 70 at 4.

The RFP here anticipated award of a fixed-price labor-hour contract (with minor cost reimbursement elements not germane to this discussion), and advised that the agency would evaluate the prices of all technically acceptable proposals for realism, reasonableness, and completeness. RFP amend 0002, Sec. M.B.4.c.-d. The RFP also included the text of the above-referenced, standard FAR clause on compensation of professional employees, which provides that "[t]he government will evaluate the [compensation] plan to assure that it reflects a sound management approach and understanding of the contract requirements." FAR Sec. 52.222-46(a). In essence, ENMAX argues that the standard FAR clause with its anticipated two-pronged review--i.e., a review of an offeror's management approach, and a review of its understanding of the contract requirements--promised something more than the Air Force provided. We disagree.

The RFP's guidance on this issue was not limited to the FAR clause, but also included the following:

In accordance with FAR 52.222-46, "Evaluation of Compensation for Professional Employees," [the] cost realism evaluation will include a review of salary and fringe benefit information provided in the proposal. It is the Government's intent to use Department of Labor, Bureau of Labor Statistics (BLS) data in support of the Professional Clause, 52.222-46, to ensure that it is reasonable for the offeror to attract and retain quality profesionals. If salaries and fringe benefits proposed appear unrealistically low, the proposal may be rejected.

RFP amend. 2, Sec. M.B.4.e (emphasis added); see also Sec. L.F.4. In addition, the RFP directed offerors to a website where the BLS data could be found. Id. Sec. L.F.4. Offerors were advised that the website contained a cross-referenced matrix linking the labor categories in the RFP to the BLS labor hour rates. Questions and Answers on RFP, question 23, p.7.

In our view, ENMAX should have been on notice from the terms of the RFP, together with the structure of the solicitation's pricing, that the Air Force might not look beyond a comparison of the proposed rates to the BLS rates. The solicitation required only that offerors submit a labor-hour rate for each of the labor categories. By comparing the proposed rates to the BLS rates for each category, the agency concluded that Carlisle's proposed rates were reasonable, realistic, and sufficiently high to hire and retain the necessary personnel in the Ogden, Utah area. This conclusion was both reasonable, and consistent with the RFP's stated evaluation scheme. /7/ Air Force Response to ENMAX Protest Concerning Government's Salary Data, Agency Report, Tab 2, attach. 22 at 3.

We also do not view the agency's conclusions as inconsistent with the review anticipated by the FAR clause. By comparing an offeror's proposed hourly rates with the BLS rates, the agency was able to reasonably conclude that the offeror was not endangering successful performance by taking liberties with professional compensation. In addition, the agency was also able to conclude that the offeror understood that the complexity of the effort requires paying appropriate professional rates. Little more was needed to allow the agency to make a judgment about proposed prices--especially in light of the competitive, and fixed-price nature of this procurement, and in light of the fact that an offeror's understanding was also gauged by the technical proposal. Accordingly, we conclude that the agency's price evaluation was reasonable and was not an abuse of its discretion. See Cardinal Scientific, Inc., supra.

RECOMMENDATION

We recommend that the agency reevaluate Carlisle's technical proposal taking into account whether the proposal should be considered acceptable overall given Carlisle's failure to show experience in [deleted], of the computing environments identified under the third technical factor. If, at the conclusion of this reevaluation, the contracting officer determines that Carlisle's proposal is not technically acceptable, we recommend that the agency terminate the award to Carlisle, and award to the offeror whose proposal presents the best value under the evaluation scheme here.

We also recommend that the protester be reimbursed the reasonable costs of filing and pursuing its protest, including attorneys fees. 4 C.F.R. Sec. 21.8(d)(1). In accordance with 4 C.F.R. Sec. 21.8(f)(1), ENMAX's certified claim for such costs, detailing the time expended and the costs incurred, must be submitted directly to the agency within 60 days after receipt of this decision.

The protest is sustained.

Comptroller General of the United States

1. The RFP also anticipated the payment of the contractor's travel costs and certain other direct costs on a cost-reimbursable basis; government estimates were used for the evaluation of these costs. RFP amend. 0002, Sec. M.B.4.c.

2. SEI/CMM is defined elsewhere in the RFP as the Software Engineering Institute's Capability Maturity Model. RFP attach. 1, Statement of Objectives at 1.

3. In its pleadings, ENMAX points to five areas where it argues that the Air Force should have concluded that the Carlisle proposal was technically unacceptable. We have reviewed each of these areas and--with the exception of whether Carlisle demonstrates experience in each of the environments identified above--we disagree. For example, while we agree with ENMAX that Carlisle's experience managing the [deleted] WAN may not be sufficient to establish experience with a "large scale (over 1 terabyte) data depot/data warehouse project" (RFP, amend. 0002, Sec. M.B.2 (factor 3)), we do not agree that the RFP required such experience. Instead, the RFP specified that the agency would evaluate experience in emerging technologies, and cited a large scale data depot/data warehouse project as an example of such experience. Id. Since Carlisle's proposal offered other examples of experience with emerging technologies it was properly found acceptable.

4. We say "at best" because the protester has correctly observed that Carlisle claims it has experience in client/server development in [deleted] because the [deleted] "WAN is built in an [deleted] environment." Carlisle Technical Proposal, supra, at 14. Instead, as ENMAX points out, [deleted] is a database application that exists within the [deleted] WAN. As the WAN administrator for a system that operates [deleted], it is not clear that Carlisle has experience in software development using [deleted].

5. Because there are no contemporaneous explanations of these pass/fail ratings, we are unable to discern whether the evaluators were willing to accept experience in only [deleted] of the four areas (or perhaps, in only [deleted] of the four areas), or whether they failed to recognize that Carlisle's proposal did not show 2 years of experience in [deleted]. We have no basis to conclude, on our own, that experience in all four areas was not required by the RFP, given that the RFP set forth the four areas in the conjunctive ("experience . . . in ORACLE, UNIX, NT, and Powerbuilder"), RFP amend. 2, Sec. M.B.2 (emphasis added), and that experience in these enumerated environments could well be critical to the successful performance of the contractor. Also, as stated above, while the checklist is not dispositive of this issue, its [deleted] consideration of this experience on a pass/fail basis provides further evidence that the agency evaluators considered experience in all four areas an important requirement under the factor.

6. In this regard, we note that the desired experience described in the past performance section of the RFP is broader than the experience set forth in the technical evaluation factors. Compare RFP amend. 0002, Sec. M.B.3.c.(1) with RFP amend. 0002, Sec. M.B.2, Factors 1-3.

7. In its comments on the agency report, ENMAX also argues that the BLS rate is too low to hire sufficiently qualified personnel for these services. ENMAX Comments, Mar. 15, 1999, at 10. Given the clear language in the solicitation stating the agency's intent to use BLS data to review professional compensation, any challenge to the use of the BLS data for this purpose had to be raised prior to the initial closing date for submission of proposals to be considered timely under our Regulations. 4 C.F.R. Sec. 21.2(a)(1) (1998).