Skip to main content

Systems Management, Inc.; Qualimetrics, Inc., B-287032.5; B-287032.6, November 19, 2001

B-287032.5,B-287032.6 Nov 19, 2001
Jump To:
Skip to Highlights

Highlights

Agency was not required to consider in the past performance evaluation the performance of all proposed subcontractors of the offerors or all references submitted by an offeror. 2. That there were strengths in the awardee's higher-priced technical proposal that warranted the award selection based on a cost/technical tradeoff. BACKGROUND The OS-21 FBS represents an integrated system of multiple weather sensors and data automation components that is designed to continually measure the conditions of the environment at military installations in support of flight safety and asset protection. The FBS weather sensors and data automation components will be integrated by the contractor into a fully automated weather observation system that will interface with a number of local communications and weather forecast systems at each site.

View Decision

Systems Management, Inc.; Qualimetrics, Inc., B-287032.5; B-287032.6, November 19, 2001

DIGEST

Attorneys

DECISION

Systems Management, Inc. (SMI) and Qualimetrics, Inc. protest an award of a fixed-price indefinite-quantity contract to Coastal Environmental Systems under request for proposals (RFP) No. F19628-00-R-0032, issued as a small business set-aside by the Department of the Air Force for a fixed-base weather observation system for the 21st century (OS-21 FBS). /1/ The protesters challenge the Air Force's evaluation of their and the awardee's proposals.

We deny the protests.

BACKGROUND

The OS-21 FBS represents an integrated system of multiple weather sensors and data automation components that is designed to continually measure the conditions of the environment at military installations in support of flight safety and asset protection. The FBS weather sensors and data automation components will be integrated by the contractor into a fully automated weather observation system that will interface with a number of local communications and weather forecast systems at each site.

The evaluation factors listed in the RFP were mission capability (technical), proposal risk, past performance and price. The RFP further provided that, when combined, the technical and past performance factors were more important than price. The agency warned that the contract might be awarded based on a higher-priced offer which offered technical superiority, if the price differential was determined to be worth the technical merit offered over a lower-priced offer. RFP at 41-42.

The agency received initial proposals in response to the RFP from Coastal, Qualimetrics, SMI, and Northern NEF, Inc. (NNEF). After written discussions, the agency awarded the contract to Coastal.

Both Qualimetrics and SMI protested the award to Coastal to our Office. In Systems Mgmt., Inc.; Qualimetrics, Inc., B-287032.3, B-287032.4, Apr. 16, 2001, 2001 CPD Para. 85, our Office sustained these protests, finding that the agency had overstated its minimum needs in requiring that the proposed systems be FAA-certified, and either waived or relaxed this requirement when it made award to Coastal, whose proposed system was not certified. We also noted that the evaluation scheme for past performance apparently inhibited a qualitative evaluation of past performance because the agency had an internal $60 million threshold before the agency would determine whether a contract was considered "very relevant," no matter how otherwise relevant the past performance. Id. at 11 n.18. We recommended that the Air Force amend the RFP to represent its actual needs concerning certification and resolicit.

In response to our decision, the agency amended the solicitation to remove the FAA certification requirement and to amend the past performance evaluation criteria. The amendment and a request for proposal revisions were transmitted to the four offerors who had originally responded to the solicitation, all of which then submitted revised proposals.

As was the case in the previous evaluation, the proposals were evaluated using a color/adjectival system with extra credit to be given to proposals that exceeded the RFP's threshold requirements. /2/ Proposal risk was evaluated to assess the risks associated with the offeror's proposed approach, as well as the likelihood of unsuccessful contract performance. /3/ RFP at 42. Past performance was reviewed and each proposal was assigned a past performance rating. /4/

The findings of the source evaluation board (SEB) are summarized as follows, showing the color rating and proposal risk assessment for the two subfactors under the mission capability factor, as well as the past performance assessments and evaluated prices:

. SMI Qualimetrics Coastal NNEF

Subfactor 1.1 Green/ Green/ Blue/ [DELETED] Technical Low Risk Low Risk Low Risk Compliance

Subfactor 1.2 Green/ Green/ Blue/ [DELETED] Interim Low Risk Low Risk Low Risk Contractor Support

Past Significant Significant Significant [DELETED] Performance Confidence Confidence Confidence

Evaluated $60.8 Million $53.2 Million $59.2 Million [DELETED] Price

Agency Report, Tab 7.7, Proposal Analysis Report (June 22, 2001), at 19.

Based on the evaluation, the agency determined to make award to NNEF. A small-business size challenge was filed with the Small Business Administration, which resulted in NNEF being disqualified when it was determined to be other than a small business concern. The agency then determined, in another detailed source selection document, that, of the remaining offerors, Coastal's highest-rated proposal offered strengths that offset the price premium, and it made award to that firm. Qualimetrics and SMI then filed these protests, contesting the evaluation of past performance and the technical merit of the proposals.

PAST PERFORMANCE

With regard to past performance, the protesters primarily contest the revised evaluation criteria in the solicitation used for determining whether a contractual effort was "very relevant," "relevant," or "somewhat relevant," and contend that this rendered the past performance evaluation irrational. This argument is untimely made under our Bid Protest Regulations because this alleged solicitation impropriety, which was apparent from the revised solicitation, was required to be protested prior to the May 21, 2001 closing date for receipt of revised proposals, but was not protested to our Office until August 15. /5/ 4 C.F.R. Sec. 21.2(a)(1)(2001).

The protesters also argue that the methodology used by the agency to evaluate past performance was improper. For example, the protesters claim that certain technical proposal evaluation results should have been, but were not, considered in order to properly assess performance risk, and, thus, no effort was made to evaluate past performance in light of the contemplated effort here.

However, under the RFP evaluation scheme, an offeror's past performance was evaluated as performance risk. RFP at 42-43. This assessment was different from the consideration of proposal risk, which considered the risk associated with the offeror's proposed approach as laid out in its proposal. See Modern Techs. Corp. et al., B-278695 et al., Mar. 4, 1998, 98-1 CPD Para. 81 at 7. Thus, in evaluating performance risk, the agency appropriately did not consider matters relating to proposal risk or the technical evaluation.

The protesters also contend that the offerors' past performance was incompletely, and thus improperly, evaluated. This is so, according to the protesters, because the past performance of Coastal's subcontractor, who developed software that is incorporated in Coastal's system was not considered. Also, SMI's performance under a subcontract with NNEF was not considered.

Where a solicitation contemplates the evaluation of offerors' past performance, the agency has the discretion to determine the scope of the offeror's performance history to be considered, provided all proposals are evaluated on the same basis and the evaluation is consistent with the terms of the RFP. USATREX Int'l, Inc., B-275592, B-275592.2, Mar. 6, 1997, 98-l CPD Para. 99 at 3. While agencies may consider the prior relevant experience of subcontractors, in the absence of a solicitation provision to the contrary, there is no broad requirement that they do so. North State Res., Inc., B-282140, June 7, 1999, 99-2 CPD Para. 60 at 7. Nor is there a requirement in all circumstances that an agency obtain or consider all of an offeror's references in the past performance evaluation. Kellie W. Tipton Constr. Co., B-281331.3, Mar. 22, 1999, 99-1 CPD Para. 73 at 6-7.

Here, there was no solicitation provision that obligated the agency to consider the past performance of all proposed subcontractors. While the record reflects that the agency did evaluate the past performance of one of Coastal's subcontractors (as well as some of other offerors' subcontractors), we find no basis to conclude that the agency abused its discretion in failing to evaluate the past performance of one particular subcontractor of Coastal.

The agency declined to consider a contract reference in SMI's proposal where NNEF was the prime contractor and SMI was a subcontractor because NNEF was an offeror in the present competition. The agency believed that requesting past performance information obtained from NNEF regarding this contract would have forced the agency to disclose the identity of SMI, a competing offeror, and that the reference may have a possible conflict of interest or bias towards the OS-21 FBS competitors (even after NNEF was out of the competition). Agency Report at 10. There is also no solicitation provision that obligated the agency to consider all of an offeror's submitted past performance references. In any event, the agency asserts that SMI's confidence rating would not have increased to high confidence, even if this contract had been evaluated, because the Air Force did not consider this contract "very relevant," given its dollar value. Agency Report at 11. We have no basis to question the reasonableness of the agency's evaluation in this regard, particularly since the agency has the discretion to determine the scope of an offeror's past performance history.

Finally, the protesters assert that the past performance evaluation was unreasonable because SMI has performed contracts totaling more than Coastal's, yet both received the same "significant confidence" performance risk rating. However, under the evaluation scheme in the RFP, the total number or value of contracts should not drive the performance risk rating, which assessed various aspects of the relevant individual past performance references, such as relevance, technical compliance, logistics and management support, currency, and type of contractual arrangement. From our review of the record, the agency reasonably rated the offerors' past performance in accordance with the RFP evaluation criteria.

TECHNICAL EVALUATION

The protesters challenge each of the strengths identified in Coastal's mission capability proposal that led to its blue/low proposal risk rating, most of which were referenced in the source selection document justifying the selection of Coastal's proposal. The evaluation of technical proposals is primarily the responsibility of the contracting agency. Our Office will not make an independent determination of the merits of technical proposals; rather we will examine the record to ensure that the agency's evaluation was reasonable and consistent with the stated evaluation criteria. Litton Sys., Inc., B-237596.3, Aug. 8, 1990, 90-2 CPD Para. 115 at 8. A protester's mere disagreement with the agency's evaluation does not render the evaluation unreasonable. SWR Inc., B-286044.2, B-286044.3, Nov. 1, 2000, 2000 CPD Para. 174 at 3. As discussed below, we find the agency evaluation of the strengths in Coastal's mission capability proposal to be reasonable.

Graphical User Interface (GUI)

The agency noted as a strength in Coastal's proposal that its proposed GUI (called Airport Weather Advisory (AWA)) on the OS-21 FBS is tailorable. Agency Report, Tab 7.7, Proposal Analysis Report, at 8; Tab 7.10, Source Selection Document, at 3. The protesters contend that the agency had no basis for its conclusion that Coastal's GUI system is more tailorable than their proposed GUI software system (called StormFront).

However, in contrast to the protesters' proposals, Coastal's proposal was replete with details concerning the tailorability of its user interface. For example, Coastal describes its AWA as follows:

[DELETED]

Agency Report, Tab 9.1, Coastal's Proposal, vol. 2, at 3.4.2.2. In addition, Coastal's proposal notes that the key to AWA's easy-to-use design is [DELETED]:

[DELETED]

[DELETED] Another significant tailorability feature of Coastal's system is the ability to add sensors through the GUI, rather than, as required by the Qualimetrics system, through software source code reconfiguration. Id. at 3.4.2.2.

In contrast, the proposal of Qualimetrics addresses GUI tailorability in the following section:

Qualimetrics, working with SMI, has developed a software system called StormFront that enables the easy configuration of custom GUI screens for any and all applications. Using StormFront, the user can define and build his own display screens.

Agency Report, Tab 8.3, Qualimetrics' Revised Proposal, at 38. SMI's proposal simply asserts the tailorability of the GUI with no significant details. See Agency Report, SMI Proposal, vol. 2, Sec. 2.8.

The tailorability of the GUI was clearly a feature that the agency greatly valued. The agency notes that the ability of the operators to reconfigure the data displays to best meet their needs is of significant importance since multiple configurations of the display may be necessary at different Air Force and Army sites due to varying complements of sensors and varying missions. Agency Report, Tab 7.7, Proposal Analysis Report, at 8. It is apparent that Coastal's proposal emphasized and described the tailorability of its system in much greater detail than the proposals of Qualimetrics and SMI. In negotiated procurements, since the agency's technical evaluation of proposal quality generally is based upon information submitted with the proposal, the burden is on the offeror to submit an adequately written proposal. See Will-Burt Co., B-250626.2, Jan. 25, 1993, 93-1 CPD Para. 61 at 3. Under the circumstances, while the protesters disagree with the evaluators' conclusions that Coastal's GUI was more tailorable than their StormFront GUI, the protesters have not provided any basis to show that the agency evaluation was unreasonable or inconsistent with the solicitation's evaluation criteria.

Use of C++

The protesters contest the agency's determination that the fact that Coastal's GUI was written in the native language of Windows, C++, which allows all of the display tab windows to update simultaneously, thereby preventing any delay or latency of data when the operator selects a different display window, represented a strength in Coastal's proposal. Agency Report, Tab 7.7, Proposal Analysis Report, at 8; Tab 7.10, Source Selection Document, at 3. The protesters argue that their proposed systems, which both used C++ in the programming, possess this same capability.

Coastal's proposal states [DELETED]. Agency Report, Tab 9.1, Coastal's Revised Proposal, at 12. The agency found, and our review confirms, that the protesters' proposals did not articulate their use of C++ in the implementation of the StormFront software system, but, more importantly, their proposals did not state that their systems provided the same capability of [DELETED]. Contracting Officer's Statement at 10. Given that Qualimetrics' and SMI's proposals do not provide the same level of detail that was provided in Coastal's proposal in this respect, we cannot find this evaluation to be unreasonable or inconsistent with the evaluation criteria.

Use of Windows 2000

The protesters disagree with the agency's conclusion that Coastal's use of Windows 2000, rather than Windows NT, in its proposed system provided enhanced security features.

While Qualimetrics' and SMI's proposed systems, which contain the protection features of Windows NT, met the agency's minimum requirements, Coastal's system using Windows 2000 was found by the agency to provide significant evidence of greater security capabilities in the areas of data storage, password protection, multiserver security, database security, and group assignments. Contracting Officer's Statement at 11; Agency Report, Tab 9.1, Coastal's Revised Proposal, at 12-17. The source selection decision stated that the security features provided by Coastal's system will greatly facilitate security accreditation and net-worthiness certification, which will lead to timely major command coordination and site-level system acceptance. Agency Report, Tab 7.10, Source Selection Decision, at 3.

The protesters contend that the security advantages of Windows 2000 are disputed and list several sources on the Internet to prove their point. Protest at 19. While these sources indicate that the security advantages of Windows 2000 versus Windows NT are a matter of dispute, these sources also provide information that supports the agency position in this regard. Under the circumstances, we will not question the agency's technical judgment on this point. Notwithstanding the protesters' assertion that the agency did not adequately quantify or discuss the relevant security advantages in Coastal's proposal, they have not shown that the agency's evaluation was unreasonable in this respect.

Unique Interfaces

The protesters also contest the strength assigned Coastal's proposal with regard to its system's unique interfaces. The agency found that Coastal's proposal highlighted the extent of the firm's open architecture from a hardware and software perspective, which allowed the system to be adaptable, flexible and expandable. This was primarily because of Coastal's unique interfaces, which permit the easy addition of new sensors. Specifically, Coastal's Universal Sensor Interfaces, the Universal Digital Interface, and the Universal Analog Interface, and its hardware/software interface design, coupled with its GUI, were found to provide a more user-friendly, flexible and scalable system for sensor reconfiguration, modification and addition. According to the agency, because of these features, future system upgrades and evolving interface requirements will be more easily implemented with respect to cost and schedule. Contracting Officer's Statement at 11; Agency Report, Tab 7.10, Source Selection Document, at 3.

In contrast, the protesters' open architecture description in this respect addresses only the hardware perspective. Contracting Officer's Statement at 11. Regarding the addition of new sensors, both Qualimetrics and SMI indicated that updates to the software that runs their systems may be required to meet the data archiving and data dissemination requirements of the new sensors. Agency Report, Tab 6.2, SMI's Evaluation Notices, at 11; Tab 6.3, Qualimetrics' Evaluation Notices, at 5.

The protesters assert that the agency is mistaken if it believes that software changes are not necessary to add new sensors under Coastal's system. However, even the protesters acknowledge that if such software changes are necessary under Coastal's system, "the means under which they are done may be minimized." Protesters' Comments at 19. For example, Coastal's system [DELETED]. Based on our review, the agency reasonably determined that Coastal's unique interfaces represented a strength in its proposal.

Interim Contractor Support (ICS)

Another strength found in Coastal's proposal was its inclusion of a sample integrated support plan, which illustrated a superior, "proactive" approach to the ICS requirements. Agency Report, Tab 7.7, Proposal Analysis Report, at 14. The agency states that "the sample ICS plan demonstrated an outstanding approach in providing ICS support that went well beyond the expectations of the ICS requirements." Contracting Officer's Statement at 12. Although the protesters' proposals did not include a sample plan, the protesters contend that this strength is a "complete mystery," because the protesters both have more experience in meeting ICS requirements than Coastal can demonstrate. Protest at 25. Based upon a review of the contents of the proposals, however, we see no mystery in Coastal's being awarded a strength in this area.

Warranty

The protesters dispute that Coastal deserves extra credit for providing additional warranties. The agency gave Coastal's proposal a strength in this area because it offered a 3-year warranty on workmanship (parts and labor) on certain system components, as compared to the 1-year warranty offered by the protesters. Contracting Officer's Statement at 12; Agency Report, Tab 6.5, Coastal's Responses to Evaluation Notices, at 2; Tab 7.7, Proposal Analysis Report, at 14. We see no reasonable basis to dispute that this is a relative strength in Coastal's proposal. The protesters nevertheless argue that the agency failed to exactly quantify the advantages that this additional warranty provides. However, the agency was not required to quantify the value of each strength.

ISO 9002

The protesters assert that Coastal's proposal should not have been given a strength for being ISO 9002 certified, given that the protesters are both ISO 9001 certified and ISO 9001 is a more stringent qualification because it also includes system design, and because one of Coastal's subcontractors is not ISO certified. The protesters' ISO 9001 certifications were reported as a strength by the agency. The agency reports that "[b]oth [9001 and 9002] certifications demonstrated to the technical evaluation team that each of these companies had progressed to the point that they were organizationally equipped to assume the responsibilities of the OS-21 program." The agency asserts that the added dimension of design in the ISO 9001 certification, which is not part of the ISO 9002 certification, is not of significant importance in this procurement since the OS-21 FSB program is a FAR Part 12 commercial acquisition, and is not a developmental effort. Contracting Officer's Statement at 12. The protesters' assertion that their ISO 9001 certifications should have been more highly regarded, because they believe some developmental effort is involved in performing this contract, constitutes mere disagreement with the agency's evaluation and does not demonstrate that awarding both the awardee's and the protesters' proposals a strength in this area was unreasonable. Moreover, the fact that one of Coastal's subcontractors is not ISO certified does not negate Coastal's certification.

Other Strengths

The protesters also contest the strengths evaluated in Coastal's proposal for "plug and play" capability and remote real-time display. They assert that these capabilities are merely features of the Windows operating system and that their Windows-based proposals should also have received similar strengths. The agency responds that these strengths are sub-elements of the overall strength in Coastal's proposal of its secure, flexible open architecture, and these strengths were not simply based on Coastal's Windows-based operating system, but also considered its specific and unique sensor interfaces and hardware/software design, coupled with its user-friendly secure user-interface. The protesters have not substantively responded to the agency's reasonable explanation.

The protesters also challenge the strength in Coastal's proposal for having multiple simultaneous language outputs, asserting that this is inconsistent with the solicitation. The agency denies this allegation, and responds that as a result of the debriefings that the protesters received in association with the previous award selection of Coastal, they were aware that the agency regarded this aspect of Coastal's proposal as a strength, yet they did not include this attribute in their revised proposals. Here too, the protesters have not responded to the agency's explanation.

COST/TECHNICAL TRADEOFF & SOURCE SELECTION

The protesters contend that the agency failed to conduct a rational cost/technical tradeoff. To the extent that this argument depends on the challenges to the Air Force's technical and past performance evaluation, it fails, since, as discussed above, we find no basis to question the reasonableness of the agency's evaluation. To the extent that the protester believes that the agency was required to quantify the dollar value of Coastal's exceptional rating, the protest is without merit. There is no requirement that an agency quantify the value of technical superiority vis-a-vis low cost/price to determine the best value to the government. See DDD Co., B-276708, July 16, 1997, 97-2 CPD Para. 44 at 8. On the record here, the agency has adequately justified its cost/technical tradeoff and award decision in finding the particular strengths in Coastal's proposal were worth the associated price premium. /6/

The protests are denied.

Anthony H. Gamboa General Counsel

1. SMI and Qualimetrics are affiliated companies.

2. The agency rated proposals with color/adjectival ratings of blue/exceptional, green/acceptable, yellow/marginal, and red/unacceptable.

3. The proposal risk ratings assigned were high, moderate, or low.

4. The past performance ratings assigned were high confidence, significant confidence, confidence, unknown confidence, little confidence, and no confidence.

5. The protester references a May 3, 2001 e-mail addressed to agency counsel raising concerns about the past performance evaluation criteria, Agency Comments (Oct. 2, 2001), attach. 1, and suggests that this correspondence constituted an agency-level protest. Protester's Comments at 3. The agency responded to the protesters' e-mail on May 4, advising that this inquiry should be directed to the contracting office. Agency's Comments (Oct. 2, 2001), attach. 1. Since the protesters did not pursue this matter further at that time, we do not consider this communication to be an agency-level protest.

6. Qualimetrics and SMI also protest the agency's affirmative determination of Coastal's responsibility, asserting that Coastal is ineligible to receive the award due to its alleged "meager financial status." Protest at 31. Our Office does not review affirmative determinations of responsibility absent a showing of possible bad faith on the part of government officials or that definitive responsibility criteria in the solicitation were not met. 4 C.F.R. Sec. 21.5(c). Since the protester does not allege bad faith by the agency or that definitive responsibility requirements were not met by Coastal, the matter is not for our review. M-Cubed Info. Sys., Inc., B-284445, B-284445.2, Apr. 19, 2000, 2000 CPD Para. 74 at 9-10.

GAO Contacts

Office of Public Affairs