Skip to main content

Matter of: Systems Management, Inc.; Qualimetrics, Inc. File: B-287032.3; B-287032.4 Date: April 16, 2001

B-287032.3,B-287032.4 Apr 16, 2001
Jump To:
Skip to Highlights

Highlights

DIGEST Protest is sustained where the procuring agency improperly relaxed for the awardee a mandatory solicitation requirement that the weather observation system be certified by the Federal Aviation Administration. The record evidences that the protesters were prejudiced because they could have proposed different systems if they had been apprised of the agency's actual requirements. The protesters contend that Coastal's proposal should have been rejected as technically unacceptable because it was not. At the time that proposals were due. The OS-21 FBS represents an integrated system of multiple weather sensors and data automation components that is designed to continually measure the conditions of the environment at military installations in support of flight safety and asset protection.

View Decision

Matter of: Systems Management, Inc.; Qualimetrics, Inc. File: B-287032.3; B-287032.4 Date: April 16, 2001

DIGEST

Attorneys

DECISION

Systems Management, Inc. (SMI) and Qualimetrics, Inc. protest an award to Coastal Environmental Systems under request for proposals (RFP) No. F19628-00-R-0032, issued by the Department of the Air Force for a fixed-base weather observation system for the 21st century (OS-21 FBS). The protesters contend that Coastal's proposal should have been rejected as technically unacceptable because it was not, as required by the solicitation, certified by the Federal Aviation Administration (FAA) or a foreign agency, at the time that proposals were due.

We sustain the protests.

The OS-21 FBS represents an integrated system of multiple weather sensors and data automation components that is designed to continually measure the conditions of the environment at military installations in support of flight safety and asset protection. The RFP, issued August 24, 2000, anticipated the award of an indefinite-quantity contract. RFP at 21.

The evaluation factors listed in the RFP were mission capability (technical), proposal risk, past performance and price. The RFP further provided that, when combined, the technical and past performance factors were more important than price. The agency warned that the contract may be awarded to a higher-priced offer which offered technical superiority if the price differential was determined to be worth the technical merit offered over a lower-priced offer. RFP at 41-42.

Proposals were required to be compliant with specific sections of the Technical Requirements Document (TRD) (RFP at 42), including the following:

The system, with the exception of Runway Visual Range (RVR) and video monitoring capability, needs to already exist as an integrated system, and needs to have been evaluated and certified by the FAA or similar foreign agency prior to submission of proposal.

RFP attach. 3, TRD Sec. 2.2.1 (emphasis added).

SMI's proposal in response to the RFP was based on its Automated Surface Observing System (ASOS) platform. The FAA had previously certified this system in accordance with FAA Circular 6000.15C. SMI included its certification letter with its proposal. See Agency Report, Tab 8.1, SMI Proposal, Executive Summary, attach. 2.

Qualimetrics' proposal was based on its Automated Weather Sensor System (AWSS), which was based on the Automated Weather Observing System (AWOS) 900 series that had been previously certified by the FAA. Qualimetrics noted in its proposal that its platform was FAA-certified and produced detailed certification test reports for its proposed system. Agency Report, Tab 8.4, Qualimetrics' Proposal, vol. 2, Sec. i, at 1.

Coastal's proposal was based on the Stand Alone Weather Sensors (SAWS) platform. This platform had not received any formal FAA certification at the time of proposal submission. In fact, the only SAWS unit that had been delivered was under a contract with the FAA, and was not yet operational and was undergoing testing at Hanscom Air Force Base (AFB). Coastal's Pre-Hearing Submission (March 9, 2001) at 4. Coastal's proposal noted that "a single runway system (SAWS) for the FAA goes in next week at Hanscom AFB." Agency Report, Tab 9.1, Coastal's Proposal, vol. 2, at 1.

Four proposals were received by the September 27 closing date. The agency conducted written discussions with the offerors and final proposal revisions were received by November 29. The proposals were evaluated using a color/adjectival system with extra credit to be given to proposals that exceed the RFP's threshold requirements. /1/ Proposal risk was evaluated to assess the risks associated with the offeror's proposed approach, as well as the likelihood of unsuccessful contract performance. /2/ RFP at 42. The findings of the source evaluation board (SEB) are summarized as follows:

Offeror SMI Offeror A Qualimetrics Coastal

Mission Color/Risk Color/Risk Color/Risk Color/Risk Capability and Proposal Risk

Subfactor 1.1 Green Green Green Blue Technical Low Low Low Low Compliance

Subfactor 1.2 Green Green Green Green Interim Low Low Low Low Contractor Support

Evaluated [DELETED] [DELETED] [DELETED] $60,038,016 Price

Agency Report, Tab 7.3, Source Selection Decision, at 4.

The agency noted as a strength in Coastal's proposal its inclusion of a graphical user interface. The agency viewed this feature as a strength because it provides "[DELETED]." Agency Report, Tab 7.2, Proposal Analysis Report, at 7. The source selection authority noted that Coastal's proposal with the graphical user interface, as compared to the other proposals submitted, "effectively leverage[d] the latest commercial information technology." Agency Report, Tab 7.3, Source Selection Decision, at 3. The source selection authority went on to note that "[DELETED]." Id. Another feature that the agency favorably viewed in Coastal's proposal was the [DELETED]. According to the agency, "[DELETED]." Agency Report, Tab 7.2, Proposal Analysis Report, at 8. The source selection authority concluded that Coastal's "significant technical merit" was worth the associated $5.1 million price premium. Agency Report, Tab 7.3, Source Selection Decision, at 4. The contract was awarded to Coastal on December 26. After receiving debriefings, Qualimetrics and SMI filed these protests.

The protesters contend that Coastal's proposal should have been rejected as technically unacceptable because Coastal's proposed system was not certified by the FAA or any foreign agency prior to the submission of proposals and it therefore did not comply with the solicitation requirement that the system be "evaluated and certified by the FAA or similar foreign agency prior to submission of proposal." /3/ The protesters argue that the agency either waived or relaxed this requirement, and that if they had known that FAA certification was not required they would have proposed either systems that were lower priced or systems that contained the very characteristics, such as hardware and software flexibility, that the agency found so attractive in Coastal's proposal. /4/

The Air Force responds that the protesters are reading this requirement in an overly restrictive manner and contends that a certified system is "a system that has been declared operational at a given location, maintenance documentation and trained personnel are available, and logistics support in place to support the fielded system." Contracting Officer's Statement at 5; see also VT at 9:52. The agency states that the purpose of the certification requirement was to obtain a vendor with experience, particularly in obtaining certifications for its systems, and a proven technology. VT at 9:38, 10:15. With regard to the proposals received in response to the RFP, the contracting officer stated:

The proposals detail that the offerors had sold several systems to the [National Weather Service], FAA, and/or foreign governments that are currently operational and, therefore, certified, thereby, satisfying the certification requirement of the TRD. Coastal . . . has produced and fielded such systems to both domestic and foreign governments in accordance with numerous contracts (e.g. according to their proposal [Coastal] has had 2200 contracts over their 18[-]year history, all related to environmental monitoring systems).

Contracting Officer's Statement at 5. The Air Force states that its understanding of the phrase "evaluated and certified" is "deliberately nonspecific," and that it determined that there were "multiple industry understandings of the term `certification.'" Air Force Pre-Hearing Submission (Mar. 9, 2001) at 1. In fact, in its reports, the Air Force uses the phrase "implied certification," not to suggest an additional kind of certification, but to contrast it to "formal FAA certification." Id. at 2 n.1. The agency concludes that because Coastal's SAWS incorporates its [DELETED] software, which was an integral part of other systems provided by Coastal, and because the SAWS successfully underwent final system review at the FAA, Coastal's system based on SAWS satisfies this requirement.

We think that a reasonable reading of the requirement that the proposed system "needs to already exist as an integrated system, and needs to have been evaluated and certified by the FAA or similar foreign agency prior to submission of proposal" is that a "certification" of the base system or platform is required from an independent organization. /5/ Contrary to the agency's argument, the term "certification" does not appear to be a vague term that connotes only that a system is "operational." /6/ In fact, as indicated by the protester, FAA Order 6000.15C (Aug. 11, 2000), which established the Airway Facilities maintenance program and is applicable to weather observation systems for airports, defines "certification" as follows:

Determination, validation, and documentation that a system, subsystem, or Service is providing or is capable of providing the advertised services to the user. The certification includes independent determination about when a system, subsystem or Service should be continued in, restore to, or removed from service, and insertion of the prescribed entry in the maintenance log.

FAA Order 5000.15C, App. 1 Definitions Sec. 8. Chapter 5 of this order establishes the FAA requirements for certification of systems, subsystems, and national airspace systems (NAS) infrastructure services provided to users of the NAS. This chapter states the requirement for a certification to include an "independent determination, which ascertains the quality of advertised services, and a validation, which officially confirms and documents the determination in the maintenance log." FAA Order 6000.15C Sec. 501. The certification process is said to be applicable to systems, subsystems, and services when they "provide essential meteorological information for takeoff and landing aircraft at airports," such as the OS-21 FBS. Id. Sec. 504. While the FAA order does not establish the certification procedures applicable to particular systems and leaves "the choice of methods used of certification determination . . . to the professional judgment of the certifying System Specialist," there is no doubt that the FAA certifies weather observation systems; indeed, the FAA-certified systems are listed at www.faa.gov/ats/aaf/aop/300/6000.15/ at App. 3, /7/ and the protesters' ASOS and AWSS systems were among those listed when proposals were received under this RFP. /8/

Moreover, the record shows that the agency was certainly aware that FAA certifications were extant and relevant. Indeed, a number of the evaluators mentioned FAA certifications as strengths in SMI's and Qualimetrics' proposals. For example, the evaluators noted as a strength in SMI's proposal that its ASOS system was certified by the FAA to replace the human observer. /9/ Agency Report, Tab 7.2, Proposal Analysis Report, at 6.

On the other hand, it is now clear that the agency does not want to impose the "overly restrictive" requirement of a certification by the FAA or similar foreign agency on the competition. /10/ The Air Force program manager stated in a hearing that was held in relation to this case that "[o]ur system for the Air Force is not required to be certified by the FAA. To have said flatly only an FAA-certified system would have been enforcing a standard that's not justifiable in our minds and been overly restrictive to competition." VT at 9:41. Later, she went on to state that "[w]e've tried to keep this from being overly restrictive, but yet somewhat restrictive to keep up the experience level and the commercial off-the-shelf [COTS] element but it probably could have been worded a little bit better." VT at 9:47. In this regard, the agency indicates the purpose of this language was to procure a COTS system, which was not a developmental item, from an experienced contractor who has gone through some "certification process" with a government agency. /11/ VT at 9:38. This system could have been provided to the Air Force, National Weather Service, FAA, another arm of the Department of Defense, or to a foreign agency. VT at 9:52, 10:14.

Thus, it is clear that the agency overstated its minimum needs in requiring that systems be FAA-certified, and that it then either waived or relaxed this requirement. It is a fundamental principle of government procurement that competition must be conducted on an equal basis, that is, offerors must be treated equally and be provided with a common basis for the preparation of their proposals. W.D.C. Realty Corp., B-225468, Mar. 4, 1987, 87-1 CPD Para. 248 at 5; Cylink Corp., B-242304, Apr. 18, 1991, 91-1 CPD Para. 384 at 4. When, either before or after receipt of proposals, the government changes or relaxes its requirements, it must issue a written amendment to notify all offerors of the changed requirements. Federal Acquisition Regulation Sec. 15.206(a). The same principle applies where a protester was misled into believing that a solicitation required it to meet certain stated requirements, whereas the agency evaluated competitors' proposals on the basis of lesser requirements. Cylink Corp., supra; Fort Wainwright Developers; Fairbanks Assocs., B-221374.4 et al., June 20, 1986, 86-1 CPD Para. 573 at 9; Corbetta Constr. Co. of Illinois, Inc., B-182979, Sept. 12, 1975, 75-2 CPD Para. 144 at 9, aff'd, B-182979, Apr. 9, 1976, 76-1 CPD Para. 240. We will sustain a protest where an agency, without issuing a written amendment, relaxes an RFP specification to the protester's prejudice (e.g., where the protester might have altered its proposal to its competitive advantage had it been allowed to respond to the relaxed requirements). SRW Inc., B-284075, B-284075.2, Feb. 16, 2000, 2000 CPD Para. 43 at 3.

As detailed above, Qualimetrics and SMI have stated that had they known of the agency's actual requirements, that FAA certification was not necessary, they each would have submitted different systems that contain the very characteristics that the agency found so attractive in Coastal's proposal or which were lower priced. /12/ See SMI's Protest at 7; Qualimetrics' Protest at 4-5; Qualimetrics/SMI's Pre-Hearing Submission (March 8, 2001) at 3-5. The Air Force does not question the protesters' statements in this regard. /13/ Under the circumstances, the agency should have revised the solicitation to reflect the relaxed requirements and permitted all potential offerors an opportunity to compete on that basis, and we sustain the protest on this basis. Cylink Corp., supra, at 5.

The agency and intervenor have raised certain alternate arguments in response to this protest ground. Specifically, in its report on the protests, the agency points out a letter from the project manager regarding the past performance on Coastal's ongoing FAA contract in which he stated that "SAWS used the latest technology in its design as a certified system for the [FAA]." /14/ Agency Report, Tab 9.3, Coastal's Past Performance Documents. The agency claims that this is evidence that FAA has certified Coastal's system.

However, there is no doubt from the record that Coastal's SAWS system had not been certified by FAA prior to the submission of proposals or when this past performance reference was submitted to the Air Force. Indeed, as noted above, at the time of proposal submission the only SAWS unit that had been produced was not yet operational and was still undergoing testing at Hanscom AFB. Coastal's Pre-Hearing Submission (March 9, 2001) at 4. Moreover, when this sentence is read in the context of the entire paragraph in which it appears, it seems clear that no certification had been done on the SAWS system, but the SAWS system had undergone technical review "prior to the First Article and the Factory Acceptance Test," and it was designed to become a certified system. /15/

The invervenor points to the "interim certification" that its SAWS system received from the FAA on November 13, 2000, after the submission of initial proposals, but before final proposals were due, and claims that this satisfies any applicable FAA certification requirement imposed by the RFP. Agency Report, Tab 9.6, FAA Interim Certification of SAWS System.

However, the Air Force has stated that the interim certification played no part in the agency's evaluation of Coastal's proposal. /16/ VT at 10:25. Moreover, this certification was not in existence "prior to the submission of the proposal," as required by the RFP. /17/ We are also unsure whether the interim FAA certification would satisfy the certification requirement in any case; in this regard the protester submitted an affidavit from a former FAA official who had determined, in his official capacity, that the SAWS needed to be certified, and who stated the following regarding Coastal's interim certification:

This interim certification was issued to ensure that field technicians had the appropriate updates in their maintenance manuals for testing and verifying that the deployed SAWS units were being properly maintained and functioning. The interim was issued in advance of the system's deployment and in advance of the system having been tested and accepted. The interim certification was not validating that the system has passed its acceptance testing. It was issued only to ensure that a step in the certification process was eventually met. It was unusual for such an interim certification to be issued because frequently the materials necessary to monitor and validate deployed systems were delayed until after verification that the system was operational.

The interim certification is not a statement that SAWS has met any of its operational requirements, that it satisfied any of its testing criteria, or that it performed its advertised services. It was done strictly to ensure filed maintenance documentation would be available.

Qualimetrics/SMI's Final Submission (Mar. 23, 2001), attach. 2, Affidavit of Former FAA Official.

Thus, we find the RFP overstated the agency's minimum needs to the prejudice of the protesters. We recommend that the Air Force amend the RFP to represent its actual needs concerning certification and resolicit. /18/ If a firm other than Coastal is the successful offeror, the agency should make award to that firm and terminate Coastal's contract for the convenience of the government. We also recommend that SMI and Qualimetrics be reimbursed their protest costs, including reasonable attorney's fees. 4 C.F.R. Sec. 21.8(d) (2000). The protesters should submit their certified claims for costs, detailing the time expended and the costs incurred, directly to the agency within 60 days of receipt of this decision. 4 C.F.R. Sec. 21.6(f)(1).

The protests are sustained.

Anthony H. Gamboa General Counsel

1. The agency rated proposals with color/adjectival ratings of blue/exceptional, green/acceptable, yellow/marginal, and red/unacceptable.

2. The performance risk ratings assigned were high, moderate, or low.

3. As indicated, a weather observation system includes a set of sensors that feed information into a data acquisition unit or platform and provides some type of output. Videotape Transcript (VT) at 9:49. The sensors may be tailored or vary depending on the particular site at which the system operates. Air Force's Post-Hearing Comments (Mar. 19, 2001) at 4. Thus, according to the protester, the only reasonable reading of TRD Sec. 2.2.1 is that the system to be certified under TRD Sec. 2.2.1 is the underlying platform of the proposed system. Qualimetrics/SMI's Comments (Feb. 26, 2001) at 2 n.1.

4. SMI has stated that but for the Air Force's insistence that offerors use a FAA-certified platform, it would have proposed another of its systems known as the Portable Automated Surface Observing System (PASOS). This system is a derivative system of its ASOS system and does not carry the FAA certification and approvals. SMI states that the "PASOS has many of the quality features of SMI's ASOS system and is easily adapted to meet the [Air Force]'s needs under OS-21." SMI points out that the PASOS costs approximately 15 percent less than its ASOS system. Qualimetrics/SMI Pre-Hearing Submission (March 8, 2001) at 3.

Qualimetrics, the lowest-priced offeror, has stated that if it had not been required to provide an FAA-certified platform, it would have proposed its standard AWOS system and included a Windows operating system and graphical user interface at no additional cost. Qualimetrics goes on to state that "[t]his would also have allowed Qualimetrics to offer many options including those for bringing in data from other systems such as Radar plots, [High Resolution Photograph Transmission] satellite pictures that Qualimetrics commonly uses on its International AWOS, greater flexibility and expandability, as well as the advanced security options available with Windows." The FAA had previously declined to certify Qualimetrics system if it used a Windows-based system or one containing a graphical user interface as part of its certified AWSS system because FAA regarded such applications as not compliant with the FAA AWSS TRD and because of security concerns. Qualimetrics/SMI's Pre-Hearing Submission (March 8, 2001) at 4-5, attach., Affidavit of Qualimetrics' Vice President.

5. According to the agency and the intervenor, the Air Force's intent to obtain an experienced vendor with an established technology, rather than some formal certification, was communicated by the agency to prospective offerors in the answers that it gave regarding this RFP posted on the Internet. For example, Coastal cites the following exchange:

2) Is it correct to assume that automated weather systems previously validated by the Government are still considered validated?

If by validated, you mean certified, the answer is yes, with respect to the site and use of the system. The certification is provided for the site, based on the site's operational functionality. When the system is installed, the site is evaluated with the new system, and certified. Information relating to the certification of proposed system should therefore reference the site or sites where the system was installed, the procuring agency, and a description of the level of service provided by the system.

Agency Report, Tab 11.3, Industry Day, at 1. Rather than showing that the Air Force was not interested in certifications, in our opinion, this response indicated that the agency expected each offeror to supply information about the "certifications" related to its proposed system.

6. In addition, as the protester points out (Qualimetrics/SMI's Final Submission (Mar. 23, 2001) at 4), the fact that the only SAWS unit that was produced was not operational at the time of the submission of proposals indicates that Coastal's SAWS system did not even meet the Air Force's definition of "a system that has been declared operational at a given location." Contracting Officer's Statement at 5.

7. Paragraph 3.a of the introduction to FAA Order 6000.15C Appendix 3 states that each listed system requires certification in accordance with the associated maintenance technical handbook.

8. We note that the weather observation/integration items certified and listed by FAA in Appendix 3 are data acquisition units or platforms.

9. The agency and intervenor seek to draw a distinction between formal FAA certifications and "certified by the FAA" as used in the RFP, which we find to be unsupported and unpersuasive.

10. The Air Force submitted an affidavit from an FAA official in which he states his understanding of the Air Force's needs, that is, that the Air Force was not requiring "offerors to demonstrate compliance with a specific formal FAA commissioning/certification process." We note that this individual never states that an FAA certification was not required by the RFP or not applicable to the systems being procured here; rather, he states that the certification process varies from system to system based on certain factors. Air Force's Pre-Hearing Submission (Mar. 9, 2001), attach. 1, Affidavit of FAA Official.

11. The intervenor has made the argument that none of the systems provided in response to this solicitation are FAA certified because each offeror had to adjust the sensors on their proposed platforms to meet the specific requirements of this RFP. Coastal Comments (Feb. 26, 2001) at 6-7. However, the record before us indicates that platforms for weather observation systems are subject to certification by FAA and does not indicate that the use of different sensors invalidates these basic certifications. We also note that the Air Force program manager testified, in the context of explaining what needs to be certified, "that swapping a sensor does not make it [the proposed system] a non-COTS product." VT at 9:51.

12. Since the RFP informed prospective offerors that FAA certification was a mandatory requirement, it may well have been that other prospective offerors were deterred from competing because of this requirement. See MTS Sys. Corp., B-238137, Apr. 27, 1990, 90-1 CPD Para. 434 at 5.

13. While the intervenor questions whether either of these offerors' proposals would have been selected if they had changed their proposals as they claim, the fact is that these offerors did not have the opportunity to do so and we see no basis to speculate about how such different proposals would have been evaluated vis vis Coastal's proposal.

14. We note that this memorandum was not referenced in the contemporaneous notes of the evaluators, the summary evaluator scores or the source selection authority's memorandum in support of the award. Indeed, the Air Force program manager testified that this letter did not form the basis for determining that Coastal's proposed system satisfied the certification requirement, but when it came to the evaluators' attention, it only confirmed their view that Coastal's system had been certified. VT at 10:28.

15. The paragraph reads as follows:

During the Final System Review (FSR) conducted on May 31, 2000 Coastal demonstrated its outstanding engineering and technical ability. The FSR was a major technical review of the final system design prior to the First Article and Factory Acceptance Test (FA/FAT). SAWS used the latest technology in its design as a certified system for the Federal Aviation Administration.

Agency Report, Tab 9.3, Coastal's Past Performance Documents.

16. The agency received this document on January 18, 2001. VT at 10:23.

17. As explained by the agency, the certification requirement was to make sure that it acquired a system that was certified and established, something that was not "just put together." VT at 9:46. The Air Force's program manager stated at the hearing that, "we wanted to make sure that before they submitted a proposal that they had some proven technology of taking sensors to a data acquisition unit and providing successful data output of a similar type to the OS-21 system." VT at 9:46.

18. In view of our decision and recommendation, we need not resolve the remainder of the protest grounds. However, we do note that the evaluation scheme for past performance, as implemented by the agency, apparently inhibited a qualitative evaluation of the various contracts that the offerors had performed. In this regard, the agency had an internal $60 million threshold before it would determine whether a contract was considered "very relevant," no matter how otherwise relevant the past performance. The protesters contend, not without some merit, that if a contract was below the $60 million threshold, assuming it met other relevancy criteria, it was rated the same regardless of whether it was a $50,000 effort or a $50 million effort, so that all offerors received the same rating regardless of their relative experience. Qualimetrics/SMI's Comments (Feb. 26, 2001) at 10.

GAO Contacts

Office of Public Affairs