IMLCORP LLC; Wattre Corporation

Decision

Matter of: IMLCORP LLC; Wattre Corporation

File: B-310582; B-310582.2; B-310582.3; B-310582.4; B-310582.5

Date: January 9, 2008

DIGEST

Protests challenging the testing methodology employed by an agency in product sample testing is denied, where the solicitation did not specify a particular testing methodology and the protesters have failed to show that the agency's test was an unreasonable method of assessing how well the firms' products would meet the agency's needs.

IMLCORP LLC and Wattre Corporation protest the award of a contract to American Technology Corporation (ATC) under request for proposals (RFP) No. N00164--07-R-8555, issued by the Department of the Navy, for acoustic hailing devices.

We deny the protests.

The RFP was issued as a commercial item acquisition in accordance with Federal Acquisition Regulation part 12.6, and provided for the award of a fixed-price contract for acoustic hailing device systems. This device is a –rugged and lightweight loudspeaker system with very high directivity that is intended for long-range hailing and warning— and will be used on Navy vessels to warn other vessels that they are entering the Navy's 500-yard exclusion zone. RFP, Performance Specification, at 3. Operational and reliability requirements were identified in the performance specification, including that the acoustic hailing device must have –100% voice intelligibility at 500 yards with 88 dB [decibels] of ambient background noise.[1] Id. at 7-8.

The RFP provided for award on the basis of a cost/technical tradeoff and identified the following evaluation factors and subfactors, each listed in descending order of importance:

RFP amend. 1, at 2-3.

The RFP required the submission with proposals of a product sample and –any optional accessories or upgrades that provide claimed improvements in functionality, performance, or additional capabilities.— RFP at 38. In this regard, offerors were informed:

The product sample shall be tested in a non-destructive manner by Government personnel. Product sample testing is planned to occur in a laboratory and field environment at [the NavalSurfaceWeaponsCenter in Crane, Indiana]. The Government may test the product sample against the technical requirements set forth in the Performance Specification contained in this solicitation.

RFP at 38.

With respect to the past performance factor, the RFP required offerors to identify up to five contracts, which the offeror had performed, or was performing, within the last 3 years for the same or similar products, and complete a –contractor performance data sheet— for each. RFP at 39, 41. Offerors were informed that the past performance quality subfactor would be assessed using the information provided on the contractor performance data sheet.

The proposals received from ATC, IMLCORP, and Wattre were evaluated as follows:

Agency Report (AR), Tab 16, Source Selection Evaluation Board (SSEB) Consensus Report, at 4-5, 7-8; Tab 17, Pre-Negotiation Business Clearance Memorandum, at 4.[3]

ATC's higher overall technical rating primarily reflected the firm's highly satisfactory ratings under the two most important subfactors--product sample and specification compliance--under the most important evaluation factor--capability. IMLCORP's and Wattre's satisfactory and unacceptable overall technical ratings, respectively, primarily reflect the ratings assigned those firms' product samples.

Product sample testing was conducted in accordance with the agency's –Shout-Off Test Plan.—[4] AR, Tab 3, Shout-Off Test Plan. That test consisted of placing the firms' acoustic hailing devices side-by-side on the edge of a lake, and projecting pre'recorded voice and sound messages over the lake to two small outboard-motor powered boats each containing three evaluators and each located 500 yards from the hailing devices. The pre-recorded messages used in this testing were the messages provided by the offerors with their product samples, and in conducting the testing the Navy used the offeror's own messages to test that particular firm's device. Each device was the subject of six tests, three with the boats stationary and three with the boats moving. The evaluators assigned a numerical score from 1 to 10 for voice intelligibility with 10 being the most intelligible for each of the six tests. In calculating the firm's intelligibility percentage, the Navy took the highest ratings assigned by the evaluators for one of the six tests, and averaged those ratings. IMLCORP AR at 9-10. Based upon this methodology, ATC's product sample received a 100-percent voice intelligibility rating over the 500-yard distance, IMLCORP's product sample received a 91.66'percent rating, and Wattre's received a 66.6-percent rating. AR, Tab 16, SSEB Consensus Evaluation Report, apps. A, B, and C (Product Sample Ratings).

Award was made to ATC, based upon that firm's higher technical rating and lowest evaluated price. These protests followed, which raise numerous challenges of the Navy's evaluation of the firms' product samples and the evaluation under the other evaluation factors.[5]

The protesters first complain that the Navy's evaluators were not trained, experienced, –operational military personnel— and that the hearing of these evaluators was not tested before the product sample tests were conducted. IMLCORP Supp. Protest at 2; Wattre Protest at 8. The Navy responds that the solicitation did not require the use of operational military personnel or identify any other experience requirements for these evaluators and that, in any event, the SSEB chair is an electrical engineer with significant experience with acoustic hailing devices. The Navy also asserts that there was no requirement that the evaluators' hearing be tested prior to the product sample evaluation and that the evaluators reflected the average hearing that would be expected of actual users of the shipboard devices. See, e.g., WattreAR at 10.

We find that the protesters' speculative challenges to the qualifications of the Navy's evaluators provide us with no basis to question the agency's product sample evaluation. See Philadelphia Produce Mkt. Wholesalers, LLC, B-298751.5, May 1, 2007, 2007 CPD para. 87 at 5 n.2. Moreover, we have long found that the selection of individuals to serve as evaluators is a matter within the discretion of the agency, and, accordingly, we do not review allegations, such as these, concerning the evaluators' qualifications or the composition of evaluation panels absent a showing of possible fraud, conflict of interest, or actual bias on the part of evaluation officials, none of which have been alleged or shown here. See Eggs & Bacon, Inc., B-310066, Nov. 20, 2007, 2007 CPD para. 209 at 4; Glatz Aeronautical Corp., B'293968.2, Aug. 10, 2004, 2004 CPD para. 160 at 3 n.1.

IMLCORP and Wattre also object that the Navy's shout'off test was not an objective, scientific test of the firms' acoustic hailing devices. Specifically, the protesters argue that numerous entities, such as American National Standards Institute, the Department of Defense's Joint Non-Lethal Weapons Program, and the Applied Research Laboratory of Pennsylvania State University, have established standardized tests for the evaluation of acoustic hailing devices, which the Navy should have, but did not, use here. In this regard, IMLCORP has provided the statement of a professor of mechanical engineering, who contends that the Navy's use of a subjective scale for assessing voice intelligibility was not an objective test of voice intelligibility and therefore was unreasonable. IMCORP Supp. Comments, attach. E, Affidavit of Professor of Mechanical Engineering, at 3.

The Navy disagrees that there exists an industry standard test for assessing these devices and notes, in this regard, that the RFP did not identify or require any particular testing methodology. See IMLCORP Supp.AR at 4; WattreAR at 3-4. The Navy recognized that there are a variety of other tests that could have been employed,[6] and states that the agency's shout-off test used elements similar to those employed by the Applied Research Laboratory of Pennsylvania State University as modified to more closely apply to the agency's requirements (for example, testing over water instead of in a building) and to account for the testing resources that the agency had available.[7] See IMLCORP Supp.AR at 3-4. The Navy also contends that neither protester has shown that the use of a different testing methodology would have resulted in a different evaluation conclusion; in other words, the agency contends that neither protester has provided any data or test results that show that its acoustic hailing device would have been assessed as being superior to that offered by ATC, even if a different testing methodology were used.

It is well established that it is a procuring agency's responsibility to determine its needs and how best to meet them. See CardioMetrix, B-270701, Mar. 13, 1996, 96'1 CPD para. 149 at 3. This responsibility includes determining the amount and type of testing necessary to evaluate a product's suitability. Terex Corp.; Caterpillar Tractor Co., B-217053; B-218535, July 24, 1985, 85-2 CPD para. 76 at 6. Our review of an agency's evaluation tests, as with other agency evaluation judgments, does not include conducting our own testing or reevaluating proposals; rather, our review is limited to examining the record to determine whether the agency's judgment was reasonable and in accord with the stated evaluation criteria and applicable procurement laws and regulations. See Abt Assocs., Inc., B-237060.2, Feb. 26, 1990, 90-1 CPD para. 223 at 4.

We find no basis in the record to disagree with the agency's view that, although there are a variety of tests available to assess voice intelligibility of devices such as the acoustic hailing devices, there is not an standard industry test, which offerors could reasonably expect would be used in this procurement. Although the protesters' believe that some other test would be more reasonable, their disagreement alone does not establish that the shout-off test used by the agency was an unreasonable method of assessing how well the firms' products would meet the agency's needs.

The protesters nevertheless argue that the way the Navy conducted the shout-off test resulted in an unequal evaluation and that the test was also not conducted in accordance with the solicitation's specifications. Specifically, the protesters object that the Navy's use of the firms' own pre-recorded messages was unreasonable because the firms' product samples were not all assessed using a standard message and that the Navy used pre-recorded voices rather than live voices in the test.

The Navy responds that the firms were permitted to submit, and in fact did submit, their own messages, which the agency presumed were chosen to optimize the performance of that firm's acoustic hailing device. Moreover, the Navy states that using

multiple sound files reduces the likelihood that a listener will become familiar with the phrases and begin anticipating words in the phrase in later test runs. Utilizing the same phrase repeatedly can cause the listeners to anticipate the words in the phrase and distort the results for later test runs.

IMLCORP Supp. Report at 5. The agency also contends that the use of a pre'recorded message, rather than broadcasting a live message, more accurately reflected the agency's needs, given that the –primary use of the [acoustic hailing devices] will be in danger zones outside U.S. territorial waters— and therefore these –messages are particularly useful due to the language and dialect variations where U.S. Naval vessels may anchor.— Id.In other words, the agency reasonably anticipates that its actual use of the acoustic hailing devices would include the transmission of pre-recorded messages.

We do not agree with the protesters that they were treated unreasonably in the shout-off test, even though the agency did not use the same message, or set of messages, to assess the voice intelligibility of the firms' devices in the shout-off tests and only used recorded messages. The protesters have not shown the agency's explanation for doing so to be unreasonable.[8] We also find that the Navy otherwise equally tested the firms' devices, given that these products were evaluated at the same time and place, operated by the same individuals, and evaluated by the same group of evaluators.

The protesters also complain that Navy did not conduct the shout-off test in accordance with the RFP's specification requirements, given that the test was conducted with a background noise between 58 to 82 decibels, but the solicitation's performance specification provided for 100-percent voice intelligibility at 500 yards with a background noise of 88 decibels.[9] While the Navy did not assess the voice intelligibility of the product samples against the background noise level identified by the RFP, the protesters have not shown that they were prejudiced in this regard, given that they have not explained how their voice intelligibility rating would improve and surpass that of ATC, if a higher background noise were employed.[10] The record also shows that the Navy assessed the offerors' compliance with this specification requirement under its evaluation of the written technical proposals under the specification compliance subfactor, and ATC's proposal demonstrated compliance with the requirement for 100-percent voice intelligibility at 500 yards with a background noise of 88 decibels. See AR, Tab 10, ATC Technical Proposal, at 22-23.

IMLCORP also protested the Navy's evaluation of the firm's proposed acoustic hailing under a number of the solicitation's specification requirements. We have carefully reviewed each of IMLCORP's complaints in this regard and find them to be without merit. For example, IMLCORP complains that the Navy assessed a weakness with respect to the weight of IMLCORP's proposed product, but the record shows that no weakness was assigned to IMLCORP's proposal for the weight of its proposed acoustic hailing device. See AR, Tab 16, SSEB Consensus Report, at 8. As another example, IMLCORP complains that the Navy assessed a weakness under the –audio main lobe— specification because its proposed product exceeded the 30'degree maximum dispersion pattern required by the RFP; IMLCORP admits however that its proposed device does exceed this requirement by 1 degree. Although IMLCORP contends that this was a minor deviation, this does not establish that the Navy could not reasonably assess a weakness for failing to meet this specification requirement.

IMLCORP also challenges the Navy's evaluation of ATC's and IMLCORP's past performance, arguing generally that the two firms should not have received the same favorable, low risk rating under the past performance factor because ATC's acoustic hailing device had received a lower performance rating than IMLCORP's in –limited military utility assessment— (LMUA) testing performed by Alion Science and Technology and the Applied Research Laboratory of The Pennsylvania State University. See IMLCORP Comments, attach. A. However, the RFP provided for an assessment of offerors' past performance quality based upon an evaluation of the offerors' performance of prior contracts for the same or similar products, whereas the LMUA testing, to which IMLCORP refers concerns other entities' assessments of the quality of the products themselves, and not the offerors' past performance quality. We agree with the Navy that the RFP provided for the quality assessment of the offerors' proposed acoustic hailing devices under the capability evaluation factor, and that there was no requirement that the Navy consider other entities' testing of the acoustic hailing devices under this evaluation factor.

The protests are denied.[11]

Gary L. Kepplinger
General Counsel