Matter of: Recon-Optical, Inc. File: B-286529 Date: January 18, 2001

Matter of: Recon-Optical, Inc. File: B-286529 Date: January 18, 2001

DIGEST

Attorneys

DECISION

Recon-Optical, Inc. protests the terms of request for proposals (RFP) No. N00019-99-R-1567, issued by the Naval Air Systems Command, for imaging sensors and associated work to be installed in the Shared Reconnaissance Pod (SHARP) system.

We deny the protest.

The Navy is seeking a high and medium altitude reconnaissance capability to replace the Tactical Air Reconnaissance Pod System it currently uses on the F-14 aircraft for reconnaisance. To this end, the RFP was issued for sensor and sensor suite fabrication, integration, installation, and testing as a Program Engineering and Manufacturing Development (EMD) effort. RFP attach. 1, Statement of Work (SOW), at 1.0. Offerors were informed that that they could provide either separate and interchangeable medium and high altitude sensors or a single sensor; the RFP stated that a single sensor meeting all medium and high altitude requirements was desirable. RFP attach. 2, Performance Specification, at 3.2. The sensors will eventually be installed in pods to be attached to the F/A-18E/F aircraft, six of which the Navy is procuring under a separate procurement action. Agency Report at 2.

Sensor imaging requirements were among the performance requirements identified in the RFP. These requirements are stated in terms of threshold (minimum) and objective National Imagery Interpretability Rating Scales (NIIRS) at various altitudes and various angles of view. For example, for visible resolution at a medium altitude overflight (imaging within 10 degrees or less of nadir) the RFP requested sensors that would provide a threshold rating of NIIRS 6 and an objective rating of NIIRS 7. The NIIRS is used by the aerial imaging community, and others, to define the quality of images and performance of imaging systems. Under NIIRS an image is assigned a number from 0 to 9 (with a lower number indicating less image recognition) to reflect a level of image recognition and interpretability. See Alternative Dispute Resolution (ADR) Conference Videotape (VT) at 10:04-05; /1/ see also L.A. Maver, C.D. Erdman, K. Riehl, Imagery Interpretability Rating Scales, 20 Society for Information Display (SID) Digest at 117-20 (1995), reprinted at >. Simply stated, a NIIRS rating represents a subjective judgment of what can be humanly perceived. VT at 10:26. For example, "[a]t NIIRS 1 for visible imagery, the criterion indicates the ability to detect a medium sized port facility. At NIIRS 9, the requirement is to differentiate cross-slot from single-slot heads on aircraft skin panel fasteners." General Image Quality Equation (GIQE) User's Guide, version 4.0 (December 1996) at 2 (italics in original).

The RFP included fixed-price line items for the sensors and cost-reimbursement line items for engineering/integration and logistics support services. RFP amend. 1 Sec. B at 3-4. With respect to the sensors, the RFP requested fixed-prices for basic and option quantities as follows:

Offer of Two Separate Sensors

Quantity Price

Base Quantity:

High Altitude Sensor 1 unit

Option Quantity:

Medium Altitude Sensor 1 unit

Sensor Suite /2/ 1-2 Suites

2-4 suites

Offer of Single Combined Sensor

Base Quantity:

Combined High/Medium Altitude 1 unit Sensor

Option Quantity:

Combined High/Medium Altitude 1-2 units Sensor 3-4 units

5-6 units

7-8 units

RFP amend. 1 Sec. B at 3-4. Offerors were informed that in evaluating price the agency would add the price for the base quantity to the price for the option quantity. RFP Sec. M at 57.

The RFP provided for award on a cost/technical tradeoff basis, and identified the following evaluation factors in descending order of importance: technical, price/cost, logistics, management, and past performance. Subfactors were stated for each of the non-price/cost evaluation factors. For example, under the technical factor, offerors were informed that the agency would evaluate the maturity of the offeror's design with emphasis placed upon the maximum use of commercial-off-the-shelf (COTS)/nondevelopmental item (NDI) sensors and/or components that have been demonstrated in a flight environment. Also under the technical factor, the RFP stated that the proposed design would be evaluated to determine if performance claims are "substantiated via analysis, lab, and flight data." RFP Sec. M at 61.

Proposal preparation instructions were provided in the RFP. Among other things, offerors were requested to provide any laboratory or flight test data to establish the maturity of its proposed system. With respect to system performance, offerors were instructed to provide analysis which demonstrates how the proposed design(s) meets or exceeds the imaging requirements, and address any assumptions used in the analysis. The analysis shall include an analysis of predicted imaging performance using the General Imaging Quality Equation [GIQE], as specified in paragraph 3.4.1.3 of the Performance Specification for SHARP Imaging Sensors. If requirements cannot be met, the offeror should address trade-off's and provide a recommended approach. The offeror should support the analyses with lab[oratory] or flight performance data.

RFP Sec. L at 49. Paragraph 3.4.1.3 of the Performance Specification, as amended, provided that for the purpose of system design analysis, the NIIRS performance predictions would be made using the GIQE, in accordance with the GIQE User's Guide, version 4.0, which was available on the agency's website at action="">.

The GIQE was developed under the auspices of the government's Imagery Resolution Assessment and Reporting Standards Committee and is designed to predict NIIRS ratings for visible and infrared images based on knowledge of electro-optical and infrared system design and operating parameters. See GIQE User's Guide at ii, 1, 3; see also J.C. Leachtenauer, W. Malila, J. Irvine, L. Colburn, and N. Salvaggio, General Image-Quality Equation: GIQE, 36 Applied Optics 8322-28 (Nov. 1997). The equation "is a regression-based model that accounts for perceptual quality attributes of scale, resolution or sharpness, contrast, and noise." /3/ GIQE User's Guide at 7. Application of the GIQE algorithmic formula results in a numerical NIIRS rating for the sensor. VT at 10:18 (testimony of the Navy's expert).

Recon-Optical complains that the requirement that offerors use the GIQE to establish their proposed sensor's performance is overly restrictive because the equation tends to favor one imaging technology over another. Specifically, Recon-Optical states that the GIQE underpredicts the NIIRS rating actually achievable by framing technology (the technology that Recon-Optical would offer) because, among other things, the equation provides insufficient weight to the framing technology's inherent ability to enhance signal-to-noise ratio (SNR). Recon-Optical asserts that the GIQE was created around linear or scanning technology, and thus does not adequately recognize the benefits of framing technology. /4/ In this regard, Recon-Optical states that the GIQE is not a generally accepted method of predicting performance and argues that the current industry measure for aerial image resolution is the use of the ground resolvable distance (GRD) analysis, which is a laboratory analysis that uses a rating system different from the NIIRS scale. /5/ Recon-Optical also complains that the application of the GIQE to this procurement is ambiguous because the Navy has not specified uniform parameters for offerors to use in applying the formula.

The Navy responds that the GIQE is an established and reasonable predictor of sensor performance and in fact is the only recognized method of predicting performance of sensors in terms of a numerical NIIRS rating. Supplemental Declaration of Navy's Expert (Dec. 4, 2000) at 2. The agency also states that GRD, which the protester advocates, will not predict a sensor's performance in terms of a NIIRS rating. Id. In any event, the Navy states that the RFP allows an offeror to provide whatever information or explanation the firm wishes to substantiate performance claims for proposed sensors, so long as the firm also provides the GIQE analysis establishing the claimed NIIRS rating for the proposed sensors. With respect to the argument that the GIQE is ambiguous because the agency had not specified uniform parameters for offerors to use in the GIQE assessment, the Navy states that the GIQE User's Guide explains the parameters and their use, and notes that the inherent flexibility of the GIQE allows offerors to select their own parameters (with supporting analysis) to maximize the performance of their proposed sensor(s).

We find no basis from our review of the record to object to the Navy's proposed use of the GIQE formula as one means of assessing a proposed sensor's predicted performance. Contrary to Recon-Optical's arguments, the GIQE is an accepted method of reasonably assessing imaging sensors' performance in terms of NIIRS. See, e.g., Leachtenauer, Malila, Irvine, Colburn, and Salvaggio, General Image-Quality Equation: GIQE, 36 Applied Optics at 8328, which concludes:

It thus appears that, in terms of NIIRS-prediction accuracy, the GIQE is at least equal to, and probably better than, other available metrics.

See also Maver, Erdman, Riehl, Imagery Interpretability Rating Scales, 20 SID Digest at 120, which states:

The NIIRS has also become a standard by which the performance of new imaging systems are specified in procurements and evaluated in acceptance testing. The NIIRS is frequently used in algorithm evaluations [such as the GIQE] to measure the impact of degradations resulting from data compression or enhancements such as sharpening algorithms.

Furthermore, the GIQE has been used in other federal procurements concerning imaging sensors; for example, the GIQE was used by the Department of the Air Force in the Global Hawk unmanned aerial vehicle program as a means of predicting NIIRS performance of that vehicle's electro-optical and infrared sensors. Declaration of Navy's Expert (Oct. 31, 2000) at 2-3.

We are also not persuaded by Recon-Optical that the GIQE unduly favors scanning technology over framing technology. The protester and agency have provided competing technical arguments concerning whether the GIQE formula will fairly capture the performance of Recon-Optical's sensor, specifically, or the performance of framing technology sensors, generally. Ultimately, these submissions demonstrate that the parties disagree, but do not establish that it was unreasonable for the agency to require offerors to submit the GIQE analysis of offerors' proposed sensors. /6/ As demonstrated by the Global Hawk procurement mentioned above, where the awardee (Teledyne Ryan Aeronautical) /7/ offered sensors based upon framing technology, the use of GIQE does not necessarily mean that offers based on framing technology will not receive the award. Id. Moreover, as noted, the GIQE is currently the only method of assessing imaging sensors' peformance that provides a numerical NIIRS rating as the result of the application of the equation. Supplemental Declaration of Navy's Expert (Dec. 4, 2000) at 2. Finally, as indicated above, other observers have found that, in terms of predicting NIIRS accuracy, the GIQE is "at least equal to, and probably better than, other available metrics." /8/ Leachtenauer, Malila, Irvine, Colburn, and Salvaggio, General Image-Quality Equation: GIQE, 36 Applied Optics at 8328.

Of significance to our decision is that the GIQE analyis, although required, is not the only data and analyses which offerors can provide and the agency will consider. The RFP provides that the agency will assess an offeror's predicted imaging performance and whether the offeror's performance claims were substantiated. RFP Sec. M at 61. To accomplish this, offerors were required to provide whatever data, analyses, or laboratory results the firm wished to provide (so long as the firm also provided the requested GIQE analysis) to establish or explain claimed performance. See RFP Secs. L, M at 49, 61. Thus, Recon-Optical is permitted to provide the GRD data that it states the agency should be considering. It also is permitted to provide data supporting its claim that the GIQE assessment does not completely capture its particular sensor's performance. In short, we think the solicitation request for any manner of data and analyses supporting a sensor's claimed peformance can reasonably be expected to provide the Navy with a solid technical foundation upon which to exercise its business judgment in selecting which offer reflects the best value to the government. /9/

We also do not agree with Recon-Optical that the use of the GIQE formula is ambiguous here because the agency did not define uniform parameters or factors that offerors were to use. As explained by the Navy, the RFP provided that the offerors would themselves specify the values (along with analyses supporting the values chosen) of the GIQE factors depending upon the sensor(s) proposed, which allowed offerors to maximize the claimed performance capable from a proposed sensor. Definitions and computational guidance for applying the GIQE formula are provided in the GIQE User's Guide referenced by the RFP. /10/ Based on our review, we find that the RFP's instructions for the use of GIQE are not ambiguous.

Recon-Optical also protests the RFP pricing scheme, which had different line items to be completed depending upon whether an offeror proposed a single sensor or multiple sensors to satisfy the RFP requirements. The protester complains that this pricing scheme favors single sensor offerors that are allowed to price a greater base and option quantity of sensors (eight total) than multiple sensor offerors (four sensor suites). Multiple sensor offerors are disadvantaged, Recon-Optical states, because there are fewer items against which the offeror can allocate its developmental and project costs.

The Navy responds that the RFP does not require pricing for a greater quantity of single sensors, but provides for the acquisition of an essentially equivalent number of sensors. That is, the Navy states that a single sensor offeror would be pricing a total of 9 sensors (base and option quantities) and a multiple sensor (a high and a medium sensor) offeror would be pricing a total of 10 sensors (base and option quantities). The Navy contends that this pricing scheme was required to allow vendors to offer either a single sensor or multiple sensor approach.

We find no merit to the protester's complaint concerning the RFP's pricing scheme. As stated by the agency, the RFP was structured to allow offerors flexibility to propose varying approaches (single sensor as compared to multiple sensors in sensor suites) to satisfying the agency's requirements. As noted above, the agency has a need for high/medium altitude sensors or sensor suites to install in the six pods that it is acquiring under a separate procurement action (with the remaining sensors being purchased as spares). Thus, it has set up a scheme that provided for the pricing of essentially equivalent numbers of sensors recognizing the two possible approaches of satisfying the agency's requirements. Under the circumstances, we think that the agency has devised a reasonable approach to allow single sensor and multiple sensor vendors to compete without unfairly evaluating price/cost.

The protest is denied.

Anthony H. Gamboa Acting General Counsel

1. An ADR conference was conducted to provide settlement assistance to the parties. At the conference, we received legal argument and testimony from the protester's technical staff and the Navy's expert (a privately employed research scientist in the field of reconnaissance sensor development). The parties were unable to reach agreement resolving the protest.

2. According to the RFP, "[a] sensor suite is defined as that combination of High Altitude and Medium Altitude Sensors that best meets the mission requirements as contained in the attached Statement of Work and Specification." RFP amend. 1 Sec. B at 3.

3. The GIQE formula consists of five factors (or parameters): ground sampled distance (a measure of scale and resolution), relative edge response (a measure of perceived sharpness), edge height overshoot (edge sharpening, if image processing techniques used), noise gain (increase in noise due to edge sharpening), and signal-to-noise ratio (a measure of noise). Agency Report, Declaration of Navy's Expert (Oct. 31, 2000) at 2; see Leachtenauer, Malila, Irvine, Colburn, and Salvaggio, General Image-Quality Equation: GIQE, 36 Applied Optics at 8324-25. The GIQE User's Manual provides definitions and computational procedures for each of these factors.

4. There are two basic technologies used for aerial imaging sensors: framing technology and scanning technology. Framing technology receives light or infrared energy from each section of a field of view simultaneously. Scanning technology divides up a field of view into squares arranged into rows and columns, and each square is integrated into a picture. See Protest at 6.

5. GRD, simply stated, is a physical test under which a target (such as a chart of bars) is used to determine at what distance the bars on the chart can be distinguished by a sensor. From this test, other imaging parameters of the sensor can be extrapolated. VT at 9:36-37, 10:08-09.

6. Although the record indicates that there are differences between the two technologies, see VT at 10:06-07 (each technology has its own strengths and weaknesses), the Navy's expert testified that the GIQE assessment reasonably captures the sensor's performance characteristics regardless of technology used. VT at 10:07.

7. Teledyne Ryan Aeronautical is now Northrop Grumman Ryan Aeronautical Center.

8. Recon-Optical's challenge to the use of the GIQE to predict sensor performance is, in part, based upon the firm's apparent belief that the RFP is seeking a COTS/NDI system and that therefore the use of a predictive model is unwarranted. Although the RFP requested that offerors propose NDI components to the maximum extent practicable, the procurement is an EMD effort and not an acquisition of a COTS/NDI system.

9. To the extent that Recon-Optical is complaining that the Navy will not reasonably consider or weigh the data and analyses provided by the offerors, in addition to the GIQE analysis, this challenge to possible, future agency action is speculative and premature, and will not be considered at this time.

10. Recon-Optical also complains that the GIQE model does not specifically explain the relationship between relative edge response and modulation transferability function. The Navy responds that this relationship is provided in the GIQE User's Guide at pages 10 to 11.