Matter of: Northrop Grumman Corporation; ITT Gilfillan File: B-274204; B-274204.2; B-274204.3; B-274204.4; B-274204.5; B-274204.6; B-274204.7, Date: November 27, 1996 * Redacted Decision

B-274204,B-274204.4,B-274204.6,B-274204.7,B-274204.5,B-274204.2,: Nov 27, 1996

Additional Materials:

Contact:

Edda Emmanuelli Perez
(202) 512-2853
EmmanuelliPerezE@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

Is denied where solicitation indicated agency's intent to conduct its own assessment and. Protest that agency improperly failed to disclose during discussions the mathematical formula used to calculate mean time between corrective maintenance action (MTBCMA) is denied. Since nondevelopmental system was required and system operational testing already had taken place. Offerors would not have been in a position to redesign their proposed systems so as to significantly improve their MTBCMAs and their competitive standing. The DASR radar system is intended to replace Department of Defense (DOD) and Federal Aviation Administration (FAA) analog radar systems (at low-to-medium density air traffic control facilities) first fielded 25 years ago.

Matter of: Northrop Grumman Corporation; ITT Gilfillan File: B-274204; B-274204.2; B-274204.3; B-274204.4; B-274204.5; B-274204.6; B-274204.7, Date: November 27, 1996 * Redacted Decision

Protest that agency improperly determined most likely mean time between corrective maintenance action (MTBCMA) for each offeror's proposed airport surveillance radar system, rather than accepting the proposed MTBCMAs, is denied where solicitation indicated agency's intent to conduct its own assessment and, moreover, accepting offerors' proposed figures would render this area of the evaluation meaningless, since offerors would be expected to employ the most favorable assumptions in their calculations, leaving the agency with no basis for comparing the proposed MTBCMAs. Protest that agency improperly failed to disclose during discussions the mathematical formula used to calculate mean time between corrective maintenance action (MTBCMA) is denied; such methodologies for evaluating proposals under the stated evaluation criteria need not be disclosed by agencies and, in any case, since nondevelopmental system was required and system operational testing already had taken place, offerors would not have been in a position to redesign their proposed systems so as to significantly improve their MTBCMAs and their competitive standing.

Attorneys

DECISION

Northrop Grumman Corporation (NG) and ITT Gilfillan protest the Department of the Air Force's award of a contract to Raytheon Company, under request for proposals (RFP) No. F19628-95-R-0007, for digital airport surveillance radar (DASR) systems. NG and ITT challenge the agency's conduct of discussions and evaluation of proposals.

We deny the protests.

BACKGROUND

The solicitation contemplated the award of a contract for site survey, site design, site preparation, production, installation and logistics support for up to 213 DASR radar systems. The DASR radar system includes a primary surveillance radar (PSR) to perform two-dimensional detection of aircraft and weather intensity and an integrated monopulse secondary surveillance radar (MSSR) to perform three-dimensional aircraft position location and identification via communication with an aircraft transponder. The DASR radar system is intended to replace Department of Defense (DOD) and Federal Aviation Administration (FAA) analog radar systems (at low-to-medium density air traffic control facilities) first fielded 25 years ago, which are nearing the end of their life cycle and are now subject to sporadic loss of airport surveillance radar coverage. The DASR radar system will "improve aircraft detection in clutter, provide National Weather Service calibrated six level weather, interface to new digital automation systems, improve reliability, and reduce support costs." (System Requirements Document (SRD), paragraph 1.0.)

Of particular importance for this protest, and as noted in the solicitation's executive summary, the solicitation sought the proposal of "Non-developmental Item (NDI) radars." Toward this end, it provided that proposed radars would undergo "[a]n Operational Capability Demonstration (OCD) conducted to verify that the offeror has a functional, NDI system," and that "the Offeror's NDI product is representative of the DASR system proposed" and "exists as a 'system' consisting of all integrated subsystems and other items needed to operate and maintain the system per the SRD (except for development required by the contract)." In this regard, the solicitation's statement of evaluation criteria specifically cautioned offerors that "[v]ariations between the design proposed in written proposals and the design/equipment actually demonstrated at the OCD may be considered a weakness and may add to the evaluated proposal risk rating."

Award was to be made to the responsible offeror whose offer was most advantageous to the government under three broad criteria: (1) technical, which was equal in importance to (2) the offeror's capabilities (as shown by the results of a past and present performance evaluation, expanded pre-award survey, and possible in-plant executive review of the offeror's facilities), both of which were more important than (3) cost/price (for a best estimated quantity of 89 DASR systems). The technical area included three factors of equal weight: (1) system performance and system specification, (2) reliability, maintainability, availability and supportability, and (3) the offeror's proposed statement of work (SOW), contract data requirements list (CDRL) and integrated master plan (IMP). Under both the technical factors for system performance/system specification and for reliability/maintainability/availability, the solicitation generally provided for proposal strengths to be given to the extent an offeror's proposed system specification "captures the desired (i.e., 'should') SRD requirements in a manner beneficial to the Government"; "[s]pecial emphasis" was to be given to the extent that the offeror's proposed system offered certain specified "desired characteristics the Government believes add significant value to the Offeror's product." (Among these latter characteristics was a desired mean time between corrective maintenance action (MTBCMA) of 1,070 hours.) Under the cost/price area, the solicitation provided for calculation of a total cost/price and for evaluation "of the realism of the Offeror's proposed cost/price(s)," including an evaluation of "the extent to which proposed prices and supporting cost data . . . are consistent, indicate a clear understanding of the solicitation requirements, and reflect a sound approach to satisfying those requirements."

Four initial technical proposals from three offerors--Raytheon, ITT and NG (which submitted two independent proposals, only one of which is relevant here)--were received by the closing time on December 19, 1995. OCDs for each offeror were conducted from February 6 to February 23, 1996. The Air Force then opened written discussions with all offerors on April 5, with written responses required by April 22, and conducted oral discussions during May. Complete proposals, including cost/price proposals and revised technical proposals, were received by June 7. Following additional written discussions with offerors, the Air Force requested best and final offers (BAFO).

Based upon the evaluation of BAFOs and the offerors' capabilities evaluation, the source selection evaluation board (SSEB) and the source selection advisory council reported to the source selection authority (SSA) that, while the evaluation of offerors' past performance and the results of the expanded pre-award survey indicated a low performance risk for all three offerors, Raytheon's technical proposal was superior to that of the other offerors. The specific results of the evaluation were as follows:

Raytheon NG ITT

TECHNICAL

System Exceptional Acceptable/ Acceptable/ Performance/ (Blue)/Low Risk Moderate Risk High Risk Specification

Reliability, Exceptional/ Acceptable/ Acceptable/ Maintainability, Low Risk Low Risk Moderate Risk Availability and Supportability

SOW/CDRL/IMP Acceptable Acceptable/ Acceptable/ (Green)/Low Risk Low Risk High Risk

MTBCMA 1,134 Hours 380 Hours 481 Hours

OVERALL Exceptional/ Acceptable/ Acceptable/ TECHNICAL Low Risk Moderate Risk High Risk

CAPABILITY Low Risk Low Risk Low Risk

COST/PRICE $[DELETED] Million $[DELETED] Million $[DELETED] Million

Based upon the evaluation record, the SSA concluded that Raytheon's offer was most advantageous. The SSA noted that Raytheon's technical proposal was evaluated as exceptional/low risk and superior to those of the other two offerors under the system performance/system specification factor. In this regard, the SSA specifically noted that Raytheon's proposal, which had no significant weaknesses in this area, offered several beneficial strengths, including superior MSSR operating characteristics, easy upgrade to a Mode S radar capability, [1] a Surveillance Data Translator design which allowed for easy growth and installation, [2] and a third DASR system to support the in-plant test program. (In addition, Raytheon's low risk rating was partly based on the fact that the system demonstrated at OCD was a current production model pulled from the production line and destined for a [DELETED] customer.) In contrast, the SSA noted that, although ITT's proposed Surveillance Data Translator likewise allowed for easy growth and installation and its technical proposal was considered acceptable under this factor, the proposal was evaluated as high risk under the system performance/system specification factor on the basis that ITT had significantly underestimated the manhours required to accomplish the proposed software modifications and had failed to demonstrate a thorough understanding of the throughput requirements with respect to interfacing with the existing automation system. Likewise, although NG's proposed system offered several beneficial operating characteristics, a weather channel design based upon operationally proven algorithms, and a third in-plant DASR test system, its proposal was evaluated as moderate risk under this factor on the basis that NG had significantly underestimated the manhours required to accomplish the proposed software modifications.

The SSA further noted that Raytheon's technical proposal was evaluated as exceptional/low risk and superior to those of the other two offerors under the reliability/maintainability/availability factor. In this regard, the SSA specifically noted that Raytheon's proposal, which had no significant weaknesses in this area, offered several beneficial strengths, including a system with a MTBCMA of 1,134 hours (that is, in excess of the desired 1,070-hour MTBCMA), a [DELETED] transmitter and a mature, easy-to-use human interface, and likely significant software maintenance cost savings from the use of [DELETED] software development tools. While NG also offered a [DELETED] transmitter, its proposed system was evaluated as having an MTBCMA of only 380 hours, significantly below the desired level (and its proposed MTBCMA of 734 hours). ITT likewise offered a [DELETED] transmitter, but its system was evaluated as having an MTBCMA of only 481 hours, again, significantly below the desired level of 1,070 hours (and its proposed MTBCMA of 1,436 hours). In addition, ITT's proposal was rated as moderate risk under the reliability/maintainability/avail ability factor on the basis that the system architecture included a significant amount of hard-to-support [DELETED] software and was characterized in significant measure by a custom design--rather than an open architecture design--such that, should the need arise to replace the [DELETED] processors (which the agency feared would become unavailable) or associated older hardware, significant redesign would be required. Further, while both Raytheon's and NG's proposals were evaluated as acceptable/low risk under the SOW/CDRL/IMP factor, the SSA noted that ITT's proposal, although considered acceptable, was rated high risk on the basis of its unrealistically aggressive test schedule, insufficient test labor hours, and limited understanding of the site activation process.

The SSA concluded that the overall technical superiority and lower risk of Raytheon's proposal, which was rated exceptional/low risk while NG's and ITT's proposals were rated only acceptable/low risk and acceptable/high risk, respectively, was worth its somewhat higher cost/price. Upon learning of the resulting August 9 award to Raytheon, NG and ITT filed these protests with our Office.

NG and ITT raise numerous arguments concerning the agency's conduct of the procurement generally and the evaluation specifically. We discuss a number of their most significant arguments below.

MTBCMA

As used by the Air Force, MTBCMA generally measured the mean time between required immediate, unscheduled corrective maintenance visits to DASR sites, that is, requirements for maintenance visits other than the regularly scheduled preventative maintenance visits. For purposes of calculating MTBCMA, the SRD stated that:

"Corrective maintenance actions are those required to maintain user confidence in the integrity of the DASR System. Those would include repair of failed redundant units considered critical to meeting the DASR System mission of providing PSR, weather and MSSR data to the automation system."

Although the agency accepted the reliability block diagrams and component failure rates furnished by each offeror in support of their estimated MTBCMAs--finding that the failure rates were based on appropriate reliability analyses and reflected the best estimates of the actual likely rates of component failures--the agency noted that each offeror's MTBCMA was based on different assumptions as to what failed components in their proposed system architecture would compromise user confidence in the integrity of the DASR system and when they should be replaced. Accordingly, when discussions failed to eliminate the differences with respect to the assumed government maintenance approach, and in order to assure a common basis on which to measure and compare the MTBCMAs of the proposed systems, the agency "rationalized" each proposed MTBCMA using its intended maintenance approach. Thus, in computing the evaluated MTBCMA of each proposed DASR system, the Air Force essentially assumed that failures of units in series and in one-of-two redundant configurations--that is, where only one of the two units is necessary for performance of the function and the second unit is redundant--would be repaired immediately; it assumed that repair of a configuration with greater redundancy would be delayed until the failure of the last redundant unit, such that failures for a subsystem consisting of three units in which only one was necessary for operation would not be repaired until two units failed. This rationalization resulted in evaluated MTBCMAs for NG and ITT significantly below those estimated by the offerors--that is, an evaluated 380 hours for NG (734 hours proposed) and 481 hours for ITT (1,436 hours proposed).

As noted above, in making his source selection decision, the SSA expressly took into account that NG's and ITT's evaluated MTBCMAs were significantly below the desired level and Raytheon's MTBCMA. In addition, the SSA was briefed on the likely cost of the greater or lesser number of unscheduled corrective maintenance visits to DASR sites to be expected as a result of MTBCMAs below or above the target MTBCMA (which was likely to result in eight unscheduled corrective maintenance visits per year per system). Specifically, the SSA was advised that NG's MTBCMA of 380 hours was likely to result in an additional 24,276 unscheduled corrective maintenance visits over the expected 20-year life of the DASR systems, for an additional total labor cost of approximately $20 million, and that ITT's MTBCMA of 481 hours was likely to result in an additional 16,372 unscheduled visits, for an additional labor cost of approximately $13.5 million. In contrast, the fact that Raytheon's evaluated MTBCMA of 1,134 hours was likely to result in 755 fewer unscheduled corrective maintenance visits relative to those likely at the target MTBCMA of 1,070 hours was expected to result in an approximate $700,000 savings relative to the target MTBCMA (and thus approximately $20.7 and $14.2 million savings relative to NG's and ITT's MTBCMAs, respectively). (The numbers of additional or fewer visits to be expected at the various MTBCMAs which were briefed to the SSA were mistakenly based on 82 DASR systems and not the correct total of 89 systems. When corrected to a total of 89 DASR systems, the resulting number of additional visits would have been 26,461 visits for NG and 17,845 for ITT, while Raytheon's evaluated MTBCMA was likely to result in 822 fewer visits.) [3]

Notice of Evaluation Approach

NG and ITT challenge several aspects of the Air Force's calculation of the MTBCMAs for the proposed DASR systems and its determination of the related impact on future maintenance costs. As an initial matter, NG challenges the Air Force's rationalization of the offerors' proposed MTBCMAs. NG argues that the agency failed to adequately advise offerors of its approach to evaluating MTBCMA and that the agency's action in rationalizing proposed MTBCMAs amounted to the imposition of an unstated evaluation criterion; the protester essentially maintains that the agency instead was required to accept the proposed MTBCMAs.

We find that the RFP adequately advised offerors of the agency's intended approach in evaluating MTBCMA. The solicitation provided for evaluation of "[t]he Offeror's approach to meeting and substantiating the DASR SRD RMA [reliability/maintainability/availability] and supportability requirements," with "[p]roposal strength [to] be given for the extent to which the Offeror's proposed system specification captures the desired (i.e., 'should') SRD requirements in a manner beneficial to the Government" and for "[s]pecial emphasis" to be given for certain listed desired SRD requirements. One of the "special" requirements included in the SRD as a reliability/maintainability/availability requirement was a desired DASR system MTBCMA "greater than or equal to 1070 hours." The evaluation was consistent with this scheme--the agency considered the offerors' ability to meet the desired MTBCMA of 1,070 hours. The solicitation also generally provided for the agency to "assess the risk associated with the Offeror's proposed approach as it relates to accomplishing the requirements of this solicitation" and specifically required that "[a]ll [reliability/maintainability/availability] claims . . . be substantiated through the use of the Reliability Model for all elements of the proposed DASR system." We think this was a clear indication that the agency intended to assess the likelihood that the offeror's proposed MTBCMA would be achieved in actual operation.

NG's position that the agency was required to accept the offerors' estimated MTBCMAs--even where such numbers differed from those most likely to be achieved in actual operation of the proposed systems--is simply untenable. First, there is no language in the RFP suggesting that the agency intended such an approach. More importantly, such an approach would render this area of the evaluation meaningless, since offerors would be expected to employ the most favorable assumptions in their calculations, leaving the agency with no basis for comparing the proposed MTBCMAs.

Inadequate Discussions

NG and ITT argue that the Air Force should have furnished more detailed information concerning its approach to rationalizing offerors' proposed MTBCMAs.

The solicitation did not furnish a precise definition of when corrective maintenance would be required and how to calculate the MTBCMA of the proposed DASR systems. The definition of MTBCMA in the SRD did not specify what maintenance actions were required "to maintain user confidence in the integrity of the DASR System" other than to indicate that they "would include repair of failed redundant units considered critical to meeting the DASR System mission of providing PSR, weather and MSSR data to the automation system." Further, as noted by the agency, MTBCMA as defined in the SRD "is a non-standard parameter in the sense that it is not specifically addressed in the usual handbooks and standards that are commonly found in nearly every reliability engineer's tool box." Indeed, the SSEB's technical adviser testified at the hearing conducted on this protest that he had never seen the term MTBCMA used before. Hearing Transcript (Tr.) at 646, 647.

However, the Air Force did advise offerors during discussions, where necessary, of the need for immediate corrective maintenance actions with respect to failed critical units and of the agency's position concerning the number of components to be characterized as critical. For example, the agency issued written clarification requests to Raytheon and NG advising that their assumption that all failed redundant units could be replaced at fixed intervals (rather than immediately) conflicted with their proposed system specifications, modeled on the SRD MTBCMA provision, which stated that:

"Corrective maintenance actions are those required to maintain user confidence in the integrity of the [DASR] System. These [include] repair of failed redundant units considered critical to meeting the [DASR] system mission."

In addition, NG and Raytheon were furnished with two scenarios to illustrate the agency's position in this regard. Scenario 1 concerned a multiple redundancy configuration in which the first failure did not require immediate corrective maintenance, while Scenario 2 illustrated a one-of-two-redundancy configuration in which immediate corrective maintenance was required, as set forth below:

"Scenario 2

--One of two redundant antenna drive motors fail.

--Impact: Only one drive motor operating; integrity of DASR compromised.

--Immediate corrective maintenance action IS required."

NG's consulting engineer for the DASR proposal (and development manager in the air traffic control area) acknowledged in testimony that, as a result of this clarification request and subsequent May 20-22 oral discussions with the agency, NG was aware of "what's critical, what's not," and of the need for immediate, not deferred, maintenance of critical units. Tr. at 137. This understanding of the agency's position as to when corrective maintenance action would be required was evidenced by NG's subsequent reduction in its proposed MTBCMA from the 1,130 hours previously proposed to 734 hours in its BAFO.

NG complains that the agency did not disclose that the mathematical formula used to calculate MTBCMA included the failure rates of both the on-line and the redundant unit in a one-of-two redundancy configuration (and not just the on-line unit), resulting in a significant reduction to the MTBCMA for NG's proposed system; on the contrary, the agency allegedly indicated during discussions that only the failure rate for one of the units would be counted. Tr. at 146-147.

Contracting agencies are not required to disclose their methodology for evaluating proposals under the stated evaluation criteria. Lexis-Nexis, B-260023, May 22, 1995, 95-2 CPD Para. 14; see DynCorp et al., B-257037.2 et al., Dec. 15, 1994, 95-1 CPD Para. 34. The Air Force thus was not required to disclose to offerors the formula to be used in its MTBCMA calculations. Moreover, it does not appear that offerors were in a position after the OCD to redesign their proposed systems so as to significantly improve both their MTBCMA and their competitive position. Again, the DASR solicitation sought the proposal of NDI radars and required the proposed radars to undergo an OCD "to verify that the offeror has a functional, NDI system" that "exists as a 'system' consisting of all integrated subsystems and other items needed to operate and maintain the system per the SRD (except for development required by the contract)." Although offerors were not prohibited from changing their proposed systems after the OCD, which occurred before discussions commenced, as indicated above, the solicitation specifically cautioned that variations between the proposed design and the design/equipment actually demonstrated at the OCD could be considered a weakness and add to the evaluated proposal risk. [4] NG argues that it could have retained its proposed design and improved its evaluated MTBCMA by substituting more optimistic assumed component failure rates. This argument is unpersuasive--if more optimistic rates legitimately could have been used, it is unclear why NG did not do so when it realized that its MTBCMA could be no higher than 734 hours, that is, well below the desired 1,070 hours. Moreover, NG's sudden adoption of significantly more optimistic component failure rates--after having submitted what presumably was its best, good faith estimate of likely failure rates--reasonably would have been considered suspect.

ITT argues that its discussions were inadequate because, unlike NG's and Raytheon's clarification requests, ITT's clarification request did not include Scenario 2, which set forth a requirement for immediate corrective maintenance in a hypothetical system where one of two antenna drive motors fail. However, discussions with different offerors need not be identical; on the contrary, discussions should be tailored for different offerors, since the need for revision ordinarily will vary with each proposal. Delta Data Sys. Corp., B-260791, July 21, 1995, 95-2 CPD 42; The Pragma Corp., B-255236 et al., Feb. 18, 1994, 94-1 CPD 124. The Air Force explains that it was not necessary to furnish Scenario 2 to ITT because its proposal indicated that, unlike NG and Raytheon, it understood the requirement for immediate corrective maintenance when critical components fail; the only question about ITT's proposal in this area concerned the firm's understanding as to which components were considered critical, and the discussions with ITT addressed this concern. [5] The discussions therefore were adequate.

Reliability of Redundant Configurations

ITT argues that the Air Force improperly used a model for determining the reliability of configurations with multiple redundancies (that is, with more than one redundant or backup unit) which had the effect of understating ITT's MTBCMA; according to the protester, had the agency used the correct model, ITT's evaluated MTBCMA would have increased from 481 hours to 651 hours (still 785 fewer hours than ITT proposed).

The Air Force (and Raytheon) maintains that ITT's calculation of the effect of applying a different model, although purportedly based on the agency's expected maintenance approach, nevertheless departs from that maintenance approach by failing to account for the impact of requiring immediate corrective maintenance when the last redundant element in a multiple redundant configuration fails and that ITT's calculation instead assumes deferring maintenance until the next scheduled preventative maintenance visit.

Our review of the record confirms the Air Force's position that ITT's calculations fail to take certain aspects of the required maintenance approach into consideration. For example, although the configuration of ITT's [DELETED] system is shown to have a two-of-four redundancy--two of the four units are redundant--in the worksheets supporting ITT's claimed 651 hour MTBCMA, the worksheets do not appear to include any allowance for the required immediate repair when the second unit fails. Taking these discrepancies into account, the record suggests that, at best, use of ITT's suggested reliability model would have increased its evaluated MTBCMA to no more than 497 hours (as calculated by Raytheon). (Furthermore, it is not clear why use of ITT's suggested model would not also have resulted in some offsetting increase in the MTBCMAs for the other offerors, thereby eliminating or reducing any improvement in ITT's relative competitive position in this regard.) Thus there is no basis to conclude that the reliability model used by the agency resulted in an understated MTBCMA for ITT.

Raytheon Reliability Data

NG challenges the Air Force's acceptance of Raytheon's estimated component failure rates, which were based on historical factory and field test data for the radars proposed. According to the protester, the assumed component failure rates instead should have been based on the predictions in Military Handbook No. 217 (MIL-HDBK-217), "Reliability Prediction of Electronic Equipment," as were NG's proposed component failure rates. DOD's instructions for use of the handbook, however, state that "[i]t should be emphasized that if similar equipment field historical data are available in sufficient quantity and quality it should always be used in lieu of a prediction. If the operating or environmental conditions are different, then modifications can be performed using MIL-HDBK-217 adjustments." MIL-HDBK-217 Use and Application (Technical Brief, April 1990). These instructions are consistent with testimony by the SSEB's technical adviser, who possesses extensive experience in reliability prediction, that data obtained from the operation of similar equipment in a similar environment is the most reliable predictor of the reliability of a radar system and is "a much more reliable predictor and gives a higher confidence in the prediction" than handbook reliability predictions. Tr. at 610-612.

Raytheon's estimated component failure rates were based on [DELETED] hours of experience--more than [DELETED] years of continuous operation--with its proposed MSSR, and [DELETED] hours--nearly [DELETED] years of continuous operation--with its proposed PSR. Testimony by government engineers confirms that there was sufficient similarity between the factory and field environments in which Raytheon's radars were tested to justify combining the total test data; there was a sufficient quantity of test data to form the basis for a reasonable reliability prediction; and there was sufficient similarity between the radars tested and those proposed by Raytheon to render the historical data a reliable predictor of reliability. Tr. at 612-618, 648-649, 716-734. (If anything, the Raytheon historical data may understate the likely reliability of its radars; normally, the reliability of electronic systems improves over time from when it is first introduced until mid-life. Tr. at 649-650.) Thus, we find no basis to question the agency's acceptance of Raytheon's estimated component failure rates.

Additional Corrective Maintenance Visits

In calculating the average labor cost of an unscheduled corrective maintenance visit to a DASR site, the Air Force made a number of assumptions, some of which appear to have overstated to some extent the likely cost of a visit. [6] For example, one of the most significant determinants of the cost of a visit is the number of technician man-hours required. It is undisputed that when working on radar systems accepted safety practice requires the presence of at least two workers. A working group comprised of persons familiar with radar system maintenance from the FAA and DOD considered in detail each activity to be performed in connection with a corrective maintenance visit, including closing off the current activity of the technicians after notification of a fault, identifying the faulty unit (usually to within one of three possible units), obtaining replacement units from logistics control, travelling to the DASR site from the maintenance center, accessing the site, coordinating the repair with operations/maintenance control, repairing the fault, securing the DASR site, returning to the maintenance center, returning units to logistics control, and closing out the maintenance action. The working group concluded that an average corrective maintenance visit by two technicians to a DASR site will take 6 hours from notification of fault to close out of the maintenance action, for a total of 12 technician man-hours.

NG challenges the Air Force's estimate of the cost of the additional unscheduled corrective maintenance visits to DASR sites--$20 million for NG and $13.5 million for ITT--expected as a result of the lower evaluated MTBCMAs. NG argues that the required work actually can be performed in less than 6 hours and with fewer than 12 technician man-hours. [7]

Our review of the record, including testimony by an experienced military maintenance technician, suggests that the agency's estimate of 12 technician man-hours may in fact be somewhat overstated. This testimony indicated that (1) after allowing for the fact that not all of the tasks to be undertaken before departing for the DASR site and after returning to the maintenance center require two technicians, and (2) given that the potential for task sharing and concurrent task performance exists, it may be possible to accomplish the required work with no more than approximately 7-1/2 technician man-hours (and perhaps with only a little more than 7 hours). Tr. at 702-710. However, the military maintenance technician also testified that when two technicians visit a radar site, which as indicated above is a safety requirement when working on radar systems, "[t]ypically one person does it all and the other person watches." Tr. at 711. The record furnishes no basis for questioning the technician's assessment in this regard, which was based on approximately 16 years of experience maintaining air traffic control radar systems. Thus, the historical record suggests that the potential for task sharing and concurrent task performance may not in fact be realized in practice. While we cannot precisely quantify the hours required under this latter scenario, it appears from our review of the record, including the numbers suggested by the agency, NG and Raytheon, that after allowing for the fact that some of the tasks to be undertaken before departing for the DASR site and after returning to the maintenance center only require one technician, an average corrective maintenance visit may require as much as approximately 9-3/8 technician hours.

In addition, although the agency assumed that the technicians repairing the DASR system will be higher cost civilian employees rather than military personnel, the agency reports that the FAA will man no more than 67 of the 89 sites--its own as well as some of the DOD sites--and it appears reasonable to assume that the remaining 22 sites will be serviced by lower cost military technicians. Further, it appears unreasonable for the agency to have calculated a work-year of 1,800 hours by reducing the available time of 2,080 hours--40 hours times 52 weeks--by expected holidays and vacations, and then allocating the fully burdened cost of a technician, including a factor for leave, over 1,800 hours. In so doing, the agency appears to have double-counted the effect of leave in calculating a burdened hourly rate for the technicians.

After allowing for military manning of some sites and correcting the productive man-year to 1,800 hours, and assuming a continuation of historic working patterns among air traffic control radar maintenance technicians such that the average corrective maintenance visit requires approximately 9-3/8 technician hours, it appears that the additional labor cost to the government of NG's and ITT's lower MTBCMAs relative to the target MTBCMA (and after giving ITT credit for an MTBCMA of 497 hours) could total as much as approximately $12.7 million for NG and $8.1 million for ITT. [8] Furthermore, whatever the precise labor cost of the additional maintenance visits, the very fact that NG's and ITT's lower evaluated MTBCMAs can reasonably be expected to result in a requirement for thousands more unscheduled corrective maintenance visits to DASR sites--26,461 additional visits for NG's system and 17,845 additional visits for ITT's--than desired by the agency or required by Raytheon's system could reasonably be evaluated as a major competitive disadvantage. [9]

IN PRODUCTION STATUS

ITT and NG argue that the Air Force improperly failed to recognize that their proposed DASR systems, like Raytheon's, were "in production." As noted by the agency, however, ITT's proposed DASR system was a combination of modified production subsystems and engineering model subsystems--e.g., its [DELETED]--integrated as a complete system for the first time at OCD, while NG's proposed DASR system, although it included many components currently in production, required a newly developed [DELETED] and was comprised of subsystems that must be integrated for the first time to meet the DASR requirements. In contrast, Raytheon demonstrated at OCD a fully integrated production system taken directly off the production line (and scheduled for delivery to [DELETED]), which required only minor modification to meet the DASR system requirements. The agency thus reasonably assigned Raytheon's system a relative advantage with respect to production status (and also reasonably concluded that the mature character of the system would result in lower risk).

MODE-S UPGRADE/SURVEILLANCE DATA TRANSLATOR GROWTH CAPABILITY

NG and ITT challenge the Air Force's determination that the ease with which Raytheon's proposed DASR system could be upgraded to Mode-S capability was a discriminator offering a significant benefit to the government. As noted by the agency, ITT's approach to upgrading to Mode-S was to [DELETED], while NG's approach required [DELETED]. In contrast, Raytheon's Mode-S upgrade approach essentially consisted of [DELETED]. We find nothing unreasonable in the agency's determination that the greater ease with which Raytheon's proposed DASR system could be upgraded to Mode-S capability represented a relative advantage. Although NG suggests that its Mode-S approach offers other unique, offsetting advantages based on a more advanced Mode-S configuration, its position essentially amounts to mere disagreement with the agency as to the relative desirability of various features and in no way demonstrates that the agency's focus on the greater simplicity of Raytheon's approach was unreasonable.

NG also challenges the Air Force's assignment of a strength to Raytheon's proposal on the basis of the ability of its Surveillance Data Translator to support more [DELETED] display indicators, an ability the agency viewed as desirable since it will eliminate the necessity for the purchase of additional translators at some sites. NG contends that its proposal should have received similar credit because its proposed translator likewise can support more [DELETED] display indicators. However, the contracting officer reports, and the agency's Proposal Analysis Report confirms, that Raytheon's strength in this area also was based on the greater ease with which its translator can support additional display indicators. Specifically, the agency states that Raytheon's translator [DELETED]. There is no basis to object to the agency's determination that Raytheon's proposal was superior on this basis.

[DELETED]

NG and ITT challenge the Air Force's determination that Raytheon's use of [DELETED] was a discriminator offering a significant benefit to the government (because the use of [DELETED] increases efficiency during software development and produces more maintainable code). The protesters argue that they likewise deserved credit in this regard since they also proposed to use [DELETED] during software development. As noted by the agency, however, while [DELETED]. The agency reasonably rated Raytheon's proposal superior based on Raytheon's greater experience with [DELETED] and the greater proportion of its existing code that was developed using [DELETED].

ITT SYSTEM ARCHITECTURE

ITT challenges the Air Force's determination that its system architecture warranted a rating of moderate risk under the reliability/maintainability/availability factor. The rating is unobjectionable. ITT's designs for its [DELETED] and its [DELETED] use a custom architecture--as opposed to an open, modular architecture which facilitates substitution of functionally similar parts that is--designed around [DELETED] processors and uses a total of [DELETED] lines of [DELETED] software code. (In contrast, ITT's remaining processors, and all of Raytheon's and NG's processors, are of an open architecture design.) [DELETED] generally is relatively more difficult to maintain and support, since it typically is more difficult to understand and requires as much as three times more lines of code than higher order languages to develop the same functionality, with more lines of code resulting in higher maintenance costs. In addition, the agency determined that should the need arise to replace the processors (which at least one prior manufacturer had ceased to produce) or associated older hardware (such as [DELETED]), or to otherwise upgrade the system during its expected 20-year life using new processors, ITT's custom design would require significant, costly and time-consuming hardware and software redesign. Tr. at 505-514, 553-563. [10]

ITT asserts that, had the agency raised this matter during discussions, it could have advised it of the existence of several manufacturers still producing [DELETED] processors (or their equivalent), at least one of which reportedly possesses a significant inventory of the processors. However, the current availability of the processors is not a substitute-- and does not establish as unreasonable the agency's preference--for an open system architecture that facilitates any processor upgrades that might be required during the 20-year life of the system. [11] (In practice, the service life of radar systems of this type usually extends to 25 to 30 years. Tr. at 506.) There thus is no basis to question ITT's moderate risk rating.

COST/TECHNICAL TRADEOFF

In a negotiated procurement, the government is not required to make award to the lowest-cost, technically acceptable offeror unless the RFP specifies that cost will be determinative. General Servs. Eng'g, Inc., B-245458, Jan. 9, 1992, 92-1 CPD Para. 44. Cost/technical tradeoffs may be made in selecting an awardee, subject only to the tests of rationality and consistency with the established evaluation factors. Varian Assocs., Inc., B-238452.4, Dec. 11, 1990, 90-2 CPD Para. 478.

Raytheon's technical proposal was evaluated as exceptional/low risk and superior to those of the other two offerors. First, Raytheon's technical proposal was evaluated as exceptional/low risk and superior to NG's and ITT's under the system performance and system specification factor. Although NG and ITT challenge their moderate (NG) and high (ITT) risk ratings under this factor, we note that their proposals were considered as only acceptable, while Raytheon's was evaluated as exceptional under this factor on the basis of a number of significant, beneficial strengths, including superior Mode-S upgrade and Surveillance Data Translator growth capabilities. In addition, the agency viewed the fact that the Raytheon system demonstrated at OCD was a current production model to be a significant advantage indicative of the lower risk associated with its proposal. These advantages would remain notwithstanding the elimination of NG's and ITT's higher risk ratings. Second, Raytheon's technical proposal was evaluated as exceptional/low risk and superior to those of the other two offerors under the reliability/maintainability/availability factor. Third, while both Raytheon's and NG's proposals were evaluated as acceptable/low risk under the SOW/CDRL/IMP factor, ITT's proposal was rated high risk. Furthermore, Raytheon's proposed DASR system was evaluated as having a MTBCMA of 1,134 hours, significantly higher than NG's and ITT's, leading the agency reasonably to conclude that award to Raytheon would result in fewer unscheduled corrective maintenance visits and significant labor cost savings. Our review of the arguments raised by NG and ITT furnishes no basis to question the agency's determination of the technical superiority of Raytheon's proposal. Since cost/price was less important than the technical criterion and, in any case, the evaluated costs of Raytheon's proposal were not substantially higher than NG's and ITT's, we conclude that the Air Force reasonably found Raytheon's proposal to be most advantageous.

The protests are denied.

Comptroller General of the United States

1. The Mode S radar is designed to (1) identify, locate, and track aircraft by using radar signals to obtain information from up to 700 individual aircraft at a time and (2) provide users with a communications channel between aircraft and ground facilities.

2. The surveillance data translator converts the digital target and weather information from the DASR system and reformats the information into a form acceptable for use in current air traffic control automation computers and analog displays.

3. The agency did not calculate the parts costs associated with the various MTBCMAs.

4. To the extent that NG believes that the agency was required to furnish a more detailed statement of its rationalization approach in the solicitation, such argument had to be raised prior to the closing time for receipt of initial proposals in order to be timely. EG&G Management Sys., Inc., B-270509.2, Apr. 1, 1996, 96-1 CPD Para. 195.

5. Specifically, the Air Force issued ITT a clarification request stating that:

"You have identified only the [DELETED]and the [DELETED] as critical elements which need immediate fixing. This seems to be in conflict with the SRD definition stating the MTBCMA critical elements include more than just those two. Please clarify and update availability calculations as required."

6. Although the SSA was briefed on the likely labor cost of the additional or fewer unscheduled corrective maintenance visits to DASR sites expected as a result of the evaluated MTBCMAs and was presented with a briefing chart which showed the labor cost of the additional visits added to the cost of the NG and ITT systems, the record read as a whole (including the source selection decision) indicates that the additional labor cost essentially was viewed by the SSA as a quantification of the impact of the MTBCMAs for purposes of the technical evaluation. According to the SSA, "the quantification of future potential cost savings was not used by me as a dollar-for-dollar offset against the evaluated cost/prices in making my source selection decision." See also Tr. at 83-84, 502-503.

7. In addition, NG argues that the elapsed time of an average visit actually must be less than 3 hours if the operational availability requirements for the DASR system are to be met. However, in view of the agency's detailed review of each activity to be performed in connection with a corrective maintenance visit, undertaken by a working group comprised of persons familiar with radar system maintenance, we do not find such a general allegation to be persuasive.

8. The above numbers for the additional labor costs resulting from NG's and ITT's lower MTBCMAs--$12.7 million for NG and $8.1 million for ITT-- and the lower labor costs resulting from Raytheon's higher MTBCMA ($400,000) were derived from calculations performed by the parties in response to various scenarios suggested by our Office based on the record here.

9. NG also argues that the Air Force improperly took into account an inflation factor (2 percent per annum) in calculating future labor costs. The agency defends its use of an inflation factor and, in fact, asserts, with some justification, that an inflation factor of 3 percent per annum would have been more accurate. Even if NG were correct in its position, the consequent approximate 19-percent decrease in estimated labor cost it calculates would not affect the reasonableness of the award decision.

10. Although agency evaluators also noted that there were [DELETED] Raytheon items for which there was only a single source, the agency did not view this as a matter of significant concern since the open architecture of Raytheon's system facilitated the substitution of functionally equivalent items if that should prove necessary.

11. Further, since redesigning this aspect of its system to move from a custom to an open architecture and rewriting the extensive body of [DELETED] code in a higher order software language would have required a significant effort and redesign, especially in the context of an NDI procurement, Tr. at 563, the agency was not required to raise the matter during discussions; agencies are not required to point out weaknesses derived from an inherent aspect of an offeror's approach which would require substantial revision to resolve the agency's concern. Northrop Worldwide Aircraft Servs., Inc., B-262181, Oct. 27, 1995, 95-2 CPD Para. 196; Tracor Flight Sys., Inc., B-245132, Dec. 17, 1991, 91-2 CPD Para. 549.

DOCUMENT FOR PUBLIC RELEASE A protected decision was issued on the date below and was subject to a GAO Protective Order. This version has been redacted or approved by the parties involved for public release.