Skip to main content

Matter of: Hughes Missile Systems Company File: B-272418; B-272418.2; 272418.3 Date: October 30, 1996 * Redacted Decision

B-272418,B-272418.2 Oct 30, 1996
Jump To:
Skip to Highlights

Highlights

Protest against evaluation of past performance is denied where information reasonably available to the procuring agency supported the report of the agency administering the protester's contract to the effect that protester was deficient in its cost/schedule performance on that contract. Is denied where offerors were advised prior to the closing date for receipt of proposals that agency viewed program similarity as more important than product similarity. BACKGROUND The JASSM missile will be an autonomous. Which can be launched from beyond the range of enemy air defenses and will provide the capability to strike heavily defended. Offerors were advised that the agency considered the need for such a standoff attack capability to be urgent and compelling.

View Decision

Matter of: Hughes Missile Systems Company File: B-272418; B-272418.2; 272418.3 Date: October 30, 1996 * Redacted Decision

Protest against evaluation of past performance is denied where information reasonably available to the procuring agency supported the report of the agency administering the protester's contract to the effect that protester was deficient in its cost/schedule performance on that contract. Protest that agency did not adequately consider the complexity of the products being developed and/or procured when evaluating past cost/schedule performance and arriving at an overall rating, and instead placed undue emphasis on program complexity, is denied where offerors were advised prior to the closing date for receipt of proposals that agency viewed program similarity as more important than product similarity.

Attorneys

DECISION

Hughes Missile Systems Company protests the Department of the Air Force's award of contracts to Lockheed Martin Integrated Systems, Inc. (LM) and McDonnell Douglas Aerospace Corporation (MD), under request for proposals (RFP) No. F08626-96-R-0002, for the definition and development of the Joint Air-to-Surface Standoff Missile (JASSM). Hughes challenges the past performance, cost and technical evaluations.

We deny the protest.

BACKGROUND

The JASSM missile will be an autonomous, precision strike standoff weapon, to be carried aboard a number of types of aircraft, which can be launched from beyond the range of enemy air defenses and will provide the capability to strike heavily defended, high value targets. Offerors were advised that the agency considered the need for such a standoff attack capability to be urgent and compelling. The solicitation contemplated the award of two cost-plus-fixed-fee contracts for a 24-month Program Definition and Risk Reduction (PDRR) phase, which would include priced options for a follow-on cost-plus-incentive-fee Engineering, Management and Development (EMD) phase, to be exercised on the basis of a downselect competition between the two PDRR contractors. The downselect contractor will then commence production, which is expected to include 10 annual production lots. Offerors were required to furnish cost and pricing information for the PDRR and EMD phases and cost estimates for the production phase; during the downselect competition, the PDRR contractors will have the opportunity to update their EMD phase pricing and to submit cost and pricing information for the production quantities.

Award was to be made to the offerors whose proposals were most advantageous to the government under three broad criteria: (1) past performance--including past technical performance (with factors for product performance, computer software, and aircraft integration) and past affordability performance (with factors for manufacturing and cost/schedule), which were of equal weight--which was equal in importance to the aggregate of (2) technical performance (key performance parameters, other requirements, and integrated master plan and schedule) and (3) affordability, which were of equal weight. The affordability criterion included four factors: (1) average unit procurement price (AUPP) for production lots 1-5 (the most important factor); (2) mission/cost effectiveness--including the expected number of missiles to kill the specified JASSM target set, total cost (number of missiles times most probable AUPP) to kill the JASSM target set, average cost to kill a JASSM target, and number of aircraft sorties (missions)--which was equal in importance to (3) total contract price for the PDRR and EMD phases; and (4) AUPP for production lots 6-10 (the least important factor).

Five proposals--including Hughes's, LM's, and MD's--were received by the closing time on April 29, 1996. All were included in the competitive range. Following written and oral discussions with offerors, the Air Force requested PDRR and EMD "price" proposals.

Based upon the evaluation of offerors' past performance and their written and oral submissions, the source selection advisory council emphasized the following considerations in its report to the source selection authority (SSA): (1) while LM had the strongest past performance and MD's performance was "consistently good," Hughes's performance was characterized by a weakness with respect to ability to perform to cost/schedule; (2) all offerors' proposed missiles exceeded the minimum key performance parameters; and (3) while Hughes's offer had the best mission/cost effectiveness rating, with the lowest AUPP and average cost per target, LM's and MD's proposed and evaluated AUPP were below the agency's objective unit price of $[DELETED] and their offers had lower development (PDRR and EMD) costs. The specific results of the evaluation were as follows:

Hughes LM MD

Past Performance

Product Exceptional (Blue) Acceptable Acceptable

Software Acceptable (Green) Acceptable Acceptable

Aircraft Acceptable Exceptional Exceptional Integration

Manufacturing Acceptable Exceptional Acceptable

Cost/Schedule Marginal (Yellow) Acceptable Acceptable

Technical

Key Exceptional Exceptional Exceptional Performance /Moderate Risk /Moderate Risk /Moderate Risk

Other Exceptional Exceptional Exceptional Requirements /Moderate Risk /Moderate Risk /Moderate Risk

Plan/Schedule Acceptable Exceptional Exceptional /Moderate Risk /Moderate Risk /Moderate Risk

Affordability

Most Probable $[DELETED] $[DELETED] $[DELETED] AUPP (Lots 1-5)

Mission/Cost Threshold/ Threshold/ Threshold/ Effectiveness Objective Objective Objective

Total Missiles [DELETED] [DELETED] [DELETED] Sorties [DELETED] [DELETED] [DELETED] $ per Target $[DELETED]/ $[DELETED]/ $[DELETED]/ $[DELETED] $[DELETED] $[DELETED]

Total Contract $[DELETED] $[DELETED] $[DELETED] (PDRR and EMD)

Most Probable $[DELETED] $[DELETED] $[DELETED] AUPP (Lots 6-10)

Based upon the evaluation record, the SSA concluded that LM's and MD's offers were most advantageous. Specifically, the SSA noted that only LM and MD were rated as consistently good-to-excellent performers and, in particular, only those two were rated acceptable with respect to past cost/schedule performance. The SSA considered their good performance in the cost/schedule area to be "a very key discriminator because the schedule on this program is of paramount importance to the user due to the critical operational shortfall this system fills." In contrast, the SSA noted that while Hughes's past performance was generally good, it was characterized by a significant weakness with respect to cost/schedule performance as a result of schedule problems experienced on the Advanced Medium Range Air-to-Air Missile (AMRAAM) program and the Tomahawk (Cruise Missile) Baseline Improvement Program (TBIP), and significant cost overruns experienced on the TBIP program. The SSA found the TBIP cost and schedule problems to be "particularly troubling as this program is of similar scope to JASSM." In addition, the SSA noted that only LM's and MD's proposals were rated as superior (exceptional) with moderate risk under all three technical performance factors. The SSA specifically concluded that MD's proposed design was "the lowest risk approach due to their use of a proven off-the-shelf [DELETED] and existing warheads," while LM's design was "of somewhat higher risk, principally due to the [DELETED]." In contrast, the SSA considered Hughes's design to have "the highest technical risk of these three contractors, driven" by their proposal of a [DELETED] target seeker and new ([DELETED]) warhead.

In the affordability area, the SSA viewed the fact that LM's and MD's proposals had the lowest total contract prices for the PDRR and EMD phases to be more significant than Hughes's "roughly [DELETED] percent" advantage with respect to estimated AUPP for production lots 1-5, noting that he "did not consider this to be a significant difference at the beginning of the PDRR phase" and that LM's higher AUPP was based on a [DELETED]. As for Hughes's advantage with respect to average cost per target killed, the SSA noted that this was primarily due to their proposal to develop a new, more powerful warhead, which made unnecessary for many targets the greater accuracy afforded by a target seeker and thereby enabled Hughes to propose a higher proportion of less expensive non-seeker missiles; he considered this approach to be characterized by higher risk than MD's offer of an off-the-shelf, proven warhead. The SSA concluded that LM's and MD's superior past performance, superior technical proposals and lowest total contract price overshadowed Hughes's evaluated advantage with respect to AUPP and average cost per target.

Upon learning of the resulting June 17 awards to LM and MD, Hughes filed this protest with our Office, challenging numerous aspects of the evaluation. We discuss its primary arguments below.

PAST PERFORMANCE

Background

Hughes challenges the evaluation of past performance. As noted above, past performance was the single most important evaluation area, with a weight of approximately 50 percent. Past performance was evaluated based on a maximum of three contracts/programs for each of the five past performance factors--product performance, computer software, aircraft integration, manufacturing and cost/schedule. The RFP generally stated that "[t]he Government will evaluate past performance using the most recent and relevant performance data" and that "[w]henever possible, the Government will gather past performance [data] on contracts/programs which are similar to the JASSM program in product types, program phase, complexity, scope, competitive environment, and contract type." In addition, for each of the five past performance factors, the solicitation specifically described the prior contracts/programs that would be considered similar. For example, with respect to cost/schedule performance, the evaluation of which is at issue here, the solicitation described a similar contract/program as one involving "a complex missile program that used streamlined acquisition initiatives, such as limited government oversight or small industry-Government teams." Potential offerors were notified (prior to the issuance of the final RFP) of the contracts/programs selected for evaluation.

The solicitation stated that the agency would evaluate past performance under the selected contracts using information from such sources as Contractor Performance Assessment Reports (CPAR), the Defense Contract Management Command (DCMC), program offices, other service organizations and the offerors. It required each offeror to identify in its offer "data sources and points of contact used to substantiate his strengths, what he has done to overcome any previous problems encountered . . . and any mitigating factors that precluded the offeror from meeting his contract cost, schedule or performance requirements." Although the solicitation limited the past performance proposal to 20 pages, potential offerors were advised prior to issuance of the solicitation that they could furnish data to DCMC supporting their past performance proposals. (Although advised by the Air Force on three occasions and by DCMC on a fourth that it could furnish additional supporting data, Hughes, unlike the other four offerors, declined the opportunity afforded it to submit additional data prior to the closing date.) In addition, where the agency questioned an offeror's past performance in a particular area on a particular contract, it raised the matter during discussions and afforded the offeror an opportunity to respond in writing and orally. (While the agency's points for negotiation (PFN) included limits on the length of responses, Hughes was afforded an additional 10 pages for responding to the PFNs concerning its past performance.)

TBIP

As noted above, in evaluating Hughes's past cost/schedule performance, the agency received reports that Hughes experienced schedule problems on the AMRAAM air-to-air [1] and TBIP cruise missile programs and significant cost overruns on the TBIP program. These reports led the agency to evaluate Hughes's overall cost/schedule performance as marginal (yellow).

Hughes challenges this evaluation to the extent it is based on Hughes's TBIP performance, primarily asserting that the agency undertook only a superficial investigation in this regard and, furthermore, unreasonably concluded that cost growth on the TBIP program would inevitability result in schedule delays, when, according to Hughes, this has not proven to be the case.

We will review an evaluation of an offeror's performance risk to ensure that it was reasonable and consistent with the stated evaluation criteria. See Dragon Servs., Inc., B-255354, Feb. 25, 1994, 94-1 CPD Para. 151. An evaluation of past performance may be based on the agency's reasonable perception of inadequate prior performance, even where the contractor disputes the agency's interpretation of the facts. See Cessna Aircraft Co., B-261953.5, Feb. 5, 1996, 96-1 CPD Para. 132; Rockwell Int'l Corp., B-261953.2; B-261953.6, Nov. 22, 1995, 96-1 CPD Para. 34; Pannesma Co. Ltd., B-251688, Apr. 19, 1993, 93-1 CPD Para. 333.

The Air Force reasonably determined that Hughes's cost/schedule performance on the TBIP program had been materially deficient. The record shows that the Air Force relied on information received from the Navy TBIP program office (including documentation generated by Hughes) and from Hughes, which reasonably indicated that Hughes had in fact experienced cost and schedule problems on the TBIP program.

Specifically, we note that in its April 29 response to the Air Force's request for an evaluation of Hughes's performance, the Navy TBIP program office rated both Hughes's cost and schedule performance as a "2" on a scale of 1-4, with "4" indicating an ability to perform with little or no government oversight and "1" indicating an inability to perform; the TBIP program office reported that Hughes had demonstrated the ability to execute program plans, including implementation of any changes or recovery initiatives, within cost and schedule only "with substantial Government oversight." [2] The agency also obtained specific information and documents from the TBIP program office which supported the Navy's determination that Hughes had encountered significant cost/schedule problems in its performance. For example, although awarded the TBIP cost contract in the amount of $226 million in September 1994, Hughes encountered sufficient cost growth in the first year of the contract that in July 1995 it requested Navy approval for an Over Target Baseline (OTB) of $294.8 million (ultimately $298.7 million). Indeed, the TBIP program office furnished Air Force evaluators information indicating that in February 1996 Hughes had identified the "'potential' for additional $42.2M [million] cost growth," and that based on Hughes's April 1996 briefing it appeared that the OTB was "becoming obsolete." Hearing transcript (Tr.) at 1218-1219. (In a July 1996 "TBIP Baseline Status Review," Hughes indicated an estimate at completion (EAC) of $381 million for the TBIP program.) In this regard, the TBIP program manager testified at the hearing held on this protest that Navy source selection officials had concluded during the TBIP source selection that Hughes had significantly underestimated the cost of performance and that, as a result, the Navy had added an additional $70 million to the TBIP budget beyond the $226 million proposed by Hughes. Tr. at 1019, 1094.

Further, the information available to the Air Force at the time it was evaluating Hughes's TBIP performance (April/May 1996) reasonably indicated that Hughes was encountering performance delays that could ultimately delay delivery of the TBIP improvements. For example, Hughes was unable to hold the preliminary design review (PDR) on the TBIP contract in September 1995, when originally scheduled, and ultimately was not ready for PDR until February 1996. In the view of the TBIP program office, Hughes's delay in this regard created the risk that Hughes would be unable to meet the required date for initial operating capability (IOC). Tr. at 1013-1015, 1125-1127. In addition, by letter dated February 26, 1996, Hughes advised the TBIP program office that the funds allocated to the TBIP contract--which was incrementally funded on an annual basis and contained a contractual limitation on the funds available for any given period--would be insufficient at current rates of expenditure to continue performance for the last approximately 3 months of the current contract period. Further, in an April 1996 briefing of the TBIP program office, Hughes warned the Navy that "TBIP Program IOC will slip 12 months and FOC [full operating capability] 15 months." Although Hughes referred to only a 10-month slip in IOC in a May 1996 electronic mail to the TBIP contracting officer, it also referred to a "15 month replan schedule," while the TBIP program manager testified that he anticipated a 15-month delay in IOC (and not merely the 10- or 12-month delay forecast by Hughes). Tr. At 1054-1055. Further still, the TBIP program office furnished Air Force evaluators information indicating that Hughes was proposing to stop work on several system capabilities. Tr. at 1218-1219. The fact (as pointed out by Hughes) that the schedule problems had not yet caused Hughes to miss any contractually established performance dates did not diminish the seriousness of Hughes's schedule problems; the only date established by the TBIP contract was IOC at the conclusion of the contract in the year 2000 and, again, as noted in an April 1996 briefing chart furnished to the TBIP program office, Hughes was advising contracting officials that it assumed that "TBIP Program IOC will slip 12 months." Tr. at 1013.

When questioned by the Air Force during discussions about its performance on the TBIP contract, Hughes's response led the agency to believe that Hughes was experiencing serious cost/schedule problems for which it bore significant responsibility. In this regard, the Air Force's PFN in this area noted that there was an "approximate 32% ($72M [million]) [cost overrun] plus another $42M problem identified by contractor in latest financial review," and a "[p]otential 3 months schedule delay due to lack of funding plus another 12-month slip presented by contractor in latest financial review." In its response, Hughes appeared to concede some responsibility for the cost overruns, stating that:

"[a]dmittedly, Hughes's bid reflected specific optimistic assumptions in its cost estimate, in particular related to the [TBIP] seeker subcontract and engineering labor. In addition, it is Hughes's position that added work comprises $41.9M of the $72M cost overrun cited in the PFN."

(Hughes's response also noted that the Navy did not agree with its position.)

The Air Force reasonably viewed Hughes's statement as an admission that it had underestimated its costs when competing for the contract and that the contract was at least $30.1 million over cost irrespective of any work allegedly added by the Navy. Tr. at 1211, 1235. The agency's conclusion in this regard is consistent with a Hughes summary of the TBIP cost history obtained by the agency, in which Hughes attributed the cost growth to an "aggressive bidding strategy," as well as to "requirements clarifications/evolution." It is also consistent with the TBIP program manager's position that Hughes "was largely responsible for the cost overruns" on the TBIP contract. Tr. at 1019-1020. Further, in its PFN response, Hughes did not deny the existence of these potential 3 - and 12- month schedule slips. In our view, the Air Force could (and did) reasonably attribute the potential 3 -month funding-based delay in the first instance to Hughes's failure to properly cost its proposed TBIP contract effort. Although Hughes attributed the "potential 12 month slip" to the cancellation of another missile program under which its seeker subcontractor was developing the target seeker Hughes proposed for TBIP, the information available to the Air Force from the Navy indicated that the choice of the seeker to propose had been up to Hughes, and suggested that, in any case, Hughes had unduly delayed in selecting a replacement seeker. Tr. at 1037, 1045-1046, 1056, 1273.

Having reasonably determined that Hughes was experiencing serious cost and schedule problems on the TBIP contract for which it bore significant responsibility, and in view of Hughes's 9-month schedule delay in its performance on the AMRAAM program, the Air Force reasonably evaluated Hughes's past cost/schedule performance as marginal. Further, given the fact that its cost/schedule problems had occurred on programs that, like the JASSM program, involved missiles, and the fact that the schedule on the JASSM program was of paramount importance "due to the critical operational shortfall" the system was to fill, the SSA, in his cost/technical tradeoff, reasonably viewed Hughes's deficient past cost/schedule performance as a significant weakness.

Product Similarity

Hughes more generally argues that the evaluation of its past cost/schedule performance reflects an overall problem in the evaluation of past performance. Essentially, Hughes contends that the agency unreasonably failed to consider the complexity of the products being developed and/or procured when evaluating past performance and arriving at an overall rating. It specifically contrasts in this regard the evaluation of its past cost/schedule performance on the TBIP program with the evaluation of LM's performance on the Joint Direct Attack Munition (JDAM) program.

The Air Force examined LM's cost/schedule performance on three programs that it had determined would furnish useful data as to LM's likely JASSM cost/schedule performance: (1) JDAM, an inertial guidance/global positioning system tail kit added to bombs to guide them to their target; (2) the Hellfire II program, an upgrade to the Hellfire air-launched anti- tank missile, with a semiactive laser seeker for guidance; and (3) the Low Altitude Navigation and Targeting Infrared System for Night (LANTIRN) program, consisting of a navigation pod and a targeting pod mounted on aircraft and primarily used for targeting laser-guided air-to-ground missiles. Although LM experienced schedule delays during the development and initial production of LANTIRN and Hellfire II, and cost overruns of approximately 60 percent during the development of Hellfire II, the Air Force concluded that these problems were overshadowed by the facts that (1) major problems no longer existed on the LANTIRN program and LM was performing ahead of schedule, resulting in a LANTIRN program manager rating of "4," the top rating, (2) there had been no schedule problems on the JDAM program, and (3) LM was performing or had performed without cost problems during the Hellfire II production phase and generally on the LANTIRN and JDAM programs. In particular, with respect to JDAM, the agency noted that the JDAM program manager reported that LM had never experienced any cost/schedule problems and, indeed, had performed ahead of schedule in some instances notwithstanding a "very challenging" schedule; according to the JDAM program manager, LM's JDAM performance was "absolutely superb." As a result, the Air Force rated LM's past cost/schedule performance acceptable.

Noting the importance of LM's JDAM performance in LM's acceptable rating, and pointing out that unlike TBIP or AMRAAM, JDAM includes neither an engine, power supply nor warhead, Hughes contends that the agency has failed to account for the fact that Hughes's marginal cost/schedule performance rating was earned for products that were considerably more complex than LM's JDAM (or even LANTIRN). According to Hughes, the agency's approach resulted in an overemphasis on program similarity at the expense of product similarity.

Hughes's position is without merit. The Air Force determined that LM's performance on the JDAM program would furnish a reliable predictor of its likely cost/schedule performance on JASSM based primarily on similarities in program complexity, not product complexity. Tr. 721-747. This determination was consistent with the agency's approach to past performance as announced prior to the due date for receipt of initial proposals. Specifically, worksheets provided to potential offerors along with the draft RFP (for purposes of soliciting offeror input into the selection of the contracts to be evaluated) advised that the Air Force would evaluate the relevance of contracts for consideration under the cost/schedule factor using seven criteria: (1) product similarity, with the greatest weight to be accorded to contracts/programs involving cruise missiles, and lesser (but still significant) weight to be given to guided missiles and guided bombs; (2) program phase; (3) system complexity; (4) subcontractor integration; (5) contract type; (6) contract environment; and (7) whether the contract/program involved a streamlined acquisition approach. At most, only two of these factors (product similarity and system complexity), accounting for only 35 of 105 rating points, related to product complexity. (Moreover, even for those factors, it was clear that the agency may consider a guided bomb like JDAM to be somewhat similar to a cruise missile like TBIP since, although a full 20 points were available under the product similarity factor for a cruise missile, 12 points were available for a guided bomb.) In general, the worksheets indicated the agency's position that for purposes of evaluating past cost/schedule performance, program similarity would be more significant than product similarity. Tr. at 748, 773-775. Given the agency's previously announced position in this regard, the fact that the agency's approach resulted in programs for products of dissimilar complexity being considered similarly relevant for purposes of serving as a predictor of future cost/schedule performance on JASSM based on similarities in program complexity is not objectionable.

TARGET X

Hughes challenges the evaluation of mission/cost effectiveness under the affordability factor. The solicitation provided for evaluation of the probable mission/cost effectiveness of the proposed JASSM missile based on consideration of the expected number of missiles required to kill the specified JASSM target set, total cost (number of missiles times most probable AUPP) to kill the JASSM target set, average cost to kill a JASSM target, and number of aircraft sorties (missions) required to kill the target set. In this regard, the ultimate overall JASSM target set adopted by the Air Force and furnished to offerors for use in preparing their designs included 20 target types: a threshold target set of 17 target types and 686 targets, and an objective target set comprised of all 20 target types and a total of 717 targets. In other words, 3 of the target types and 31 of the targets were simply objective targets. While the solicitation provided for the entire JASSM target set, including objective targets, to be taken into account in evaluating mission/cost effectiveness, only threshhold target types were considered in the evaluation of compliance with key technical performance parameters. One of the objective target types, code-named Target X for purposes of this protest, was a relatively large target that was most vulnerable to warheads emphasizing blast effectiveness--e.g., through use of a thinner warhead casing and more explosive--over penetrating capability. [3] Target X was not included in the initial draft target set but, rather, was added in October 1995, approximately 5 months before the final solicitation was issued.

The prescribed approach adopted to measure missile effectiveness against most of the target types, including Target X, was the Department of Defense's Joint Munitions Effectiveness Manual (JMEM) Open End Methods model. Tr. at 35, 167-168. For a particular warhead against a particular target, this program calculates a single shot probability of damage (Pd), that is, the percentage of the target that a single warhead is likely to destroy. The solicitation provided that "[t]he offeror shall use only Government provided methodology for computing Pd."

Prior to issuance of the final solicitation, Hughes, MD, and LM advised the Air Force that destroying Target X would be difficult. Hughes advised the agency that the Target X class of targets "drives the need for" high explosive, and that without its proposed new [DELETED] warhead, which offered additional blast effectiveness, "it would cost you another thousand missiles" to destroy the Target X class of targets. Tr. at 517-518. Likewise, MD cautioned the agency that Target X "sucks up warheads like crazy," and that if it relied on a penetrator type of warhead against Target X, it would require significantly more missiles. Notwithstanding these warnings, the Air Force retained Target X in the JASSM target set.

Upon evaluating the effectiveness of offerors' proposed warheads, however, the Air Force noted that the JMEM methodology predicted that large numbers of missiles would be required to destroy the Target X targets. Although this was true for the warheads proposed by all of the offerors, it was particularly true for LM's warhead, which apparently [DELETED]. Indeed, the agency's initial evaluation indicated that approximately [DELETED] more LM missiles would be required to destroy the objective target set (which included Target X) than Hughes or MD missiles. However, the Air Force concluded that the data indicated that the outputs from the JMEM model with respect to the Pd and number of weapons required to destroy Target X were unreliable due to Target X's vast size. Specifically, the agency maintains that where the Pd value--that is, the probable single shot percentage of destruction of the target--is sufficiently small, inherent uncertainty in the JMEM model renders comparison among warheads unreliable. Tr. at 47-65. [4] Given the perceived unreliability of the data with respect to Target X, and the fact that the solicitation provided for the agency to evaluate the realism of the primary parameters used to calculate missile effectiveness, the agency essentially decided to normalize the results to prevent Target X from becoming a discriminating factor in the source selection; the agency adjusted the Target X effectiveness of all warheads to that achieved by the most effective warhead, which was [DELETED].

Hughes argues that it was unreasonable for the Air Force to abandon the prescribed method (the JMEM model) for calculating missile effectiveness against Target X, and to instead equalize the warheads' effectiveness against Target X. Hughes maintains that this had the effect of improperly eliminating from consideration the likely superior performance of Hughes's warhead against Target X relative to the performance of LM's warhead.

It is clear that the Air Force has determined that using the prescribed evaluation approach--evaluating the effectiveness of warheads against Target X using the JMEM model--would produce a result which does not represent the agency's actual minimum needs. According to the agency, given the relative ineffectiveness of all of the warheads against Target X, warfighters would be unlikely to mount a broad, area-based attack with JASSM missiles; they instead likely would either launch a precision attack against critical elements of Target X--about which insufficient information was provided to offerors to permit them to calculate such an attack--or attack Target X with a different weapon system.

Upon determining that the prescribed evaluation approach would lead to an overemphasis on Target X and thereby produce a result which did not represent the agency's actual minimum needs, the agency was faced with a choice between effectively eliminating Target X from the evaluation without reopening negotiations or reopening negotiations on the basis of a revised evaluation approach. In this regard, as a general matter, source selection officials do not have the discretion to announce in the solicitation that they will use one evaluation plan and then follow another without informing all offerors of any significant changes in the evaluation scheme and affording them an opportunity to respond to the modified evaluation approach. See DynCorp, 71 Comp.Gen. 129 (1991), 91-2 CPD Para. 575; Colonial Storage Co.; Paxton Van Lines, Inc., B-253501.5 et al., Oct. 19, 1993, 93-2 CPD Para. 234, aff'd, Colonial Storage Co.-- Recon., B-253501.8, May 31, 1994, 94-1 CPD Para. 335. Here, the Air Force decided to eliminate Target X from the evaluation but not to reopen negotiations.

Hughes argues that the Air Force made a significant change in the stated evaluation approach such that it was required to reopen negotiations so as to afford offerors an opportunity to respond by modifying their proposals. Specifically, Hughes claims that had it been advised of the agency's decision, essentially, to remove Target X from the target list, it would have altered its warhead approach to account for this, perhaps by choosing a nondevelopmental warhead (and thereby reducing its evaluated warhead development risk and saving on the cost of developing the [DELETED]), or perhaps by redesigning the [DELETED] to optimize its performance against the remaining targets. (Hughes's chief engineer for the JASSM proposal estimated that choosing a nondevelopmental warhead could save approximately $[DELETED] in development costs and reduce Hughes's AUPP by approximately $[DELETED], while redesigning the [DELETED] could reduce Hughes's total contract price by approximately $[DELETED] and its AUPP by approximately $[DELETED]). Tr. 507-518, 540-548, 571-572. (We note, however, that in its prior submission dated July 25, 1996, Hughes stated that "Hughes likely would have proposed to use a significantly less expensive off-the-shelf warhead were it not for Target X.")

Where the record establishes a procurement deficiency, we will sustain a protest on that basis only where it resulted in competitive prejudice; where an agency improperly changes its stated evaluation approach after receipt of offers, such prejudice exists where the record shows a reasonable possibility that the protester would have altered its proposal to its competitive advantage had it been given the opportunity to respond following the change. Federal Computer Corp., B-239432, Aug. 29, 1990, 90-2 CPD Para. 175; see Global Assocs. Ltd., B-271693; B-271693.2, Aug. 2, 1996, 96-2 CPD Para. 100; Akal Sec., Inc., B-261996, Nov. 16, 1995, 95-2 CPD Para. 216.

The record in this case is extensive, including contemporaneous internal documentation of Hughes's design and warhead selection process and testimony by the Hughes chief engineer for the JASSM project. Based on a close review of this record, we conclude that, at most, had the Air Force sought revised proposals, Hughes could have reduced its development costs by a relatively small amount. While it is possible that a change in warhead could have resulted in a less expensive Hughes missile, i.e., a lower AUPP, additional Hughes missiles would be required to destroy the JASSM target set, so that overall mission/cost effectiveness would not have materially improved and might actually have suffered. Given the advantages both LM and MD had in past performance and technical performance, together worth 75 percent of the evaluation, we share the Air Force's opinion that Hughes could not have altered its proposal to its competitive advantage. We reach this view even assuming that given an opportunity to revise their proposals to reflect the elimination of Target X, LM, and MD would not have improved their own competitive standing.

As an initial matter, Hughes has not specified which warhead it would have substituted for the [DELETED]; according to the protester, a reexamination of its warhead approach "would take thousands of hours because that's how many hours it took for our engineers to do the original to determine the best warhead." Tr. at 517, 542, 567. This inability to specify the particular replacement warhead Hughes would have selected left the Air Force and the awardees unable to address specifically Hughes's claim of prejudice, but it also left Hughes with only speculation about a possible warhead redesign to contradict the existing record, which leads us to conclude that Hughes would not have offered a different warhead. In this regard, the JASSM target list included other blast-intensive targets (besides Target X) and internal Hughes documentation produced in response to an agency document request shows that Hughes's proposed [DELETED] warhead clearly would be the most effective warhead--i.e., it would require fewer total missiles--against the JASSM target set even after the elimination of Target X. Tr. at 525-528, 533-539, 549-551, 574-576, 927-928.

For example, Hughes proposed the [DELETED] as a backup in case developmental problems were encountered on the [DELETED] warhead. Tr. at 499-500. Approximately 23 percent more JASSM missiles with [DELETED] warheads than missiles with [DELETED] warheads would be required to destroy the JASSM target set even after the elimination of Target X. [5] Further, the Hughes documentation indicates that the [DELETED] was viewed by Hughes as having a higher risk ("high" risk) and unit cost than Hughes's [DELETED] warhead design (which was viewed as "low" risk) from which the [DELETED] design was derived. [6] In this regard, the Hughes documentation indicates that the [DELETED] was viewed as having the lowest risk and unit price of any of the warhead options discussed in the documentation and Hughes's chief engineer testified that the [DELETED] warhead had the same unit cost and risk ratings as the [DELETED]. Tr. at 550-551. [7] (According to the agency, however, since the warhead on Hughes's [DELETED] design was smaller than the one on the [DELETED] design Hughes's missile effectiveness ratings would have suffered had it substituted the [DELETED] for the [DELETED], while the evaluated development risk would have remained the same.) [8] As for the possibility that Hughes could have begun developing still another new warhead had it been advised of the removal of Target X from the target set, we note that this would not have eliminated the evaluated risk associated with Hughes's proposal to develop the [DELETED] warhead, but instead could very well have increased that risk given Hughes's belated start on designing such a warhead. Since the record--including Hughes's contemporaneous documentation--indicates that Hughes's [DELETED] warhead remained its most effective warhead even after removal of Target X from the JASSM target set, and given the substantial effort required to change warheads, we simply do not find credible Hughes's claim that elimination of Target X would have led it to alter its proposal to its significant competitive advantage.

Furthermore, even if Hughes had selected a different warhead--for example, the [DELETED]--and, as a result, had been able to reduce its total contract price for the development phases of the JASSM program by up to Hughes's estimate of $[DELETED] for the [DELETED] (leaving its price at $[DELETED], considerably higher than the $[DELETED] and $[DELETED] prices offered by LM and MD), and also lowered its estimated AUPP for the [DELETED] production quantities, there is no basis for concluding that Hughes would have been in line for an award. Hughes offered no contradiction to agency assertions that the [DELETED] would have markedly reduced the Hughes mission effectiveness numbers. Similarly, a $[DELETED] reduction in development cost that Hughes asserts would result from redesign of the [DELETED], and a reduction in AUPP, would not have placed Hughes in line for award. Here, too, agency assertions that Hughes's overall missile effectiveness numbers would have suffered--because a reduction in warhead size (which Hughes suggested) would have increased the number of missiles needed to destroy the JASSM target set--were effectively uncontradicted by the protester. Hughes's proposal was reasonably evaluated as having a significant weakness with respect to past cost/schedule performance, and being less advantageous than LM's or MD's proposals with respect to overall past performance, which was worth 50 percent of the evaluation. In addition, Hughes's proposal was also evaluated as being of higher risk and less advantageous with respect to technical performance, which was worth another 25 percent of the evaluation. [9] Further, its total contract price for the development phases of the JASSM program under any scenario would have remained substantially higher than either LM's or MD's prices. [10] In these circumstances, even assuming no improvement in LM's and MD's proposals had the Air Force reopened negotiations, we see no reasonable possibility that any improvement in Hughes's position with respect to the affordability criterion, which accounted for only 25 percent of the evaluation, would be sufficient to offset its evaluated disadvantages under evaluation criteria worth 75 percent of the evaluation.

The protest is denied.

Comptroller General of the United States

1. Information in CPARs (including yellow performance ratings) and from DCMC indicated that Hughes had experienced schedule delays in its performance on the AMRAAM program. In response to the PFN issued in this regard, Hughes conceded that a 9-month schedule delay had been encountered on an AMRAAM contract, but pointed out that this had not resulted in a delay on a subsequent production contract and that its current CPAR performance ratings on the AMRAAM program were either blue or green. The agency nevertheless considered the delay to be relevant since it appeared that a delay in production had been avoided only because the production schedule was able to accommodate the delay under the preceding contract.

2. Although, as pointed out by Hughes, the TBIP program manager at one point advised the Air Force that "[i]n general, [he] was pleased with the performance of both McDonnell Douglas and Hughes" on the overall Tomahawk program, the record indicates that the program manager concurred in the evaluation that Hughes had encountered cost and schedule problems on TBIP and, as noted below, he specifically testified that Hughes "was largely responsible for the cost overruns" on the TBIP contract. Tr. at 1012-1024, 1112-1113.

3. Since some of the information concerning the JASSM target set is classified, our discussion of the Target X issue is somewhat limited.

4. We note, however, that the record indicates that the warheads in fact possessed different blast capabilities. In this regard, an agency engineer who participated in the JASSM lethality assessment testified that the inherent uncertainties in the JMEM model when used to evaluate the effectiveness of the proposed JASSM warheads against large targets such as Target X were such that it was "[n]ot necessarily" the case that the LM warhead would be "considerably less effective" against Target X than the Hughes warhead. Tr. at 70, 73, 75, 608. However, he also conceded that there was indeed a difference between Hughes's and LM's warheads; according to the engineer, the fact that Hughes's warhead [DELETED] will result in Hughes's warhead producing a bigger blast than LM's, Tr. at 213-214, 237, 605, and a warhead with a greater blast effectiveness was likely to be more effective against Target X. Tr. at 607-608. The agency engineer further conceded that using the effectiveness data for the [DELETED] warhead would not furnish a more reliable predictor of warhead effectiveness against Target X than the effectiveness data supplied by LM for its own warhead. Tr. at 260-261.

5. The [DELETED] warhead's approximate 23 percent advantage in this regard is based on a comparison between Hughes's effectiveness numbers for the [DELETED] APW and the evaluated effectiveness of the [DELETED] at a low circular error probable (CEP). (The CEP denotes the radius around the target in which 50 percent of the missiles can be expected to land.) The JASSM program manager testified that the agency likewise had concluded during its evaluation that the effectiveness numbers for Hughes's missile would have "declined markedly" and the unit price increased had Hughes substituted the [DELETED] warhead for the [DELETED] warhead. Tr. at 915, 925-928, 979-982.

6. The [DELETED] warhead was [DELETED] inches longer, included approximately [DELETED] pounds more high explosive, and was viewed as a better penetrator than the [DELETED] warhead. Tr. at 506-507.

7. In this regard, Hughes's chief engineer testified as follows:

"Question: And does this chart show that of these [warhead] options, the lowest unit cost was for the [DELETED]?

Answer: Yes.

Question: And do you know if the unit cost of the [DELETED] also would be the lowest unit cost in comparison with these other warhead options?

Answer: Yes.

Question: If Target [X] were removed from the target set, would that change the unit cost on any of these warhead options?

Answer: These were estimates, but no. . . .

Question: I see the risk assessments for the [DELETED] warhead candidate, and the risk assessments across the row are all low. Do you see that?

Answer: Correct.

Question: Would these risk assessments be the same for the [DELETED] warhead?

Answer: Probably pretty close, yes.

Question: Would any of these risk assessments change as a result of the removal of Target [X] from the target set?

Answer: No."

Tr. at 550-551.

8. The Air Force notes that when during discussions it questioned Hughes's proposed development schedule, Hughes responded that it had "initiated work on analysis and design of the JASSM [DELETED] series warhead configurations in September of 1995," that is, prior to the addition of Target X to the JASSM target set. Hughes's chief engineer, however, testified that this reference was only "an after-the-fact term we gave to the entire development series for the [DELETED] type warheads that we were looking at, like the [DELETED]"; according to the engineer, Hughes only started looking at the [DELETED] design "when . . . the target set changed, and that was in October," and "actually arrived at the design in January." Tr. at 508-510.

9. Although Hughes's selection of a nondevelopmental warhead could have reduced its risk under the technical criterion, its evaluated higher risk in this area was based on additional considerations beyond the development risk associated with its new [DELETED] warhead and, in any case, its substitution of a nondevelopmental warhead such as the [DELETED] would have resulted in a marked decline in the effectiveness numbers for its missile.

10. Hughes also challenges the cost/technical tradeoff, primarily on the ground that the SSA failed adequately to take into account Hughes's affordability advantage. The record shows that the SSA did not consider Hughes's "roughly [DELETED] percent" lower AUPP for production lots 1-5 significant in light of the fact that the AUPP for LM and MD were not viewed as excessive and the speculative nature of the AUPP at this stage in the program (in that the program is only at the beginning of the PDRR phase and production is years away, while the offerors besides Hughes are not committed to their current production pricing and Hughes is only committed to its current pricing for the first 165 missiles of the planned 2,400 missile purchase). Nothing in the solicitation precluded the SSA from considering the significance of Hughes's AUPP advantage in light of these factors.

DOCUMENT FOR PUBLIC RELEASE A protected decision was issued on the date below and was subject to a GAO Protective Order. This version has been redacted or approved by the parties involved for public release.

GAO Contacts

Office of Public Affairs