Skip to main content

Matter of: Allied-Signal Aerospace Company, Bendix Communications Division File: B-249214.4 Date: January 29, 1993

B-249214.4 Jan 29, 1993
Jump To:
Skip to Highlights

Highlights

General Accounting Office will look to the entire record. To determine whether evaluation and selection are supportable. That review is not limited to the question of whether the evaluation and selection decision were properly documented and supported at the time they were made. Agency was not obligated to analyze that information in any greater detail than was necessary to assure the realism of cost proposals. PROCUREMENT Competitive Negotiation Offers Evaluation Technical equality Cost realism Award of cost-plus-fixed-fee contracts to offerors with significantly lower costs than the protester where the difference between the technical proposals is considered small. Is appropriate under request for proposals which indicated that.

View Decision

Matter of: Allied-Signal Aerospace Company, Bendix Communications Division File: B-249214.4 Date: January 29, 1993

PROCUREMENT Competitive Negotiation Offers Cost evaluation Records GAO review In reviewing an agency's cost evaluation and source selection, General Accounting Office will look to the entire record, including statements and arguments made in response to a protest, to determine whether evaluation and selection are supportable; that review is not limited to the question of whether the evaluation and selection decision were properly documented and supported at the time they were made. PROCUREMENT Competitive Negotiation Offers Evaluation Cost realism Analysis Although solicitation instructed offerors to submit with proposals detailed cost information, agency was not obligated to analyze that information in any greater detail than was necessary to assure the realism of cost proposals. PROCUREMENT Competitive Negotiation Offers Evaluation Technical equality Cost realism Award of cost-plus-fixed-fee contracts to offerors with significantly lower costs than the protester where the difference between the technical proposals is considered small, is appropriate under request for proposals which indicated that, although technical merit was the most important consideration, cost could become relatively more important in the selection decision as the difference in technical scores decreased. Although the protester challenges the cost realism analyses based on a lack of a detailed comparison of the cost estimates, such a detailed analyses is not necessary where there is a significant disparity among the overall cost estimates. Rather, the cost analyses were reasonable since the agency assured itself that each firm proposed a technical approach that meets all the RFP requirements and that each firm fairly and reasonably reflected the costs represented by that approach in its cost estimate.

Attorneys

DECISION Allied-Signal Aerospace Company, Bendix Communications Division protests the award of contracts to Raytheon Company and Wilcox Electric, Inc. under request for proposals (RFP) No. DTFA01-90-R-07409, issued by the Federal Aviation Administration (FAA) for the design, development, and test of six Category II/III (CAT II/III) Microwave Landing Systems (MLS). Allied-Signal argues that the FAA failed to evaluate the proposals in accordance with the solicitation and failed to properly analyze the realism of the cost proposals.

We deny the protest.

BACKGROUND

The solicitation contemplated the award of two or more cost-plus-fixed- fee contracts for the design and development of the MLS, which the FAA considers to be a possible replacement for current instrument landing systems. The MLS is expected to provide more precise guidance to aircraft executing landings, greater flexibility in operations, freedom from radio frequency interference and greater stability, integrity and continuity of service. The solicitation criteria explained that only the contractors selected for award of these design and development contracts would be eligible for follow-on production contracts.

The MLS includes a device which measures the distance of a landing aircraft from the runway: the Precision Distance Measuring Equipment, or DME/P, and another device which measures the angle of the aircraft with respect to the extended centerline of the runway: the Angle/Data Equipment, which consists of an Azimuth antenna and an Elevation antenna. The solicitation statement of work (SOW) detailed extensive requirements for program management, engineering management, systems engineering, software development, production, test and evaluation, turnkey installation and technical support for the MLS.

The SOW required that the contractors "conduct and manage a fully integrated systems engineering effort in accordance with military standard MIL-STD-499," which was developed by the Air Force to assist the government and contractors in defining the system engineering efforts required to support defense programs. The SOW also included extensive guidance on the development of MLS software which was required to be in accordance with Department of Defense (DOD) standard DOD-STD-2167A, the purpose of which is to establish standardized requirements for software development.

The solicitation included 31 contract line items (CLINS). CLINS 1 through 11 called for design and development of a Category II/III MLS, and included in subCLIN 1a all nonrecurring design costs for the system including systems engineering and software development costs. CLINS 12 through 31 covered options for training, additional systems, related hardware, data, software maintenance, testing and logistics support and other services.

The awards were to be based on the results of an evaluation of technical, cost and management proposals, in descending order of importance. The solicitation also stated that as the difference between technical scores decreased, cost could become relatively more important in the award selection.

Technical proposals were to be evaluated in accordance with a series of technical evaluation factors set forth in the RFP. In addition, the solicitation stated that offerors who submitted acceptable proposals would have their software engineering capabilities evaluated using the Software Engineering Capability Assessment (SECA) methodology. The SECA process was to include evaluation of information submitted by the offerors as well as on-site reviews. The SECA evaluation was to establish a software engineering process maturity level rating for a contractor in the range of 1 to 5; a higher rating indicates a lower risk for software development. The solicitation stated that the SECA results would be combined with the scores under one of the technical evaluation factors listed in the RFP: System Software and Firmware Technical Approach."

The solicitation evaluation factors stated that "[c]ost proposals will be evaluated to determine the consistency of approach in accordance with Section L-18 and to determine the reasonableness and realism of all proposed costs/fees." Section L-18 entitled, "SUPPLEMENTAL INSTRUCTIONS TO OFFERORS FOR PREPARATION OF COST OR PRICING DATA," stated that "[c]ost proposals must be submitted in accordance with the Work Breakdown Structure (WBS) herein included." The WBS contained seven levels, each level representing a further breakdown in contract effort and the solicitation instructed offerors to estimate costs at the lowest levels of the WBS and "roll up" costs by summarizing them at the higher levels.

In addition, section L-18 told offerors to provide detailed cost estimates for each WBS level under each CLIN on "Format C" documents which were included in the solicitation. The Format C documents were to include summarized estimated costs for labor, and materials as well as other costs. Further, section L-18 stated that offerors were "to submit any additional data, supporting schedules or substantiation that are reasonably required for the conduct of an appropriate review and analysis." The RFP also stated in section L-18 that "it is essential that there be a clear understanding of . . . matrix and accompanying rationale that traces the offeror's proposed cost from the work breakdown structure to the contract line item(s)."

Five firms submitted proposals. The agency evaluated them and informed the offerors of proposal ambiguities and allowed the offerors to submit clarifications. The technical proposals were scored and offerors then were given an opportunity to revise their proposals in response to deficiencies and weaknesses identified by the agency. The agency again scored the proposals and created a competitive range which included all five firms. Following discussions, the agency requested, and all five offerors submitted best and final offers (BAFO).

All five BAFOs were considered technically acceptable. Out of a possible score of 10, the highest final technical scores assigned were [deleted] for Allied-Signal, [deleted] for Wilcox, and [deleted] for Raytheon. The final technical scores included consideration of the SECA process maturity level which had been assigned to the offerors. The SECA ratings were: [deleted] for Raytheon, [deleted] for Wilcox and [deleted] for Allied- Signal. As explained above, a higher rating indicates a lower risk for software development.

The FAA reports that it evaluated the cost proposals to ensure that proposed costs were an accurate reflection of the technical approach of each offeror.

The BAFO costs of each of the offerors were as follows:

Offeror

Basic Items (CLINS 1-11)

Options (CLINS 12-31)

Total cost proposed by offeror

Fee (included in total cost proposed by offeror)

Govt. estimated items

Total evaluated cost

The agency's cost evaluation team report and the report of the source evaluation board (SEB) each noted the large disparity in proposed costs between the two awardees and the three other offerors. The cost evaluation team report states:

"The question of cost realism becomes immediately apparent: if the prices proposed by Raytheon and Wilcox are `realistic' then the others would appear to be drastically overstated. Conversely, if the offers of [Allied- Signal], Norden and AT&T are considered to be realistic, the Raytheon and Wilcox offers could be so low as to appear to be buy-ins."

The report continues, however:

"The cost team proposes that all of the offers are realistic, dependent upon the Offeror's technical approach and prior experience with microwave landing systems. [Allied-Signal], however, although having prior MLS experience, was found by the Q&Q reviewers to have very large development costs in the Basic phase. This accounts for the [deleted] difference between Raytheon and [Allied-Signal] in their estimates for the Basic CLINs 1-11."

The SEB report also notes the disparity in proposed cost between Raytheon and Wilcox and the other offerors and accepts the cost evaluation team's conclusion that each of the offerors' proposed costs were fair and reasonable and accurate reflections of their particular technical approach. The SEB report states that although Allied-Signal has MLS experience and was anticipated to have costs consistent with that experience, its proposed costs were in the range of offers with little prior MLS experience, with high start-up costs. In resolving this anomaly, the SEB report concludes that "[a]lthough [Allied-Signal] has prior MLS experience, their software approach had to be newly-developed to meet the MLS CAT II/III requirements, raising their costs substantially." The SEB report also states that Allied-Signal's need to establish an effective software development management capability "is reflected in the SECA [deleted] achieved by [Allied-Signal] and contributes significantly to its submission of the highest nonrecurring engineering costs of all the offerors."

Also, the SEB report states that Allied-Signal proposed a high cost for developing hardware "apparently due to the fact that a large percentage of previously designed [Allied-Signal] equipment cannot support the MLS CAT II/III processing and software requirements." The report also notes that Allied-Signal's proposal included a license fee of [deleted] and a recurring charge for the DME/P portion of the system.

In selecting Wilcox and Raytheon for award, the source selection authority relied upon the SEB report and the cost evaluation team report. The source selection authority noted that the difference in technical scores among all the offerors was small and thus he concluded that cost should become important in the selection decision.

PROTEST OVERVIEW

Allied-Signal primarily argues that the FAA conducted a faulty cost realism analysis and failed to question or adjust the unrealistically low proposed costs of Raytheon and Wilcox. According to Allied-Signal, the solicitation required that the cost evaluation be based on the detailed WBS information and, in spite of this mandate, there is no evidence that the agency evaluated the cost proposals at the lower levels of the WBS or made a reasonable attempt to determine the cost realism of the proposals. Consequently, the protester argues that the FAA did not meaningfully address the [deleted] differences in the cost proposals between the awardees and the other offerors and thus could not have made a reasonable selection decision.

ANALYSIS

When an agency evaluates proposals for the award of a cost reimbursement contract, an offeror's proposed estimated costs of contract performance are not controlling, since the offeror's estimates may not provide valid indications of the actual costs which the government is, within certain limits, required to pay. See Federal Acquisition Regulation Sec. 15.605(d). Consequently, the agency must perform a cost realism analysis to determine the extent to which an offeror's proposed costs represent what the contract should cost assuming reasonable economy and efficiency. Arthur D. Little, Inc., B-243450, July 31, 1991, 91-2 CPD 106. Our review of an agency's exercise of judgment in this area focuses on whether the agency's cost realism analysis is reasonably based. Grey Advertising, Inc., 55 Comp.Gen. 1111 (1976), 76-1 CPD Para. 325. Lack of Adequate Documentation of the Evaluation

Allied-Signal first argues that the FAA failed to adequately document its cost evaluation. It argues that there is no evidence in the record of an adequate analysis of the actual costs of performance proposed by Raytheon and Wilcox. According to the protester, we should sustain the protest simply on this basis.

The FAA did not create a detailed contemporaneous record of all the aspects of the cost evaluation process. For example, the FAA's Q&Q team, which was responsible for assuring the realism of the cost proposals, did not fully document the process of verifying whether each cost proposal included sufficient costs for each element of the offeror's proposed technical approach. The agency explains that when such cost realism issues arose, if those matters could be resolved by checking the cost proposal against the technical proposal, no record was kept.

While we think that the FAA should have made more of an effort to document the totality of its cost analysis, we do not agree with Allied- Signal that the selection decisions should automatically be voided simply because of this. See JSA Healthcare Corp., B-242313; B-242313.2, Apr. 19, 1991, 91-1 CPD Para. 388; Burnside-Ott Training Center, Inc.; Reflectone Training Sys. Inc., B-233113; B-233113.2, Feb. 15, 1989, 89-1 CPD Para. 158. In reviewing cost evaluations and selection decisions, we look to the entire record, including statements and arguments made in response to the protest, so that we may determine whether they are supportable; we do not limit our review to the question of whether they were properly documented at the time they were made. Id.

In this case, we conducted a hearing concerning the cost evaluation and other issues raised in the protest. At the hearing, various FAA officials who were involved in the cost and technical evaluations testified and were subject to cross-examination. Our decision concerning the cost evaluation includes consideration of testimony from the hearing, the written evaluation record and the FAA's explanations of the evaluation in its protest submissions. Based on our examination of all this evidence, we conclude, for the reasons stated below, that the cost evaluation-which included a determination that each offeror's proposed costs were sufficient for its proposed technical approach-was reasonably based.

The Cost Evaluation

In evaluating the cost proposals, the FAA used an independent government cost estimate (IGCE) of approximately [deleted] million as a yardstick to compare CLIN prices and to create questions for cost discussions with the offerors. The analysis also included an audit of each offeror's initial cost proposal by the Defense Contract Audit Agency (DCAA), which examined the rationale used by each offeror in estimating direct labor hours, direct material requirements, other direct cost quantities, labor rates and overhead. DCAA also audited or provided audit support for major proposed subcontractors. Although the audits were qualified since they were performed before the technical evaluations were completed, the agency explains that the audit reports, in addition to the IGCE, provided a base line for comparison with the proposed costs.

The primary work of determining cost realism was performed by the FAA's Q&Q team, which was composed of individuals who had been involved in the technical evaluation and were knowledgeable about each offeror's technical approach. The cost evaluation team report states that the Q&Q team "conducted the review of the cost proposals to ensure that the costs were an accurate reflection of the technical approach for each offeror.

The team also examined the proposed hours and materials for reasonableness, and identified areas of concern regarding possible contractor misunderstanding of the requirements."

The Q&Q team reviewed the WBS submissions of the offerors with the assistance of International Management, Development and Training, Inc. (IMDT), the FAA's cost evaluation support contractor. The FAA reports that IMDT formatted the cost information submitted in the proposals for use by the Q&Q team in the cost realism analysis and verified each offeror's cost proposal for consistency with the provisions of RFP section L-18, including the WBS instructions, and also for completeness and mathematical accuracy. The cost evaluation team report states that the Q&Q team used the WBS submissions and other cost information, including that prepared by IMDT, to examine the relative level of effort and cost allocated to such areas as program management in relation to the total MLS effort in each proposal. The team also identified how each offeror assigned costs within the various tasks required by the statement of work. In addition, agency officials testified that the Q&Q team used the WBS submissions to verify the realism of each offeror's costs against its own technical approach. Hearing Transcript (Tr.) 74-79, 107-125. As the chairman of the Q&Q team explained at the hearing, "[O]ur purpose was to go back over to the cost proposal and say that the activities that have already been blessed and accepted by the technical evaluation team are being properly costed out, that the costs involved--or the cost of the cost proposal reflect the effort that he's describing in his technical proposal." Tr. 122-123.

Allied-Signal maintains that this cost evaluation effort was inconsistent with the solicitation and inadequate. First, the protester argues that the agency could not have considered the detailed WBS submissions required by the RFP instructions since Raytheon and Wilcox did not submit complete WBS information. The protester points out that the RFP stated that cost proposals would be evaluated to determine consistency with section L-18, including the WBS instructions, and that the FAA's BAFO Instruction Letter dated January 27, 1992, reiterated the requirement for cost information at the WBS level for BAFOs. Allied-Signal argues that in spite of these requirements, Raytheon did not submit lower level WBS information during discussions or with its BAFO even though it made major technical changes and increased its level of effort. Thus, according to the protester, any cost evaluation of Raytheon's BAFO covered only the information in its original cost proposal. In addition, Allied-Signal argues that Wilcox took exception in its BAFO to many WBS items and that the BAFOs of both Wilcox and Raytheon did not contain the matrices or cross references between CLINS and WBS items that were necessary to allow the FAA to trace the WBS item costs to CLIN prices, as required by section L-18.

Further, Allied-Signal states that there was little or no significant analysis of the cost proposals below the CLIN level. According to the protester, although there should be considerable documentation relating to the analysis of each offeror's WBS information, in fact, no documentation created by the cost evaluation team or the Q&Q team exists which indicates an evaluation of cost proposals at the WBS level.

We do not agree with the protester that the FAA conducted a cost evaluation that was inconsistent with the RFP evaluation criteria, that showed disregard for the WBS information submitted with the proposals or was otherwise inadequate.

The evaluation scheme in section M of the RFP did not specify to what extent the WBS information must be used in the cost analysis; rather, it simply required that cost proposals "be evaluated to determine consistency of approach in accordance with section L-18." Section L-18 stated that detailed WBS information should be submitted and the FAA reports that both Wilcox and Raytheon provided such information in their initial proposals to the agency's satisfaction. Further, while Allied-Signal argues that the Raytheon and Wilcox BAFOs did not include the updated WBS information called for by the FAA's BAFO request, we think that the adequacy of the cost information submitted should be assessed in the context of the cost evaluation actually conducted, not on whether it conformed to a format or whether each offeror's cost proposal conformed precisely with the RFP instructions. In our view, the solicitation evaluation criteria did not mandate that the FAA use any particular method in conducting its cost analysis and, while there is little or no documentation of the analysis of the cost proposals below the CLIN level, we do not think that by instructing offerors to submit detailed WBS cost information, the agency was obligated to analyze that information in any greater detail than was necessary to assure the realism of the cost proposals. An agency is not required to conduct an in-depth analysis or to verify each item in conducting a cost realism analysis. PRC/VSE Assocs. Joint Venture, B-240160 et al., Oct. 30, 1990, 90-2 CPD Para. 348. The purpose of a cost realism analysis is to determine what, in the government's view, it would realistically cost the offeror to perform given the offeror's own technical approach. SRS Technologies, 69 Comp.Gen. 459 (1990), 90-1 CPD Para. 484. We think that the record shows that the FAA did that here.

In the FAA's contemporaneous evaluation reports, in the agency's protest submissions and during the hearing, FAA officials have repeatedly represented that the Q&Q team, with the assistance of the agency's support contractor, used each offeror's WBS information and technical proposal to identify how the offeror assigned costs within the various tasks required by the statement of work and to verify the realism of the offeror's costs against its own technical approach. We have no basis to question these representations.

Further, the cost evaluation team report includes a reasonably detailed analysis of the agency's determination that both Raytheon and Wilcox could perform their contracts at essentially the costs which they proposed. With respect to the Raytheon proposal, although the cost evaluation team report concludes that Raytheon understated the cost of software support by approximately [deleted] the report also states that the proposal was overstated by approximately [deleted] in other areas. Most important, the report concludes that the proposed costs were "fair, reasonable and realistic . . . based upon the technical approach and the offeror's prior experience with microwave landing systems."

With respect to Wilcox, the cost evaluation team report states that the firm's BAFO included [deleted] million in unsupported "no cost" or "cost not applicable" entries. Other than these items, however, the cost evaluation team report states that "the estimated costs submitted by Wilcox appear to reflect the effort described in the offeror's technical proposal. The estimated total cost is considered to be fair, reasonable, and realistic in light of the offeror's approach and experience with MLS." Finally, although the cost report states that the proposals of both awardees were well below the government's estimate, it notes that the IGCE was based on an offeror with little MLS experience and both Wilcox and Raytheon have substantial experience.

Throughout this protest, Allied-Signal has had access to the cost and technical proposals submitted by Wilcox and Raytheon; yet other than to point out obvious differences in the amounts proposed by the awardees for various aspects of the effort as compared to the much higher amounts proposed by the protester, Allied-Signal has not pointed out where either of those firms failed to propose any specific effort or materials required by the statement of work and has not demonstrated that either of the awardees failed to include costs for any required effort. The protester also has not shown that the labor hours or labor rates proposed by the awardees for any aspect of their proposed technical approaches were understated.

In our view, the record here shows that the cost analysis conducted of the individual offerors' cost estimates was reasonable.

The Cost Disparity

Allied-Signal argues that whatever the merits of the method of analysis used by the FAA in considering the cost proposals of the individual offerors, the agency has not explained the large difference in the costs of the two awardees as compared to the other offerors and the IGCE. The protester argues that the FAA's failure to explain the wide cost disparity demonstrates that the agency's analysis did not establish a reasonable basis for the selection.

As pointed out earlier, the record shows that the FAA found that each of the offerors could perform the contract essentially at the cost which it proposed and that based on the agency's evaluation of the technical proposals, which the protester has not disputed, the agency concluded that there were no significant differences in technical merit between the proposals of Allied-Signal, Wilcox and Raytheon. The award selection was then based upon the lower proposed costs of Raytheon and Wilcox. The conclusion that each of the offerors could perform at its proposed cost was based primarily upon the detailed Q&Q review of the cost elements of each proposal in the context of the particular technical approach proposed. The FAA evaluators at that stage of the process did not compare one offeror's cost elements (WBS-level cost information) with the proposed costs for the same task by another firm.

The comparison analysis was performed at the CLIN and overall cost level. In this analysis, the evaluators first considered the differences between the estimates for the performance of the entire MLS project submitted by Raytheon and Wilcox and the IGCE and concluded that since the IGCE was based on a hypothetical inexperienced firm and set forth "worst case" or maximum cost solutions, the proposed costs of both firms, though significantly lower than the IGCE, were realistic since each offeror's proposal evidenced extensive MLS experience. The evaluators acknowledged that Allied-Signal also possessed MLS experience; however, that firm's overall cost estimate was more in line with the IGCE and the other firms in the competition which had limited MLS experience. After noting that Allied-Signal had the highest "start-up" costs (CLIN 1a) of any of the five offerors, the evaluators expressed the view that Allied-Signal was required to propose the greatest effort, and associated costs, in order to improve its ability to develop software under the contract and that this contributed "substantially" to its higher costs. The SEB report states that Allied-Signal's need to "establish an effective software development management capability . . . is reflected in the SECA [deleted] achieved by [Allied-Signal] and contributes significantly to its submission of the highest nonrecurring engineering costs of all offerors." The FAA reports that Allied-Signal's SECA rating showed that it had a relatively low level of maturity in its software development processes compared to Raytheon, with a level [deleted] SECA rating and Wilcox, with a level [deleted] SECA rating.

The evaluators also noted in the comparison analysis that the protester's need to redevelop a large percentage of its hardware to meet the MLS processing and software requirements may also have driven up costs.

Allied-Signal disputes the FAA's rationale for the cost disparity arguing that the reasons suggested by the FAA during the protest process as possible causes for the cost difference between it and the awardees, are either unsupported or insignificant in view of the vast disparity between its overall cost estimate and those of Raytheon and Wilcox. According to Allied-Signal, the problem is that the cost estimates of the awardees are unrealistically low, not that its estimate is high.

It is true that the explanations offered by the FAA for the cost difference between Allied-Signal and the awardees are no more than informed speculation based upon the offerors' CLIN cost estimates and the agency's evaluation of the technical proposals. The FAA has not argued that it conducted a comparative analysis of the offerors' cost proposals at the cost element (WBS) level; it states that its work at that level was confined to a verification of the proposed cost elements against the offeror's technical approach to ensure that each portion of the technical proposal had a corresponding and reasonable counterpart in the cost proposal.

In order for a cost analysis to be reasonably based, an agency need not necessarily conduct a detailed cross-proposal comparison of each cost element where, as here, there is a significant disparity among the overall cost estimates. For a cost analysis to be reasonable under these circumstances, the agency must, as the FAA did here, assure itself that each firm has proposed a technical approach that meets all of the RFP requirements and that each of the firms has fairly and reasonably reflected the costs represented by that approach in its cost estimate.

Once this has been accomplished, it is sufficient for the agency to recognize the disparity in the overall cost estimates and, if it is confident in view of the cost and technical evaluations that the cost differentials are due to the offerors' divergent technical approaches, capabilities, experience, efficiencies and skills, it need not do more.

Under the circumstances here, where the FAA decided to make award to the low cost offerors rather than an essentially technically equal offeror with much higher costs, the FAA was not obligated to conduct an investigation of the voluminous cost proposals involving thousands of entries in order to ferret out a dollar-for-dollar explanation of all of the hundreds of cost differences between Allied-Signal and the awardees.

The protest is denied.

GAO Contacts

Office of Public Affairs