Is denied where the record shows that the agency evaluation was reasonable. Protester's assertion that the agency failed to hold meaningful discussions regarding two areas where its proposal was assessed moderate risk is denied where the record shows that in both instances the agency clearly advised the protester of the factual underpinning of its concerns. Symetrics argues that the evaluation of proposals was unreasonable. That discussions were improper. The earlier protest was sustained because the record showed there had been a material change in the agency's requirements after the solicitation was issued and before the award date. Because the change in requirements may have significantly affected the evaluation.
Matter of: Symetrics Industries, Inc. File: B-274246.10 Date: September 17, 1998 * Redacted Decision
Symetrics Industries, Inc. protests the award of a contract to Tracor Aerospace, Inc. by the Department of the Air Force, pursuant to request for proposals (RFP) No. F33657-96-R-0001, issued for the purchase of Lots V through VII of the AN/ALE-47 Countermeasures Dispenser System (CMDS). Symetrics argues that the evaluation of proposals was unreasonable, that discussions were improper, and that the selection decision lacked a reasonable basis.
We deny the protest.
This protest challenges the reselection of Tracor for award by the Air Force following an earlier decision by our Office sustaining Symetrics's protest of the initial selection of Tracor. The earlier protest was sustained because the record showed there had been a material change in the agency's requirements after the solicitation was issued and before the award date, and because the change in requirements may have significantly affected the evaluation. Symetrics Indus., Inc., B-274246.3 et al., Aug. 20, 1997, 97-2 CPD Para. 59 at 8. In response, the Air Force amended its solicitation, requested a new round of proposals, held discussions with the three offerors still participating in the competition, and reevaluated best and final offers (BAFO).
As explained in our earlier decision, the AN/ALE-47 CMDS is an electronic warfare system used by the Army, Navy and Air Force to protect aircraft from hostile missile attacks. The system discharges chaff cartridges and decoy flares to distract ground-launched missiles aimed at aircraft. Five distinct line replaceable units (LRU) comprise the system in varying numbers and configurations depending on the aircraft involved. These LRUs are a control-display unit, a programmer, a switch assembly, a digital sequencer, and a dispenser assembly.
The RFP here, first issued in October 1996, contemplated the award of a 4-year fixed-price, indefinite-delivery/indefinite-quantity (ID/IQ) contract to the offeror whose proposal provided the best value to the government. /1/ The RFP explained that the proposal with the best value would be "the most advantageous offer, price and other factors considered . . . providing the best mix of utility, technical quality, business aspects, risks, and price for a given application." RFP Sec. M.1.0.
The RFP advised that each proposal would be evaluated in four areas, in descending order of importance: technical, schedule, cost/price--most probable life cycle cost, and management. Under these four factors, the technical factor included four subfactors of equal weight: manufacturing/quality assurance, integrated logistics support, systems engineering, and testing. In addition to the four factors, the RFP also listed four general considerations of equal weight, all of which were less important than the four evaluation factors. The general considerations were pre-award survey, executive in-plant review, plant visits, and RFP terms and conditions. RFP Sec. M.2.0.
Offerors were also advised that two of the evaluation factors--technical (including the four subfactors) and management--would be assigned a color/adjectival rating, a proposal risk assessment, and a performance risk assessment. /2/ They were also advised that the color/adjectival rating and the two risk ratings would receive equal consideration. RFP Sec. M.6.0. The remaining two evaluation factors--schedule and cost/price--were not rated but were assigned a performance risk assessment.
After the evaluation of BAFOs in the most recent round of this procurement, Tracor's proposal received two blue ratings (exceptional) and two green ratings (acceptable) under the four equally-weighted technical subfactors, a green rating under the management factor, and low risk ratings in every assessment of proposal and performance risk. However, Tracor's proposal had the highest price of any offeror, $19.8 million with a most probable life-cycle cost (MPLCC) of $78.2 million. Symetrics's proposal received all green ratings under the technical and management factors. While Symetrics received low risk ratings in all of the assessments of performance risk, it received two moderate risk ratings and three low risk ratings under the assessments of proposal risk. Specifically, Symetrics received a moderate risk rating under the technical subfactor of manufacturing/quality assurance, and under the management evaluation factor. Symetrics's proposal offered the lowest price of the three remaining competitive range offerors, $18.7 million with a MPLCC of $77.4 million. A summary of these results is shown in the table below:
Color Prop Perf Color Prop Perf Rating Risk Risk Rating Risk Risk
--Mfg./QA Blue Low Low Green Mod. Low
--Int. Logistics Support Blue Low Low Green Low Low
--Systems Eng'g Green Low Low Green Low Low
--Testing Green Low Low Green Low Low
Schedule ----- ----- Low ----- ----- Low
Price (MPLCC) $78.2M ----- Low $77.4M ----- Low
Management Green Low Low Green Mod. Low
In selecting Tracor over Symetrics, the source selection authority (SSA) noted that Tracor had a higher price and MPLCC, but concluded that the price difference was insignificant given Tracor's superior manufacturing processes and integrated logistics support capability, reflected in its blue ratings under the manufacturing/quality assurance and the integrated logistics technical subfactors. In addition, the SSA expressed concern that Symetrics might not effectively manage this program because of a recent change in Symetrics's ownership and corporate structure. Accordingly, the SSA concluded that Tracor's proposal offered the best value to the government and awarded to Tracor. /3/
Symetrics disputes each evaluation conclusion that provides a basis for distinguishing between its proposal and Tracor's. Specifically, Symetrics argues that there was no reasonable basis for awarding Tracor a higher rating under the manufacturing/quality assurance and the integrated logistics technical subfactors, and no reasonable basis for concluding that Symetrics's proposal presented a moderate level of risk under the manufacturing/quality assurance technical subfactor, and under the management factor.
In considering protests against an agency's evaluation of proposals, we will examine the record to determine whether the agency's judgment was reasonable and consistent with the stated evaluation criteria and applicable statutes and regulations. ESCO, Inc., B-225565, Apr. 29, 1987, 87-1 CPD Para. 450 at 7. Here, we have considered each of Symetrics's contentions, the evaluation materials, the proposals, and the Air Force's response to Symetrics's arguments. As a result of our review, we find no basis for concluding that the evaluation was unreasonable, or not in accordance with the stated evaluation criteria. To illustrate our conclusion, we will discuss in detail a sampling of Symetrics's arguments.
With respect to the distinction drawn between Tracor and Symetrics under the manufacturing/quality assurance subfactor, Tracor received a rating of blue (exceptional) after the evaluators identified several significant strengths in this area of its proposal. Specifically, the evaluation shows that Tracor's proposal indicates "[e]xceptional work measurement and quality assurance programs," "[a]ll known production, schedule and [quality assurance] risks have been mitigated," and "[s]killed, experienced and stable workforce in place." Under these three headings were other more detailed evaluation comments. The evaluators identified no significant weaknesses in the proposal, and no proposal risk. Addendum to Proposal Analysis Report, May 12, 1998, at 16.
Symetrics received a rating of green (acceptable) after the evaluators identified one significant strength, no significant weaknesses, and moderate proposal risk. Symetrics's significant strength was its "[s]trong understanding of work measurement," but the company was assessed a moderate proposal risk because the proposal's "labor hours appear to be low for a first time producer of a system of this complexity." Id. at 14.
The SSA made the following comments when comparing the offerors under this technical subfactor:
Tracor's rating was based, in part, upon Tracor's exceptional work measurement and quality assurance programs. Tracor's "6 Sigma" statistical process control minimizes manufacturing defects as demonstrated by the [deleted] success rate for circuit card tests currently being obtained on the AN/ALE-47 production. Tracor and [Offeror A] received a low proposal risk rating for this factor. Symetrics received a moderate proposal risk rating for this factor based upon their unrealistically low labor hours in their cost proposal for a first time producer of a system of this complexity. The hours in the cost proposal were lower than the hours detailed in their technical proposal which the evaluation team believed to be more reasonable. During the course of discussion over this difference, Symetrics revised their technical proposal to match the lower hours in the cost proposal; however, they did not offer an adequate justification for the difference between the two portions of their proposal.
Source Selection Decision, May 14, 1998, at 2-3.
Symetrics argues that it was unreasonable for the agency to give additional credit to Tracor for its Six Sigma Program, a quality assurance process, because the program is not as highly regarded as the Air Force believes; because the Air Force mistakenly concluded that Tracor's performance under the Six Sigma Program was exemplary; and because giving credit to Tracor for this expensive quality program constructively eliminates small businesses from the competition in violation of congressional direction that a full and open competition be held for this system. Symetrics also argues that the Air Force improperly overlooked favorable past performance information for Symetrics contained in performance reports from 1994, 1995, 1996, and 1997 (among other places). In addition, Symetrics argues that, under the guidelines in the Air Force's own evaluation plan, Symetrics should have received a rating of blue, not green, under this factor.
As an initial matter, magazine, newspaper, and on-line articles submitted by both the Air Force and Symetrics show that the Six Sigma Program--for which Tracor's proposal received credit--is a quality assurance process that has received considerable favorable review in the mainstream and manufacturing press. While the articles overall indicate that the Six Sigma Program can lead to significant improvements in manufacturing quality, Symetrics offers two articles that question whether the program is worth its cost, or is merely the latest in a series of industry fads related to improving quality.
We need not reach a conclusion about whether the Air Force or Symetrics is correct about the merits of the Six Sigma Program. /4/ The record amply demonstrates that the program is a mainstream effort to improve manufacturing quality and that a number of major corporations have implemented portions of the program. In our view, it was reasonable for the Air Force evaluators to recognize and value Tracor's attempt to implement this kind of quality improvement program. In addition, we do not view the evaluation's recognition of Tracor's Six Sigma Program as a constructive elimination of small businesses from consideration here. While Symetrics may be correct in its contention that the cost of implementing the program is generally beyond the reach of small businesses, the solicitation here did not require the program. Instead, the evaluators viewed the program as a benefit of Tracor's proposal, and included the program in the list of discriminators when deciding which proposal represented the best value to the government. We see nothing improper about the evaluation in this regard.
Symetrics also argues that the Air Force's award of a blue rating to Tracor under this factor was unreasonable because Tracor's achievements under its Six Sigma Program were overstated. Specifically, Symetrics argues that the Air Force was wrong when it stated that "Tracor is currently operating at an average of [deleted] in the six sigma program which puts their manufacturing facility in line with the top facilities in the world." Addendum to Proposal Analysis Report, supra, at 17. Symetrics points to one of the articles on the Six Sigma Program reprinted in the agency report and urges that under the standards set forth there Tracor's [deleted] rating puts the company "at the top end of the industry average." Symetrics's Comments on the Agency Report, July 31, 1998 at 15. Thus, in Symetrics's view the evaluation conclusion was unreasonable.
In our view, even if we accept Symetrics's characterization of Tracor's [deleted] quality rating within the Six Sigma Program, the distinction between the Air Force's comments that Tracor's facility is "in line with the top facilities in the world" and Symetrics's comments that Tracor is "at the top end of the industry average" does not compel us to conclude that the evaluation was unreasonable and should be overturned. The minor difference between these two assessments is well within the realm of the agency's discretion to make reasonable assessments of proposals. See ESCO, Inc., supra.
We also reject Symetrics's contention that the distinction drawn between its proposal and the proposal submitted by Tracor was unreasonable because past performance reports for Symetrics show very high quality ratings in its manufacture of a related system. As shown above, in the agency's assessment of past performance, both Symetrics and Tracor received ratings of low performance risk. However, the proposal ratings were not based on past performance. Instead, the RFP at section M.3.1.1 explained that the agency would evaluate the offeror's manufacturing management program and quality system approaches. The evaluators concluded that Symetrics had a strong understanding of the work measurement requirements of the solicitation, and awarded the company a rating of green (acceptable). Favorable reports about Symetrics's past performance do not establish that this rating was unreasonably low.
As a final evaluation matter, Symetrics contends that the Air Force's own evaluation plan requires that Symetrics receive a rating of blue, not green, under the manufacturing/quality assurance subfactor. Specifically, Symetrics points out that the Air Force defined the rating of blue (exceptional) as "[e]xceeds specified performance or capability in a beneficial way to the Air Force and has no significant weakness." /5/ According to Symetrics, since the definition of a blue rating uses the word "way" in the singular, and since the evaluators noted a strength under this subfactor (with no significant weakness), the agency's own evaluation scheme mandates a rating of blue (exceptional).
As a preliminary matter, we note that the standardized color ratings used by the Air Force here are set forth in the Air Force Federal Acquisition Regulation Supplement (AFFARS), Appendix BB, Sec. BB-304. These rating definitions reflect internal instructions to agency personnel and do not provide outside parties with legal rights. Rockwell Int'l Corp., B-261953.2, B-261953.6, Nov. 22, 1995, 96-1 CPD Para. 34 at 13 n.16. Instead, our review focuses not on the Air Force's internal definitions but on whether the evaluations at issue are consistent with the RFP's evaluation scheme. Loral Aeronutronic, B-259857.2, B-259858.2, July 5, 1995, 95-2 CPD Para. 213 at 9-10; Sabreliner Corp., B-242023, B-242023.2, Mar. 25, 1991, 91-1 CPD Para. 326 at 12 n.16.
Here, section M.3.1.1 of the RFP advised offerors that the agency would consider numerous elements of the proposal's manufacturing management program and its quality system. We have reviewed Tracor's evaluation under this subfactor and the record clearly shows that several strengths are identified. In addition, the presence of these strengths reasonably supports the distinction between Tracor's and Symetrics's proposals. Under these circumstances, we have no basis to conclude that the evaluation here was improper. Moreover, there is no requirement that an agency view as exceptional a proposal for which the evaluators have identified a strength and no significant weaknesses, nor is this result mandated by the evaluation guidelines. /6/ The difference between the assessment that a proposal is acceptable rather than exceptional is a matter of judgment that we will not disturb without a showing that the agency's application of the ratings was unreasonable or unequal in some way.
In a similar vein, Symetrics argues that the Air Force failed to hold meaningful discussions because the agency violated its own internal regulatory guidance about discussions. In particular, Symetrics argues it was not given a clear indication during discussions that the agency was considering assessing a moderate risk against the company's proposal under the manufacturing/quality assurance subfactor, and under the management factor. /7/
As stated above, while we will not consider whether the Air Force violated its internal guidelines in this area, we will address the issue of whether discussions were meaningful. Our review of the adequacy of discussions seeks to ensure that agencies point out weaknesses that, unless corrected, would prevent an offeror from having a reasonable chance for award. Ann Riley & Assocs., Ltd., B-271741.2, Aug. 7, 1996, 97-1 CPD Para. 120 at 9, recon. denied, B-271741.3, Mar. 10, 1997, 97-1 CPD Para. 122.
Here, there is no dispute that while the Air Force did not indicate the specific action it was considering based on its concern--i.e., assigning a moderate risk rating to the proposal in two areas--the factual basis for its ultimate conclusion was clearly raised with the protester during discussions. For example, in the area of labor hours, the Air Force asked Symetrics why the labor hours in its cost volume did not match those in its technical proposal, and gave the company an opportunity to explain the discrepancy. Given that the factual issues that formed the underpinning for the evaluation concerns were clearly raised with Symetrics, we conclude that there was no violation of the more general requirement to point out all weaknesses that prevent an offeror from having a reasonable chance for award. Robbins-Gioia, Inc., B-274318 et al., Dec. 4, 1996, 96-2 CPD Para. 222 at 16-17.
Finally, Symetrics argues that the source selection statement lacked a reasonable basis because the evaluation conclusions upon which it was based were in error, because portions of the statement were inconsistent with the evaluation conclusions, and because the magnitude of the cost/technical tradeoff was obscured by the comparison of MPLCC, rather than each offeror's proposed price.
With respect to Symetrics's challenge that the source selection statement was based on erroneous evaluation conclusions, our review of the record, including the issues discussed above, leads us to conclude that the evaluation was reasonable, and thus provided a reasonable basis for the SSA's decision.
With respect to Symetrics's contention that portions of the source selection statement were inconsistent with the evaluation conclusions, we have reviewed each of Symetrics's arguments and we conclude that there is no basis to find that the statement was unreasonable. For example, the agency's evaluators concluded that Symetrics's proposal posed a moderate risk under the manufacturing/quality assurance subfactor because the proposal's "labor hours appear to be low for a first time producer of a system of this complexity." Addendum to Proposal Analysis Report, supra, at 14. In Symetrics's view, there is an inconsistency between the conclusion above and the source selection statement, which indicates that Symetrics received the moderate risk rating because of the "unrealistically low labor hours in [its] cost proposal for a first time producer of a system of this complexity." Source Selection Statement, supra.
We disagree. In our view, there is no meaningful difference between the observation made by the evaluators ("hours appear to be low") and the statement by the SSA ("unrealistically low labor hours"), and nothing about the difference in these two statements leads us to conclude that the SSA's conclusions are unsupported by the record. Contrary to Symetrics's contentions, the SSA's statement does not mean there was an improper or unreasonable cost realism review here. Instead, we interpret the SSA's choice of words as within the normal range of variation in language. There is no doubt that the evaluators and the SSA are describing the same phenomenon--i.e., that Symetrics's proposed labor hours in its technical proposal raised concerns about the company's ability to produce the system as claimed, especially after the proposed hours were unilaterally lowered in response to a discussion question that directed Symetrics to discrepancies in the number of labor hours identified in the technical and cost portions of its proposal. Id. at 3.
Finally, we turn to Symetrics's claim that the magnitude of the cost/technical tradeoff was obscured by the SSA's comparison of overall life-cycle costs, or MPLCC, rather than each offeror's proposed price.
Section M.4 of the RFP sets forth the process by which the agency intended to calculate each offeror's MPLCC. In essence, the record here shows that the calculation of the MPLCC involved adding $58.4 million to Tracor's and $58.7 million to Symetrics's price. In selecting Tracor, the SSA compares the MPLCC developed for Symetrics and Tracor and concludes that "while Tracor's total MPLCC is 1.4 [percent] higher than [Symetrics's], discounting risk, I consider this difference to be insignificant." Source Selection Statement, supra, at 5.
Our review of the record shows that there is, in fact, a 5.4-percent difference between the proposal price offered by Symetrics ($18.716 million) and the price offered by Tracor ($19.771), and that the addition of virtually identical total life-cycle costs tends to obscure, rather than illuminate, the true cost/technical tradeoff performed here. Although Symetrics raises this issue in its protest, the Air Force's approach to evaluating the proposed prices has been clear since the beginning of this procurement. Not only was this matter apparent from the face of the solicitation, but it was apparent when we reviewed the earlier award decision in Symetrics Indus., Inc., supra. Accordingly, this element of Symetrics's challenge to the cost/technical tradeoff is untimely. 4 C.F.R. Sec. 21.2(a)(1) (1998).
The protest is denied.
Comptroller General of the United States
1. The appropriate method for procuring these items--after procuring them since 1988 on a sole-source basis from Tracor, the original equipment manufacturer--has been in dispute for several years. A recitation of the earlier controversies surrounding this procurement is found in our decision in Datacom, Inc.--Protests and Request for Costs, B-274175 et al., Nov. 25, 1996, 96-2 CPD Para. 199 at 2-4. In addition, after our decision sustaining Symetrics's challenge to the earlier selection of Tracor for award, Symetrics and another offeror, Varo, LLC, challenged the revised solicitation. After mediation of the dispute by our Office, the Air Force agreed to amend the solicitation. In response, Symetrics withdrew its protest, and Varo's protest was dismissed as academic. Symetrics Indus., Inc., B-274246.6, Nov. 19, 1997; Varo LLC, B-274246.8, Nov. 19, 1997.
2. The color/adjectival ratings used in the evaluation were blue/exceptional, green/acceptable, yellow/marginal, and red/unacceptable. Proposal risk was rated high, moderate or low, and was defined as the risks identified in an offeror's proposed approach. Performance risk was rated high, moderate, low, or not applicable, and was defined as an assessment of the offeror's present and past work record to gauge confidence in the offeror's ability to successfully perform as proposed.
3. The SSA also concluded that the third offeror, whose MPLCC was higher than Tracor's, offered the government no additional benefit commensurate with its higher long-term cost.
4. We note, however, that one of the two articles cited by Symetrics appears under the heading "Unconventional Wisdom," which suggests that the Air Force's view of the benefits of Tracor's Six Sigma Program is squarely within the mainstream of conventional thought on this issue.
5. Air Force Federal Acquisition Regulation Supplement (AFFARS), Appendix BB Sec. BB-304. In contrast, a green (acceptable) rating is defined as "[m]eets evaluation standards and any weaknesses are readily corrected." Id.
6. In fact, even if we were to consider Symetrics's challenge to the application of the color rating scheme, Symetrics's argument is based on an unreasonably narrow reading of the definitions in the scheme.
7. Again, Symetrics's argument is based on a provision in Appendix BB of the AFFARS which requires that the agency prepare significant clarification requests (SCR) whenever a performance risk assessment of moderate or high is made in evaluating an offeror. AFFARS Sec. BB-307(a)(1). In the event that the agency establishes a competitive range, these SCRs are to be provided to the offeror during discussions, on the same basis as deficiency reports. AFFARS Sec. BB-307(a), (b).
* DOCUMENT FOR PUBLIC RELEASE
The decision issued on the date below was subject to a GAO Protective Order. This redacted version has been approved for public release.