BAE Systems Information and Electronic Systems Integration, Inc., of Hudson, New Hampshire, protests the award of a contract to Raytheon Company, of Waltham, Massachusetts, by the Department of the Navy, Naval Air Systems Command, under request for proposals (RFP) No. N00019-12-R-0035, for the technology development phase of the Navy's next generation jammer (NGJ) program.
DOCUMENT FOR PUBLIC RELEASE
The decision issued on the date below was subject to a GAO Protective Order. This redacted version has been approved for public release.
Matter of: BAE Systems Information and Electronic Systems Integration Inc
File: B-408565; B-408565.2; B-408565.3
Date: November 13, 2013
Thomas P. Humphrey, Esq., David Z. Bodenheimer, Esq., John E. McCarthy, Jr., Esq., Adelicia Cliffe, Esq., Stephan D. Rice, Esq., Olivia L. Lynch, Esq., Gordon Griffin, Esq., and Robert Sneckenberg, Esq., Crowell & Moring LLP, and Catherine K. Ronis, BAE Systems, Inc., for the protester.
Mark D. Colley, Esq., Kara L. Daniels, Esq., Emma V. Broomfield, Esq., Amy L. Likoff, Esq., C. Scott Morrow, Esq., and Lauren J. Schlanger, Esq., Arnold & Porter LLP, and Rebecca B. Ransom, Esq., Raytheon Company, for the intervenor.
Thomas Florip, Esq., Kim Foltz, Esq., and Gina Gascoigne, Esq., Department of the Navy, Naval Air Systems Command, for the agency.
Eric M. Ransom, Esq., K. Nicole Willems, Esq., and Edward Goldstein, Esq., Office of the General Counsel, GAO, participated in the preparation of the decision.
1. Protest is sustained where the agency failed to reasonably evaluate technical risk in accordance with the terms of the solicitation.
2. Protest challenging awardees technical evaluation is sustained where the documentary record is insufficient to permit meaningful review of the agencys basis for eliminating multiple risks and weaknesses from the evaluation.
3. Protest challenging awardees corporate experience evaluation is sustained where the agency acknowledges that it improperly credited the awardee with outdated experience.
4. Although we conclude that the agency improperly distributed post-final proposal submissions from the awardee to its technical evaluators, where the submissions were not made for the purpose of revising the awardees proposal, and were not material to the agencys evaluation of the awardees proposal, we conclude that the protester was not prejudiced by the agencys error.
BAE Systems Information and Electronic Systems Integration, Inc., of Hudson, New Hampshire, protests the award of a contract to Raytheon Company, of Waltham, Massachusetts, by the Department of the Navy, Naval Air Systems Command, under request for proposals (RFP) No. N00019-12-R-0035, for the technology development phase of the Navys next generation jammer (NGJ) program.
We sustain the protest.
The NGJ program is for the development of an airborne electronic attack platform to replace the Navys existing AN/ALQ-99 tactical jamming systems pods on the EA-18 aircraft. Electronic attack is the use of electromagnetic, directed energy, or antiradiation weapons to attack with the intent of degrading, neutralizing, or destroying enemy combat capability.
The Navy is using an evolutionary acquisition approach for the NGJ program. According to this approach, the Navy will pursue its requirements in three increments. Increment 1 relates to mid-band frequency jamming capabilities, while increments 2 and 3 relate to low-band and high-band frequency jamming capabilities, respectively. The increment 1 acquisition approach involves four phases: technology maturation, technology development, engineering and manufacturing development, and production and development. The Navy began the program by awarding technology maturation contracts to four firms, including BAE and Raytheon, in July 2010. The technology maturation contracts were later extended and ran for a total of 33 months. Following the technology maturation contracts, the Navy anticipated competitively awarding a single cost-plus incentive-fee contract for the technology development effort, and later making a sole source award to the same contractor for the engineering and manufacturing development effort. A series of contracts for the production and development phase would then follow.
As stated above, this protest concerns the Navys competitive acquisition for the technology development effort. The Navy issued an RFP for the technology development contract on July 10, 2012. As expected, the RFP anticipated a single award. The RFP provided that the award would be made on a best-value basis considering four evaluation factors: technical, past performance, corporate experience, and cost. The RFP advised that the technical factor was the most important factor, and that past performance and corporate experience were of equal importance with each being more important than cost. Additionally, the three non-cost evaluation factors, when combined, were significantly more important than cost.
The RFP provided for two subfactors within the technical factor: design approach, and program and schedules. The RFP stated that between the subfactors, design approach was more important than program and schedules. Concerning the technical factor evaluation, the RFP explained that both a technical rating and a technical risk rating would be assigned to each proposal. According to the RFP the technical rating would represent an evaluation of the offerors technical solution for meeting the Governments requirement, and would encompass an assessment of compliance with the solicitation requirements and merit which considers the benefits and detriments related to program performance and operations. Agency Report (AR), Tab A.3, RFP, at 145. In contrast, the technical risk rating would consider the risk associated with the technical approach in meeting the requirement, including assessments of potential for disruption of schedule, increase in costs, degradation of performance, the need to increase Government oversight, or the likelihood of unsuccessful contract performance. Id.
For the past performance factor and the corporate experience factor, the RFP stated that separate performance confidence assessment ratings would be assigned. Concerning past performance, the RFP stated that the evaluation would address the governments level of confidence in the offerors ability to successfully perform the required effort based on the Offerors (including principal and critical subcontractors and/or JV/SLE team members) relevant past performance and systemic improvement. Id. Concerning corporate experience, the RFP stated that the evaluation would focus on the Offerors (including principal and critical subcontractors and/or JV/SLE team members) recent and relevant corporate experience on programs with similar complexity as they relate to the tasks required to be performed in support of the solicitation requirements. Id. at 146.
Finally, with regard to the cost factor, the RFP required offerors to propose costs for both the technology development phase and the engineering and manufacturing development phase, with the proposed cost for the technology development phase being more important than the cost for the engineering and manufacturing development phase. The RFP stated that the evaluated cost for each contract phase would be either the Offerors proposed cost or the Governments Most Probable Cost . . . whichever is higher. Id. at 149.
The Navy received three proposals, including the proposals of BAE and Raytheon, on September 19, 2012. All three offerors were considered to be within the initial competitive range. Discussions began immediately after the competitive range determination and concluded on April 26, 2013. Final proposal revisions were due on May 3.
After the Navy completed its evaluation, BAEs and Raytheons proposals were rated as follows:
Good / High Risk
Good / Low Risk
Good / High Risk
Good / Low Risk
Program and Schedules
Acceptable / Low Risk
Good / Low Risk
Cost--Engg and Mfg. Dev.
AR, Tab O.2, BAE Debriefing, at 91. On July 8, the Navy awarded the technology development phase contract to Raytheon, and notified BAE that its proposal was not selected as the best value to the government. BAE received written debriefing slides on July 12, and a face-to-face debriefing on July 15. This protest followed.
BAE presents wide-ranging issues in its protest and supplemental protests of this award, including allegations that the Navy conducted an unreasonable, unequal, and inadequately documented technical evaluation; applied more lenient standards to Raytheons proposal regarding a variety of documentation and substantiation requirements; failed to evaluate the proposals in accordance with the RFPs evaluation criteria; made multiple factual and mathematical errors in its interpretation of the proposals; unequally evaluated the proposals under the corporate experience factor; failed to conduct adequate discussions with BAE; conducted improper post-FPR discussions with Raytheon; and made unreasonable adjustments to BAEs proposed cost. As detailed fully below, we agree with BAEs protests in several areas. Specifically, we conclude that the Navy failed to evaluate risk in accordance with the RFPs technical evaluation criteria, and inadequately documented its analysis of weaknesses and risks in several technical areas. Additionally, we conclude that the agency committed errors in its evaluation of the proposals under the corporate experience factor.
Technical Evaluation Errors
While BAEs protests present a variety of allegations concerning the Navys technical and engineering judgments with regard to multiple aspects of its own, and Raytheons technical evaluations, our Offices review of the record in this case reveals straightforward errors in the Navys application of the RFPs stated evaluation scheme, and in its documentation of the evaluation record. First, our review of the record demonstrates that the Navy failed to conduct an assessment of proposal risks concerning the Aeromechanical/Air Vehicle sub-element of the design approach technical subfactor, which was required pursuant to the terms of the RFP. Second, our review has identified multiple technical areas in which a weakness or risk concerning Raytheons proposal was identified, was not resolved through discussions, but nonetheless disappeared from the evaluation without explanation.
The evaluation of an offerors proposal is a matter within the agencys discretion. VT Griffin Servs., Inc., B-299869.2, Nov. 10, 2008, 2008 CPD ¶ 219 at 4; IPlus, Inc., B-298020, B-298020.2, June 5, 2006, 2006 CPD ¶ 90 at 7, 13. In reviewing a protest of an agencys evaluation of proposals, our Office will examine the record to determine whether the agencys judgment was reasonable and consistent with the stated evaluation criteria and applicable procurement statutes and regulations. Shumaker Trucking & Excavating Contractors, Inc., B-290732, Sept. 25, 2002, 2002 CPD ¶ 169 at 3. While we will not substitute our judgment for that of the agency, we will sustain a protest where the agencys conclusions are inconsistent with the solicitations evaluation criteria, undocumented, or not reasonably based. DRS ICAS, LLC, B-401852.4, B-401852.5, Sept. 8, 2010, 2010 CPD ¶ 261 at 4-5.
With regard to the Navys risk evaluation, as stated above, the RFP set forth a technical evaluation scheme under which the Navy was required to evaluate proposals for both technical merit and technical risk. As set forth in the RFP, the technical risk evaluation was to consider the risk associated with the technical approach in meeting the requirement, including assessments of potential for disruption of schedule, increase in costs, degradation of performance, the need to increase Government oversight, or the likelihood of unsuccessful contract performance. AR, Tab A.3, RFP, at 145.
Turning to the specific Aeromechanical/Air Vehicle sub-element evaluation, the RFP stated that, under the design approach evaluation criteria:
The Government will evaluate and assess the Offerors overall system approach and its ability to meet the requirements of this solicitation (i.e., System Performance Specification), as well as its approach to System Overview, Built-in-Test (BIT), Platform Integration, Aeromechanical/Air Vehicle, Open Systems Architecture, Supportability and IPS Management, Key System Performance Criteria, EIRP, Frequency and Spatial Coverage, Jamming Techniques, System and Assignment Management, and Size and Weight.
Id. at 146 (emphasis added). The RFPs system performance specification (SPS), referenced above, also set forth multiple specific requirements related to the aeromechanical/air vehicle sub-element, including requirements related to aircraft stability and control, airworthiness, as well as a prohibition on modification of the EA-18 aircraft or aircraft systems.
In the evaluation of BAEs and Raytheons proposals under the aeromechanical/air vehicle sub-element, the Navy evaluator raised numerous concerns applicable to both offerors designs. For example, the evaluator found that each offerors NGJ pod was [DELETED], and that:
[DELETED] incurs a risk that the flying qualities requirements will not be met (SPS 885), preventing certification throughout the carriage envelope (SPS 873), with the required store loadouts (SPS 800), unless modifications are made to flight control system control laws (which is disallowed by SPS 889).
AR Exhibit Book, Tab T.4, BAE Evaluator Worksheet, at 1; AR, Tab U.2.i.18, Raytheon Evaluator Worksheet, at 1.
However, the evaluator also found additional challenges concerning Raytheons proposed NGJ pod. For example, while the evaluator found that BAEs pod was [DELETED], Raytheons proposed pod was [DELETED]. Id. In fact, the evaluator noted that for Raytheons proposed pod:
[DELETED]. The aerodynamic impacts of [DELETED] may make airworthiness certification [DELETED] more difficult [DELETED].
AR, Tab U.2.i.18, Raytheon Evaluator Worksheet, at 1. Further, while the evaluator noted that both proposed pods posed the risk of flight envelope restrictions in [DELETED], the evaluator concluded that the Raytheon pod design also posed some risk [DELETED] because we have never [DELETED]. AR Exhibit Book, Tab T.4, BAE Evaluator Worksheet, at 2; AR, Tab U.2.i.18, Raytheon Evaluator Worksheet, at 2.
Notwithstanding the evaluators assessment of the above concerns with the offerors NGJ pod designs, in the final version of the evaluators worksheet the entire analysis reviewed above was struck from the evaluation. In explanation of the decision to remove the analysis, the evaluator wrote:
Weakness closed. [. . .] This is a risk that should be tracked by the program through [technology development] and [engineering and manufacturing development], but it would be nearly impossible to fully assess compliance with the requirements without a full [flying qualities] analysis program. As the program is currently planned, we will be running [computational fluid dynamics] in the [technology development] phase to get an early look at this risk. In the [engineering and manufacturing development] phase, we will be running wind tunnel tests followed by flight tests. This weakness is closed, but I recommend that the issue is tracked as a risk in the [technology development] phase.
Despite closing the weakness and removing the above analysis from the evaluation, the evaluator went on to acknowledge that Raytheons discussions response on the aeromechanical/air vehicle issues does not provide additional data that eliminates this weakness. AR, Tab U.2.i.18, Raytheon Evaluator Worksheet, at 2. In contrast, the evaluator noted that BAEs proposal has done the most they can do to minimize the aerodynamic impact of the pod, aside from [DELETED]. AR Exhibit Book, Tab T.4, BAE Evaluator Worksheet, at 2. Ultimately, however, the record shows that no comparative assessment of these risks or notice of any aeromechanical/air vehicle risks whatsoever was included in any report or briefing presented to the source selection authority (SSA) in this procurement.
In its protest, BAE asserts that the Navys actions in this regard demonstrate that the Navy failed to conduct a reasonable risk evaluation of the aeromechanical/air vehicle sub-element in accordance with the RFP. The Navy responds that its evaluation did assess risk under the aeromechanical/air vehicle sub-element, as demonstrated by its evaluators worksheet, but that the evaluator reasonably determined in the final review that the concerns no longer merit a weakness given the early stage of the development program, thus the weakness was closed (removed), as is evidence from the evaluator worksheet language itself as well as its strike-through font. Supplemental AR, Legal Memo, at 233.
According to the Navy, the RFP required only limited aeromechanical information concerning pod drag, pod loads, corrosion prevention, size, weight, center of gravity, and mass moment of inertia--and no requirement to show detailed analysis of aerodynamic interactions between the proposed pods and the aircraft. On the basis of this limited information, the Navy asserts that it reasonably determined that in the cases of both offerors flight risks, the full extent of the risk in either design could not be fully characterized and assessed until during the execution of the [technology development] and [engineering and manufacturing development] phases of the program. Id. at 234.
We disagree that the agencys decision to defer consideration of aeromechanical/ air vehicle technical risk until contract performance was reasonable. The RFP required the Navy to conduct an analysis of the technical risks posed by the offers design approach for the purposes of awarding the technology development contract. It did not allow the Navy to defer analysis of the risks posed by the competing designs until after the award had been made. The Navys failure to consider these risks was inconsistent with the terms of the RFP.
Further, while it is understandable that definitive assessments of an offerors compliance with specific SPS requirements may be impossible due to limited information in the early stages of a development effort, assessing the impact of that ambiguity is the very purpose of a technical risk analysis. Conducting an assessment of the positive and negative possibilities raised by limited developmental data is at the very core of a reasonable risk analysis. Here, the Navy evaluators original analysis of the risks inherent in the aeromechanical/air vehicle sub-element plainly demonstrates that the Navy possessed sufficient information about the two proposed designs to identify areas of the offerors proposed pod designs that increased the potential for disruption of schedule, increase in costs, degradation of performance, the need to increase Government oversight, or the likelihood of unsuccessful contract performance. AR, Tab A.3, RFP, at 145. We therefore conclude that the Navys decision to essentially ignore the RFPs required technical risk analysis for the aeromechanical/air vehicle sub-element was unreasonable, and represents a failure to follow the RFPs stated evaluation criteria.
Next, our review of the record reveals that the agency failed to adequately document its resolution of weaknesses and risks in several areas of the technical evaluation. Specifically, under the platform integration sub-element and size and weight sub-element of the design approach subfactor, the evaluation documents contain risks and weaknesses that were attributed to Raytheons proposal, were not resolved, but nonetheless disappeared from the evaluation record without contemporaneous documentation. Although an agency is not required to retain every document generated during its evaluation of proposals, the agencys evaluation must be sufficiently documented to allow our Office to review the merits of a protest. Apptis, Inc., B-299457 et al., May 23, 2007, 2008 CPD ¶ 49 at 10. Where an agency fails to document or retain evaluation materials, it bears the risk that there may not be adequate supporting rationale in the record for us to conclude that the agency had a reasonable basis for its source selection decision. Navistar Def., LLC; BAE Sys., Tactical Vehicle Sys. LP, B-401865 et al., Dec. 14, 2009, 2009 CPD ¶ 258 at 13.
As reviewed above, platform integration was among the elements for evaluation by the Navy under the RFPs design approach evaluation criteria. As relevant, under the platform integration sub-element, the Navy was to assess the offerors pods compliance with safe stores separation and stores compatibility requirements of the SPS.
In the evaluation of Raytheons proposed pods compliance with these requirements, the Navy evaluator recorded a weakness due to risks associated with [DELETED]. Specifically, the evaluator found that:
[DELETED] represents a risk to safe separation, particularly [DELETED]. The Offerors current analysis predicts [DELETED] which is a violation of [DELETED] safe separation acceptance criteria [DELETED].
AR, Tab U.2.i.13, Evaluator Worksheet, at 3. In discussions, Raytheon proposed to mitigate the violation by [DELETED]. However, the evaluator found that risks remained. Specifically, the evaluator concluded that:
[DELETED] will have to be evaluated and may increase the number of [DELETED] safe separation test points required during EMD. [DELETED]. Ground and flight test evaluation of [DELETED] may be beyond the scope of the government's notional EMD schedule contained in the SOO. The impact (e.g. on repeatability, wind tunnel correlation, mission representativeness, etc.) of [DELETED], is unknown and may complicate the safe separation evaluation.
Id. The evaluator also wrote that [b]oth the original approach and the option of [DELETED] to ensure safe separation criteria of [DELETED] warrant weaknesses either individually or combined. Id. The evaluator worksheet plainly shows that these concerns remained weaknesses after the conclusion of discussions. However, this assessment did not appear in the evaluation summary sheets, nor did the assessment appear in later evaluation documents presented to the SSA.
Similarly, under the size and weight sub-element of the design approach subfactor, the Navy evaluator recorded a weakness in Raytheons proposal where [DELETED]. Specifically, the evaluator stated that:
The Government assesses risk against the proposed design meeting the weight requirements of SPS-894 (2,377 lbs. threshold). The Governments per pod weight estimate [DELETED] . . . is [DELETED] than the Offerors proposed NGJ per pod weight [DELETED].
AR, Tab U.2.i.47, Evaluator Worksheet, at 4. In this instance, the above weakness appeared in the unrated summary sheet for the evaluation area, in the form of a significant weakness stating that [t]he Offerors proposed design is assessed to have a significant risk of being [DELETED] heavier per pod than the Offerors allocation, resulting in a total system weight that is [DELETED] above the required maximum weight of 2,377 lbs. (SPS-894). AR, Tab U.2.iii.2, Unrated Worksheet, at 7.
However, in the Navys rated summary worksheet for this area--containing the signatures of two of the same evaluators as the unrated summary worksheet and signed on the same date as the unrated summary worksheet--the significant weakness was removed without any contemporaneous explanation. See AR, Tab U.2.ii.2, Rated Worksheet, at 13. As was the case with the platform integration sub-element, the concerns set forth in the size and weight sub-element evaluation did not appear in later evaluation documents presented to the SSA.
BAE asserts that there is no reasonable basis in the record for the disappearance of these documented risks and weaknesses from the evaluation record. The Navy now contends that these weaknesses did not appear in later evaluation documents because they were reasonably deemed insignificant. For example, the Navy argues that because Raytheon had proposed a risk mitigation for the stores separation risk--[DELETED]--that did not [DELETED], the weakness was reasonably deemed not significant and therefore properly did not roll up to the [later evaluation documents]. Supplemental AR, Legal Memo, at 249.
Nothing in the contemporaneous record documents or supports the Navys contention that the above weaknesses were reviewed and found minor. Specifically, concerning the safe stores separation risk, the Navy evaluators worksheet plainly demonstrates that Raytheons pod design posed risks even considering Raytheons proposed mitigation, and the evaluator specifically stated that [b]oth the original approach and the option of [DELETED] to ensure safe separation criteria of [DELETED] warrant weaknesses either individually or combined. AR, Tab U.2.i.13, Evaluator Worksheet, at 3. Concerning the size and weight sub-element weakness, that weakness was specifically documented as a significant weakness in the Navys unrated summary worksheet for the evaluation area, only to disappear in the rated summary worksheet--apparently signed by two of the same individuals on the same day. On the basis of the evaluation record here, we cannot conclude that the Navy reasonably decided that the above weaknesses and risks were minor. There is simply no contemporaneous explanation of the resolution of these risks and weaknesses in the record here. Accordingly, on the basis of this record, we are unable to conclude that these matters were reasonably considered, and set aside by the agency evaluators.
Additionally, with regard to the size and weight sub-element evaluation, the contemporaneous record supports BAEs contention that the Navy failed to consider the weight impact of certain Raytheon design changes made in response to evaluation notice (EN) T-RTN-245A, shortly before the date for the submission of final proposal revisions. While the Navy directs our Office to multiple documents in the record that it asserts included consideration of these final changes to Raytheons design, all of the documents cited by the agency appear to consider information through EN T-RTN-246, dated March 22, and none include a citation to EN T-RTN-245A, which prompted the final design changes, and is dated April 23. Accordingly, on the basis of the record in this case, we cannot conclude that the Navys weight evaluation considered all elements of the actual final NGJ pod design proposed in this procurement.
The Navy argues that BAE was not prejudiced where both Raytheon and BAE had risks that were struck from the aeromechanical/air vehicle evaluation, and where BAE had significant risks under the size and weight sub-element, as well as other technical sub-elements, resulting in its technical high risk rating. Our decision here does not discuss the Navys evaluation of multiple technical areas that resulted in BAEs high risk rating because we find no error in the agencys evaluation of those areas; however, we note that the NGJ technical development proposals were, in essence, tradeoffs between power generation, cooling systems, jamming systems, and size and weight.
Concerning the aeromechanical/air vehicle evaluation, BAE, in its proposal, achieved [DELETED]. Thus, despite BAEs overall higher risk rating, we nonetheless consider it prejudicial for the Navy to neutralize a technical evaluation area in which [DELETED].
We also find the Navys errors prejudicial where weaknesses and risks disappeared from Raytheons evaluation, and where, in the source selection decision, the SSA emphasized that there were no evaluated weaknesses in Raytheons proposal. Specifically, the SSA stated that:
BAE [has] significant weaknesses [DELETED], which further challenge [its] ability [DELETED]. Raytheon on the other hand, has no significant weaknesses [DELETED], and in fact has no significant weaknesses at all.
AR, Tab N.2, Source Selection Decision Document, at 2 (emphasis original). Finally, [DELETED]. Id., at 3.
Where [DELETED], and where a reasonable and adequately documented evaluation may have decreased the risk advantage of Raytheons proposal, we cannot say whether the SSAs reconsideration of the proposals relative technical risks and merits would [DELETED]. In such circumstances, we resolve any doubts regarding prejudice in favor of a protester since a reasonable possibility of prejudice is a sufficient basis for sustaining a protest. See Kellogg, Brown & Root Servs., Inc.--Recon., B-309752.8, Dec. 20, 2007, 2008 CPD ¶ 84 at 5.
BAE challenges the Navys evaluation of the corporate experience factor, specifically, the Navys evaluation of three of the five corporate experience subfactors: hardware development, aircraft integration, and logistics support experience.  AR, Tab A.3, RFP, at 145, and Protest, at 87. As discussed below, the record reflects that the Navy erred in its evaluation under the aircraft integration sub-factor.
The aircraft integration experience sub-factor included four elements: (1) obtaining U.S. government flight clearances under the NAVAIR 4.0P process; (2) aircraft stores integration; (3) operations in a ships electro-magnetic environment; and (4) integration with onboard AEA or EW systems. With regard to the first element, obtaining U.S. government flight clearances under the NAVAIR 4.0P process, Raytheon received a rating of extensive corporate experience based on the Navys conclusion that Raytheon had [DELETED] years of substantiated experience with obtaining government flight clearances, whereas BAE received a rating of only corporate experience based on having substantiated only [DELETED] years of experience in obtaining flight clearances.
The Navy, however, concedes that its evaluation under this element was erroneous in two respects. First, the Navy acknowledges that it failed to credit BAE with [DELETED] experience for its work on the [DELETED]. The Navy evaluators found that BAE gained [DELETED] experience in obtaining flight clearances in the course of its work on the [DELETED]. AR, at 248. The Navy, however, inadvertently failed to credit this experience in the corporate experience rating summary sheet, and as a result, the SSEB Report did not include any reference to this experience. Id. Accordingly, BAE should have been credited with [DELETED], rather than [DELETED] years of experience under this element.
Second, the Navy admits that its evaluation of BAE and Raytheon under the element was unequal. In this regard, the record reflects that both BAE and Raytheon claimed experience associated with obtaining flight clearances under the NAVAIR 4.0P process for work performed before 2004, as well as work performed after 2004. As explained by the Navy, the evaluation team decided to use 2004 as a cut-off for recent experience in obtaining clearances because changes were made to the NAVAIR flight clearance process and policy in 2004 which, in the Navys view, rendered pre-2004 experience not relevant. Supplemental AR, Tab Z, Evaluator Declaration, Aug. 17, 2013, at 6-7. Based on the 2004 cutoff, the Navy did not credit BAE with [DELETED] years of substantiated experience from [DELETED]-2004. AR, Tab L, SSEB Report, at 151. The record reflects, however, that the Navy credited Raytheon with experience which it gained prior to 2004. AR, at 246. According to the Navy, Raytheon should not have received credit for its pre-2004 experience.
The precise impact of this error on Raytheons evaluation under this element is unclear. The SSEB report indicates that Raytheon and its subcontractors substantiated more than [DELETED] years of combined experience in all areas of aircraft integration. AR, Tab L, SSEB Report, at 175. With respect to obtaining clearances, however, the SSEB report identifies only [DELETED] contracts accounting for a total of [DELETED] years of experience under this element--[DELETED]. Based on the cut-off established by the Navy, none of the work under the [DELETED] performed before 2004 should have been credited to Raytheon.
Thus, based on the contemporaneous record provided to our Office, Raytheon would be left with only [DELETED] years of experience, [DELETED], when accounting for the agencys error. In an affidavit, however, the corporate experience team lead asserts that the Navy credited Raytheon with [DELETED] years of experience in obtaining government clearances, and that [DELETED] years of that experience was obtained prior to 2004. Supplemental AR, Tab Z, Evaluator Declaration, Aug. 17, 2013, at 8. Accordingly, to the extent Raytheon was initially credited with [DELETED] years of experience, and [DELETED] of those years were found to be in error, as the agencys evaluator asserts, Raytheon would be left with only [DELETED] years of experience, [DELETED] less than BAE. In any event, as explained above, the record reflects, and the agency acknowledges, that both BAE and Raytheon were misevaluated under the aircraft integration subfactor element obtaining U.S. government flight clearances under the NAVAIR 4.0P process.
Raytheon Head Start Program
In their proposals in response to the RFP, both BAE and Raytheon proposed to undertake certain pre-award efforts in order to mitigate elements of risk under the technical factor, program and schedules sub-factor. BAE, in its proposal, set forth a ready on day 1 program, under which it would [DELETED] prior to the time of contract award. Raytheon, in its proposal, set forth a head start program, under which it would begin [DELETED] prior to the award decision, and would [DELETED].
In response to Raytheons proposal, the Navy advised Raytheon that it would consider the head start program as a potential risk reducer, but requested that Raytheon provide [DELETED], and provide a bi-weekly status report to the contracting officer concerning [DELETED]. Raytheon submitted its first head start submission to the Navy on February 28, and its final bi-weekly status report on July 2--more than 2 months after the May 3 deadline for submission of FPRs. The record demonstrates that all of Raytheons head start reports were reviewed by multiple Navy procurement officials, including five reports received after the closing date of discussions/FPR deadline.
BAE argues that the review of Raytheons post-FPR head start submissions constituted improper post-FPR discussions or, at minimum, unequal exchanges that favored Raytheon. BAE also asserts that prejudice should be presumed under these circumstances, citing our Offices decision in Radiation Oncology Group of WNY, PC, B-310354.2, B-310354.3, Sept. 18, 2008, 2009 CPD ¶ 136. The Navy responds that while procurement officials did review Raytheons post-FPR head start reports, the contents of the head start reports were not considered in Raytheons design approach subfactor evaluation and, with regard to the program and schedules sub-factors, were only assessed to determine whether Raytheons proposal [DELETED].
We agree with BAE that the Navy procurement officials review and consideration of the contents of Raytheons post-FPR head start status reports was improper. In fact, at least one of the Navys evaluation personnel agreed that the post-FPR head start reports should not have been reviewed. In an email chain discussing the head start reports, the evaluator wrote I disagree [that we can consider the reports]. Unless it was information received on an existing contract, I believe it is considered proposal information received after FPR. AR, Tab R.1.iii, Evaluator Email, May 8, 2013.
However, in this case, Raytheons continued submission of bi-weekly status reports did not constitute an effort to revise its proposal in any way. The contents of the status report did not expand on or provide additional detail relating to Raytheons technical approach, and our review of the record shows that no evaluation reports were altered as a result of the Navys review of post-FPR head start reports. Accordingly, we conclude that the Navys distribution of the reports to the evaluators, although in error, did not impact the competition.
We have considered BAEs emphasis on Radiation Oncology Group of WNY, PC, supra, in support of its contention that prejudice should be presumed where post-FPR communications provide, and the agency considers, additional supporting material regarding [the offerors] technical proposal. Id. at 3. BAE argues that under these circumstances, the agency bears the burden of affirmatively proving that it did not consider [post-FPR materials] in its evaluation, which the Navy has failed to do. BAE Supplemental Comments at 13.
We disagree that Radiation Oncology is applicable here. In that protest, the awardee submitted late information explicitly revising its technical proposal by providing additional quality/performance evaluations. The agency in that case did not argue that the additional materials were not material to the evaluation, and the source selection decision was not sufficiently documented to establish the extent to which the late materials were considered in the technical evaluation.
In contrast to Radiation Oncology, the reports at issue here were not an attempt to make late revisions to Raytheons proposal, but were instead submissions required by the agency as a condition of its consideration of Raytheons timely-proposed head start program. Further, while it is clear from the record that multiple Navy evaluators reviewed the head start reports, BAE has not itself identified material technical data in any head start report, nor identified any technical evaluation document that may have been modified as a result of the head start reports. Rather, BAE alleges only that the status updates in the head start reports could have provided assurance to Navy evaluators concerning Raytheons design approach and program and schedules. Where the information provided in Raytheons head start reports was not intended to modify or revise Raytheons proposal, and has not been shown to be material to the technical evaluation, our Offices decision Radiation Oncology is not applicable.
We recommend that the Navy document a reevaluation of the proposals that is reasonable and consistent with the RFPs evaluation criteria. Upon completion of the reevaluation, the Navy should perform and document a new cost/technical tradeoff analysis with the rationale for any tradeoff made. If the Raytheon proposal is not found to represent the best value to the government in accordance with the RFPs evaluation factors, the agency should terminate the award to Raytheon and award a new contract to the offeror representing the best value to the government.
We also recommend that BAE be reimbursed its costs of filing and pursuing the protest. Bid Protest Regulations, 4 C.F.R. § 21.8(d)(1) (2013). The protesters certified claims for such costs, detailing the time expended and costs incurred, must be submitted to the agency within 60 days after receipt of this decision.
The protest is sustained.
Susan A. Poling
 The record in this protest includes classified documents. However, our Office has concluded that no classified documents in the record bear on the issues discussed in this decision, and this decision therefore makes no further reference to classified materials.
 We discuss a selection of BAEs protest allegations in this decision. To the extent that allegations raised by BAE are not discussed, they have been reviewed by our Office and found to present no basis to sustain the protest. As a general matter, we note many of BAEs protest grounds challenging the Navys technical and engineering judgments rely on expert testimony that essentially disagrees with the Navys evaluation. A protesters mere disagreement with the agencys conclusions does not render the evaluation unreasonable. Visual Connections, LLC, B-407625, Dec. 31, 2012, 2013 CPD ¶ 18 at 4.
 Aircraft stores include pods, tanks, and missiles that may be equipped on the aircraft. Safe stores separation concerns whether the proposed NGJ system may prevent the safe separation (tank jettison or missile firing) of adjacent or other stores. Stores compatibility relates to whether the proposed NGJ system meets Navy aircraft integration requirements.
 We note that the Navys arguments were not supported with affidavits from the technical evaluators; rather, they were advanced entirely by Navy counsel in legal argument.
 The corporate experience factor is comprised of five sub-factors: (1) hardware development; (2) aircraft integration; (3) software development; (4) program management; and (5) logistical support. We have reviewed all of BAEs challenges with regard to the hardware development and logistics support experience sub-factors and conclude that they are not supported by the record.
 BAE has challenged the agencys decision to discount experience prior to 2004; however, we do not find the agencys decision in that regard to be unreasonable. Generally, an agency has broad discretion to determine whether a particular contract is relevant to the evaluation where a solicitation requires the evaluation of an offerors experience. K-Mar Industries, Inc., B-400487, Nov. 3, 2008, 2009 CPD ¶ 159 at 3.
 The team lead further asserts that, despite this error, Raytheon demonstrated [DELETED] years of experience obtained after 2004. Supplemental AR, Tab Z, Evaluator Declaration, Aug. 17, 2013, at 8. The Navy, does not, however, identify this additional [DELETED] years of experience, and there is nothing in the SSEB Report to substantiate it.
 The Navy contends that BAE was not prejudiced by the alleged errors because neither offerors overall adjectival rating under the aircraft integration subfactor would change as a result of correcting the errors. We do not find this persuasive as we have held that point scores and adjectival ratings are merely guides for intelligent decisionmaking. Information Ventures, Inc., B-407478.4, July 17, 2013, 2013 CPD ¶ 176, at 8. Additionally, we have held that selection officials should reasonably consider the underlying bases for ratings, including the advantages and disadvantages associated with the specific content of competing proposals and quotations, in a manner that is fair and equitable and consistent with the terms of the solicitation. Id. As a result, and in light of other errors in the evaluation discussed elsewhere in this decision, it is impossible to know how errors in the record related to this subfactor may have impacted the selection decision in this case.