Shaw-Parsons Infrastructure Recovery Consultants, LLC; Vanguard Recovery Assistance, Joint Venture

DOCUMENT FOR PUBLIC RELEASE
The decision issued on the date below was subject to a GAO Protective Order. This redacted version has been approved for public release.

Decision

Matter of: Shaw-Parsons Infrastructure Recovery Consultants, LLC; Vanguard Recovery Assistance, Joint Venture

File: B-401679.8; B-401679.9; B-401679.10

Date: September 8, 2010

DIGEST

Protest that agency's reevaluation of protester's past performance was unreasonable is sustained where the agency's reevaluation was based on a numerical scoring system, which had the effect of penalizing protester based on its submission of performance information for additional less relevant contracts.

Shaw-Parsons Infrastructure Recovery Consultants, LLC, of Alexandria, Virginia, and Vanguard Recovery Assistance, Joint Venture, of Washington, D.C., protest the award of architect/engineering (A/E) services contracts to Fluor Enterprises, Inc., AECOM Services, Inc., Nationwide Infrastructure Support Technical Assistance Consultants (NISTAC), and CH2MHill-CDM PA-TAC Recovery Services (CCPRS), pursuant to solicitation No. HSFEHQ-09-R-0411, issued by the Department of Homeland Security, Federal Emergency Management Agency (FEMA), to support the Public Assistance program under the Robert T. Stafford Disaster Relief and Emergency Assistance Act, 42 U.S.C. sections 5121-5206. Both Shaw-Parsons and Vanguard challenge the agency's reevaluation of offerors' past performance, which was undertaken by the agency in response to a decision by our Office sustaining a prior protest filed by Shaw-Parsons.

We sustain the protest filed by Vanguard and deny the protest filed by Shaw-Parsons.

BACKGROUND

On March 10, 2010, our Office sustained a protest filed by Shaw-Parsons in connection with FEMA's award of the above contracts. Shaw-Parsons Infrastructure Recovery Consultants, LLC; Vanguard Recovery Assistance, Joint Venture, B'401679.4 et al., Mar. 10, 2010, 2010 CPD para. 77.[1] In sustaining Shaw-Parsons' protest, we concluded that FEMA failed to consider in its past performance evaluation information contained in past performance questionnaires (PPQ), which was "too close at hand" for FEMA to have ignored. We recommended that FEMA conduct a reevaluation, giving reasonable consideration to the PPQs it received for the firms remaining in the competition as the information related to the quality of their past performance.

As explained in our prior decision, interested firms were invited to submit a completed standard form (SF) 330 (A/E qualifications statement) detailing their qualifications to provide the services required by the Sources Sought Notice (SSN) issued in connection with the procurement.[2] Firms were advised that their qualifications would be evaluated under the following five factors: (1) specialized experience and technical competence; (2) capacity to accomplish work within the required time; (3) professional qualifications; (4) past performance; and (5) location in the general geographic area of the project. The SSN, which was amended several times, specified that factors 1, 2, and 3 were of equal importance and more important than factors 4 and 5. Evaluation factor 5 was identified as more important than factor 4. In addition, factor 1 consisted of the following three subfactors: subfactor 1A, experience developing reliable cost estimates for multi-million dollar construction projects and/or infrastructure repair projects; subfactor 1B, experience in evaluating projects for compliance with environmental regulations and preparing environmental documents; and subfactor 1C, experience in staffing at the levels in evaluation factor 2.

Under the past performance factor, the SSN provided that FEMA would evaluate firms' past performance in performing contracts of "similar size, type, and scope . . . in terms of project management, accuracy of costs estimates, cost control, quality control, completion of projects within budget, and compliance with performance schedules." SSN at 3. Firms were required to provide "references for at least five contracts within the past three (3) years with names, affiliations, and telephone numbers, with a narrative discussion." Id. The SSN also advised that FEMA "reserves the right to use information outside of the response in evaluating past performance, including agency knowledge of the firm[']s performance." Id. at 4.

After firms had submitted their SSNs, FEMA sent them PPQs, which were to be completed by the firms' past performance references and returned to FEMA. The PPQs allow references to rate a firm's performance as "Superior," "Acceptable," or "Unacceptable" in the following four categories: (a) Quality of Product or Service; (b) Cost Control; (c) Timeliness of Performance; and (d) Business Relations. The PPQs also allow the references to supplement their adjectival ratings with narrative comments.

FEMA explains that after receiving our decision, the Source Evaluation Board (SEB) reconvened for the purpose of reevaluating the firms' past performance giving consideration to information contained in their SF330s and PPQs. According to FEMA, the SEB reevaluated the information contained in the firms' SF330s and reviewed the ratings and comments provided in the firms' PPQs as the information related to the areas established for consideration by the SSN (project management, accuracy of costs estimates, cost control, quality control, completion of projects within budget, and compliance with performance schedules). Agency Report (AR), Exh. 6, Source Selection Board Official Consensus Final Report, May 5, 2010, at 17'18. The record reflects that FEMA assigned firms separate adjectival ratings ("Superior," "Acceptable," or "Unacceptable") for their SF330s and their PPQs, and considered the information contained in the SF330s and the PPQs to be "equally important" when assigning firms an overall past performance rating. Id. at 19. From these ratings and the associated evaluations, FEMA derived an overall past performance rating using the same adjectival rating scheme.

The record reflects that when the SEB evaluated and rated the firms' PPQs, it evaluated and rated two components: (1) the information contained in the PPQ narratives, which supported the adjectival chart ratings; and (2) the particular adjectival chart ratings themselves. Because the four evaluation categories set forth in the PPQs did not precisely match the six areas for evaluation established in the SSN, the SEB reviewed the narrative comments and considered any comments which it believed to be relevant to the six past performance areas set forth in the SSN. The SEB documented its consideration and evaluation of these narratives in a chart, which identified, for each firm, any comments from the PPQs that the SEB considered to be a strength or a weakness and the SSN category that best corresponded to the strength or weakness. The difference between the PPQ rating categories and the SSN past performance categories also led the SEB to attribute less weight to the particular adjectival chart ratings when assigning an overall PPQ rating. Id.

For the purpose of evaluating firms' PPQ chart ratings, the record reflects that the SEB utilized a mathematical scoring system. Specifically, for each Superior and Acceptable rating a firm received (none of the firms received Unacceptable ratings) FEMA assigned 10 points and 5 points, respectively. The SEB, however, decided to reduce the number of points for any Superior rating (to 7.5) where the rating was given in connection with a PPQ for a contract that the SEB considered to be less relevant--the contract was not sufficiently similar to the SSN requirements in terms of size, scope, type, and complexity. Based on the total number of Superior and Acceptable adjectival ratings received, FEMA calculated a total score for each firm. This total was then divided by the total number of PPQs submitted on behalf of the firm, resulting in an average score. Because there are four rating categories for each PPQ, the highest rating a firm could possibly achieve on any one PPQ was 40 points (10 points for a Superior rating in each of the four PPQ rating categories); thus, the highest average a firm could achieve was 40 points, assuming it received all Superior ratings for all its PPQs, and none of the PPQs were for contracts identified by the SEB as less relevant. The record reflects that, at a minimum, an overall average PPQ score of 36 resulted in a Superior rating for the PPQ chart component of the PPQ evaluation. See Supplemental AR, at 13 (indicating that a firm with an average score of 36 or 39 points would have both received a Superior rating for their PPQ chart information).

As it relates to the current protests, the number of PPQs received and considered by the SEB for any particular firm ranged between two and six. With respect to Shaw'Parsons, FEMA received five PPQs, all of which were deemed to be relevant. The ratings for these five PPQs reflects 19 ratings of "Superior" and one "Acceptable" rating. This resulted in an average PPQ chart rating score of 39 points, which, as noted above, resulted in a "Superior" rating for this component of the PPQ rating. Id., Exh. 2. Shaw-Parson's PPQ chart rating score was second only to one of the awardees, Fluor, which had a perfect score of 40 points, albeit based on only two PPQs. Notwithstanding its Superior PPQ chart rating, Shaw-Parsons received an overall Acceptable rating for its PPQ information (combining the PPQ narrative evaluation with the PPQ chart rating), an Acceptable rating for its SF330 information, and an overall past performance rating of Acceptable.

In this regard, based on its review of the narrative comments contained in Shaw'Parsons' PPQs, the SEB identified various strengths for three of the six past performance evaluation categories established by the SSN. The SEB also identified one weakness as it related to Shaw-Parsons' cost control performance where one of its references wrote that "Shaw provides nominal cost controls. They could do more to reduce costs and use economy of scale, especially in their program management." AR, Exh. 6, Source Selection Board Official Consensus Final Report, May 5, 2010, Attach. E, Shaw-Parsons Evaluation, at 4. The SEB did not identify any client comments regarding the quality of Shaw-Parsons' performance as it related to accuracy of costs estimates, quality control, or completion of projects within budget. In addition, the SEB indicated that for all but one evaluation category, accuracy of costs estimates, the information contained in Shaw-Parsons' SF330 was largely based on quotes from prior clients and not sufficiently specific. Id. at 5.

Vanguard also received an overall past performance rating of Acceptable. FEMA received six PPQs for Vanguard. The SEB determined that two of the PPQs were for contracts that were not multi-million dollar contracts of similar size, type, and scope. Id., Attach. F, Vanguard Evaluation, at 6. The PPQ charts for the four relevant PPQs reflected 13 Superior ratings and three Acceptable ratings. The other two PPQs (for the less relevant contracts) reflected all Superior ratings. Using the point scoring methodology described above, the SEB calculated Vanguard's average PPQ chart rating score as 34.83[3] and assigned Vanguard a rating of Acceptable for its PPQ chart ratings.

The SEB, however, assigned Vanguard a Superior rating for the narrative information contained in its PPQs. In this regard, based on the PPQ narratives, the SEB identified various strengths for all six of the past performance evaluation categories established by the SSN and no weaknesses. Regarding the information contained in Vanguard's SF330, the SEB highlighted several statements which it considered to be strengths, but also determined that there were areas where the information provided by Vanguard was somewhat general in nature. For example, under compliance with performance schedule, the SEB noted that, according to Vanguard's SF330, on a contract with the U.S. Army Corps of Engineers, it had completed over 95 percent of the task orders within initial schedule and remaining task orders involved scope changes. The SEB also indicated, however, that the other information contained in Vanguard's SF330 for this element was general in nature and lacked sufficiently specific detail to determine whether Vanguard's performance was successful. The SEB assigned Vanguard an Acceptable rating for its SF330 past performance information.

At the conclusion of the SEB's reevaluation, none of the firms' overall past performance ratings changed from the prior evaluation. The SEB's final past performance ratings were as follows:

Firm	SF 330	PPQ		Overall Past Performance Rating
		Factor 4 Criteria	PPQ Chart Ratings
AECOM	Superior	Superior	Acceptable	Superior
CCPRS	Superior	Acceptable	Acceptable	Superior
FLUOR	Acceptable	Acceptable	Superior	Acceptable
SHAW-PARSONS	Acceptable	Acceptable	Superior	Acceptable
NISTAC	Acceptable	Acceptable	Acceptable	Acceptable
PB	Acceptable	Acceptable	Acceptable	Acceptable
VANGUARD	Acceptable	Superior	Acceptable	Acceptable

AR, Exh. 6, Source Selection Board Official Consensus Final Report, supra, at 21.

Since the firms' past performance ratings did not change, the SEB did not alter the firms' relative rankings, again determining that Fluor was the highest-rated firm, with AECOM and CCPRS tied for second. Since NISTAC, Shaw-Parsons, and Vanguard remained tied for the fourth position based on the spread of overall adjectival ratings, the SEB, as it had in the prior iteration, used the firms' relative rankings under subfactor 1A to break the tie, placing NISTAC in the fourth position. The SEB again recommended that the four most highly-rated firms be considered for award. The overall ratings and rankings were as follows:

Firm Ranking		Factor Ratings
		1A	1B	1C	1	2	3	4	5
1	Fluor	S	S	A	S	S	S	A	S
2	AECOM	A	S	S	A	S	S	S	S
3	CCPRS	S	S	A	S	S	A	S	S
4	NISTAC	S	A	A	S	S	A	A	S
5	Shaw-Parsons	S	A	A	S	A	S	A	S
6	Vanguard	S	A	A	S	S	A	A	S
7	PB	A	S	A	A	A	A	A	S

Id. at 22.

The SEB forwarded its final report to the Source Selection Authority (SSA), who signed the report. When the SSA made her prior selection decision (before FEMA took corrective action in response to our decision), she documented her selection decision in a separate memorandum. Second AR, Exh. 4, Source Selection Decision, Oct. 6, 2009. In that decision, the SSA independently compared and contrasted the evaluation findings under each factor for the three firms tied for the fourth position (NISTAC, Shaw-Parsons, and Vanguard). Id. at 4-5. Based on this assessment, the SSA identified NISTAC for the fourth award position, concluding that it was more highly technically qualified than Shaw-Parsons and Vanguard. Id. As part of this prior assessment, the SSA concluded that there was nothing to distinguish these firms under the past performance factor. Since the firms' past performance ratings were unchanged, and believing that her previous selection decision had adequately addressed her assessment of the firms' relative merits, the SSA believed it unnecessary to re-document her analysis since it would merely have been repetitive or duplicative of her previous analysis. Second AR, Exh. 3, SSA Declaration, July 27, 2010. Upon learning the results of the agency's reevaluation, Shaw'Parsons and Vanguard filed these protests.

DISCUSSION

Shaw-Parsons and Vanguard allege numerous problems with the agency's past performance reevaluation. Generally, they allege that the analytical framework used by the agency in reevaluating firms' past performance was inconsistent with the terms of the solicitation, our prior decision, or that it was in some way unreasonable or unfair. As explained below, we sustain the protest filed by Vanguard as it relates to the agency's methodology for evaluating the PPQ chart ratings component of the overall PPQ evaluation.

As a general matter, the evaluation of an offeror's past performance is within the discretion of the contracting agency, and we will not substitute our judgment for reasonably based past performance ratings. MFM Lamey Group, LLC, B-402377, Mar. 25, 2010, 2010 CPD para. 81 at 10. Where a protester challenges the past performance evaluation and source selection, we will review the evaluation and award decision to determine if they were reasonable and consistent with the solicitation's evaluation criteria and procurement statutes and regulations, and to ensure that the agency's rationale is adequately documented. Wackenhut Servs., Inc., B-400240, B-400240.2, Sept. 10, 2008, 2008 CPD para. 184 at 6; S4, Inc., B-299817, B'299817.2, Aug. 23, 2007, 2007 CPD para. 164 at 9. A protester's mere disagreement with the agency's determinations as to the relative merit of competing proposals, and its judgment as to which proposal offers the best value to the agency, does not establish that the evaluation or source selection was unreasonable. Wackenhut Servs., Inc., supra, at 6.

Vanguard's Protest

Acceptable PPQ Chart Rating

As noted above, the record reflects that the SEB divided its evaluation of the firms' PPQs into two parts, one based on the narrative information contained in the PPQs and the second based on the individual PPQ chart ratings. For the purpose of evaluating the PPQ chart ratings, the SEB, as explained above, used a numerical scoring system, one which assigned points for Superior and Acceptable ratings; using this scoring system, the SEB calculated an average score for each firm. In this regard, an average score of 36 points, or higher, translated to a Superior rating. Under this scheme, Vanguard received an Acceptable rating for its PPQ chart ratings'-it had an average score of less than 36 points. While Vanguard received a narrative rating of Superior for its PPQs, it only received an Acceptable rating for its PPQs overall, and an overall past performance rating of Acceptable.

In its protest, Vanguard argues that its Acceptable PPQ chart rating was unreasonable because the agency's scoring methodology effectively penalized Vanguard for having submitted PPQs for two less relevant contracts. Vanguard does not challenge the notion of giving less relevant contracts less weight in the evaluation per se; rather, Vanguard contends that the way in which the SEB accounted for the less relevant contracts was fundamentally flawed because the SEB effectively (and unreasonably) penalized Vanguard by diminishing the value of its Superior ratings on its relevant contracts. Specifically, in considering Vanguard's two less relevant PPQs, the SEB assigned 7.5 points as opposed to 10 points for each Superior rating, and using these scores calculated an average score for Vanguard. Vanguard maintains that had these less relevant contracts not been submitted, its past performance rating score would have been 36.25, thus leading to a Superior rating, yet when the less relevant contracts were included in the average, its score was reduced to 34.83, even though it received Superior ratings across-the-board on the PPQs for the two less relevant contracts. According to Vanguard, any reasonable consideration of its less relevant contracts could only have increased its score since it received all Superior ratings for these contracts.[4] We agree.

In Olympus Bldg. Servs., Inc., B-285351, B-285351.2, Aug. 17, 2000, 2000 CPD para. 178, we sustained a protest where the agency used a scoring methodology to evaluate offerors' experience which was similar to the one used by FEMA to evaluate firms' past performance in this case. Specifically, the agency in Olympus assigned different point scores depending on the degree of relevance of an offeror's experience, deducted points for less relevant experience, added the scores, and calculated an average. The application of this formula penalized the protester for two extra experience references, albeit for less relevant experience. We noted that had the protester's score been based on only the comparable references, the protester would have received a higher score than it did when these references were averaged with the less relevant references. We concluded that the agency's scoring methodology improperly penalized the protester for having included extra references for its experience.

Here, Vanguard submitted six PPQs, two of which were identified as less relevant. The agency deducted points for each of the Superior ratings Vanguard received on these contracts, which, when averaged with Vanguard's ratings for its relevant contracts, had the effect of lowering its overall PPQ rating score, notwithstanding the fact that Vanguard received the highest adjectival ratings on its less relevant PPQs. Thus, as in Olympus, the agency effectively penalized Vanguard for having submitted references for additional, less relevant contracts and used what was an arbitrary score for the purpose of its evaluation.[5] See also United Paradyne Corp., B'297758, Mar. 10, 2006, 2006 CPD para. 47 (sustaining protest where the agency's approach to evaluating past performance was unreasonable because it had the effect of penalizing offerors with relevant experience for their non-relevant experience).

The agency argues that Vanguard was not materially impacted by this issue since the PPQ chart ratings were given minimal weight and any change in those ratings would not have affected the result in any event. Setting aside the fact that such post hoc assessments made by agencies in the heat of the adversarial process are given little weight by our Office, see Boeing Sikorsky Aircraft Support, B-277263.2, B-277263.3, Sept. 29, 1997, 97-2 CPD para. 91 at 15, the agency's conclusions are not supported by the underlying record. Vanguard received a Superior rating for the narrative information contained in its PPQs--information which FEMA claims was more important to the overall evaluation--yet Vanguard only received an overall rating of Acceptable for its PPQs. This was apparently due to its Acceptable rating for its PPQ chart information, which, as explained above, was derived from the agency's faulty scoring system. Moreover, if Vanguard had received an overall Superior rating for its PPQs, there is a reasonable possibility that it would have received an overall past performance rating of Superior.[6] Thus, we cannot conclude, as the agency suggests, that Vanguard was not prejudiced by this error.[7]

Other Issues

Vanguard challenges the agency's best value decision arguing it was based on an erroneous consideration of Vanguard's performance.[8] Given that we are sustaining Vanguard's protest as it relates to the agency's reevaluation of its own proposal and we are recommending that the agency reconsider is reevaluation of Vanguard's past performance information, we need not address these issues.

Vanguard also argues that FEMA should have downgraded Fluor based on the fact that its references returned only two PPQs.[9] FEMA argues that it would have been unfair to penalize a firm for the failure of its references to return the PPQs. FEMA also asserts that the relative lack of information in fact was considered to the extent the firm had a smaller base from which to establish the quality of its performance. We agree with Vanguard's general contention that an agency may reasonably have less confidence regarding the quality of a firm's past performance where it has less information upon which to base an assessment. While there is nothing in the contemporaneous record to suggest that FEMA specifically accounted for the fact that it received only two PPQs from Fluor's references, we do not believe that doing so would have had any meaningful impact on the competition since the SEB rated Fluor's past performance overall as only Acceptable based on the narrative comments and its SF330 information, notwithstanding the fact that it received all Superior chart ratings for its two PPQs. In addition, we note that Fluor had the highest ratings under the four most important evaluation factors.[10]

Shaw-Parsons' Protest

Shaw-Parsons challenges numerous aspects of FEMA's reevaluation process as well as the reasonableness and fairness of FEMA's past performance judgments. We have reviewed all of the issues raised by Shaw-Parsons and conclude that they do not provide a basis to sustain its protest.

As an initial matter, Shaw-Parsons maintains that FEMA's evaluation was contrary to the solicitation because it applied unannounced evaluation criteria by dividing the PPQ information into what were essentially discrete subfactors (separately evaluating the SF330 information, and the two components of the PPQ information, the PPQ narratives and PPQ chart ratings). Shaw-Parsons notes that the SSN did not establish such a process for the past performance evaluation and failed to disclose the relative importance of these elements in the agency's evaluation. In this regard, Shaw-Parsons also contends that the SSN did not reasonably put firms on notice that FEMA would assess the quality of their performance based on statements in their SF330. Had it known this, Shaw-Parsons argues, it would have provided more information regarding the quality of its performance in response to the six SSN past performance categories.[11]

The agency's evaluation was not inconsistent with the solicitation. First, the SSN identified six areas that the agency would consider in evaluating past performance and required firms to provide a "narrative discussion" for each of their past performance references. It is apparent from the record that all the firms' SF330 narrative discussions, including Shaw-Parsons', contained information regarding the quality of their performance under each of the six SSN categories. See, e.g., Shaw'Parson's SF330, at 56-59. To the extent Shaw-Parsons maintains that it would have done a better job had it known the significance the agency gave the SF330 narrative information, that is Shaw-Parsons' error, not FEMA's, since the solicitation put firms on notice that the agency would use the narratives as part of the past performance evaluation. Second, agencies are not required to specifically identify subfactors comprising an evaluation criterion where the subfactors are reasonably related to the stated evaluation criterion, nor are they required to disclose the specific evaluation methodology that they intend to use in making evaluative judgments regarding firms' proposals. Olympus Bldg. Servs., supra, at 5. Since the information contained in the firms' SF330s, and the PPQ narrative and PPQ chart rating information, reasonably relate to the past performance factor, FEMA was not required to specifically disclose how it intended to consider or integrate these elements when conducting its evaluation. Thus, Shaw-Parsons' arguments in this regard are without merit.

Shaw-Parsons also argues that the process used by FEMA in its reevaluation was unreasonable because it was not integrated--FEMA considered firms' SF330 information separately from their PPQ information, and within the PPQ evaluation, FEMA considered the PPQ narrative information separately from the PPQ ratings. According to Shaw-Parsons, this failure to perform an integrated assessment was compounded by the fact that the weights FEMA ascribed to the various components of the past performance information were fundamentally unreasonable. Shaw'Parsons maintains that to be consistent with our prior decision sustaining its protest, FEMA should have given the PPQs greater weight as compared to the information contained in the firms' SF330s. In addition, Shaw-Parsons argues that FEMA should have given more weight to the PPQ chart ratings than it did since the chart ratings formed the basis for the narrative comments. Shaw-Parsons asserts that, had the agency conducted an integrated assessment, one which considered the SF330 information and PPQs in their proper context, it would have received an overall Superior rating for the past performance factor.

The record reflects that the SEB specifically reevaluated all the firms' SF330 information--identifying strengths, as well as those areas where the agency could not assess a firm's performance due to a lack of information. The SEB also reviewed the PPQs, which included looking behind the adjectival chart ratings in an effort to discern any narrative comments relating to the firms' qualitative performance and noting where the PPQs did not shed light on their performance under the six SSN performance categories. FEMA's evaluation team also considered the firms' qualitative chart ratings, although giving the ratings less weight in the evaluation since they did not precisely correspond to the various past performance evaluation categories set forth in the SSN.

Notwithstanding Shaw-Parsons' belief that the process used by FEMA to structure its reevaluation yielded unreliable evaluation results, the record reflects that the agency did in fact consider all the past performance information for all firms as part of its reevaluation, and, with the exception of the matter discussed above in connection with the agency's evaluation of Vanguard's past performance information, FEMA's reevaluation was not, on the whole, unreasonable.

In reevaluating Shaw-Parsons' past performance as acceptable overall, FEMA found that Shaw-Parsons' SF330 lacked sufficient information to assess its performance for all but one past performance category (accuracy of cost estimates) established by the SSN. The SEB's assessment in this regard was reasonably based where the record reflects that the information in its SF330 was limited to general comments from client references. See Shaw-Parsons' SF330, at 56-58. In addition, the SEB determined that for three areas (accuracy of cost estimates, quality control, and completion of projects within budget), Shaw-Parsons' PPQs also did not include narrative information detailing the quality of its performance and one PPQ raised a concern regarding Shaw-Parsons' cost control, noting Shaw-Parsons' nominal cost controls and indicating that Shaw-Parsons could have done more to reduce costs under the contract. While Shaw-Parsons takes issue with the agency's assessments, pointing to numerous statements in its PPQs, which it believes demonstrate that it deserved a Superior rating, we have reviewed all of them and have no basis to conclude that the agency's evaluation was unreasonable.

By way of example, regarding quality control, Shaw-Parsons argues that the superior quality of its performance was reflected in statements indicting that it has provided "superior services and products," that it "has consistently met and exceeded the expectations of the [agency]," that its "work was of the highest quality," and that it "was very conscientious at making sure only eligible debris was removed, thus protecting [the City's] ability to claim reimbursements." Shaw-Parsons' United States Postal Service Contract and East Baton Rouge Debris Monitoring Contract PPQs. The record reflects that the agency did not attribute weight to such general performance statements. While Shaw-Parsons takes the agency to task for the narrowness of its past performance assessment, there was nothing inherently unreasonable with its decision not to credit Shaw-Parsons with strengths for comments of such a general nature.

Similarly, Shaw-Parsons contends that one of its cost control strengths should have been considered as reflecting superior performance on two other areas as well (accuracy of its costs estimates and its completion of projects within budget). Specifically, Shaw-Parsons points to a narrative comment justifying Shaw-Parsons' superior rating for cost control, which noted how Shaw-Parsons' actual billable cost for monitoring debris removal was only $10.3 million, yet the initial estimated cost was approximately $14 million. First, on its face, we fail to see how this discrepancy between the estimated and final cost reflects superior cost estimating ability. Second, the agency reasonably captured this strength under the factor that it believed was most appropriate given that the comment was drafted in support of the cost control rating and the narrative did not directly address any of the other SSN categories. To the extent Shaw-Parsons argues that FEMA unfairly gave NISTAC credit for essentially the same strength under two categories--accuracy of cost estimates and cost control--the record reflects that the narrative comments at issue, which were written in support of a Superior rating for NISTAC's cost control, directly address both aspects of NISTAC's performance; specially, they highlight NISTAC's ability to "properly estimate costs" through its use of "value engineering, careful project oversight and knowledge of construction trends" and details NISTAC's ability to effectively manage resources, thereby allowing the agency to stay within its budget. See NISTAC, U.S. Department of Labor, PPQ.

Regarding the question of whether FEMA gave reasonable weight to the past performance information in its possession, Shaw-Parsons argues that giving equal weight to the firms' SF330 information and the information contained in their PPQs was contrary to our prior decision sustaining its protest, which noted that "[c]ommon sense dictates that an offeror's self-assessment regarding the quality of its past performance, as reflected in its SF330, is, by its nature, of less value as compared to disinterested assessments provided by third parties." Shaw-Parsons Infrastructure Recovery Consultants, LLC; Vanguard Recovery Assistance Joint Venture, supra, at 7. This statement must be understood in its proper context. In sustaining Shaw-Parsons' protest, we found fault with the agency's evaluation to the extent it did not give any consideration to the PPQs it received, instead basing its past performance assessment entirely on the firms' SF330 information. We did not address the relative weight to be given this information as part of any reevaluation. Thus, the statement upon which Shaw'Parsons relies merely refuted FEMA's contention that it acted reasonably in deciding not to give any consideration to the PPQs, which it had received. To the extent our decision suggested that it would have been reasonable to give greater weight to the performance assessments of disinterested third parties as set forth in the PPQs, it does not stand for the proposition that it is unreasonable to afford the information contained in the firms' SF330s and their PPQs equal weight, as FEMA did in its reevaluation. Rather, given the agency's discretion in conducting its evaluation, we have no basis to conclude that the agency's methodology in this regard was improper.

We also reject Shaw-Parsons' contention that FEMA should have given greater weight to the PPQ chart ratings, as compared to the narrative comments in the PPQs, since they formed the basis for the narrative comments, and the related argument that FEMA gave the PPQ narrative comments too much weight by considering them outside the context of their underlying PPQ chart ratings. According to Shaw'Parsons, given that the narrative comments in its PPQs were nearly all in support of Superior ratings, it should have received a Superior rather than an Acceptable rating for its narrative information. Shaw-Parsons asserts that using FEMA's approach, a firm with extensive narrative comments in support of all Acceptable PPQ chart ratings could have achieved a higher rating than a firm which received all Superior chart ratings but no narrative comments. Shaw-Parsons maintains that this hypothetical situation is in fact reflected in one instance where the SEB gave CCPRS a strength for a narrative comment that was written in support of one of its Acceptable ratings.

In our view, FEMA had reasonable concerns regarding the utility of the adjectival chart ratings in evaluating past performance since the charts contained only four rating categories, which did not directly correspond to the six areas for evaluation established by the SSN past performance categories. Nevertheless, in an effort to give full and meaningful consideration to the firms' PPQ information, FEMA's evaluation team evaluated the underlying narrative comments, discerning where they in fact related to the six categories, and thus gave greater consideration to this directly relevant information when assessing the quality of each firms' performance. We see nothing inherently unreasonable with a process which focuses on the substantive comments regarding a firm's performance, as opposed to bare adjectival ratings in a chart without any context.

Moreover, the hypothetical situation identified by Shaw-Parsons did not in fact occur and thus does not support Shaw-Parsons' contention that the agency's methodology was unreasonable. To the extent Shaw-Parsons complains that the SEB assigned CCPRS a strength for a narrative comment in a PPQ which was in support of an Acceptable rating, such a finding is not inherently unreasonable since, as previously explained, the agency reasonably gave the substantive comments more significance in its evaluation.[12] Similarly, we see nothing unreasonable with Shaw-Parsons' overall Acceptable rating for the narrative information in its PPQs, notwithstanding its almost universal Superior PPQ chart ratings. The record shows that, in looking behind the chart ratings, the SEB determined that one of the evaluation comments reflected a weakness and that the comments provided were relevant to only three of the six SSN past performance evaluation categories. Given this record, we have no basis to conclude that the agency's evaluation was unreasonable.

As a final matter, Shaw-Parsons argues that the agency's decision to select NISTAC for the fourth award was not documented; the decision failed to account for Shaw'Parsons' superior past performance as compared to NISTAC's past performance; and that the agency improperly used the evaluation of subfactor 1A in selecting NISTAC as the fourth contract awardee.

While it is true that the SSA did not draft a new selection decision, the SSA continued to rely on the prior decision since the overall past performance ratings did not change, and because the past performance factor was the least important factor, there would have been no change in the resulting award decision. In the prior award decision, the SSA, unlike the SEB, ranked firms for the fourth position on a consideration of all evaluation factors, not just subfactor 1A.[13] See SSA Declaration, July 27, 2010. In this decision, the SSA concluded that there were no meaningful differences between firms under the past performance factor and that NISTAC was superior to Shaw'Parsons under factors 1, 2, and 5. Given that NISTAC and Shaw-Parsons continued to be tied adjectivally after the reevaluation, and, under the terms of the solicitation, factors 1, 2, and 5 were more important than the past performance factor, the SSA reasonably concluded that there was no need to again look behind the ratings. Notwithstanding their equivalent adjectival ratings, any difference between NISTAC and Shaw-Parsons under the past performance factor would have been overtaken by NISTAC's advantages on the three other more important factors, which the SSA had previously documented and, which were not affected by the agency's reevaluation. Thus, Shaw-Parsons' challenge to the agency's selection decision is without merit.

RECOMMENDATION

We recommend that the agency, consistent with our decision, reevaluate Vanguard's past performance information. Based on that reevaluation, we recommend that the agency make a new source selection determination. We also recommend that the agency reimburse Vanguard the costs of filing and pursuing the protest, including attorneys' fees. Bid Protest Regulations, 4 C.F.R. sect. 21.8(d)(1). In accordance with section 21.8(f) of our Regulations, Vanguard's claim for such costs, detailing the time expended and the costs incurred, must be submitted directly to the agency within 60 days after receipt of the decision.

Vanguard's protest is sustained; Shaw-Parsons' protest is denied.

Lynn H. Gibson
Acting General Counsel