IBM-U.S. Federal

DOCUMENT FOR PUBLIC RELEASE
The decision issued on the date below was subject to a GAO Protective Order. This redacted version has been approved for public release.

Decision

Matter of: IBM-U.S. Federal

File: B-407073.3; B-407073.4; B-407073.5; B-407073.6

Date: June 6, 2013

DIGEST

1. Protest is sustained where agency’s evaluation of protester’s price under one of the solicitation’s price scenarios was not calculated in such a way as to result in evaluation on a common basis.

2. Protest is sustained where, during post-selection negotiations, which were anticipated by the solicitation, the agency materially relaxed a solicitation term regarding software certification; relaxing this term for only the awardee resulted in a failure to treat offerors equally.

3. Protester’s contention that the agency’s evaluation of the past performance of a commercial cloud services provider improperly ignored information from news reports of service outages is denied where the evaluators were aware of the reports, but concluded that the reports lacked sufficient detail concerning the offeror’s adherence to service level agreements to justify downgrading the offeror’s past performance rating.

IBM U.S. Federal (IBM), of Bethesda, Maryland, protests the Central Intelligence Agency’s (CIA) award of a contract to Amazon Web Services, Inc., of Seattle, Washington, under request for proposals (RFP) No. 2012-12041000001, for commercial cloud services (C2S). IBM challenges the evaluation of proposals and the selection decision.

We sustain the protest in part and deny it in part.

BACKGROUND

The RFP contemplated the award, on a “best value” basis, of a single fixed-price indefinite-delivery/indefinite-quantity (ID/IQ) contract for commercially-managed cloud computing services for the Intelligence Community to be provided through task orders. The contemplated ID/IQ contract was to include a 270-day period to achieve initial operating capability, a 4-year base ordering period, a 3-year option, and a 2-year option, with a maximum value of $600 million over the base period.

According to the Federal Cloud Computing Strategy, “cloud computing describes a broad movement to treat [information technology (IT)] services as a commodity with the ability to dynamically increase or decrease capacity to match user needs.” Statement of Objectives at 1. In this regard, cloud computing delivers computing services via networks with the potential to provide IT services more quickly and at a lower cost. Cloud computing provides users with on-demand access to a shared and scalable pool of computing resources with minimal management effort or service provider interaction. Information Technology Reform: Progress Made but Future Cloud Computing Efforts Should be Better Planned, GAO-12-756 at 1 (July 2012). By leveraging shared infrastructure and economies of scale, cloud computing allows users to control the computing services they access, while sharing the investment in the underlying IT resources among consumers. Statement of Objectives at 1.

Here, the contractor generally was to provide a copy of its existing public cloud (modified where necessary) to be installed on government premises and operated by the provider. Performance is to include both: (1) infrastructure as a service, in which the provider (contractor) is responsible for networking, storage, servers and virtualization, and the consumers (government agencies) are responsible for the operating system, middleware, runtime, data, and applications, System Requirements Document (SRD) Fig. 2-2; and (2) platform as a service, which encompasses the same components as infrastructure as a service, but only applications and data are the responsibility of the consumer.[1] Id.

In addition, the required intelligence community cloud was to provide for auto-scaling, SRD § 3.9.1, that is, a mechanism that allows computing infrastructure to automatically scale up or down the number of computer resources that are being used to support an application, based on the needs of that application at a specific time. Contracting Officer’s Statement (COS) at 8. Without this capacity, customers must anticipate when surge capacity is required and provision it in advance. Id. Further, of significance for this protest, the solicitation provided that the intelligence community cloud system (for both infrastructure as a service and platform as a service) was to permit consumers (here, the intelligence community agencies) to run their own applications (those not provided by the cloud provider) on the cloud, and the cloud was to provide “scalable” solutions, with respect to applications, including those provided by the consumer. See SRD §§ 3.4.11, 3.9.1.

The solicitation provided for an initial mandatory qualification evaluation to verify that an offeror was an established commercial cloud service provider with an existing, large-scale public offering. This was to be followed by the evaluation of proposals under four factors: (1) technical/management, including subfactors for technical approach (evaluated under a demonstration/oral presentation element and a written element), service level agreements, and management; (2) past performance; (3) security, evaluated on a pass/fail basis; and (4) price.

Price was to be evaluated for completeness, reasonableness, and to determine the offeror’s understanding of the government’s requirements. Evaluation Criteria at 5. Offerors were required to submit a fixed price for task order 1 for program management to achieve initial operating capability, and a guaranteed minimum price for task order 2 for the provision of cloud services for the first year after initial operating capability. In addition, offerors were to provide a Cloud Services Catalog Price List containing fixed prices for various services. Also, the solicitation set forth six representative service-type ordering scenarios; using their proposed catalog prices, offerors were to calculate the total costs of orders for each scenario, on a yearly basis, for the base performance period. The total evaluated price was to be comprised of the sum of the task order 1 pricing and the prices for the six ordering scenarios for the base period. Price Instructions at 1; Evaluation Criteria at 5.

Five offerors, including IBM and Amazon, submitted proposals by the July 13, 2012, closing time. Shortly thereafter, two offerors (AT&T Corporation and Microsoft Corporation) filed protests with our Office challenging various aspects of the mandatory qualification requirements, and a third offeror withdrew its proposal. When the agency elected to take corrective action by amending the RFP to remove the qualification language challenged in the protests, we dismissed both protests as academic (B-407073, B-473073.2, Aug. 17, 2012).

Based upon the subsequent demonstrations/oral presentations and the evaluation of proposals by the technical/management evaluation team, past performance evaluation team, and price evaluation team, the agency decided to include the proposals of IBM, Amazon, and a third offeror in the competitive range. After discussions with the competitive range offerors, the agency requested submission of final proposal revisions (FPR).

In its evaluation of FPRs, the technical/management evaluation team assigned a deficiency to IBM’s proposal under the demonstration element (within the technical approach subfactor) as a result of language in the firm’s FPR that appeared to indicate that its existing public cloud could not provide auto-scaling for applications provided by the consumer. The technical/management evaluation team also assigned a significant weakness under the written element of the technical approach subfactor for IBM’s failure to make clear in its proposal how IBM would modify its existing commercial cloud to provide the required auto-scaling capability for the intelligence community.

In addition, because the offerors appeared to have adopted materially different interpretations of the scenario requirements, the price evaluation team adjusted offerors’ prices (both up and down) using each offeror’s proposed catalog prices in order to “provide a common basis for comparing price.” Report on Final Cost Proposals at 5. The agency’s adjustment of IBM’s FPR scenario 5 price increased IBM’s price for this scenario from approximately [deleted] to approximately [deleted].

The final consensus evaluation for IBM and Amazon was as follows:

Source Selection Decision (SSD) at 1.

Based upon the above evaluation results, the source selection authority (SSA) concluded that Amazon’s proposal represented the best value. In this regard, while IBM’s proposal offered an evaluated [deleted] price advantage over 5 years, the SSA concluded that this advantage was offset by Amazon’s superior technical solution. SSD at 8.

Prior to contract award, and in accordance with the terms of the RFP, the agency conducted post-selection negotiations with Amazon to address certain matters in its proposal. RFP at 47. Upon learning of the ultimate award to Amazon and after receiving a debriefing, IBM filed this protest.

DISCUSSION

IBM raises numerous challenges to the evaluation of its technical and price proposals, the conduct of discussions (including the post-selection negotiations with Amazon), and the evaluation of Amazon’s past performance. Based upon our review of the record, we find for IBM on its challenges to the agency’s price evaluation under one of the solicitation’s price scenarios, and the conduct of post-selection negotiations. For these reasons we sustain the protest. We deny IBM’s challenges to the agency’s evaluation of Amazon under the past performance evaluation factor. We have considered all of IBM’s arguments and discuss the most significant below.

Scenario 5 Evaluated Price

As set forth above, offerors were required to address six notional scenarios. Scenario 5, for data analytics, was identified as platform as a service (everything but applications and data are the responsibility of the contractor), and included the following description:

This scenario centers around providing a hosting environment for applications which process vast amounts of information in [parallel] on large clusters (1000s of nodes) of commodity hardware in a reliable, fault-tolerant manner (MapReduce). The solution to this scenario should automatically provision clusters of compute for the segmentation and [parallel] processing of input datasets via the MapReduce framework (3.4.1) where the vendor is responsible for the management of the OS [operating system] and MapReduce implementation. Assume a cluster large enough to process 100TB [terabytes] of raw input data. Assume input data set was loaded from available object-based storage that realizes 6 reads/second and 2 writes/second. Assume 100% duty cycle on all virtual machines associated with this scenario.[ [2]] (Total storage for each order shall be 100 TB object. Read/writes and IOPS [input/output operations per second] should be calculated per order).

RFP, amend. 4, Scenario 5 Description (emphasis added).

Pointing to the solicitation reference to a “100% duty cycle” and the pricing instruction’s request for yearly prices, the agency explains, in its report and at the hearing our Office conducted in this matter, that it was requesting under scenario 5 the prices for simultaneously running a large number of orders, each consisting of 100 TB of data, repeatedly throughout the year. COS at 28-29; Hearing Transcript (Tr.) 269-71. Because IBM’s FPR price was based on a single run processing 100 TB of data, multiplied by the specified number of orders for the period (30 for the first year), the agency used IBM’s proposed catalog pricing and the services identified in its FPR for this scenario, to adjust the firm’s price to represent 1 year of continual processing for each order.

IBM asserts that the agency’s adjustment of its scenario 5 price was unreasonable. In this regard, IBM maintains that the agency imposed an unstated requirement that each 100 TB data analytics unit provided under this scenario was to be continually repeated throughout the entire year. In contrast, under the protester’s interpretation (as reflected in its FPR), it was only required to propose a solution capable of processing a single 100 TB data run with no specified duration. IBM First Supplemental Protest at 40. IBM explains that it therefore proposed an “optimal” price for a level of services (e.g., computer capacity) with a processing time of approximately [deleted], which represented a single run of 100 TB of data. Tr. at 410.

As an initial matter, we find untimely IBM’s challenge to the agency’s interpretation that scenario 5 called for repeated 100 TB data runs throughout the year, rather than a single run under each order. In this regard, prior to submitting its initial proposal, IBM asked two questions seeking additional information about the scenario’s technical parameters. Specifically, IBM first asked:

Are orders the number of new images of that scenario type that are loaded to the image store/catalog in the year or are they the number of instantiations/runs of that scenario type in a year?

Question/Answer No. 49. The agency responded that “[a]s outlined in the scenario, the servers should be treated as operating on 100% duty cycle and should be priced out as simultaneous orders.” Id.

IBM’s second question was more detailed, requesting detailed information concerning the scenario requirements. Among other things, IBM’s second question asked if concurrent data analytics users would be running jobs, what would be the anticipated average number of runs of each scenario type daily, monthly, etc., and how many analytic jobs were expected to be concurrently executing. Question/Answer No. 50. In response, the agency repeated its answer to Question No. 49, and added:

The contractor should propose commercial best practices derived from their commercially available solution(s) to provide data analytics via the MapReduce software framework to concurrent users from multiple organizations.

Id.

Without further inquiry, IBM initially priced scenario 5 at approximately [deleted], based on the specified number of orders, with 12 months of availability. Thus, IBM’s initial approach, consistent with the manner in which all other offerors in the competitive range prepared their pricing, appeared to be based upon continual 100 TB data runs throughout the year. COS at 32-33; IBM Initial Proposal at VI-17 (proposal of services at 12 months each); Tr. at 428-29 (“7/24/365” capacity approach). However, in its FPR, IBM (but not the other offerors) reduced its scenario 5 price to approximately [deleted], based on a single run for each of the specified number of orders.

Under our Bid Protest Regulations, a solicitation defect apparent on the face of the solicitation must be protested prior to the time set for receipt of initial proposals or quotations, when it is most practicable to take effective action against such defects. 4 C.F.R. § 21.2(a)(1) (2013). Furthermore, an offeror who chooses to compete under a patently ambiguous solicitation does so at its own peril, and cannot later complain when the agency proceeds in a way inconsistent with one of the possible interpretations. Wackenhut Servs., Inc., B-276012.2, Sept. 1, 1998, 98-2 CPD ¶ 75 at 5; CardioMetrix, B-274585, Nov. 18, 1996, 96-2 CPD ¶ 190 at 3; Watchdog, Inc., B-258671, Feb. 13, 1995, 95-1 CPD ¶ 69 at 5.

Here, by its own actions, IBM evidenced its recognition that the scenario 5 instructions were ambiguous as to the frequency of the expected 100 TB data runs. IBM requested clarification of the requirements in this regard and then, not having received meaningful clarification, first adopted one interpretation (that is, continual runs) in its initial proposal and then a different interpretation (a single run per order) in its FPR. Having chosen to compete despite its recognition of the patently ambiguous nature of the solicitation in this area, IBM cannot now complain when the agency proceeds in a manner inconsistent with one of the possible interpretations and adjusts IBM’s price to match the government’s (and Amazon’s apparent) interpretation of the requirement. See, e.g., Wackenhut Servs., Inc., supra.

That said, while we view as untimely IBM’s challenge to the agency’s interpretation that continual runs (100% duty cycle for 1 year) were required under scenario 5, its challenge to the agency’s calculation of an evaluated price for scenario 5 is not untimely.

In this regard, our Office will review the reasonableness or consistency of the evaluation method the agency employs. See Federal Computer Int’l Corp., B-276885, July 29, 1997, 97-2 CPD ¶ 35 at 3. In meeting the requirement to consider cost or price to the government in evaluating competitive proposals, 41 U.S.C. § 3306(c)(1)(B) (2011), an agency’s chosen method of evaluation must include some reasonable, common basis for evaluating or comparing the relative costs of proposals. See Aalco Forwarding, Inc., et al., B-277241.15, Mar. 11, 1998, 98-1 CPD ¶ 87 at 11. Ordinarily, normalization--which is the term used by the parties here--involves the measurement of offerors’ costs against the same baseline where there is no logical basis for differences in approach or where there is insufficient information provided with the proposals, leading to the establishment of common estimates by the agency. See Bendix Field Eng’g Corp., B-246236, Feb. 25, 1992, 92-1 CPD ¶ 227 at 17.

Here, the record indicates that the agency lacked sufficient information to ensure that proposals were evaluated on a common basis with respect to scenario 5. In this regard, the agency attempted to ensure that offerors were evaluated on a common basis with regard to scenario 5 by scaling IBM’s single-run price upwards to equal one year of continual processing for each order. Generally, this resulted in assuming that each IBM 100 TB data run, indicated by IBM to take [deleted], would be repeated approximately [deleted] times in a year. Tr. at 361, 375.

However, while Amazon’s scenario 5 price indicated that it, like every other offeror besides IBM, assumed repeated 100 TB data runs throughout the year, an agency adviser to the price evaluation team (and author of scenario 5) testified that based on the information in the proposal, the agency did not know how long a single 100 TB data run would take using Amazon’s proposed solution. Tr. at 314, 378. Specifically, Amazon’s proposal made various assumptions about running 100 TB data sets at “100% duty cycle” and “leveraging [its] [deleted],” and “priced [its] [deleted]” instance type for 100% of the time, but it did not propose a specific time for a single 100 TB data run. Amazon Proposal at VI-32-33. Thus, there was no way from the information available concerning Amazon’s solution to ascertain how many 100 TB data runs were included in Amazon’s scenario 5 pricing.

Further, the agency recognized that solutions (and prices) could vary based on the makeup of virtual machines--the amount of locally available storage, the amount of memory, the amount of virtual compute cores proposed--and how fast a unit processes 100 TB of data, all of which would have an effect on performance, including how many times a solution could process that amount of information in a given period. Tr. at 344, 347. In this regard, while Amazon’s proposed scenario 5 solution included [deleted] compute units or [deleted] virtual cores (i.e., divisions within the virtual computer available for separately processing information without interfering with one another), IBM’s solution included [deleted] virtual cores; [deleted]. Tr. at 386-88.

In sum, given the agency’s uncertainty regarding just what performance (e.g., number of 100 TB data runs) was included in each evaluated price, there is no basis for concluding that Amazon was evaluated for scenario 5 using the same or otherwise comparable level of performance as included in IBM’s adjusted price. Thus, we find the price evaluation to be unreasonable and we sustain the protest on this basis.

Post-Selection Negotiations

The RFP included the following provision regarding software certification:

The Contractor certifies that it will undertake to ensure that any software to be provided . . . under this contract will be provided . . . free from computer virus, which could damage, destroy, or maliciously alter software, firmware, or hardware, or which could reveal to unauthorized persons any data or other information accessed through or processed by the software.

RFP Commercial Clause § 152.204-706(a). In its post-selection negotiations with Amazon, Amazon proposed, and the agency accepted, modifying the above solicitation clause as follows: “The Sponsor agrees that ‘only software developed and provided by [Amazon] would be subject to this requirement.’” Memorandum of Understanding, Feb. 12, 2013, at 3.

IBM asserts that in agreeing to this modification, the agency relaxed a material requirement of the solicitation. The agency responds that post-selection negotiations were contemplated by the RFP, RFP at 47 (“The Government intends to select for final negotiations a contractor(s) resulting from this solicitation whose proposal represents the best value . . . .”), and that this modification does not represent a material change to the RFP’s requirements or Amazon’s obligations. We disagree.

It is a fundamental principle of government procurement that competition must be conducted on an equal basis; that is, offerors must be treated equally and provided with a common basis for the preparation of their proposals. Systems Mgmt., Inc.; Qualimetrics, Inc., B-287032.3, B-287032.4, Apr. 16, 2001, 2001 CPD ¶ 85 at 8. When, either before or after receipt of proposals, the government changes or relaxes its requirements, it must issue an amendment to notify all offerors of the changed requirements and give them an opportunity to respond. Diebold, Inc., B-404823, June 2, 2011, 2011 CPD ¶ 117 at 4; Systems Mgmt., Inc.; Qualimetrics, Inc., supra; see Cardkey Sys., B-220660, Feb. 11, 1986, 86-1 CPD ¶ 154 at 2 (If it becomes apparent that the contract being negotiated differs significantly from the requirements stated in the RFP, the contracting agency must amend the RFP or, at the least, advise offerors of the change during discussions and seek new offers.) We will sustain a protest where an agency, without issuing a written amendment, materially alters the solicitation’s requirements to the protester’s prejudice. See Systems Mgmt., Inc.; Qualimetrics, Inc., supra.

Here, the record indicates that the modification of the RFP clause to the negotiated language materially reduced Amazon’s obligations from those imposed in the RFP. In this regard, the RFP clause required the offeror to certify its undertaking to ensure that “any software” provided will be virus free, while the modified language covered “only software developed and provided” by Amazon. This is significant because Amazon’s proposal contemplated the provision of third party and open-source software that is not developed by Amazon and thus would not fall within its modified certification. See, e.g., Amazon Proposal at 1-4,1-9 (“rich ecosystem of . . . third-party security tools”), and at 1-13, 1-23, 1-26, 2-38 (proposed use of “MySQL” and “Red Hat Enterprise Linux,” an open-source database and software product, respectively).

During the hearing before our Office, the agency admitted that, at the time of the post-selection negotiations, it did not consider the impact of the modified language; it understood that Amazon was certifying all software and did not realize that the change might be a restriction on the certification. Tr. at 486-88. Indeed, the contracting officer testified that had the agency focused more closely on the modification, it would have taken exception to the revision. Tr. at 502-03. Thus, the hearing testimony suggested that the new language represented a material change to the terms of the underlying competition.

Both at the hearing and in briefs filed after the hearing, however, the agency asserts that this change was not significant because Amazon is still required to comply with other provisions of its contract. For example, the agency points out that SRD § 3.13.2 required C2S infrastructure and services to be “assessed and authorized in accordance with Intelligence Community Directive (ICD) 503 (reference 3),” which provides for the agency to incorporate standards, policies, and guidelines issued by the National Institute of Standards and Technology and the Committee on National Security Systems (CNSS). Agency Post Hearing Comments at 20. One aspect of these standards includes “malicious code protection,” and Amazon’s detailed proposal to meet these requirements was incorporated into the contract. Tr. at 470-72; Amazon System Security Plan, Appendix B-91--B-93.

We see no basis in the contemporaneous record to support a conclusion that the agency made its decision to allow the modification based on these other provisions of the solicitation, or on Amazon’s agreement to them.[3] In this regard, while we consider the entire record in resolving a protest, including statements and arguments in response to a protest, in determining whether an agency’s selection decision is supportable, we accord greater weight to contemporaneous evaluation and source selection materials than to new judgments made in response to protest contentions. Boeing Sikorsky Aircraft Support, B-277263.2, B-277263.3, Sept. 29, 1997, 97-2 CPD ¶ 91 at 15. We accord lesser weight to post hoc arguments or analyses because we are concerned that judgments made in the heat of an adversarial process may not represent the fair and considered judgment of the agency, which is a prerequisite of a rational evaluation and source selection process. Id. Here, given the contracting officer’s testimony that, had the agency focused on the modification requested by Amazon, it would have taken exception, we find no basis for accepting the agency’s current assertion that the modification was not material.

In any case, the agency’s current position does not show that the modification was not material. In this regard, the agency now contends that the only aspects not covered by Amazon’s agreement to the certification requirements were (1) the certification itself, which the agency believes would be satisfied by Amazon signing its contract; and (2) the requirement to immediately notify the contracting officer of an issue with malicious code. Agency Post Hearing Comments at 20. However, regardless of the agency’s position with respect to the certification, it is unclear why elimination of Amazon’s obligation to immediately notify the contracting officer about malicious code would not represent a change in a material requirement.[4]

In sum, we find that the agency, without issuing a written amendment, materially relaxed the solicitation’s requirements for Amazon without affording the other offerors an opportunity to propose to the modified requirements. Accordingly, the protest is sustained on this basis as well.

Auto-Scaling Evaluation

During the agency’s final evaluation, the evaluators assigned IBM’s proposal a deficiency under the demonstration factor because they found that IBM’s FPR introduced new language that cast doubt on whether the firm’s existing public cloud provided auto-scaling for all applications, both IBM-supplied and agency consumer-supplied. COS at 9. While the evaluators found that IBM’s proposed C2S cloud for the intelligence community included auto-scaling capability for all applications, they also assigned a significant weakness because there was a lack of detail on how this additional capability (relative to the existing public cloud) would be accomplished. Id. at 13. IBM asserts that the agency’s evaluation was irrational and that its proposal made clear that its public cloud provided auto-scaling capability for all applications whether supplied by IBM or agency consumers.

In considering protests challenging an agency’s evaluation of proposals, we will not reevaluate proposals; rather, we will examine the record to determine whether the agency’s evaluation conclusions were reasonable and consistent with the terms of the solicitation and applicable procurement laws and regulations. Gonzales-Stoller Remediation Servs., LLC, B-406183.2 et al., Mar. 2, 2012, 2012 CPD ¶ 134 at 5. A protester’s disagreement with a procuring agency’s judgment is insufficient to establish that the agency acted unreasonably. Sig Sauer, Inc., B-402339.3, July 23, 2010, 2010 CPD ¶ 184 at 6.

Here, the proposal instructions required each offeror to “clearly demonstrate its understanding of the technical requirements and suitability of the proposed technical approaches that will be employed.” Evaluation Criteria at 2. Moreover, it is an offeror’s obligation to submit an adequately written proposal for the agency to evaluate. See Independence Constr., Inc., B-292052, May 19, 2003, 2003 CPD ¶ 105 at 5.

Our review of the record shows that the RFP clearly required offerors to demonstrate the ability to auto-scale computer resources both in their existing public cloud and in the C2S cloud solution. Specifically, the cloud was to provide “dynamic workload management . . . across the cloud infrastructure,” SRD § 3.4.11, and “shall provide application services with . . . automated scaling.” SRD § 3.9.1. While IBM’s initial presentation demonstrated its auto-scale capability for applications from its existing public cloud structure, in its FPR, IBM revised its written proposal in support of the demonstration so as to indicate that its ability to auto-scale in its existing public cloud structure was limited to applications provided by IBM. In this regard, in referring to its capability to auto-scale, IBM’s FPR drew a distinction between [deleted]. IBM FPR at 1-25.[5] This distinction was continued in a figure from its FPR indicating that, [deleted].[6] Id. at 1-26. Thus, IBM’s FPR suggested that the ability to auto-scale in its existing public cloud structure was limited and did not apply to all services.

As explained by the agency, it had “grave” concerns about IBM’s auto-scaling capability when it considered these limiting references in conjunction with a subsequent statement in IBM’s FPR that an application was to be selected based on “catalog choices.” IBM FPR at 1-28; Tr. at 50-51. According to the agency, it interpreted this and other uses of “catalog” in IBM’s proposal to refer to the C2S service catalog, a listing of IBM’s fixed prices for cloud services. Supplemental COS at 3. While agency consumers could add their own applications to an overall “library” for their use (SRD § 3.4.13), the IBM service catalog could not be altered by agency consumers. Id. Thus, given the indications in IBM’s FPR that auto-scaling in its existing public cloud structure was limited to applications from IBM’s service catalog, and that consumer applications could not be added to that catalog by the consumer, the evaluators could not conclude that IBM’s existing public cloud structure, as described in its FPR, provided a mechanism through which an application furnished by an agency consumer could be auto-scaled. Final Technical Evaluation Report at 8.

In response to the agency’s arguments, the protester identifies a number of places in its FPR where it used the word catalog in a more generic sense, thus indicating that its reference to selecting applications from the “catalog” was not intended to mean the service catalog provided by IBM. For example, IBM’s FPR stated that the “[intelligence community] can build custom applications for inclusion in the image library or catalog.” FPR at 2-7, 2-8. In addition, at a later stage of the demonstration, where IBM showed its ability to add consumer applications to a library, the IBM presenter stated that “that image now exists in the catalogue.” Tr. at 113. IBM also notes that even the RFP uses the term catalog other than in the context of the services catalog. Specifically, it refers to the requirement that an offeror provide an image library, a repository for virtual images and VM templates “to be managed and selected from the catalog.” SRD § 3.4.13.

Our review of the record indicates that these various references suggest that IBM may in fact have auto-scaling capability for consumer applications. However, while the cited passages in IBM’s FPR established, e.g., that consumer applications can be added to the “library,” and that “applications” can be auto-scaled in IBM’s existing public cloud structure, there is no place in the FPR where IBM clearly and affirmatively stated that its existing public cloud structure can auto-scale all applications, whether provided by IBM or the consumer. Tr. at 70, 220. Likewise, there is no clear statement in the FPR defining what the applications catalog is as distinguished from the IBM-supplied service catalog.

We recognize that IBM’s proposed cloud for the intelligence community--i.e., the contract at issue here--promised the capability to auto-scale all applications. However, the above deficiency was assigned for failure to demonstrate the existence of this capability in IBM’s existing public cloud, a different element of the evaluation. Further, since we find the agency reasonably determined that IBM failed to clearly establish the capability of its existing public cloud to auto-scale all applications, there is no basis to find unreasonable the agency’s assignment of a significant weakness, under a different evaluation element (the written element), as a result of IBM’s failure to explain how it would add the capability to auto-scale all applications in the cloud it proposes to provide for the intelligence community.

We see no basis to question the reasonableness of the agency’s concerns (expressed as a deficiency and a significant weakness under the technical approach subfactor) that IBM’s proposal failed to adequately explain its auto-scaling capabilities. Accordingly, this aspect of the protest is denied.

Past Performance Evaluation

IBM also argues that the past performance evaluation of Amazon’s proposal was flawed because the agency failed to take into account news reports about five service outages experienced by some of Amazon’s public cloud clients between April and December 2012, while the agency was conducting this procurement.[7] IBM First Supplemental Protest at 57. IBM contends that this information was relevant to the past performance evaluation, noting that the second most important element under the past performance factor was whether an offeror’s “performance history demonstrates success in meeting management, delivery, and performance requirements within the terms of a Service Level Agreement [SLA].” Evaluation Criteria at 3-4. IBM argues that had the agency considered this information, Amazon’s past performance rating would have been lower than high confidence.

Our Office has recognized that in certain limited circumstances an agency evaluating an offeror’s past performance has an obligation (as opposed to the discretion) to consider outside information bearing on an offeror’s past performance. New Orleans Support Servs. LLC, B-404914, June 21, 2011 CPD ¶ 146 at 5. Where we have charged an agency with responsibility for considering such outside information, the record has demonstrated that the information in question was “too close at hand” to require offerors to shoulder the inequities that spring from an agency’s failure to obtain, and consider, the information. Id.; see e.g., International Bus. Sys., Inc., B-275554, Mar. 3, 1997, 97-1 CPD ¶ 114 at 5; G. Marine Diesel, B-232619.3, Aug. 3, 1989, 89-2 CPD 101 at 4-6 (contracting officer who was personally aware of the awardee’s continuing difficulties in performing a contract for services related to the subject solicitation, and who considered those difficulties in deciding not to exercise the remaining options, erred in not considering the difficulties when evaluating proposals).

The record here shows that the contracting officer, the SSA, and other members of the evaluation team, were aware of the news reports regarding the service outages, Addendum to Supplemental Agency Report at 1, but concluded that the reports and their knowledge of the circumstances surrounding the outages were not sufficiently detailed to take the outages into account in the past performance evaluation. Id.; Contracting Officer’s Declaration, Apr. 25, 2013; SSA Declaration, Apr. 26, 2013. For example, as noted by the SSA, none of the cited news reports addressed Amazon’s success in meeting the terms of its SLAs with the affected customers. SSA Declaration, Apr. 26, 2013.

In addition, the record shows that one of Amazon’s past performance references acknowledged outages in Amazon’s system. Specifically, the National Aeronautics and Space Administration, Jet Propulsion Laboratory (JPL), rated Amazon as very good for meeting SLAs, despite reported outages, commenting:

[Amazon] has experienced some outages, especially in their Virginia region. However, this has had no impact on JPL, and as a percentage of the overall availability, these outages are miniscule. All cloud vendors have outages. All private clouds and private data centers have outages. The difference is that we hear about them when they happen in a public cloud, such as [Amazon]. JPL operates mission critical services in the [Amazon] cloud and we have had 100% uptime since inception. To achieve this, we implement failover and elastic load balancing but it’s simple and inexpensive and very much worth it.

JPL Past Performance/Questionnaire at 3.

In conclusion, we see no basis to find that the agency improperly evaluated Amazon’s proposal under the past performance factor.

Evaluation of Guaranteed Minimum Prices

IBM asserts that the agency improperly considered the effect of its guaranteed minimum price on its total evaluated price. In this regard, IBM relies on the RFP language that the guaranteed minimum would not be used as part of the total evaluated price. Evaluation Criteria at 5. IBM’s assertion is without merit.

The guaranteed minimum price feature of this solicitation was intended to allow the offerors to recover a portion of the upfront infrastructure expenditures and to reduce their cost risk. Source Selection Evaluation Board (SSEB) Recommendation at 19. The RFP specifically provided that the guaranteed minimum proposed by each offeror would be evaluated for completeness and reasonableness and would “be used within the Government’s overall risk rating involved with this acquisition.” Evaluation Criteria at 5. The SSEB, in its discussion of the offerors’ guaranteed minimum prices, addressed the risk involved with IBM’s minimum by noting that [deleted]. SSEB Report at 20. The SSEB also found “serious cost risk” associated with IBM’s proposed contract clause which indicated that the contractor would [deleted]. Id. The SSEB observed that if the guaranteed minimum was “considered as part of the likely cost,” it would reduce the price difference between IBM and Amazon. Id. The SSA made similar observations in her source selection decision. SSD at 9.

In our view, the evaluation in this regard was not inconsistent with the RFP’s evaluation criteria. The record shows that IBM’s evaluated price was not increased by the amount of its guaranteed minimum. See, e.g., SSD at 8. Rather, the SSEB’s evaluation and the SSA’s tradeoff analysis simply considered the risk involved in IBM’s guaranteed minimum and future position on negotiations. While IBM objects to consideration of the magnitude of its guaranteed minimum, clearly the magnitude of the guarantee was a factor in the extent of the resulting risk, which under the RFP was to be evaluated.

RECOMMENDATION

We sustain the protest because the agency’s adjustment of scenario prices was unreasonable in that it did not result in evaluation on a common basis, and because the agency materially relaxed a solicitation term for the awardee during post-selection negotiations. We recommend that the agency reopen the competition and amend the RFP as necessary to ensure that proposals are prepared and evaluated on a common basis, consistent with the issues discussed in this decision. We further recommend that the agency conduct discussions with offerors, obtain and evaluate revised proposals, and make a new source selection decision. Finally, we recommend that the protester be reimbursed its costs of filing and pursuing the protest, including reasonable attorneys’ fees. 4 C.F.R. § 21.8(d)(1).

The protester’s certified claim for costs, detailing the time expended and costs incurred, must be submitted to the agency within 60 days after receipt of this decision. 4 C.F.R. § 21.8(f)(1).

The protest is sustained.

Susan A. Poling
General Counsel