ERP Services, Inc.

Decision

Matter of:  ERP Services, Inc.

File:  B-419315

Date:  February 24, 2021

DIGEST

Protest that agency applied unstated evaluation criteria and unreasonably evaluated quotations is denied where the record reflects that the evaluation was reasonable and consistent with the terms of the solicitation.

ERP Services, Inc., of Sterling, Virginia, challenges the award of a contract to Akamai Technologies, Inc., of Cambridge, Massachusetts, under request for quotations (RFQ) HQ0034-20-R-0264 issued by the Department of Defense, Washington Headquarters Services, for domain name system (DNS) resolver services.  The protester alleges that the agency erred in its evaluation of quotations by applying unstated evaluation criteria.

We deny the protest.

BACKGROUND

On July 22, 2020, the agency issued the RFQ as a combined synopsis/solicitation for commercial services using the simplified acquisition procedures of Federal Acquisition Regulation (FAR) part 13.  Contracting Officer’s Statement and Memorandum of Law (COS/MOL) at 1-2.  The RFQ sought to procure DNS resolver[1] services, as well as a system to store and analyze various information logged by the DNS resolver, which the RFQ referred to as a “data lake.”  Id.  The RFQ contemplated the award of a single fixed price contract with a 1-year base period and two 1-year option periods.  Agency Report (AR) at 73-74.[2]

The solicitation provided for a three-phased competition, with each phase focusing on a different evaluation factor.  AR at 26-27.  In the first phase, vendors would submit written technical solutions that the agency would evaluate to assess whether they addressed the RFQ’s minimum requirements.  Id.  Vendors with solutions that met the RFQ’s minimum requirements would be invited to the second phase, during which the vendors would provide a live technical usability demonstration.  Id.  Finally, the agency would determine which vendors were “viable competitors” based on the first two phases, and invite those vendors to the third phase, in which they would submit pricing.  Id. at 27.

Relevant to this protest, the RFQ provided that, during the usability demonstration, vendors would need to demonstrate their ability to meet each of seven tasks with their proposed solution by “walking government evaluators through how each requirement would be completed using an in-production service, a prototype, or wireframes.”  AR at 26.  Additionally, the RFQ provided that the demonstration and solution would be evaluated “on how well [a] user can accomplish a given task in a reasonable amount of time and with minimal cognitive strain,” and “from a user-centered design/human factors perspective.”  Id. at 26-27.

The agency received seven written technical solution quotations in response to the RFQ, including quotations from the protester and intervenor.  COS/MOL at 3.  On the basis of its written technical solution, the protester was invited to the second phase to conduct a usability demonstration.  Id.  The demonstration was conducted by videoconference on September 17, 2020, and was recorded.  Id.  Following the presentation, the agency concluded that the protester did not propose a viable solution, primarily based on the agency’s review of ERP’s solution under various usability factors.  Id.  Accordingly, the agency did not invite the protester to submit pricing for phase three, and the protester was not considered in the agency’s best-value tradeoff.  Id.

The agency conducted a best-value tradeoff among vendors that reached phase three, and concluded that the intervenor’s quotation represented the best value to the government.  COS/MOL at 4.  On September 30, 2020, the agency notified the protester of the agency’s award decision, and the protester requested and received additional information from the agency.  Id.  This protest followed.

DISCUSSION

The protester challenges the agency’s conclusion that it was not a viable competitor, arguing the agency applied two unstated evaluation criteria when assessing ERP’s demonstration.  Protest at 6-12; Comments at 3-10.  Specifically, the protester disputes the agency’s negative judgments about the protester’s user interface color scheme and the use of a terminal or command-line interface[3].  According to ERP, the agency’s concerns are not related to any stated evaluation criterion.[4]  Id.

The evaluation of quotations is a matter within the agency’s discretion.  Advisory Technical Consultants, B-416981.3, June 4, 2019, 2019 CPD ¶ 209 at 3.  When reviewing a protest challenging an agency’s evaluation, our Office will not reevaluate quotations, but instead, will examine the record to determine whether the agency’s judgment was reasonable and consistent with the stated evaluation criteria and applicable procurement statutes and regulations.  OPTIMUS Corp., B-400777, Jan. 26, 2009, 2009 CPD ¶ 33 at 4.  A protester’s disagreement with an agency’s judgment, without more, is insufficient to establish that an agency acted unreasonably.  Converge Networks Corp., B-415915.2, B-415915.3, Aug. 20, 2018, 2018 CPD ¶ 334 at 5.

Moreover, where a protester challenges an evaluation as unfairly utilizing unstated evaluation criteria, our Office will assess whether the solicitation reasonably informs vendors of the basis for the evaluation.  Raytheon Co., B-403110.3, Apr. 26, 2011, 2011 CPD ¶ 96 at 5.  In that regard, procuring agencies are not required to identify every area that may be taken into account; rather, it is sufficient that the areas considered in the evaluation be reasonably related to or encompassed by the stated criteria.  Id.

User Interface Color Scheme

With respect to the protester’s first argument, the protester notes that the agency primarily based its negative judgments for six of seven tasks on the fact that the protester’s interface used fonts and buttons that do not provide sufficient contrast against a black background, and the use of an orange “confirm” button instead of a green one.  Comments at 3 (citing AR at 166-168).  While the solicitation indicated that the agency would evaluate solutions for user-centered design and human factors, ERP argues that the agency’s evaluation judgments concerning the color scheme of its demonstration prototype were not logically encompassed in that or any other stated evaluation criterion.  Id. at 3-8.  This is especially the case, the protester maintains, because the RFQ permitted offerors to demonstrate their solutions using wireframes,[5] which would lack any color whatsoever.  Id.  Accordingly, the protester contends that vendors had no reason to expect that color choices would be evaluated, and that the agency, in effect, unreasonably evaluated vendors based on unknowable personal preferences.  Id.

Alternatively, the protester contends that, even assuming for the sake of argument the solicitation reasonably can be read to allow for the evaluation of color choices, the protester’s reading of the solicitation--as not permitting the evaluation of color choices--is also reasonable, rendering the solicitation ambiguous.  Comments at 8 n.4; Supp. Comments at 7-8.  The protester contends that it was not aware of any ambiguity in this respect until it learned of the agency’s evaluation from its post-award request for additional information.  Id.

Preliminarily, we note that the protester’s characterization of the agency’s objections to its user interface to some extent mischaracterizes the agency’s concerns.  Specifically, the agency’s contemporaneous concerns were that ERP’s design choices made navigation difficult, displayed a lack of understanding of common interaction design principles, and did not conform to well-known interaction design principles.  AR at 168.  The agency cited the color and font issues as examples of that general concern.  Id. at 166-168.  More significantly, the agency’s specific concerns about color and font were not merely aesthetic; the agency evaluators noted that the interface “us[ed] fonts and buttons that do not provide sufficient contrast against a black background making it difficult to read.” Id. at 67 (emphasis supplied).

The protester is correct that the RFQ specifically permitted offerors to demonstrate their solutions using wireframes which could, conceivably, lack any color elements at all.  AR at 26.  However, the protester did not elect to demonstrate its solution using wireframes.  Rather, the protester elected to use an interface with small dark gray fonts against a black background, and used bright orange as the primary accent color for interactive elements.  See Protest at 10-11; ERP Demonstration Video, generally.  That is to say, while the RFQ did not require the protester to use color in its demonstration, the protester chose to do so.  The agency was not required to ignore that aspect of the protester’s demonstration because the protester could have hypothetically demonstrated its solution in a different way.  See, e.g., Frequentis Defense, Inc., B‑416970.2, Apr. 4, 2019, 2019 CPD ¶ 135 at 5 n.7 (argument that protester should not have been excluded from competition for failing to meet a solicitation requirement because the solicitation permitted offerors to propose alternative solutions that would not have been subject to that specific requirement is denied because the protester did not actually propose those alternatives).

After review of ERP’s demonstration video and the user interface screen captures included in the protester’s filings, we see no reason to conclude that the agency was unreasonable, as a factual matter, in finding that the protester’s interface was difficult to read.  See Protest at 10-11; ERP Demonstration Video generally.  Moreover, while the RFQ did not specify precisely how the agency would assess whether a user could complete a task with minimal cognitive strain, or what the agency would consider in evaluating human factors, the ability for a user to easily read interface text is clearly and reasonably related to those explicit requirements.  Accordingly, we cannot conclude that the agency applied an unstated evaluation criterion in this regard.

Further, we do not believe that the solicitation was ambiguous in the way the protester suggests.  When a protester and agency disagree over the meaning of solicitation language, we will resolve the matter by reading the solicitation as a whole and in a manner that gives effect to all of its provisions; to be reasonable, and therefore valid, an interpretation must be consistent with the solicitation when read as a whole and in a reasonable manner.  Crew Training Int’l, Inc., B-414126, Feb. 7, 2017, 2017 CPD ¶ 53 at 4.  An ambiguity exists where two or more reasonable interpretations of the terms or specifications of the solicitation are possible; a patent ambiguity exists where the solicitation contains an obvious, gross, or glaring error, while a latent ambiguity is more subtle.  Id.

In this case, as discussed above, the solicitation indicated that the agency would evaluate the demonstration based on whether a user could complete a task with “minimal cognitive strain,” and for “user-centered design/human factors.”  AR at 26-27. The agency argues that there are well-known design principles for user interfaces, and that this solicitation language was intended to invoke those design principles.  COS/MOL at 8.  The protester responds by arguing that this RFQ language did not enumerate any such design principles, nor did it require that quoted solutions must conform to any specific principles, well-known or otherwise.  Comments at 4.  Accordingly, the protester argues that, reading the design language together with the fact that vendors could use colorless wireframes, the protester reasonably came to the conclusion that the agency would not evaluate the color schemes of user interfaces.  Id. at 8 n.4; Supp. Comments at 7-8.

In this case, we do not believe that the protester’s reading is reasonable.  Specifically, the protester’s reading suggests that, because the RFQ permitted the use of wireframes, the RFQ also, in effect, required the agency to ignore any usability issues created by a vendor’s choice to use colors.  However, such a reading would lead to absurd results:  if a vendor demonstrated an otherwise well-designed interface with black text on a black background--that is to say, a completely unreadable interface--under the protester’s reading of the solicitation such an interface would be technically acceptable despite being entirely unusable.  This is not a reasonable reading of the solicitation, and, accordingly, we do not believe that the RFQ is ambiguous in this regard.

Terminal Window Demonstration

Next, the protester argues that the agency also applied an unstated evaluation criterion with regard to the protester’s demonstration of certain functionality using a terminal or command line interface.  Comments at 9-10.  Specifically, the protester contends that the solicitation permitted vendors to use prototypes for their demonstrations, but the agency negatively evaluated the protester’s presentation for demonstrating its prototype data lake service using a terminal interface.  Id.  In this regard, the protester notes that it explained during the presentation that it was only using a terminal interface to demonstrate that its technical solution for the data lake service was fully functional, but that the web interface for the data lake portion of the requirement was not yet developed and would be customized to meet the agency’s needs.  Id.  

We do not agree that the agency applied an unstated evaluation criterion in this respect. The solicitation specifically required that the data lake service include a web-based portal for the data lake, and also required vendors to demonstrate various functions of their proposed solution using an in-production service, a prototype, or wireframes.  AR at 26, 119.  While the protester is correct that the solicitation did not require the demonstration of a finished product, the solicitation was clear that the agency was evaluating both the usability of the proposed interface and the underlying functionality, and the solicitation required vendors to provide, at minimum, a wireframe of their proposed solution.  Id.

The protester, however, did not demonstrate its proposed web-based portal for this task, or provide a wireframe of a notional web interface.  Rather, the protester chose to demonstrate this task using a terminal query directly to its data lake service.  Contrary to the protester’s suggestion, while a terminal query can show that the data lake service could functionally accomplish a task, a terminal interface does not represent a prototype or wireframe of a web-based portal.  Because the protester only demonstrated a terminal interface, the agency evaluated the protester on that basis, and concluded that the interface actually demonstrated by the protester would only be suitable for users with a high degree of technical skill.  AR at 168.

In short, while the solicitation did not require the demonstration of a finished web portal user interface, the solicitation did require vendors to demonstrate their proposed interfaces, however notional.  The only interface demonstrated by the protester was a

terminal interface, and the agency properly evaluated the protester on that basis.  We see no reason to conclude that the agency applied an unstated evaluation criterion or otherwise erred in this respect.

The protest is denied.

Thomas H. Armstrong
General Counsel