Matter of: I.S. Grupe, Inc. File: B-278839 Date: March 20, 1998

Matter of: I.S. Grupe, Inc. File: B-278839 Date: March 20, 1998

DIGEST

Attorneys

DECISION

I.S. Grupe, Inc. protests the decision by the Agency for Health Care Policy and Research, Department of Health & Human Services (HHS), not to award it a Phase II contract under request for proposals (RFP) No. AHCPR-97-0021 for a project the firm proposed under the agency's Small Business Innovation Research (SBIR) program. The protester contends that the agency improperly determined that its proposal, entitled "Internet Multimedia Cancer Patient Education System," was technically unacceptable and would not continue to be funded.

We deny the protest.

The RFP was issued on July 10, 1997, under the SBIR program. This program was established under the Small Business Innovation Development Act of 1982, 15 U.S.C. Sec. 638 (1994), which requires certain federal agencies, including HHS, to establish SBIR programs. Under these programs, the agencies reserve a statutory percentage of their research and development budgets for award to small business concerns for research or research and development through a three-phase process. The purpose of the work performed under Phase I is to determine the scientific or technical merit and feasibility of ideas submitted under the SBIR program; under Phase II (awarded on the basis of Phase I results), to identify the potential for yielding a product or process of continuing interest to the agency; and under Phase III (involving private capital), to pursue commercial applications of the research or development. The funding vehicle for this HHS SBIR program in both Phase I and Phase II was contracts, rather than cooperative agreements or grants.

The RFP provided the following five Phase II evaluation factors and respective weights: (1) the degree to which the Phase I objectives were met and feasibility demonstrated--25 percent; (2) the scientific/technical merit of the proposed Phase II research, including the adequacy of the objectives for addressing the problem/opportunity--35 percent; (3) the qualifications of the principal investigator, supporting staff, and consultants--20 percent; (4) the potential of the proposed research for technological innovation--15 percent; and (5) the adequacy and suitability of the facilities and research environment--5 percent. Under the RFP, cost reimbursement contracts would be awarded to some or all offerors submitting technically acceptable proposals.

The three firms, including the protester, which previously had been awarded Phase I contracts, submitted Phase II proposals in response to this RFP. The basis for the protester's proposal was the completion of an interactive multimedia educational system for cancer patients which would allow an individual patient to determine the materials relevant to his/her medical problem and treatment. Proposals were evaluated by a panel of six individuals in accordance with the referenced evaluation scheme. Of relevance to this protest, one of the evaluators (whom the agency designated as evaluator No. 5) was a medical doctor and the chief executive officer of a media firm specializing in medical information for consumers. Each panel member initially assigned points to each offeror's proposal and provided narratives of the strengths and weaknesses in these proposals. The evaluators then met as a group, discussed each proposal, and modified individual scores and narratives in light of these discussions. The average of the individual scores for an offeror's proposal became the consensus score for that proposal.

The evaluators concluded that the protester's proposal, which was accompanied by a videotape showing "not the prototype product, but only a view of the prototype product," and which would not allow the viewer to actually interact with the educational system depicted, was technically unacceptable (consensus score--55.3 points out of a possible 100). The agency ultimately awarded Phase II contracts to the other two offerors, whose proposals were determined technically acceptable (consensus scores--84.7 points and 79.2 points).

The protester challenges the evaluation of its proposal, primarily expressing disagreement with the evaluators' conclusions concerning the technical merit of the proposal. The protester first complains about the evaluators' viewing of the videotape, which accompanied the firm's Phase II written proposal. /1/ The videotape contained a partial demonstration of the operational prototype developed under the protester's Phase I contract and incorporated a sampling of the multimedia objects (text, audio, images, animation, and video) generated during that project. The protester's complaint, however, is without merit because the protester, in fact, encouraged the agency to distribute copies of the videotape to the evaluators for viewing during the evaluation process. Specifically, the protester stated in the cover letter to its proposal that

[w]e have also enclosed . . . copies of a videotape which you may want to distribute with the copies of the proposal. We understand and accept that such distribution is at your sole option and that the proposal in text form represents [the firm's] response to your RFP. . . . Since we are working with a multimedia presentation, it is often easier to show things than to describe them in words, which is the reason for including the videotape. The tape shows some selected multimedia segments from the prototype as examples to support the proposal text. Only a commonplace VCR is needed to view the videotape, rather than a computer system. We believe that the videotape is a useful adjunct to the proposal, and we hope that you elect to distribute copies of the tape to the reviewers as well as copies of the text. [Emphasis added.]

In addition, in the proposal itself, the protester requested that the evaluators "please view the videotape sent with the Phase II proposal to experience the real Phase I result." Simply stated, to accommodate the protester's requests, the evaluators reasonably viewed the videotape in conjunction with the evaluation of the protester's Phase II written proposal.

The protester next complains that the evaluators, particularly evaluator No. 5, improperly downgraded the overall technical merit of its proposal. The protester maintains that evaluator No. 5 not only assigned an unreasonably low number of points to its proposal based in large measure on his views of the content and quality of the videotape, but also caused the other evaluators to lower their initial scores assigned to the firm's proposal, with the cumulative effect that the overall consensus score for its proposal was improperly skewed.

Because of the experimental and creative nature of an SBIR procurement, which is not based on design or performance specifications for existing equipment, but rather emphasizes scientific and technical innovation and has as its objective the development of new technology, the contracting agency is given substantial discretion in determining which proposals it will fund. Quantum Magnetics, Inc., B-257968, Nov. 30, 1994, 94-2 CPD Para. 215 at 4. In light of the agency's discretion, we limit our review of awards under SBIR procurements to determining whether the agency violated any applicable regulations or solicitation provisions, or acted in bad faith. Virginia Accelerators Corp., B-271066, May 20, 1996, 97-2 CPD Para. 13 at 2; Bostan Research, Inc., B-274331, Dec. 3, 1996, 96-2 CPD Para. 209 at 2. Here, the agency's conclusion that the protester's proposal was technically unacceptable is supported by the record.

The record shows that evaluator No. 5 was a medical doctor and the chief executive officer of a media firm specializing in medical information for consumers. As such, he used his particular expertise in two areas--medicine and media--as the basis for evaluating the merits of the protester's technical proposal. While this evaluator believed that the theme of the protester's proposal--providing timely medical and treatment information to cancer patients--was the proposal's greatest strength, he nevertheless found weaknesses in the underlying details of the proposal. Evaluator No. 5 assigned 18 out of a possible 100 points to the protester's proposal, and provided narratives of the strengths and weaknesses in the firm's proposal.

For example, evaluator No. 5 noted as a weakness that after viewing the videotape, he was concerned that the protester did not demonstrate a sensitivity to the needs of health care consumers--the target audience--from the perspective of language used. This evaluator referenced a tenth grade literacy level as the goal mentioned by the protester for this project, and noted that the language appeared to be at a pre-medical literacy level. The protester objects to these characterizations contending that if this evaluator had read the firm's Phase II written proposal, he would have seen the statement that a "sixth grade reading level is a reasonable comprehension target level for typical patients." However, in the Phase I Final Report, a required section of the protester's Phase II proposal, the protester included a section entitled "Operational Reviews." In that section, the protester stated that "[a]lmost all the consultants who viewed the operational prototype system [i.e., the videotape] were concerned about the intended target audience for the material and the organization of the presentation. . . . [S]ome reviewers found inconsistency with regard to the educational level assumed for the target audience." More specifically, the protester referenced the statement of a doctor/consultant for the project who stated that "'portions of the text seem comprehensible by a Grade 10 or higher audience while other portions seem to be very much like a medical school text.'" In light of the statement of the protester's own doctor/consultant, we cannot conclude that evaluator No. 5 was unreasonable in his assessment that the videotape appeared to be directed to a more educated audience.

Evaluator No. 5 also listed as a weakness that in preparing the videotape, the protester neglected to utilize the skills of those who clearly could have contributed to making the interactive multimedia educational system a higher quality product, for example, medical illustrators and instructional designers. The protester did its media work in-house in order to save money. The protester stated in its proposal that "[w]e did animation, both two-dimensional and three-dimensional internally. . . without the services of a graphic artist. . . . We didaudio internally. . . . We imported graphics, and we both imported and internally created full motion video. Production values were high throughout, yet no expensive studio facilities were used." However, given the medical and media background of evaluator No. 5, we believe he could reasonably downgrade the protester's proposal for failing to utilize the skills of various media and other professionals who could have created a more professional, interactive multimedia project to educate cancer patients. The evaluator stated that "in his eyes," the protester's failure to use media professionals "sever[ely] compromis[ed] the integrity and value of the effort . . . ."

Accordingly, we believe the record supports evaluator No. 5's downgrading of the protester's proposal because of weaknesses in the proposal details for completing an interactive multimedia cancer education system. Technical evaluators have considerable latitude in assigning ratings which reflect their subjective judgments of a proposal's relative merits. MiTech, Inc., B-275078, Jan. 23, 1997, 97-1 CPD Para. 208 at 5. A protester's mere disagreement with the particular point scores awarded to its proposal does not render the evaluation unreasonable. Id.

We also point out that the other evaluators expressed concerns similar to those raised by evaluator No. 5, noting, for example, that the protester used very advanced and medical language; the protester assumed a very high literacy level for its audience; and the protester failed to utilize the professional services of a graphic artist, an audio/visual firm, and an instructional designer or health educator. In addition, the other evaluators believed the protester did not clearly address in its proposal the target audience, that is, cancer patients, as the firm failed to spend sufficient time testing materials with this audience. Since the project was intended to educate cancer patients, the evaluators believed the protester had to focus on what a patient's needs for cancer education were, for example, by addressing treatment alternatives as experienced by actual patients. One of the evaluators even noted that a patient educator was needed as a key person for this project. Finally, with respect to the videotape, the evaluators commented that the information presented was unfocused and uneven, and the doctor who did most of the videotape presentation sounded like a "talking head."

In sum, while evaluator No. 5, as well as the other evaluators, believed that the concept proposed by the protester to educate consumers with respect to cancer treatments through the use of interactive multimedia techniques was an overall strength of the firm's proposal, the evaluators nevertheless determined there were many weaknesses in the firm's proposed approach, as evidenced by its written proposal and accompanying videotape. While the protester expresses disagreement with the weaknesses noted by the evaluators, the protester has failed to show that these weaknesses were not supported by the record.

Finally, to the extent the protester argues that evaluator No. 5 unfairly influenced and persuaded the other evaluators to lower the scores initially assigned to the protester's proposal, thereby resulting in a lower overall consensus score assigned to the firm's proposal, we point out that agency evaluators may discuss their individual evaluations with each other in order to reach a valid consensus score since such discussions generally operate to correct mistakes or misperceptions that may have occurred in the initial evaluation. Wastren, Inc., B-276093, May 12, 1997, 97-1 CPD Para. 175 at 3 n.2. A consensus score need not be the score the majority of the evaluators initially awarded; a score may reasonably be determined after discussions among the evaluators. The overriding concern in these matters is whether the final scores assigned accurately reflect the relative merits of the proposals. Id. In this case, based upon the record, including the protester's written Phase II proposal and the videotape, we have no basis to question the agency's decision not to continue to fund the protester's project because the proposal submissions were reasonably determined technically unacceptable.

The protest is denied.

Comptroller General of the United States

1. Our Office viewed a copy of the videotape, which lasted approximately 36 minutes.