Matter of: Genovation Inc. File: B-251714 Date: April 21, 1993

Matter of: Genovation Inc. File: B-251714 Date: April 21, 1993

PROCUREMENT Competitive Negotiation Offers Evaluation errors Evaluation criteria Application Protest challenging one reviewer's evaluation of a proposal submitted under Small Business Innovation Research Program is denied where the record shows that although two other reviewers gave the proposal higher scores, the evaluation in question reasonably reflected weaknesses in the proposal.

Attorneys

DECISION Genovation Inc. protests the rejection of its proposal by the Department of Education under request for proposals (RFP) No. 92-024, which invited offers for research into various topics relating to special education and rehabilitative services. The protester contends that one of the evaluators that reviewed and scored its proposal did not evaluate it properly, resulting in the rejection of its proposal. We deny the protest.

The RFP was issued on January 10, 1992, under the Small Business Innovation Research Program (SBIR). This program was established under the Small Business Innovation Development Act of 1982, 15 U.S.C. Sec. 638 (1988), which requires certain federal agencies, including the Department of Education, to establish SBIR programs. Under these programs, the agencies reserve a statutory percentage of their research and development budgets for award to small business concerns for research or research and development through a three-phase process. The purpose of the work performed under Phase I is to determine the scientific or technical merit and feasibility of ideas submitted under the SBIR program, for a typical period of 6 months; under Phase II (awarded on the basis of Phase I results), to identify the potential for yielding a product or process of continuing interest to the agency; and under Phase III (involving private capital), to pursue commercial applications of the research or development. The funding vehicle for the Department of Education's SBIR program in both Phase I and Phase II is contracts, rather than cooperative agreements or grants. As offerors were advised in the RFP, the agency has the sole discretion to select SBIR topics and awardees, and to award several contracts or no contracts at all under a given topic.

The RFP sought research proposals for Phase I for any of 10 topics that were listed in the solicitation. Offerors were advised to respond to only one of the listed topics in each proposal they submitted.

The RFP disclosed the six evaluation factors that would be followed in scoring Phase I proposals, and the maximum number of points that could be awarded under each factor. The factors and points assigned were as follows: (1) importance of the problem and anticipated benefits of research (15 points); (2) adequacy of Phase I effort to demonstrate feasibility of concept (15 points); (3) the scientific and technical quality of the Phase I proposal and its relevance to the particular topic, with special emphasis on innovation and originality (40 points); (4) qualifications of principal, other senior or key personnel, and consultants (20 points); (5) adequacy of facilities and equipment to conduct research (5 points); and (6) adequacy of budget (5 points).

The solicitation stated that the Department of Education would select for award those proposals offering the best value to the government according to these evaluation factors. Offerors were also advised that while the agency expected to make approximately 20 Phase I awards, it had discretion to award multiple contracts or no contracts at all under any one of the 10 topic areas. Final decisions regarding funding would be made by the agency based upon the evaluation criteria and consideration of other factors, including possible duplication of other research and legislative restrictions on program funds.

The agency received 269 proposals in response to the RFP. Forty of these, including one from Genovation, responded to Topic 8, "Development or Adaptation of Innovative Technologies to Enhance Learning and Development of Children with Disabilities." Each of the proposals was individually reviewed and rated by three evaluators who were selected from rosters of qualified individuals maintained by the agency. Reviewers assigned a numerical score for each of the evaluation criteria and generally included comments in their evaluations, identifying strengths and weaknesses under each criterion. They also provided a short summary of their evaluation, and selected one of the following recommendations: to fund the project, to possibly fund it if major changes were made, or not to fund it. The three evaluators' overall scores were then averaged for each proposal, and the proposals were ranked according to the consensus score. Overall, 23 of the 269 proposals received a score of 91 or more and were recommended for funding; 2 of the ones that were submitted under Topic 8 were in this group. Upon learning that its proposal had not been selected for funding, Genovation requested and received copies of the evaluation sheets that each of the evaluators had completed for its proposal. Genovation filed a protest against the evaluation with the agency, which was denied. This protest followed.

Genovation's proposal, entitled "An Interactive Infra Red Communication System to Enhance the Learning of Disabled Children," was to research, design and develop the specifications for a very low-cost infrared communications system that would provide the severely disabled student with the ability to communicate with the teacher in real time in a normal class environment. Under the proposed system, the disabled child would use an infrared communicator configurable to the specific disability of the user, and the teacher would use an infrared transceiver portable computer that would graphically present the average class understanding of the teacher's presentation, as well as the individual level of understanding for each student using the system. Under Phase I, Genovation proposed to produce the system design, application studies, preliminary circuit prototypes, and software adequate for determining final working system specifications.

The three evaluators assigned the proposal individual scores of 94, 91, and 70, resulting in an average score of 85 points. The protester asserts that the lowest of these scores was the result of a flawed evaluation, and speculates that the reviewer responsible for that score "did not understand our proposal, did not bother to read it closely, and misdirected his/her limited technical knowledge."

In reviewing protests against an agency's allegedly improper technical evaluation and consequent rejection of a proposal, we examine the record to determine whether the agency's judgment was reasonable and consistent with the stated evaluation criteria and applicable statutes and regulations. See S&H Sys., B-250561, Feb. 4, 1993, 93-1 CPD Para. . We find no basis to object to the agency's evaluation in this case.

The agency asserts that it conducted its evaluation in accordance with all applicable statutes and regulations, and argues that the protest merely reflects Genovation's disagreement with the reviewer's opinion. The Department of Education points out that while two of the reviewers were generalists in the areas of special education and communication, the reviewer who gave Genovation's proposal the lower score (identified as "#R62") was a "highly technically trained, Ph.D.-level, computational linguist at a major university." The agency notes that reviewer R62 was best qualified to appraise proposals for this topic, able to identify technical strengths, as well as weaknesses, that were not identified by the other reviewers.

We have reviewed the evaluations that the three reviewers completed for Genovation's proposal. Comments entered by reviewer R62 under each of the evaluation criteria consistently reflect a concern that while the proposal addresses technical and engineering issues quite effectively, it does not provide much information to show the system's feasibility in an instructional or educational context. Overall, R62 questioned how teachers would use the technology being proposed and whether, in fact, teachers would be willing to use it. As a specific example, under evaluation criterion 3 (the single most heavily weighted evaluation criterion), reviewers were to assess the scientific and technical quality of the proposal and its relevance to the particular topic addressed, with special emphasis on innovation and originality. Reviewer R62 acknowledged that the proposed infrared technology is innovative and its application to the disabled learner in the classroom fairly original, but noted "however, the issues in education are not only technical engineering issues. Issues of content and use in educational . . . [setting] are inadequately discussed or planned for." We have reviewed Genovation's proposal, and think that the reviewer could reasonably reach this conclusion. The proposal provides a fairly detailed description of the technical approach being proposed, addressing the system design, software requirements, etc., but offers very little information concerning its use in an educational setting or its applicability to classroom curricula. While the protester objects to this assessment on the basis that the system first has to be developed, before exploring whether teachers would accept the technology for classroom use, we think the reviewer could reasonably criticize the proposal for omitting any consideration of this aspect of the project in its proposal.

For criterion 4, assessing the qualifications of the principal investigator and other personnel proposed, R62 pointed out that the proposed staff is experienced in infrared technology "but somewhat less so in educational methodology. Specific roles of key personnel unclear." The two key personnel identified in Genovation's proposal have degrees in electrical engineering, finance, systems management, mechanical engineering and aerospace engineering; their work experience is also listed in areas such as marketing, technological consulting, product development, engineering and management. There is no mention in the proposal of any education-related experience or qualifications for this "management team." Regarding the use of consultants, the proposal states generally that the firm "will draw personnel resources from its current staff, and from the local consulting community, if necessary." While a number of professionals are listed, showing credentials or experience relating to the education of disabled students, it is not clear what involvement they would have in the project. Furthermore, it appears that a number of persons listed are currently employed in various academic positions, further calling into question the extent to which they would be involved in this project. We therefore conclude that reviewer R62's assessment was reasonable in this area.

We will not describe the evaluation of each criterion in detail. Under the RFP's evaluation scheme Criteria 3 and 4, discussed above, were worth 60 percent of the maximum evaluation points available and were clearly the most important factors. In these two areas alone, Genovation lost sufficient points to rank its proposal below the two awardees. While some of reviewer R62's criticisms may have been harsher than the other two reviewers, they were not unreasonable, nor was R62's overall evaluation or funding recommendation without any reasonable basis. In this regard, we note that the score R62 assigned Genovation's proposal was not exceptionally low or out of proportion with the evaluator's criticisms, and R62's recommendation was that the proposal could be funded if major changes were made. Finally, we note that this was a competitive procurement, and the record shows that the two proposals for Topic 8 (with scores of 91 and 93 overall) that were selected for funding were reasonably rated higher and were very strong overall.

The protest is denied.