Matter of: JWK International, Inc. File: B-251125 Date: March 4, 1993

Matter of: JWK International, Inc. File: B-251125 Date: March 4, 1993

PROCUREMENT Competitive Negotiation Offers Evaluation Personnel experience PROCUREMENT Competitive Negotiation Offers Evaluation Technical acceptability Where solicitation required offerors to document relevant experience of proposed personnel, as well as corporate experience, and to discuss potential problems in planned study involving screening of 3.5 million people, and to identify solutions to potential problems, evaluators reasonably found protester's proposal unacceptable where proposal contained no evidence of experience with studies involving more than 5,000 subjects and failed to address or propose solutions to problems associated with mass mailing and follow-up of persons who did not respond. PROCUREMENT Socio-Economic Policies Small business set-asides Withdrawal Propriety Contracting officer properly withdrew a small business set-aside where all small business proposals received were technically unacceptable.

Attorneys

DECISION JWK International, Inc. protests the rejection of its proposal under request for proposals (RFP) No. NCI-CN-25433-51, issued as a small business set-aside by the National Cancer Institute, National Institutes of Health, and the subsequent cancellation of the RFP. The protester contends that the decision to reject its proposal and ultimately to resolicit on an unrestricted basis resulted from an evaluation of JWK's best and final offer (BAFO) that was unreasonable and inconsistent with the evaluation of its initial proposal.

We deny the protest.

The RFP, issued on February 14, 1992, contemplated award of a cost- reimbursement contract for a coordinating center for a prospective cohort study of diet and cancer in members of the American Association of Retired Persons (AARP) over a period of 6 years. The contractor was to furnish all necessary services, qualified personnel, material, equipment, and facilities needed to establish a central data coordinating center to administer the issuance of a questionnaire to approximately 3.5 million AARP members, construct a cohort of 350,000 meeting certain criteria specified in the statement of work, and conduct follow-up on that cohort for a period of 5 years. The study, which is to be the largest cohort study of diet and cancer ever conducted, is highly sensitive and expected to provide key information about the relationship of diet to major cancers, particularly those of the breast, lower bowel, and prostate.

In the course of performance, the contractor was to establish and maintain a centralized office from which mail and telephone interviews could be conducted and in which data could be stored. The contractor would be responsible for modification of the agency's Health Habits and History Questionnaire (HHHQ) to produce a baseline questionnaire; contractual arrangements with state cancer registries; pilot and validation studies; construction of a cohort; design of further questionnaires; communication with hospitals, physicians and other data sources; integration of cancer incidence and mortality information into the data base; quality control; and development of programs to process data. The study would require rapid analysis of voluminous data, as well as assurance that the drop-out rate of respondents would be kept to a minimum in the course of the study.

The agency advised potential offerors that technical proposals would "receive paramount consideration" in the selection decision, and provided for an evaluation based upon the offeror's demonstrated capability in relation to the needs of the project as set forth in the RFP. In addition to mandatory criteria for establishing the centralized office, the solicitation included three evaluation criteria, as follows: personnel, 40 points, of which 20 points were for the project director; method/approach, 35 points; and corporate experience and resources, 25 points.

Under the personnel factor, an offeror was to demonstrate that it could provide the necessary support staff for all aspects of the scope of work, including a project director, programmer/analysts, technical support personnel, and administrative/clerical support, and document the relevant experience of its proposed personnel. The agency would evaluate the project director's experience managing personnel and directing studies involving development and mailing of questionnaires, selection of subsamples for pilot and validation studies, interviewing and tracing study subjects, following up nonrespondents, obtaining reports from hospitals and doctors' offices, use of cancer registries, coding and assessment of data, and editing and analyzing responses to complex questionnaires.

Under the methods/approach factor, the offeror was to describe the responsibilities of key team members and the methods and procedures proposed for meeting study goals, and to discuss potential problems and quality control concerns, including methods for addressing and resolving such problems. The agency requested documentation supporting procedures for modifying the HHHQ and generating a machine-readable instrument; for subcontracting with state cancer registries; for analysis of data from the pilot and validation studies; for mailing and obtaining responses to approximately 3.5 million baseline questionnaires in the most cost- efficient manner; for construction of the cohort, including mailing and return of follow-up questionnaires; for linking the cohort to cancer registries; for obtaining reports from hospitals; and for maintaining quality control including accuracy and completeness of data.

Finally, under the corporate experience and resources factor, the offeror was to document its corporate experience and resources relevant to the statement of work, including large-scale epidemiological research, especially studies involving similar work.

The agency received two proposals on March 30 and referred them to an initial technical evaluation group (ITEG), which identified strengths and weaknesses in the proposals; four of five members of the group rated the protester's proposal as technically acceptable. The agency then referred the evaluations and the proposals to its source evaluation group (SEG), which reviewed the initial evaluation findings and identified further weaknesses and concerns for discussion. This review resulted in the elimination of the other offer from the competitive range.

The SEG prepared a list of discussion questions, which the agency provided to the protester by letter dated August 5; the protester submitted a BAFO on August 19. Upon review of the protester's responses to the technical questions, the SEG concluded that the protester's proposal did not contain sufficient evidence that JWK had the organizational experience to conduct a study of such import and magnitude. In view of the lack of technical quality of proposals and the high prices received, the agency concluded that the solicitation had not generated adequate competition and decided to cancel and re-issue it on an unrestricted basis. This protest followed.

The protester contends that the agency's determination that its proposal was technically unacceptable was unreasonable. The protester asserts that the ITEG awarded JWK a total of 3,145 of 5,000 points available, for an average score of 629 out of 1,000 points, and argues that the agency has provided no rational justification for lowering the protester's score to 1,090 out of 3,000 points available, or 363 out of 1,000 points, in the SEG review. The protester contends that the agency did not adequately review its technical proposal and responses to discussion questions.

The fact that one of the two committees that reviewed the protester's proposal believed the proposal to be acceptable does not establish that the second evaluation, which found the proposal unacceptable, was unreasonable. The difference in point scores between the two panels is not controlling; numerical point scores, when used for proposal evaluation, are useful only as guides to intelligent decisionmaking, and may be expected to reflect the disparate, subjective judgments of the evaluators. Bunker Ramo Corp., 56 Comp.Gen. 712 (1977), 77-1 CPD Para. 427. Our chief concern is whether, standing on its own, the SEG's conclusions leading to its finding that JWK's proposal was technically unacceptable are reasonable, supported by the record, and consistent with the listed evaluation criteria. See Marine Animal Prods. Int'l, Inc., B-247150.2, July 13, 1992, 92-2 CPD Para. 16.

Regarding the reasonableness of the evaluation in the area of personnel, the ITEG found significant weaknesses in the protester's proposal-- specifically, the limited expertise of certain team members, particularly in the area of statistics and programming for large-scale projects; the principal investigator's lack of experience in leading research projects of a scope even remotely approaching the one contemplated; and lack of experience with mass-mailing requiring the selection and tracing of extremely large statistical samples. The protester did not address common techniques for insuring maximum response throughout the survey period, and failed to demonstrate an appreciation of the problems in dealing with the specified data sources, such as cancer registries and records (often handwritten) of physicians, who may be unwilling to honor long-term blanket waivers such as the protester proposed. The SEG was concerned about the protester's lack of experience with the food frequency questionnaire, which was to be a primary instrument for the project; the lack of publication in peer journals by the principal investigator; the lack of experience in tracing and following individuals in an epidemiological study; the lack of job stability for proposed team members; and the over-commitment of a key staff member.

The agency addressed 12 questions to the protester in the area of personnel, several seeking evidence of the team's experience in carrying out large-scale epidemiological studies.[1] The agency asked for plans to maximize response rates and follow-up procedures and advised the protester of its concerns related to personnel turnover and over-commitment of key personnel. In the final evaluation, the SEG identified the principal investigator's lack of experience with large epidemiological cohort studies; JWK's general lack of experience with large-scale studies; and concerns related to the commitment of key personnel as significant weaknesses rendering the proposal unacceptable.

In the area of methods/approach, the ITEG expressed concern over the protester's failure to consider problems associated with missing data and exclusion criteria. While the protester proposed developing strong links to state registries and hospitals, the only registry with which the protester actually had links was not qualified to participate in the study; the protester failed to address problems associated with the time lag for data entries, or with cohort members moving into and out of registry areas. There was little detail on follow-up plans, and the proposal did not address problems in gaining access to hospital and physician records. The SEG had an additional concern regarding the statistical inefficiency of the method of selecting populations for study, and noted that the precise roles of the protester's personnel were unclear. Overall, the SEG found that the discussion of quality control, particularly in the area of follow-up, was inadequate; that familiarity with the problems of dealing with hospitals and registries was not demonstrated; and that the proposed dietary assessment method was weak.

The agency addressed 10 questions in this area to the protester, some duplicating the questions in the personnel area, as in its concerns over response rates. The agency asked about plans for missing data, for maximizing responses, and addressing problems with physicians, registries and hospitals; the agency asked the protester to justify the statistical method for drawing its sample. In the final evaluation, the agency concluded that the method for drawing a representative sample was simply ill-conceived; that the protester exhibited no method for insuring maximization of responses, offering no innovative solutions and omitting many routine ones; that JWK did not appreciate the primary importance of the state registry data; and that the methods for ensuring accurate and complete responses to the survey were simply not addressed in the proposal.

In the area of corporate experience and resources, the ITEG found the protester to have a variety of survey experience, but none with projects of the scope contemplated for the diet and cancer study. While the protester had considerable experience with dietary assessments, it had virtually no cancer study experience. As one more example of the pro- tester's failure to appreciate the logistics of this massive effort, JWK provided no details on printing, mailing, and optical scanning of questionnaires, layout techniques, quality control, or subcontracting arrangements. The SEG took exception to certain favorable portions of the ITEG's evaluation, challenging the ITEG's finding that the protester had "considerable" experience with dietary assessment; further, that group emphasized more strongly the gap between the protester's past experience and the magnitude of the instant effort. The SEG also disagreed with the ITEG concerning the benefits of the protester's multiple locations (which the initial panel considered of benefit in tracing and following cohort members), pointing out that the protester had presented no plan for using these offices in maintaining contact with members of the cohort. The SEG found that JWK's proposed use of a special data entry program showing decoded ICD-9 (International Classification of Disease, 9th version) information was insufficiently detailed; nor did the protester provide sufficient information on physical space requirements for storage and organization of the data forms, which were quite voluminous.

The agency addressed five questions to the protester in this area, again asking for documentation of the protester's experience, as well as information on available computer hardware, subcontractor experience, and physical space. After review of the protester's responses, the evaluators found no evidence of experience with large epidemiological surveys or those involving mass mailing of questionnaires and long-term follow-up of subjects, cross-checked with other record repositories; there were insufficient details regarding the subcontracting of printing, mailing and optical scanning of questionnaires.

Rather than expressly addressing the evaluators' specific findings, JWK argues generally that the agency misevaluated its answers to the 27 discussion questions; based on its contentions that its original proposal was acceptable, the protester contends that the additional information provided no basis for changing the rating from acceptable to unac- ceptable. The protester's assertions fall into three general categories: first, that the project is unique and that no offeror has experience with comparable projects; second, that the items not addressed in its proposal were not required by the solicitation; and third, that contrary to the evaluators' conclusions, it provided the information requested.[2]

Regarding the scope of the project, the protester concedes that it has no experience with a project as large and presents nothing to demonstrate experience with a project even similar in scale; we thus see no basis for finding the agency's conclusion in this area unreasonable.[3] To the extent that the agency asked JWK to elaborate on some of its solutions, such as its statistical method for drawing the sample for the validation approach, we do not find it unreasonable for the agency to ask an offeror to demonstrate its understanding of the method that it did propose, to show its technical capability. Finally, to the extent that the protester contends that the evaluators are simply wrong and that it provided sufficient information to demonstrate its experience and its understanding of the problems that might be encountered, such mere disagreement with the opinions of the evaluators does not render the evaluation unreasonable. See Marine Animal Prods. Int'l, Inc., supra, at 5-6.

The protester also contends that the evaluation was not adequately documented, in that there was no justification for the lowering of its initial technical score and that none of the evaluators, apart from the SEG chairman, provided an explanation for the technical scoring. An agency must be able to document its technical judgments in order for us to determine whether those judgments are reasonable. U.S. Def. Sys., Inc., B-245563, Jan. 17, 1992, 92-1 CPD Para. 89. It is not improper, however, for evaluators to discuss their findings, for the purpose of generating a consensus rating; the overriding concern is that the evaluation reflect the actual merits of the proposals, not that it be mechanically traceable back to the scores initially given by the individual evaluators. Schweizer Aircraft Corp., B-248640.2; B-248640.3, Sept. 14, 1992, 92-2 CPD Para. 200. Here, we find the consensus reports prepared by the SEG chairman adequate to establish the reasonableness of the evaluation.

In sum, the record shows that the agency found that JWK's proposed personnel had no experience with projects even approaching the magnitude of the contemplated effort; that the qualifications of some personnel and their commitment to the project were questionable; that the protester failed to present a carefully considered method for selection of the validation and cohort groups; and that the proposal did not reflect understanding of the logistics of mass mailing and the difficulties of follow-up and coordination with physicians, hospitals, and state cancer registries. The SEG was of the unanimous opinion that the protester could not perform the work for this project; we find this conclusion reasonable, consistent with the evaluation criteria, and supported by the record.

The conclusion of the SEG, that the protester could not perform the work as proposed, was essentially a finding that the proposal was technically unacceptable. Given this finding, and since no technically acceptable offers were received, the contracting officer properly decided to with- draw the small business set-aside and resolicit on an unrestricted basis. Femme Comp Inc., 69 Comp.Gen. 664 (1990), 90-2 CPD Para. 121.

The protest is denied.

1. For example, questions Nos. 1 through 3 asked the protester to document the experience of its proposed personnel in carrying out large- scale studies, its principal investigator's experience in leading a research project of this scope, and its experience with high-volume mailed questionnaires; question No. 6 asked for experience in dealing with hospital and physician records.

2. The protester does not address certain weaknesses in its proposal apparent from the evaluation, such as its use of state registries, and only peripherally addresses concerns encompassed by, but not directly asked by, the discussion questions related to its experience with hospital and physician records, quality control, its cancer experience as distinct from its dietary assessment experience, and details regarding printing, mailing and optical scanning. In fact, although the protester presents arguments defending its response to each of the 27 questions, the agency concedes that some of the responses were acceptable. Elsewhere, as in addressing personnel stability, the protester has failed to acknowledge the agency's actual concern even after reviewing the agency report.

3. The largest project referenced in the proposal encompassed 4,204 responses, with a 70 percent response rate. There are no other projects with more than 3,000 responses, although the response rates in two of the larger projects approached 100 percent. In contrast, the instant solicitation contemplates selection of a cohort 100 times that size from a target population 1,000 times that size.