Matter of: TSM Corporation File: B-252362.2 93-2 CPD 13 Date: July 12, 1993

Matter of: TSM Corporation File: B-252362.2 93-2 CPD 13 Date: July 12, 1993

PROCUREMENT Competitive Negotiation Offers Competitive ranges Exclusion Administrative discretion PROCUREMENT Competitive Negotiation Offers Competitive ranges Exclusion Competitive sufficiency Agency properly eliminated protester's proposals from the competitive range where agency reasonably determined that the proposals contained multiple weaknesses and deficiencies and would require major revisions in order to become eligible for contract awards.

Attorneys

DECISION TSM Corporation protests that the Department of the Army improperly excluded TSM's proposals from the competitive range under request for proposals (RFP) No. DABT60-92-R-0005, for the development of training materials to be used by Army schools and other governmental agencies.

We deny the protest.

BACKGROUND

The RFP was issued on August 18, 1992, and was divided into three sections: contract line item numbers (CLINs) 0001-0035 were set aside for small businesses; CLINs 0036-0060 were set aside for small disadvantaged businesses;[1] and CLINS 0061-0115 were unrestricted. Under the small business set-aside CLINs (0001-0035), the RFP called for proposals to provide basic computer-assisted instruction (CAI) and "level III" interactive videodisc (IVD) courseware,[2] "in accordance with MIL-STD 1379D."[3] Under the unrestricted CLINs (0061-0115), the RFP similarly called for proposals to provide basic CAI and "level III" IVD courseware; in addition, the unrestricted CLINs called for more complex CAI and "level IV" IVD courseware.[4]

Section M of the RFP listed the technical factors under which proposals would be evaluated, stating: "the major areas of evaluation, listed in descending order of importance are: factor I -sample materials; factor II -management proposal; factor III -company experience and technical qualifications of proposed personnel."[5] Section L of the RFP contained a recommended outline for technical proposals that coincided with the evaluation factors identified in Section M. Regarding the submission of sample materials, Section L required offerors to submit, among other things, an "instructional media design" and sample lesson. Specifically, Section L stated:

"(2) Instructional Media Design (Design Strategy & Sample Prototype Lesson). The offeror shall develop and submit an Instructional Media Design (IMD). The IMD will be evaluated to determine the offeror's ability using the SAT [systems approach to training] process. The offeror shall use the information in the MIL-STD 1379D, DI-ILSS-81091, and the attached Sample Materials for interactive courseware (ICW) Design Strategy and lesson prototype (attach 8) to design and develop the IMD. The bid sample prototype lesson will include detailed video descriptions . . . ."

By the October 13 closing date, the agency received 29 proposals from 22 offerors.[6] TSM, a small business, submitted two proposals: one responding to the CLINs set aside for small businesses, and one responding to the unrestricted CLINs. Initial proposals were evaluated by a technical evaluation board (TEB) between October 14 and November 6. Based on that evaluation, the TEB recommended that both of TSM's proposals be eliminated from the competitive range.[7] By letters dated January 28, 1993, the agency advised TSM that its proposals would not be included in the competitive range. The agency's letters identified multiple weaknesses and deficiencies in each of TSM's proposals.[8] This protest followed.

DISCUSSION

TSM protests that the agency should have retained its proposals in the competitive range, arguing generally that any agency concerns regarding weaknesses and deficiencies in TSM's proposals should have been made subjects for discussions. The agency responds that TSM's proposals contained multiple weaknesses and deficiencies under each of the evaluation factors which, in the aggregate, would have required TSM's proposals to be substantially re-written in order to qualify for contract awards.

Regarding the evaluation of TSM's technical proposal under factor I, the agency evaluators found that TSM's instruction media design was vague and ambiguous, explaining that it "left too many loopholes." For example, in discussing its "module pretest,"[9] TSM's proposals stated: "The [pre]test item content will be randomized where possible." (Emphasis added.) In discussing "module posttest,"[10] TSM's proposal similarly stated: "The [post]test item content will be randomized where possible" and added, "where possible, some scenarios will be comprehensive in nature . . . ." (Emphasis added.) In identifying the "touch areas" incorporated into its design strategy,[11] TSM's proposal stated: "multiple touch areas will be separated by a minimum of 1 inch where possible." (Emphasis added.) In discussing the use of "visuals," TSM's proposal stated: "visuals will be changed frequently to maintain interest." (Emphasis added.) The agency evaluators found TSM's instructional media design unacceptable because it merely listed options or possibilities without identifying a specific course design.

The agency further found TSM's instructional media design to be unacceptable for failure to include certain required flow charts,[12] because it did not contain "front matter" as required,[13] and because it was internally inconsistent and inconsistent with the sample lesson TSM submitted.[14] The agency also found that the sample lesson TSM submitted often provided for the same "feedback/remediation" in response to either a wrong response or a correct answer.[15]

In responding to the agency's evaluation, TSM first complains that there were "shortcomings" in the RFP regarding the data required to prepare an instructional media design.[16] To the extent TSM is now protesting that the solicitation was defective, its protest is untimely. 4 C.F.R. Sec. 21.2(a)(1) (1993). Having failed to timely protest these alleged "shortcomings," TSM may not now use them as an excuse for deficiencies in its proposal.

Regarding the agency's criticism that TSM's design was vague and failed to specify a particular proposed design, TSM states:

"Use of the terms such as `may,' `when appropriate,' and `where possible' recognize that the design strategy documents the plans for the courseware but that there may be instances in which (a) the instructional developer must apply common sense to a specific situation within a lesson; or (b) the course manager/instructor has the means to impact delivery of the courseware."

Regarding the required flow charts, TSM responds that:

"TSM's interpretation of the solicitation was that the detailed flow diagrams were due and would be delivered with the interactive courseware. . . . If the [agency's] desire is to include the detailed flow diagram with the [instructional media design] . . . TSM can do that."

Regarding its omission of "front matter," TSM states:

"the items referenced [by the agency] were not considered relevant to the sample Instructional Media Design Report. The front matter is generally associated with publications such as Technical Manuals and some Field Manuals. . . . Normally, such a minor omission is addressed during the discussion phase of this type of solicitation."

Regarding the agency's criticism that the "feedback/ remediation" in its sample lesson was often the same for both a right and wrong answer, TSM acknowledges that, "the government comment is essentially a true statement." In general, TSM argues that, while its proposals may have contained flaws, those flaws should have been made the subject of discussion rather than considered bases for exclusion of its proposals from the competitive range.

The evaluation of proposals and the resulting determination as to whether a particular offer is in the competitive range are matters within the discretion of the contracting agency, since it is responsible for defining its needs and determining the best method of accommodating them. Smith Bright Assocs., B-240317, Nov. 9, 1990, 90-2 CPD Para. 382. Our Office will not substitute its judgment for the agency's regarding the relative merits of proposals but, rather, will examine the proposals and agency's evaluation to ensure that the evaluation was reasonable and consistent with applicable statutes and regulations and the stated evaluation criteria. Travel Centre, B-236061.2, Jan. 4, 1990, 90-1 CPD Para. 11. The fact that a protester disagrees with the agency's conclusions does not itself establish that the agency acted unreasonably. Id.

The Federal Acquisition Regulation (FAR) requires that the competitive range consist of all proposals that have a reasonable chance of being selected for award. FAR Sec. 15.609. However, where a proposal would require major revisions or submission of effectively a new proposal to be eligible for award, the agency is not required to include that proposal in the competitive range, see Advanced Micrographics, Inc., B-245319.2, Jan. 8, 1992, 92-1 CPD Para. 36, and an agency has no duty to hold discussions with an offeror whose proposal is outside the competitive range. Id.

We have reviewed the record in this case, including the RFP, TSM's proposals, and the agency's evaluation documents, and find no basis to question either the agency's determination that TSM's proposals contained multiple weaknesses and deficiencies, or its determination that, viewed in the aggregate, these weaknesses and deficiencies were so significant that substantial revisions would have been necessary for the proposals to qualify for contract awards.

None of TSM's responses to the agency's criticisms demonstrate that the agency's determinations were unreasonable or inconsistent with the evaluation criteria. For example, the RFP contemplated, and the agency reasonably expected that TSM's instructional media design would identify a particular proposed design to be used in creating the training materials. TSM's response that it was important for the developer to "apply common sense to a specific situation, and important for the course manager/instructor to ha[ve] the means to impact delivery of the courseware" neither negates the RFP requirement, nor justifies TSM's failure in the first instance to identify a particular design strategy that it was proposing to apply. Correction of TSM's proposals in response to this criticism alone would require substantial proposal revisions. TSM's other responses similarly fail to demonstrate that the agency improperly downgraded TSM's proposals for the omission of detailed flow charts, and "front matter," or for the inconsistencies within TSM's design and between its design and its sample lesson. The fact that it may have been possible for TSM to correct or improve its proposals in numerous areas in response to discussions does not establish that the agency was required to include the proposals in the competitive range. On the contrary, on this record, the agency reasonably eliminated TSM's proposals from the competitive range. [17]

TSM also complains that the agency improperly engaged in discussions with other offerors, asserting that "all offerors were advised there would be no [discussions or] BAFO." Contrary to TSM's assertion, the RFP stated: "No discussions are planned but will be opened if necessary." Further, since TSM's proposals were properly eliminated from the competitive range, TSM is not an interested party to challenge the agency's subsequent actions with respect to the competitive range offerors in this procurement. See, e.g., Satellite Transmission Sys., Inc., 70 Comp.Gen. 624 (1991), 91-2 CPD Para. 60.

The protest is denied.

1. TSM's protest does not question the agency's action regarding the CLINs set aside for small disadvantaged businesses.

2. The RFP defined "IVD courseware" as:

"The application of videodisc and computer to the delivery of instruction wherein there is an ongoing interchange of stimulus and reaction between computer and videodisc system and the user. Most of the visual information is presented by video with the computer managing the flow and keeping student records."

3. MIL-STD 1379D contains interactive courseware protocols. As explained in the RFP, MIL-STD 1379D "resulted from the military's decision to write standards for [interactive courseware] products so that military developers would use standard specifications."

4. The RFP explained that "level III" IVD systems involve any videodisc player that interfaces with an external computer; in a "level III" system, the videodisc player is essentially a computer peripheral with its functions under the computer's control. In contrast, in a "level IV" IVD system, the videodisc is used to store computer-readable digital data as well as analog video and sound information.

5. The solicitation stated that factor I was nearly twice as important as factor II, which was slightly more important than factor III.

6. The agency received nine proposals responding to the CLINS set aside for small businesses; 11 proposals responding to the CLINs set aside for small disadvantaged businesses; and nine proposals responding to the unrestricted CLINs.

7. Eighteen other proposals were eliminated from the competitive range. Following the competitive range determination, there were four proposals in the competitive range for the small business CLINs; three proposals in the competitive range for the small disadvantaged business CLINs; and four proposals in the competitive range for the unrestricted CLINs.

8. In its January 28 letter to TSM regarding its small business proposal, the agency erroneously provided TSM with comments applicable to the proposal of another offeror. By letter dated February 23, the agency corrected this error, providing TSM with the proper list of weaknesses and deficiencies applicable to its small business proposal.

9. The "module pretest" was a test administered prior to instruction to identify course knowledge the student already possessed.

10. The "module posttest" was a test administered after course completion to measure the student's mastery of the course objectives.

11. TSM's proposal listed six "conventions" on which its design relied: text, touch, routing, audio, video or "visuals," and information presentation.

12. As provided by the RFP, the flow charts to be submitted were to be detailed to include, for example: blocks of instruction; user paths; video, graphics and text frames with reference and sequence numbers; audio, still frame, and other configurations with reference and sequence numbers; and processing information such as branching, remediation, non- scored test items, motion sequences, computer processing, performance update, scored test items, scoring routine, and database search or entry to be used. The agency assessed the flow charts submitted by TSM as generic lesson charts, with little or no detail.

13. Under MIL-STD 1379D, an offeror's instructional media design was to contain "front matter" consisting of, among other things, an executive summary, a definitions section, a forward, and a "How to Use the Training Document" section. TSM's proposal contained none of these items.

14. For example, TSM's instructional media design provided, at page 19, that "timed stills" would be used to show cause, effect and animation; however, page 34 of TSM's design stated that animation would not be used, and the sample lesson TSM submitted did not use animation.

15. In addition to the above flaws in TSM's proposal under factor I-- which the RFP provided was nearly twice as important as either factor II or factor III--the agency identified multiple weaknesses and deficiencies in TSM's proposal under the less important evaluation factors. In light of our determination that the agency properly evaluated TSM's proposal under factor I, we do not discuss the details of the agency's evaluation under factors II and III.

16. In submitting its proposals, TSM stated: "As the subject matter information [in the RFP] did not provide all the data required to complete the [instructional media] design, TSM has made a number of assumptions."

17. Our discussion above includes only examples of weaknesses and deficiencies the agency properly found to exist in both TSM's small business proposal and TSM's proposal for the unrestricted CLINs. Although the agency found additional, unique flaws in TSM's proposal responding to the unrestricted CLINs, we find the agency's rejection of TSM's proposals was justified on the basis of the factors discussed above.