B-234141, B-234141.2, B-234141.4, B-234141.5, May 17, 1989, 89-1 CPD 472

B-234141, B-234141.2, B-234141.4, B-234141.5, May 17, 1989, 89-1 CPD 472

PROCUREMENT - Competitive Negotiation - Offers - Evaluation errors - Administrative policies - Compliance DIGEST: 1. Alleged failure by contracting agency to comply with internal instructions for conducting proposal evaluation, which required a specific and detailed explanation for a proposal's unacceptability, is a matter for consideration within the agency itself rather than through the bid protest process; instead, the General Accounting Office will consider the reasonableness of the evaluation and compliance with any applicable statutes or regulations. PROCUREMENT - Competitive Negotiation - Technical evaluation boards Bias allegation - Allegation substantiation - Evidence sufficiency 2. Disparity in technical scoring among individual evaluators does not by itself cast doubt on the validity of evaluation panel's unanimous, final conclusion that protester's initial proposal was unacceptable, since it is not unusual for individual evaluators to reach disparate conclusions when judging proposals, as both objective and subjective judgments are involved. PROCUREMENT - Competitive Negotiation - Offers - Competitive ranges Exclusion - Administrative discretion 3. Agency determination that proposal is technically unacceptable and consequent exclusion from the competitive range will not be disturbed where proposal indicated limited organizational experience and contained deficiencies in required strategy for the implementation of total quality management such that the proposal had no reasonable chance for award and would require major revisions to be acceptable. PROCUREMENT - Competitive Negotiation - Technical evaluation boards Qualification - GAO review 4. The composition of technical evaluation panels is within the discretion of the contracting agency and, as such, will not be reviewed by the General Accounting Office absent a showing of possible bad faith, fraud, conflict of interest or actual bias on the part of evaluators. PROCUREMENT - Bid Protests - GAO procedures - Protest timeliness - Apparent solicitation improprieties PROCUREMENT - Competitive Negotiation - Requests for proposals - Amendments - Compliance time periods - Adequacy 5. Protest that offeror was not allowed sufficient time to prepare a revised proposal after delayed receipt of amendment to request for proposals is untimely where protest was not filed by the extended closing date for submission of proposals. PROCUREMENT - Specifications - Performance specifications - Adequacy 6. Protest that agency should have provided offerors with greater detail concerning the expected manner of compliance with requirements in request for proposals (RFP) is denied; the RFP clearly set forth the minimum elements of the required total quality management implementation plan, and there is no requirement that an agency specify precisely the manner in which offerors are to fulfill performance requirements.

Stat-a-Matrix, Inc.; National Graduate University; Stat-a-Matrix Institute; International Management Systems, Inc.:

Stat-a-Matrix, Inc.; National Graduate University; Stat-a-Matrix Institute; and International Management Systems, Inc., protest the Office of Personnel Management's (OPM's) rejection of their proposals under request for proposals (RFP) No. 88-2795.

We deny the protests.

The RFP solicited consulting and training services for the purpose of assisting federal agencies in implementing total quality management (TQM), which is a management approach that relies on principles of quality assurance and makes managers and employees alike responsible for providing quality products and services. The statement of work (SOW) in the solicitation established the following overall goals for the implementation of TQM: (1) create an organizational culture that emphasizes excellence in service delivery; (2) meet customer requirements and increase customer satisfaction; (3) achieve continuous improvement in the quality of products and services and the processes used to produce them; (4) increase productivity; and (5) achieve participative involvement of employees in improvement efforts. The solicitation required the implementation and achievement of these goals by all levels within an agency, including the senior management, middle management, and work force, and specifically called for a description of the approach and strategies that would be used to enable federal agencies and their employees to understand TQM, make a long term commitment to the improvement process, build and improve the necessary management structure, acquire the skill of systematic problem solving, and function effectively in quality improvement teams.

The solicitation contemplated the award of multiple fixed-price Federal Supply Schedule contracts against which agencies deciding to implement TQM would place delivery orders. It provided for award to be made to those responsible offerors whose offers conform to the solicitation and are most advantageous to the government, with technical quality more important than price. The RFP stated that technical proposals would be evaluated on the basis of the following evaluation criteria, listed in descending order of importance: (1) implementation strategy and relevant materials and/or services proposed; (2) relevant experience and demonstrated capability; (3) professional staff capabilities; and (4) organizational capability.

The agency received 80 proposals by the October 11, 1988 closing date. In the initial technical evaluation, the technical evaluation panel, composed of 3 members, rated 25 proposals as acceptable or capable of being made acceptable and 55 proposals as unacceptable and not reasonably susceptible of being made acceptable. The proposals of all of the protesters here were determined technically unacceptable, principally for failure to provide a TQM implementation strategy and clearly establish prior TQM experience. OPM rejected the proposals as having no reasonable chance for award. We address each of the protests separately below.

NATIONAL GRADUATE UNIVERSITY

The University's proposal was found technically unacceptable on the basis that it did not present an overall strategy for implementing TQM; the agency determined that the proposal focused primarily on providing training courses, included little discussion of providing consulting services, and did not adequately explain its strategy for the implementation of TQM principles learned in the classroom. In addition, the agency found that the proposal did not show any experience in the implementation of TQM.

The University first questions the composition of the evaluation panel. The panel consisted of: (1) the Chief of the Productivity Management Branch, Office of Management and Budget, (2) the Chief of the Resources Management Division, Internal Revenue Service (IRS), and (3) the Deputy Associate Director for Training and Investigations, OPM, who was the agency's senior career executive in charge of all OPM training programs. The University believes the panel should have included a representative from either the Federal Quality Institute, the federal office charged with introducing TQM to federal agencies, or the several quality assurance offices within the Department of Defense.

The composition of technical evaluation panels is within the discretion of the contracting agency and, as such, will not be reviewed by our Office absent a showing of possible bad faith, fraud, conflict of interest, or actual bias on the part of evaluators. New Mexico State University, B-230669.2, June 2, 1988, 88-1 CPD Para. 523. None of these factors is shown or even alleged here. Moreover, it is our view that the important and responsible positions held by the agency evaluators here constituted prima facie evidence that they were qualified to evaluate proposals. Communications and Data Systems Assocs., B-223988, Oct. 29, 1986, 86-2 CPD Para. 491.

The University next complains that the evaluation panel's written assessment of its proposal did not comply with the agency's internal guidelines for conducting the evaluation, which required a specific and detailed explanation for a proposal's unacceptability. We will not consider the matter. An agency's internal instructions and procedures do not have the force and effect of law, so that the alleged failure to comply with them in a particular instance involves a matter for consideration within the agency itself, rather than through the bid protest process. Holsman Services Corp., B-230248, May 20, 1988, 88-1 CPD Para. 484; Spectron Caribe, Inc., B-224251, Nov. 25, 1986, 86-2 CPD 609.

The University also challenge the evaluation, complaining that the agency had an inadequate understanding of the content of its proposed training, which it maintains constituted a strategy for implementing TQM, and that the agency overlooked the achievements of its faculty, which included authoring both a text on TQM and the University's proposed TQM source materials. In this regard, the protester contends that disparities in the scoring of its proposal including findings of technical unacceptability based scores of 0 and 10 out of 100 available points, and a finding of technical acceptability with a score of 95 points, demonstrate the agency's inadequate understanding of its TQM implementation strategy, as well as an insufficient review of the University's experience.

The determination of the relative merits of proposals is primarily the responsibility of the contracting agency, which must bear the burden of any difficulties resulting from a defective evaluation. Pitney Bowes, B-233100, Feb. 15, 1989, 68 Comp.Gen. ***, 89-2 CPD Para. 157. Accordingly, in reviewing complaints about the evaluation of a technical proposal and the resulting determination of whether the proposal is within the competitive range, we will not reevaluate the proposal and independently judge its merits; we will only consider whether the evaluation was reasonable and otherwise consistent with procurement laws and regulations. Educational Computer Corp., B-227285.3, Sept. 18, 1987, 87-2 CPD Para. 274. Although the competitive range of offerors to be included in discussions must include all proposals that have a "reasonable chance of being selected for award," and any doubt as to whether a proposal is in the competitive range should be resolved by inclusion, Federal Acquisition Regulation (FAR) Sec. 15.609(a), our Office will not disturb an agency's decision to exclude a firm from the competitive range where its technical proposal is reasonably considered so deficient that it would require major revisions to be acceptable. Vista Videocassette Services, Inc., B-230699, July 15, 1988, 88-2 CPD Para. 55.

We find no basis to question the determination of technical unacceptability. The University's proposal does not explain how the "off- the shelf" training it offers in the form of various courses would provide for TQM implementation, the most important technical criterion in the RFP. The RFP specifically required a description of the approach that would be used in TQM implementation and we think the agency reasonably concluded that course offerings by themselves, without explanation, do not meet this requirement. For example, the proposal states that the University's approach "will provide senior management with a framework within which TQM policies, goals and management systems can be articulated" and "will also explore ... government-unique issues affecting TQM." We agree with OPM that merely proposing to articulate and explore TQM does not constitute a specific, total strategy for implementing TQM.

Further, the University's proposal does not indicate previous specific organizational experience in TQM implementation. The RFP stated that it was "imperative that offerors demonstrate experience and capabilities directly related to the proposed instructional approaches and implementation strategies," and requested detailed descriptions, as well as references, for prior, relevant TQM efforts that would substantiate a claimed capability to perform the contract work. The University's proposal only indicated experience in providing courses primarily on statistical process control and federal procurement, not experience in TQM implementation; the fact that its faculty has written on the subject of TQM does not by itself show experience in actual TQM implementation. Although the University's professional staff appears to have some TQM implementation experience, such experience is limited; for the most part their experience is in other areas such as federal procurement.

As for the scoring disparities, we have long recognized that it is not unusual for individual evaluators to reach disparate conclusions when judging competing proposals, since both objective and subjective judgments are involved, Digital Radio Corp., B-216441, May 10, 1985, 85-1 CPD Para. 526; thus the mere fact that an individual evaluator gives a perfect score or that the scores of certain evaluators are extreme or inconsistent does not, by itself, render the scores invalid. Panuzio/Rees Assocs., B-197516, Nov. 26, 1980, 80-2 CPD Para. 395. In any case, here, the record indicates that the panel member who originally scored the University's proposal acceptable with 95 points subsequently reconsidered the evaluation and found the proposal unacceptable after concluding that the University's references in its proposal to TQM experts did not indicate that these experts were proposed employees or that the University itself had specific implementation experience. We find no indication in the record that the scoring by the technical evaluation panel reflects other than their reasoned judgment as to the University's proposal. Thus, the variance in the initial scoring by itself does not give us reason to question the scoring, particularly when all the evaluators finally concluded that the University's proposal was unacceptable.

We conclude that it was not unreasonable for the agency to determine that the proposal would require major revisions to be made acceptable, and that the University thus had no reasonable chance for award.

STAT-A-MATRIX INSTITUTE

The Institute first argues that it was not promptly notified of an extension to the closing date for submission of proposals. While an amendment extending the original October 3, 1988, closing date to October 11 was issued on September 23, the protester states it did not learn of the extension until an October 3 telephone conversation with OPM officials and did not receive a copy of the amendment until October 4. The protester complains that it thus was not given the same length of time as other offerors to prepare a revised proposal.

Our Bid Protest Regulations provide that where alleged improprieties do not exist in the initial solicitation, but are subsequently incorporated into the solicitation, the alleged improprieties must be protested no later than the next closing date for receipt of proposals following the incorporation. 4 C.F.R. Sec. 21.2(a)(1) (1988). Here, although the Institute knew or should have known of its basis for protest no later than October 4, when it received the amendment, it did not file its protest until January 13, 1989, after notification of the rejection of its proposal. Thus, its protest in this regard is untimely and will not be considered.

See Joseph H. Carter, B-227094.2, Nov. 9, 1987, 87-2 CPD Para. 463.

The Institute also questions the evaluation of its proposal. The proposal, which consisted of offered courses leading to a master of science degree in quality management, as well as non-degree courses and training material, was rejected on the basis that it did not present a total strategy for implementing TQM throughout an organization, and did not show specific TQM experience or staff capability. OPM determined that federal agencies could utilize the Institute's offerings, consisting of such courses a quality engineering, quality improvement, and statistics and process control, only after devising their own strategies for TQM implementation. The Institute contends that its course offerings constituted its strategy, that is, to educate a cadre of individuals within the organization on the concepts and benefits of TQM, so that they then could apply TQM principles to the workplace. Further, the protester maintains that its proposal in fact described previous TQM assistance it had provided to several large organizations.

We agree with the agency that the Institute proposal does not explain in detail the steps by which the principles of TQM will be implemented throughout the agencies. While course offerings may be a component of a TQM implementation strategy, we agree with OPM that courses, by themselves without some broader framework or explanation, do not constitute a clearly defined, overall strategy for implementation. Regarding prior experience, as already noted the RFP specifically required a detailed description of past experience in providing assistance, including a statement of the objectives, methodology, achievements, innovations, date and time span, and cost of the assistance, and the submission of references. While the Institute's proposal briefly described a program to introduce Food and Drug Administration personnel to TQM concepts and also stated that the Institute had assisted corporate quality institutes in becoming familiar with TQM, our review of the proposal indicates that the specific and detailed description of past experience required by the RFP was not provided, and the fact that, as noted by the protester, the Institute has been accredited to grant a master of science degree in quality management does not by itself demonstrate the existence of experience in actually implementing TQM.

An offeror must demonstrate affirmatively the merits of its proposal, and it runs the risk of rejection if it fails to do so. Vista Videocassette Service, Inc., B-230699, supra; Computer Brokers, B-226103.2, Nov. 30, 1987, 87-2 CPD Para. 526. Here, the Institute simply did not document relevant TQM experience as required and did not provide for satisfying all the solicitation requirements. Its proposal, therefore, reasonably was found to be not susceptible of being made acceptable without major revisions, and elimination of the proposal from the competitive range was unobjectionable.

STAT-A-MATRIX, INC.

The Stat-a-Matrix proposal was determined technically unacceptable on the basis that it did not present an acceptable strategy for the implementation of TQM and did not indicate organizational and staff experience in TQM implementation. The agency maintains that to accomplish the RFP objectives, a proposed implementation strategy had to address what it describes as both the quantitative and human dimensions of TQM, that is, the proposal had to demonstrate both the knowledge and skills the offeror would provide to the management and the workforce, and how the offeror would assist agencies in attaining the participative involvement of employees in TQM improvement efforts (implementation goal number five, as listed in the SOW). Although the agency determined that Stat-a- Matrix's proposed application of statistical process control techniques to management functions represented a strength with respect to the quantitative element of TQM, it found the proposal lacking with respect to how to assist people in a large organization in restructuring the environment to one conducive to TQM.

Stat-a-Matrix argues that the agency improperly evaluated the firm's proposal. First, the protester alleges that the agency evaluated the proposal on the basis of an undisclosed and undefined evaluation criterion, the human aspect of TQM implementation; it complains that the participative involvement of employees in improvement efforts was listed in the statement of work only as a goal and should not have been evaluated as a requirement.

Our review of the record indicates that the agency's technical evaluation was consistent with the evaluation factors set forth in the RFP. The RFP listed the proposed implementation strategy as the most important evaluation criterion, while the SOW specifically indicated that in TQM implementation OPM was seeking to achieve the participative involvement of employees in improvement efforts. Accordingly, although the SOW termed employee involvement a TQM "goal," it is clear from the RFP as a whole that the proposed strategy for achieving this goal was to be a primary consideration in the evaluation of proposals. We think it should have been apparent from this description of the requirements that a successful proposal would need to address the human element of TQM implementation. Although Stat-a-Matrix may have preferred added detail in the RFP, we have held that a solicitation need not specify the manner in which offeror's are to fulfill performance requirements. Pitney Bowes, B-233100, supra; see Personnel Decisions Research Institute, B-225357.2, Mar. 10, 1987, 87-1 CPD Para. 270.

Stat-a-Matrix also argues that OPM's evaluation ignored the fact that its information-driven management approach to TQM, that is, acquiring information to identify and distinguish management-controllable causes of defects from employee-controllable causes, in fact involves the human element of TQM. However, while this may be the case, our review provides no basis to question the agency's conclusion that the protester's proposal does not actually explain exactly how the firm would provide for the required participative involvement of all employees in TQM implementation. Instead, the firm's proposal emphasizes its approach as "utilizing and managing" of information. We do not think OPM was required to infer the specifics of the protester's approach.

Stat-a-Matrix's protest of the evaluation of its implementation strategy appears to arise largely from the firm's disagreement with the agency's philosophy of TQM implementation and the determination of its minimum needs. Specifically, Stat-a-Matrix questions whether the focus should be on the participative involvement of all employees in TQM implementation; according to the protester, based on its experience, it has determined that 80 to 90 percent of the opportunity for improvement is management- controllable, not employee-controllable and, moreover, that a 2-year time period is necessary to prepare an organization to deal with the TQM cultural change prior to involving the workforce. This assertion essentially challenges the statement of requirements and evaluation criteria in the solicitation, which emphasized the importance of the participative involvement of all employees, and therefore is untimely; alleged solicitation improprieties in the solicitation must be protested prior to the closing date for receipt of proposals. 4 C.F.R. Sec. 21.2(a)(1).

The protester also challenges the evaluation of its stated experience, noting that its proposal listed 20 years of TQM experience and that the firm's proposal initially received high scores under the criterion for experience from two of the three evaluators. The record, however, indicates that two of the three evaluators, including one of the evaluators who initially gave the firm a high score for experience, nevertheless determined Stat-a-Matrix's proposal to be unacceptable. Further, the panel as a whole ultimately concluded that the proposal was unacceptable, in part because the listed experience was not in TQM implementation.

Our review of the Stat-a-Matrix proposal confirms the evaluation panel's ultimate conclusion that the proposal does not indicate specific TQM implementation experience. Rather, the previous efforts described in the proposal primarily encompassed programs and training involving statistical analysis techniques, statistical process control, and the firm's trademarked program of "statistically-aided management," and did not include specific experience with the so-called human element of TQM, i.e., the participative involvement of employees in TQM implementation. While the firm may have extensive TQM implementation experience as it claims, its proposal simply did not give an indication of it. As we have stated, it is an offeror's responsibility to submit a well-written proposal which clearly demonstrates compliance with the RFP requirements. Communications and Data Systems Assocs., B-223988, supra.

Finally, Stat-a-Matrix contends that the evaluation panel was biased against small firms because the individual evaluation sheets refer to the firm as small. We do not consider the mere reference to a firm as small to constitute the hard evidence of favoritism or antagonism toward a particular offeror necessary to establish bias. Computer Brokers, B-226103.2, supra.

We conclude that the agency reasonably concluded that Stat-a-Matrix's proposal would require major revisions in order to become acceptable, and that elementation of the proposal from the competitive range therefore was proper.

INTERNATIONAL MANAGEMENT SYSTEMS, INC.

International's proposal also was rejected based on an inadequate strategy for TQM implementation and limited firm and staff experience in TQM implementation. The agency found that the proposal emphasized statistical process control and considered this to be an inadequate and incomplete strategy for implementing all aspects of TQM, including the human element. Although International proposed an outline of topics to be covered in its recommended training program, which included such areas as human relations and team concepts, the agency found that the proposal did not explain how the work force would be brought to accept and use TQM. The agency further determined that the resumes of the proposed professional staff indicated experience in areas such as quality control and industrial engineering, but very little experience in TQM.

International first contends that its proposal properly was evaluated on the basis of an undisclosed and vague criterion, the human element of TQM. As previously discussed, however, we believe it was clear from the RFP that an offeror was required to detail its approach to TQM implementation through employee participative involvement, the so-called human element of TQM.

Next, International argues that its proposal was inadequately evaluated by OPM in that the proposal in fact met all of the requirements for TQM set forth in the RFP. In this regard, the protester maintains that its proposal provided for the participative involvement of employees in steps two, three, and four of its six-step approach which, respectively, provided for organizing and planning, analysis, and the development of human resources. We have reviewed the proposal, however, and find no provision for the participative involvement of employees in TQM implementation in steps two and three of International's approach. Further, although the outline of topics for International's proposed skills training program in step four includes human relations and team concepts, the emphasis of the outline is on statistical process control. In any case, we agree with the agency that simply listing the titles of the topics to be covered does not satisfy the requirement for an explanation of how the firm will provide for the work force to be brought to accept and use TQM. With respect to the firm's previous experience, its proposal listed and described in detail the activities it conducted for only one client, and then only in the implementation of statistical process control, not TQM. Further, our review of International's proposal leads us to agree with the evaluation panel's final consensus determination that the resumes included with the proposal indicated experience primarily in areas other than specific TQM implementation, such as in statistical quality control and quality control engineering. this regard, the RFP specifically provided that resumes must be sufficiently detailed to permit an assessment of the capacity of the professional staff to do work of the kind described in the RFP. While the protester believes its proposed staff has the requisite TQM experience, this was not demonstrated in the resumes submitted with its proposal.

In view of these deficiencies and weaknesses, we find no basis to question the elimination of International's proposal from the competitive range.

The protests are denied.