B-170324, APR 19, 1971

B-170324: Apr 19, 1971

Additional Materials:

Contact:

Edda Emmanuelli Perez
(202) 512-2853
EmmanuelliPerezE@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

WAS TO LIMIT SELECTION TO THOSE FIRMS UTILIZING EXISTING TECHNIQUES. THE EVALUATION PROCEDURE SEEMED TO CENTER ON THE ASSESSMENT OF INNOVATIVE CONFIGURATIONS AND EVEN AFTER PROTESTANT'S PROPOSAL WAS DETERMINED SUBSTANTIALLY LOWER THAN OTHER OFFERORS. MEANINGFUL NEGOTIATIONS WERE NEVER CONDUCTED WITH PROTESTANT TO UPGRADE ITS PROPOSAL. WHILE NO FURTHER ACTION IS POSSIBLE AT THIS TIME STEPS SHOULD BE TAKEN TO PRECLUDE SUCH DEFICIENCIES IN FUTURE PROCUREMENTS. CARLUCCI: FURTHER REFERENCE IS MADE TO ADMINISTRATIVE REPORTS OF AUGUST 7 AND 20 AND SEPTEMBER 15. REQUEST FOR PROPOSALS (RFP) 70-107 WAS ISSUED BY PUBLICATION IN THE COMMERCE BUSINESS DAILY. COPIES OF THE RFP WERE SENT TO 20 FIRMS.

B-170324, APR 19, 1971

BID PROTEST - EVALUATION PROCEDURE - DUTY TO NEGOTIATE DECISION CONCERNING PROTEST BY BEHAVIORAL RESEARCH LABORATORIES AGAINST FAILURE TO BE SELECTED FOR PARTICIPATION BY THE OFFICE OF ECONOMIC OPPORTUNITY IN ITS "PERFORMANCE INCENTIVE REMEDIAL EDUCATION EXPERIMENT." WHILE THE PURPOSE OF THE RFP, AS STATED THEREIN, WAS TO LIMIT SELECTION TO THOSE FIRMS UTILIZING EXISTING TECHNIQUES, THE EVALUATION PROCEDURE SEEMED TO CENTER ON THE ASSESSMENT OF INNOVATIVE CONFIGURATIONS AND EVEN AFTER PROTESTANT'S PROPOSAL WAS DETERMINED SUBSTANTIALLY LOWER THAN OTHER OFFERORS, MEANINGFUL NEGOTIATIONS WERE NEVER CONDUCTED WITH PROTESTANT TO UPGRADE ITS PROPOSAL, TO THE CHANGED TECHNICAL STANDARDS. WHILE NO FURTHER ACTION IS POSSIBLE AT THIS TIME STEPS SHOULD BE TAKEN TO PRECLUDE SUCH DEFICIENCIES IN FUTURE PROCUREMENTS.

TO MR. CARLUCCI:

FURTHER REFERENCE IS MADE TO ADMINISTRATIVE REPORTS OF AUGUST 7 AND 20 AND SEPTEMBER 15, 1970, RELATIVE TO THE PROTEST OF PATTON, BLOW, VERRILL, BRAND & BOGGS, ATTORNEYS AT LAW, ON BEHALF OF BEHAVIORAL RESEARCH LABORATORIES (BRL), AGAINST THE FAILURE OF THE OFFICE OF PLANNING, RESEARCH, AND EVALUATION TO SELECT BRL FOR PARTICIPATION IN THE OFFICE OF ECONOMIC OPPORTUNITY'S (OEO) "PERFORMANCE INCENTIVE REMEDIAL EDUCATION EXPERIMENT (RFP 70-107)."

REQUEST FOR PROPOSALS (RFP) 70-107 WAS ISSUED BY PUBLICATION IN THE COMMERCE BUSINESS DAILY, ISSUE NO. PSA-5053, DATED APRIL 27, 1970, AND INVITED QUALIFIED COMPANIES IN THE FIELD OF APPLIED EDUCATIONAL TECHNOLOGY TO SUBMIT PROPOSALS IN RESPONSE TO THE QUESTIONS AND ISSUES STATED THEREIN. IN ADDITION TO PUBLICATION IN THE COMMERCE BUSINESS DAILY, COPIES OF THE RFP WERE SENT TO 20 FIRMS. OFFERORS WERE ADVISED THAT OEO INTENDED TO AWARD COST-REIMBURSEMENT CONTRACTS TO APPROXIMATELY 24 SELECTED SCHOOL DISTRICTS WHICH WOULD, IN TURN, CONTRACT WITH UP TO SIX COMPANIES SELECTED BY OEO TO CARRY OUT THE TEACHING PROGRAM; THAT THE CONTRACTS WOULD BE OF THE PERFORMANCE INCENTIVE TYPE; AND THAT THE COMPANIES SELECTED MUST HAVE A DEMONSTRATED CAPABILITY IN EDUCATIONAL TECHNIQUES AND TECHNOLOGY APPROPRIATE FOR REMEDIATING DISADVANTAGED CHILDREN IN READING AND MATHEMATICS. OFFERORS WERE ALSO ADVISED THAT THE PURPOSE OF THE EXPERIMENT IS TO EVALUATE THE RELATIVE EFFECTIVENESS OF EXISTING TECHNIQUES, NOT TO UNDERWRITE THE DEVELOPMENT OF NEW TECHNIQUES AND THAT TO BE CONSIDERED, ORGANIZATIONS MUST DEMONSTRATE THEIR CAPABILITY TO CARRY OUT AN INSTRUCTIONAL PROGRAM OF THE MAGNITUDE DESCRIBED IN THE RFP BY SEPTEMBER 1970.

THE CRITERIA FOR EVALUATING PROPOSALS WERE SET FORTH IN THE RFP AS FOLLOWS:

"THE APPROX SIX CONTRACTORS WHO WILL PARTICIPATE IN THIS EXPERIMENT WILL BE SELECTED UPON THE BASIS OF THEIR RESPONSES TO THE FOLLOWING QUESTIONS: (A) A STATEMENT OF THEIR GENERAL CAPABILITY AND A DESCRIPTION OF ALL CORPORATE AND STAFF EXPERIENCES IN THE AREA OF APPLIED EDUCATION TECHNOLOGY AND TRAINING. (B) A FULL DESCRIPTION OF THEIR PROPOSED APPROACH, I.E. THE PARTICULAR MATERIALS, PROCEDURES, TYPES OF HARDWARE (IF ANY) AND SOFTWARE USED, ETC., AND A DISCUSSION OF PREVIOUS FINDINGS USING THIS APPROACH. (C) A DESCRIPTION OF HOW THEY PROPOSE TO SUPPLY INSTRUCTIONAL STAFF I.E. WHETHER THEIR OWN INSTRUCTORS WILL BE SUPPLIED, WHETHER THEY WILL TRAIN EXISTING TEACHERS OR OTHER LOCAL PEOPLE, ETC. (D) A DESCRIPTION OF THE INCENTIVES, IF ANY, WHICH ARE PART OF THEIR APPROACH AND WHOM THEY ARE MAINLY AIMED AT (E. G. PUPILS, TEACHERS, PARENTS, THE SCHOOL SYSTEM). (E) A DESCRIPTION OF THEIR APPROACH TO SCHOOL/CONTRACTOR COOPERATION, INCLUDING TEACHER'S UNIONS. THIS SECTION SHOULD ALSO INCLUDE A DESCRIPTION OF ANTICIPATED PROBLEMS OF PROGRAM IMPLEMENTATION AND PROPOSED SOLUTIONS TO THESE PROBLEMS.

"IN ADDITION, THE PROPOSAL SHOULD CONTAIN A DESCRIPTION OF THE BASIS UPON WHICH THEY PROPOSE TO RECEIVE PAYMENT. THIS SHOULD DESCRIBE THE RELATIONSHIP BETWEEN GRADE LEVEL INCREASES TO BE ACHIEVED, LENGTH OF TIME WITHIN WHICH ACHIEVEMENT OF INCREASE IS EXPECTED, AND A SCHEDULE OF PAYMENTS GRADUATED IN ACCORD., WITH THE ACTUAL LENGTH OF TIME REQUIRED TO ACHIEVE THE GRADE LEVEL INCREASE. DUE TO THE MULTITUDE OF UNKNOWN VARIABLES AT THIS STAGE (LOCATION OF SCHOOLS, LOCAL SALARIES, NUMBER OF STUDENTS, ETC.) ONLY GENERAL APPROXIMATIONS OF THE ABOVE ARE EXPECTED: THE PROPOSAL SHOULD DISCUSS THE VARIOUS COST FACTORS AND ASSUMPTIONS WHICH WERE CONSIDERED IN ARRIVING AT THIS ESTIMATE. THE PROPOSAL SHOULD STATE THE EXTENT TO WHICH THE TECHNIQUES AND MATERIAL TO BE USED IN THIS EXPERIMENT ARE PATENTED, COPYRIGHTED, OR OTHERWISE SUBJECT TO PROPRIETARY INTERESTS. TO THE EXTENT ANY OF THE MATERIALS AND TECHNIQUES ARE SUBJECT TO THE ABOVE INTERESTS, THE PROPOSAL SHOULD SET FORTH THE TERMS UPON WHICH THE CONTRACTOR IS PREPARED TO LICENSE SUCH TECHNIQUES OR MATERIALS."

THIRTY-ONE COMPANIES, INCLUDING BRL, RESPONDED BY THE CLOSING DATE OF MAY 11, 1970, AND AFTER INITIAL REVIEW OF THE PROPOSALS BY THE OEO EVALUATING COMMITTEE, A REQUEST FOR ADDITIONAL INFORMATION WAS SENT TO EACH COMPANY WHICH HAD RESPONDED TO THE RFP. AFTER EVALUATING THE ADDITIONAL INFORMATION FURNISHED BY EACH OFFEROR, THE EVALUATING COMMITTEE DETERMINED THAT THE PROPOSALS OF THE FOLLOWING SIX COMPANIES WERE ACCEPTABLE AND WITHIN A COMPETITIVE RANGE:

1. ALPHA LEARNING SYSTEMS, INC.

2. LEARNING FOUNDATIONS INTERNATIONAL, INC.

3. PLAN EDUCATION CENTERS, INC.

4. SINGER/GRAFLEX MANPOWER TRAINING DIVISION

5. WESTINGHOUSE LEARNING CORPORATION

6. QUALITY EDUCATIONAL DEVELOPMENT, INC.

THE SELECTION PANEL EVALUATED THE BRL PROPOSAL AND CONCLUDED THAT IT WAS UNACCEPTABLE BECAUSE:

" *** IT REPRESENTED AN INSTRUCTIONAL APPROACH DEEMED NOT TO BE WORTHY OF TESTING IN THE OEO PERFORMANCE INCENTIVE EXPERIMENT. GENERALLY SPEAKING, BRL PROPOSED TO SELL SCHOOLS READING AND MATHEMATICS MATERIALS, SPEND A FEW DAYS TRAINING THE SCHOOLS' TEACHERS TO USE THE MATERIALS, PUBLISH A BROCHURE FOR PARENTS EXPLAINING THE PROGRAM AND PROVIDE SOME NONDESCRIPTIVE CONSULTING THROUGH THE YEAR (THESE SERVICES EXCLUDING COST OF MATERIALS AMOUNT TO NO MORE THAN ONE MAN YEAR EFFORT FOR EACH SCHOOL DISTRICT). THE MATERIALS THEY PROPOSE TO USE HAVE BEEN EMPLOYED IN MANY SCHOOLS FOR THOUSANDS OF CHILDREN WITH WELL-ESTABLISHED RESULTS. DOCUMENTATION SUBMITTED WITH THE BRL PROPOSAL AND SUBSTANTIATED BY OUR STAFF, REVEAL THAT THE USE OF THESE MATERIALS WILL RESULT IN GAINS FROM SOME WHERE BETWEEN .5 AND 1.0 GRADE LEVELS. SINCE WE ALREADY KNOW WHAT THIS APPROACH WILL DO AND SINCE THE PRINCIPAL PURPOSE OF THE OEO EXPERIMENT IS THE ASSESSMENT OF THE EFFECTIVENESS OF NEW INSTRUCTIONAL APPROACHES USING EXISTING TECHNIQUES, THE BRL PROPOSAL WAS NOT CONSIDERED ACCEPTABLE.

"SPECIFIC REASONS FOR FINDING BRL'S PROPOSAL UNACCEPTABLE ARE AS FOLLOWS:

"I. THE PRIMARY PURPOSE OF THE PROJECT IS TO DETERMINE THE RELATIVE COST- EFFECTIVENESS OF VARIOUS INNOVATIVE BUT EXISTING LEARNING SYSTEM CONFIGURATIONS. IMPLICIT IN SUCH A PROJECT, OEO WANTED TO USE THE PERFORMANCE CONTRACT APPROACH TO DEMONSTRATE THE POTENTIAL OF PROMISING LEARNING SYSTEM CONFIGURATIONS. BRL READILY ADMITTED THAT THEIR LEARNING SYSTEM HAD BEEN DEMONSTRATED IN NUMEROUS PLACES; HOWEVER, BY THEIR OWN EVALUATION, THE RESULTS INDICATED THAT LESS THAN ONE YEAR GRADE LEVEL GAIN HAD BEEN ACHIEVED ELSEWHERE. CORRESPONDINGLY, THE BRL PROPOSAL TO OEO SPOKE OF ONLY ONE GRADE LEVEL GAIN. WHILE IT IS TRUE THAT THE RELATIVELY SMALL GAINS OBTAINED COULD HAVE BEEN EXPLAINED BY BRLS LACK OF CONTROL OVER SCHOOL TEACHERS IN WHICH THE PROJECTS WERE CONDUCTED, IN THEIR RESPONSE, BRL AGAIN CHOSE TO FOLLOW A SIMILAR APPROACH; HENCE, THEIR SYSTEM COULD NOT BE EXPECTED TO PRODUCE GREATER RESULTS.

"II. THE ADDENDUM SENT TO THE RESPONDEES TO THE RFP, INCLUDING BRL, REQUESTED EACH FIRM TO SUBMIT AN INCENTIVE FEE FORMULA TO BE THE BASIS FOR REIMBURSEMENT. IN THE BRL RESPONSE, NO SUCH FORMULA APPEARED REFLECTING THE VARIABLES OUTLINED IN THE ADDENDUM. RATHER, THE COST FOR ONE LEVEL OF ACHIEVEMENT WAS PRESENTED. NO PENALTY FORMULA RELATING TO INTERIM PERFORMANCE OBJECTIVES WAS PRESENTED AT ALL.

"III. THE RFP SPECIFICALLY STATED THAT NO DEVELOPMENTS OF TECHNIQUES WOULD BE UNDERWRITTEN. RATHER, EXISTING TECHNIQUES WOULD BE USED. FOLLOWING THE INTENT OF THE RFP, MANY FIRMS AND IN PARTICULAR THOSE SIX FINALLY CHOSEN PROPOSED TO USE EXISTING TECHNIQUES IN NEW INNOVATIVE CONFIGURATIONS. BRL PROPOSED TO UTILIZE ITS EXISTING TECHNIQUES WITHOUT ANY MODIFICATION.

"MOREOVER, ALL OF THE SIX FIRMS SELECTED PROPOSED TO USE MATERIAL MARKETED OR DEVELOPED BY OTHER FIRMS OR SOFTWARE HOUSES ALONG WITH THEIR OWN. BRL PROPOSED TO USE ONLY MATERIAL DEVELOPED AND MARKETED BY BRL OR ITS AFFILIATES. SINCE BRL IS IN THE LEARNING MATERIALS PUBLISHING BUSINESS, THEY PROPOSE TO USE ONLY THEIR MATERIALS IN THE STUDY. THE PANEL FELT THAT THIS WOULD LIMIT THE RANGE OF INSTRUCTIONAL APPROACHES TO WHICH CHILDREN COULD BE EXPOSED AND DOES NOT REFLECT A BROAD ENOUGH RANGE OF MATERIALS WHICH CAN BE BROUGHT TO BEAR ON THE PROBLEM.

"IV. IN THEIR PROTEST BRL RAISES THE ISSUE OF COST. THEY CLAIM THAT BECAUSE THEY GUARANTEE ONE GRADE LEVEL GAIN FOR $40 (NO PAYMENT FOR LESS THAN ONE GRADE LEVEL), THEY ARE THE LOW BIDDERS. IF ONE COMPARES THE BRL COST DIRECTLY TO THE COST OF ANY OF THE OTHER OFFERORS THEY ARE CORRECT. HOWEVER, THE COMPARISON IS SPURIOUS. THE BRL COST ONLY INCLUDES THE COST OF MATERIALS AND APPROXIMATELY ONE MAN YEAR EFFORT PER SCHOOL DISTRICT (DIVIDED AMONG 2-4 SCHOOLS WITHIN THE DISTRICT) AND DOES NOT INCLUDE TEACHERS SALARIES IN THEIR ESTIMATES. ASSUMING A $9,000 AVERAGE SALARY, THE TEACHER COST PER STUDENT, PER SUBJECT, PER YEAR IS $72. THIS COST MUST BE ADDED TO THE $40 FOR A TOTAL OF $112 FOR ONE GRADE LEVEL OR MORE GAIN. THE ONE GRADE LEVEL COST FOR OTHER OFFERORS IS APPROXIMATELY THE SAME. BRL DOES NOT CHARGE FOR ANY GAINS ABOVE ONE GRADE LEVEL. THEIR ACCOMPANYING DOCUMENTATION REVEALS THAT THEY WILL BE FORTUNATE TO GET TO THE ONE GRADE LEVEL.

"SINCE BRL DOES NOT CONTROL THE INSTRUCTIONAL PROCESS, THEY ARE NOT IN ANY POSITION TO GUARANTEE MORE ACHIEVEMENT. MOREOVER, ONCE THE GUARANTEE IS REACHED, BRL RECEIVES FULL COST/PROFIT RECOVERY AND THEREFORE HAS NO INCENTIVE TO GO BEYOND THIS MINIMUM EVEN IF BRL HAD A CAPABILITY FOR EXCEEDING THE MINIMUM. THE PAYMENT INCENTIVE PLAN PROPOSED BY BRL - WITH NO CORPORATE INCENTIVE TO ACHIEVE MORE THAN ONE GRADE LEVEL - WAS NOT COMPETITIVE WITH OTHER OFFERORS WHO DID NOT BREAK EVEN UNTIL THEY REACHED APPROXIMATELY 1.6 GRADE LEVELS AND FOR WHOM AN AVERAGE MAXIMUM GAIN WAS SET AT 2.0 GRADE LEVEL. EACH OF THE OTHER OFFERORS NEEDS ABOUT 1.6 GRADE LEVELS TO BREAK EVEN AND EACH FEELS HE CAN DO BETTER.

"AGAIN, WHEN CONSIDERING THE POSSIBLE GAINS V THRUST OF THE PROGRAM, BRL IS NOT COMPETITIVE. THE MAXIMUM GAIN PROPOSED BY BRL WAS THE MINIMUM FOR MOST OTHER FIRMS. IT DID NOT OFFER TO OEO OR THE SCHOOLS THE POTENTIAL TO INVESTIGATE HIGHER LEVELS OF ACHIEVEMENT."

SUBSEQUENTLY, OEO AWARDED CONTRACTS TO 18 SCHOOL DISTRICTS AND THE SCHOOL DISTRICTS, IN TURN, AWARDED CONTRACTS TO THE SIX FIRMS LISTED ABOVE. OEO APPROVED THE CONTRACTS WITH THE SIX FIRMS.

THE DECISIONS OF OUR OFFICE HAVE CONSISTENTLY HELD THAT AN RFP MUST ADVISE ALL OFFERORS OF ALL EVALUATION FACTORS AND OF THE RELATIVE IMPORTANCE OF EACH FACTOR. 49 COMP. GEN. 229 (1969); B-169645, JULY 24, 1970; B-167054, JANUARY 14, 1970. OEO MAINTAINS THAT THE INSTANT RFP DID COMPLY WITH THE NECESSARY REQUIREMENTS IN THAT IT WAS AS SPECIFIC AS POSSIBLE, CONSIDERING THAT IT DID NOT KNOW THE TECHNOLOGICAL APPROACHES IT INTENDED TO EMPLOY BUT ONLY THAT A POSSIBLE SIX APPROACHES WOULD BE APPLIED TO DETERMINE THE RELATIVE EFFECTIVENESS OF THE VARIOUS METHODS.

ON THE RECORD BEFORE US, WE ARE NOT PERSUADED THAT OEO'S TECHNICAL EVALUATION OF THE RELATIVE MERITS AND DISADVANTAGES OF THE PROPOSALS RECEIVED WAS ARBITRARY AND CAPRICIOUS AS ALLEGED. IN OUR VIEW, THE OVERRIDING QUESTION PRESENTED BY BRL'S ALLEGATION IS THE ADEQUACY OF THE EVALUATION PROCESS EMPLOYED BY OEO, PARTICULARLY THE SUFFICIENCY OF THE RFP.

THE PURPOSE OF THE RFP, AS STATED THEREIN, WAS TO LIMIT SELECTION TO THOSE FIRMS UTILIZING EXISTING TECHNIQUES AND NOT TO THOSE FIRMS WHICH WOULD REQUIRE OEO TO UNDERWRITE THE DEVELOPMENT OF NEW TECHNIQUES. HOWEVER, OEO EVALUATION GUIDELINE NO. 2 SETS OUT ONE CRITERIA AS "IN WHAT COMPONENTS OF THE TOTAL LEARNING SYSTEMS PROPOSED DO THERE EXIST UNIQUE INNOVATIVE FEATURES WHICH OFFER GREAT PROMISE FOR INCREASING THE READING AND MATH ACHIEVEMENT OF THE TARGET POPULATION?" FURTHER, ATTACHMENT TO THE LETTER FROM YOUR ASSISTANT GENERAL COUNSEL, DATED SEPTEMBER 15, 1970, REPORT ON "PROCESS BY WHICH EDUCATIONAL TECHNOLOGY COMPANIES WERE SELECTED," STATED THAT RESPONSES TO THE RFP "WERE JUDGED IN TERMS OF THEIR SOUNDNESS FROM A THEORETICAL AND CONCEPTUAL VIEWPOINT IN TERMS OF THE INNOVATIVE FEATURES OF THE TOTAL LEARNING SYSTEM CONFIGURATION."

IT ALSO APPEARS FROM AN ATTACHMENT TO THE OEO REPORT OF SEPTEMBER 15, 1970, ENTITLED "EVALUATION OF THE BEHAVIORAL RESEARCH LABORATORY (BRL) PROPOSAL," THAT ONE OF THE SPECIFIC REASONS ASSIGNED AS PARTLY DETERMINATIVE IN FINDING BRL'S PROPOSAL UNACCEPTABLE AND THUS NOT WITHIN A COMPETITIVE RANGE WAS ITS INTENDED USE OF EXISTING TECHNIQUES WHICH PREVIOUSLY HAD BEEN IMPLEMENTED. THIS APPEARS TO BE INCONSISTENT WITH THE STATEMENT IN THE RFP THAT "IN THE SELECTION OF THE SIX ORGANIZATIONS TO COMPRISE THE EXPERIMENT, CONSIDERATION WILL BE GIVEN ONLY TO THOSE ORGANIZATIONS WHICH HAVE A DEMONSTRATED CAPABILITY IN EDUCATIONAL TECHNIQUES AND TECHNOLOGY APPROPRIATE FOR REMEDIATING DISADVANTAGED CHILDREN IN READING AND MATH." WHILE CRITERIA (B) OF THE RFP REQUESTED A DISCUSSION OF PREVIOUS FINDINGS USING AN EXISTING APPROACH, THE EVALUATION PANEL PENALIZED BRL FOR SUBMISSION OF SUCH INFORMATION AND ASSIGNED AS ONE SPECIFIC REASON FOR REJECTION OF BRL'S PROPOSAL ITS DISCUSSION AND USE OF ITS EXISTING TECHNIQUE WITH ESTABLISHED RESULTS IN MANY AREAS. IT SHOULD ALSO BE NOTED HERE THAT THE SIX COMPANIES SELECTED WERE DEEMED SUPERIOR DUE IN PART TO THEIR PROPOSED INNOVATIVE FEATURES. THEREFORE, IT SEEMS THAT THE CRITERIA FOR SELECTION HAD EVOLVED FROM THOSE STATED IN THE RFP AS EXISTING TECHNIQUES AND DEMONSTRATED CAPABILITIES TO ASSESSMENT OF INNOVATIVE CONFIGURATIONS. EVEN CONCEDING THAT INITIALLY OEO MAY NOT HAVE HAD SPECIFICALLY FORMULATED SPECIFICATIONS DUE TO THE NATURE OF THIS EXPERIMENT, ONCE SUCH SPECIFICATIONS WERE CRYSTALLIZED, AN AMENDMENT TO THE RFP SHOULD HAVE BEEN ISSUED TO ALL RESPONDING COMPANIES APPRISING THEM OF RFP CHANGES. THIS WOULD HAVE SERVED TO NOTIFY OFFERORS OF THE CRITERIA AGAINST WHICH THEIR OFFERS WERE TO BE MEASURED AND WOULD HAVE PLACED THEM ON AN EQUAL BASIS. 50 COMP. GEN. (B-169645, JULY 24, 1970); 48 COMP. GEN. 314, 318 (1968); 47 COMP. GEN. 252, 262-263 (1967).

MOREOVER, THE TWO-PHASE EVALUATION PROCESS EMPLOYED HERE WAS NOT IN CONFORMITY WITH ESTABLISHED PROCUREMENT PRACTICES. PHASE I OF THE EVALUATION WAS CONDUCTED BEFORE ANY INFORMATION HAD BEEN RECEIVED CONCERNING COST OF THE PROGRAM. IT IS REALIZED, AND OUR OFFICE HAS SO HELD, THAT IN A NEGOTIATED CONTRACT, AWARD NEED NOT BE MADE UPON A DETERMINATION OF PRICE ALONE. 45 COMP. GEN. 417 (1966); B-169908(1), JULY 31, 1970; SECTION 1-3.805-1 OF THE FEDERAL PROCUREMENT REGULATIONS (FPR). BUT IN THE OEO REPORT ON "JUSTIFICATION FOR SELECTION OF CONTRACTORS," PHASE I EVALUATION, IT IS STATED THAT BRL'S PROPOSAL, AS WELL AS OTHERS, WAS "UNACCEPTABLE TO THE GOVERNMENT AND COULD NOT POSSIBLY LEAD TO THE SUCCESSFUL COMPLETION OF THE TASKS DESCRIBED IN THE RFP." THIS STATEMENT MUST BE CONSIDERED IN CONJUNCTION WITH EVALUATION GUIDELINE NO. 3:

"IF THE LEARNING SYSTEM WERE PROVEN TO BE COST-EFFECTIVE DURING THE YEAR OF OPERATIONS, COULD IT BE INCORPORATED INTO A TYPICAL SCHOOL SYSTEM WITHOUT A RELATIVELY LARGE LOSS OF EFFICIENCY? CONSIDER THE ADMINISTRATIVE CHANGES, START-UP AND TURNKEY COSTS, AND SOCIO-POLITICAL RAMIFICATIONS OF THE TURNKEY OPERATIONS WITH THE PARTICULAR SYSTEM." THUS, THE PHASE I PORTION, THE TECHNICAL REVIEW, WAS NECESSARILY INVOLVED WITH COST.

WHILE OEO NEVER CONTENDS THAT BRL'S PROPOSAL WAS NOT RESPONSIVE, IT APPEARS THAT IT WAS THE PHASE I DETERMINATION THAT BRL WAS NOT TO BE SELECTED. EVEN AFTER SUBSEQUENT PRICE INFORMATION WAS SUBMITTED WITH THE RESULT THAT BRL'S PRICE PROPOSAL WAS SUBSTANTIALLY LOWER THAN OTHER OFFERORS, MEANINGFUL NEGOTIATIONS WERE NEVER CONDUCTED WITH BRL TO UPGRADE ITS PROPOSAL TO THE CHANGED TECHNICAL STANDARDS. WE HAVE HELD THAT SOUND PROCUREMENT PRACTICE CONTEMPLATES THAT MEANINGFUL NEGOTIATIONS BE CONDUCTED WITH ALL OFFERORS WITHIN A COMPETITIVE RANGE. B-168190(2), FEBRUARY 24, 1970. WE RECOGNIZE THAT THE PROCUREMENT ACTIVITY HAS THE RESPONSIBILITY TO DETERMINE WHICH OFFERORS ARE WITHIN A COMPETITIVE RANGE. 45 COMP. GEN. 417 (1966). NEVERTHELESS, CONSIDERING THE QUOTED PRICE OF BRL, NEGOTIATIONS WITH THAT FIRM COULD HAVE RESULTED IN CONSIDERABLE SAVINGS TO THE GOVERNMENT HAD NOT THE INITIAL DETERMINATION OF THE TOTAL UNACCEPTABILITY OF ITS PROPOSAL BEEN MADE. THE TOTALITY OF THE CIRCUMSTANCES, INCLUDING PRICE, LEADS TO A CONCLUSION THAT NEGOTIATIONS SHOULD HAVE BEEN CONDUCTED WITH BRL.

FINALLY, COMMENT IS NECESSARY ON ONE FURTHER ARGUMENT ADVANCED BY YOUR ASSISTANT GENERAL COUNSEL TO EXPLAIN THE NEGOTIATION PROCEDURES EMPLOYED HERE. IN HIS LETTER DATED AUGUST 20, 1970, HE ASSERTS, IN EFFECT, THAT THE SOLICITATION AND RESULTING SELECTION PROCESS WERE NOT SUBJECT TO THE PROCEDURES GOVERNING DIRECT PROCUREMENT BY OEO SINCE IT MERELY SERVED IN THE CAPACITY OF AGENT FOR THE SCHOOL DISTRICTS IN CONDUCTING THE SUBCONTRACT PROCUREMENT PROCESS. WE ARE NOT PERSUADED. UNDER THEIR COST- REIMBURSEMENT CONTRACTS THE SCHOOL DISTRICTS' RIGHTS IN THE SELECTION OF SUBCONTRACTORS WERE SEVERELY LIMITED BY OEO; THE SUBCONTRACT WAS SUBJECT TO OEO SUPERVISION; AND OEO WAS INSTRUMENTAL IN DRAFTING THE TERMS OF THE SUBCONTRACT. SEE ATTACHMENT I, ADDENDUM TO THE RFP, WHICH PROVIDED (1) THAT THE SUBCONTRACT WOULD BE DEVELOPED WITH THE ASSISTANCE OF AN OEO CONTRACTING OFFICER AND MANAGEMENT AND SUPPORT GROUPS UNDER CONTRACT TO OEO; (2) THAT STUDENT EXCEPTIONS BE SUBJECT TO OEO APPROVAL; AND (3) THAT THE TESTS TO MEASURE STUDENT ACHIEVEMENT WOULD BE SELECTED BY OEO. MOREOVER, UNDER THIS ADDENDUM THE FINAL NEGOTIATED SUBCONTRACT WAS REQUIRED TO BE APPROVED BY OEO.

WE BELIEVE, THEREFORE, THAT THE PROCUREMENT WAS SUBJECT TO THE FEDERAL PROCUREMENT STATUTE AT 41 U.S.C. 254, ET SEQ., AND THE IMPLEMENTING REGULATIONS IN PART 1-3, FPR (ADOPTED BY OEO IN CHAPTER 22, TITLE 41, CFR). WHILE NO FURTHER ACTION IS POSSIBLE AT THIS TIME, WE RECOMMEND THAT FUTURE SIMILAR PROCUREMENTS BE ADMINISTERED IN ACCORDANCE WITH THE PRESCRIBED PROCEDURES GENERALLY APPLICABLE TO CIVILIAN AGENCIES OF THE GOVERNMENT.