B-173265, AUG 20, 1971, 51 COMP GEN 102

B-173265, AUG 20, 1971, 51 COMP GEN 102

EQUIPMENT - AUTOMATIC DATA PROCESSING SYSTEMS - SELECTION AND PURCHASE - NEGOTIATION PROCEDURES ALTHOUGH ALL PERTINENT PORTIONS OF THE WORK STUDY REPORT USED IN THE PREPARATION OF A REQUEST FOR PROPOSALS (RFP) FOR A DATA BASE MANAGEMENT SYSTEM (DMS) SHOULD HAVE BEEN PHYSICALLY INCLUDED IN THE RFP FOR SAKE OF CLARITY SINCE THE RFP INCORPORATED THE REPORT BY REFERENCE AS WELL AS APPRISING OFFERORS OF PROCUREMENT REQUIREMENTS, THE TIME TO QUESTION THE ADEQUACY OF EVALUATION CRITERIA AND THEIR IMPORTANCE WAS PRIOR TO PROPOSAL SUBMISSION. FURTHERMORE, ON THE BASIS OF THE COST EFFECTIVENESS FORMULA IN THE REPORT, THE USE OF OPERATION AND MAINTENANCE COSTS COMPUTED ON A 5- YEAR CYCLE TO DETERMINE THE MOST ADVANTAGEOUS PROPOSAL IN A COMPETITIVE RANGE, A PROCEDURE THAT IS PER SE ACCEPTABLE IF SUCH COSTS ARE REASONABLE, WAS PROPER, EVEN THOUGH THE OPERATION AND MAINTENANCE COSTS WERE INCAPABLE OF PRECISE ASSESSMENT AND WERE ONLY PROJECTED COSTS. CONTRACTS - NEGOTIATION - AWARDS - INITIAL PROPOSAL BASIS THE FACT THAT INITIAL PROPOSALS MAY BE RATED AS ACCEPTABLE DOES NOT INVALIDATE THE NECESSITY FOR DISCUSSIONS OF WEAKNESSES, EXCESSES, OR DEFICIENCES IN THE PROPOSALS SO THAT THE CONTRACTING OFFICER MAY OBTAIN THE MOST ADVANTAGEOUS CONTRACT FOR THE GOVERNMENT, THEREFORE, WHERE THE RECORD OF AN AWARD MADE ON THE BASIS OF THE MOST FAVORABLE INITIAL PROPOSAL PURSUANT TO SECTION 1-3.805-1(A)(5) OF THE FEDERAL PROCUREMENT REGULATION EVIDENCES DISCUSSIONS WERE CONDUCTED WITH ALL OFFERORS WITHIN COMPETITIVE RANGE, PRICE AND OTHER FACTORS CONSIDERED, AND THAT ALL OFFERORS WERE TREATED SIMILARLY, IN ORDER TO ELIMINATE UNCERTAINTIES, THE DISCUSSIONS WERE "MEANINGFUL," REGARDLESS OF WHETHER THE TERM EMPLOYED DURING THE PROCUREMENT PROCEDURES WAS "DISCUSSION" OR "NEGOTIATION" SINCE BOTH TERMS ARE CONSIDERED SYNONYMOUS.

TO THE SECRETARY OF AGRICULTURE, AUGUST 20, 1971:

BY LETTERS, WITH ENCLOSURES, DATED JULY 12 AND 28, 1971, THE ACTING DIRECTOR, OFFICE OF PLANT AND OPERATIONS, FURNISHED OUR OFFICE REPORTS ON THE PROTEST OF PROGRAMMING METHODS INC., AGAINST THE FOREST SERVICE'S AWARD OF A CONTRACT ON JUNE 3, 1971, TO TRW SYSTEMS GROUP UNDER REQUEST FOR PROPOSALS (RFP) NO. 5-71.

THE SUBJECT RFP WAS ISSUED ON FEBRUARY 12, 1971, WITH A CLOSING DATE OF MARCH 29, 1971. A PREPROPOSAL CONFERENCE WAS SCHEDULED FOR MARCH 2, 1971. THE PROCUREMENT INVOLVES COMPUTER SOFTWARE, RELATED TRAINING AND DOCUMENTATION FOR A GENERALIZED DATA BASE MANAGEMENT SYSTEM (DMS) TO SUPPORT THE DEVELOPMENT, IMPLEMENTATION AND OPERATION OF A FOREST SERVICE- WIDE MANAGEMENT INFORMATION SYSTEM (INFORM). THE DMS IS INTENDED TO SUPPORT THE INTEGRATION OF DATA FROM FOREST SERVICE RECORD FILES GROUPED INTO A SERIES OF FUNCTIONAL AREAS. INFORM IS INTENDED TO FULFILL THE INFORMATION REQUIREMENTS FOR PLANNING AND CONTROL AT ALL LEVELS OF THE FOREST SERVICE ORGANIZATION. AS STATED IN PARAGRAPH 1 OF THE INTRODUCTION TO THE SPECIFICATIONS, THE FOREST SERVICE WAS "SEEKING AN EXISTING, OR AN ADAPTATION OF AN EXISTING, DMS," ONE "WHICH CAN BE MODIFIED AND EXPANDED *** AND CAN BE INSTALLED UPON HARDWARE OF VARIOUS MANUFACTURERS AND AT VARIOUS FOREST SERVICE OWNED AND LEASED COMPUTING FACTILITIES." PARAGRAPH 2 FURTHER ADVISED OFFERORS THAT:

THE FOREST SERVICE HAS LIMITED HARDWARE CAPABILITIES NOW, BUT PLANS TO EXTEND ITS CAPABILITIES IN THE NEXT TWO OR THREE YEARS TO INCLUDE A LARGE, CENTRAL COMPUTING FACILITY WHICH WILL SUPPORT NINE OR TEN REGIONAL COMPUTING CENTERS. THE REGIONAL SITES MAY HAVE SEVERAL REMOTE BATCH TERMINALS. CURRENT PLANS ARE FOR ALL SITES TO BE EVENTUALLY CONNECTED BY TELECOMMUNICATION LINES. THE HARDWARE FOR THIS EXTENSION HAS NOT BEEN SELECTED.

THE RFP WAS DEVELOPED FROM A STUDY AND REPORT DATED OCTOBER 20, 1970, MADE BY THE AUERBACH CORPORATION UNDER CONTRACT WITH THE FOREST SERVICE. THE FINAL REPORT, AUER-1834-TR-2, INFORM DATA MANAGEMENT SYSTEM STUDY, DETAILS FOREST SERVICE REQUIREMENTS IN TERMS OF RECOMMENDED DMS SPECIFICATIONS, WHICH ARE REFLECTED IN LARGE MEASURE IN THE RFP. IT ALSO DETAILS A SYSTEM FOR THE TECHNICAL AND COST EVALUATION OF PROPOSALS. THE TECHNICAL EVALUATION SYSTEM WAS BASED ON A FEATURE LIST WEIGHTED TO REFLECT FOREST SERVICE REQUIREMENTS. THE WEIGHTED FEATURES LIST FROM THE AUERBACH REPORT WAS INCLUDED IN THE RFP AS EXHIBIT 6. SECTION IV, SYSTEM EVALUATION METHODOLOGY, AND APPENDIX "E," GUIDELINES FOR THE EVALUATOR, OF THE AUERBACH REPORT, WHICH OUTLINE THE METHOD OF EVALUATION TO BE FOLLOWED, WERE NOT INCLUDED IN THE RFP. THE AUERBACH REPORT IS, HOWEVER, REFERENCED IN CLAUSE 10, AWARD OF CONTRACT, OF THE SUPPLEMENTAL INSTRUCTIONS AND CONDITIONS OF THE RFP, WHICH OUTLINED THE EVALUATION PROCEDURES AND THE BASIS FOR DETERMINING WHICH OFFER IS MOST ADVANTAGEOUS, IN THE FOLLOWING TERMS:

CLAUSE 10 - AWARD OF CONTRACT.

(I) EVALUATION OF PROPOSALS. UPON SUBMISSION OF PROPOSALS BY BIDDERS, THE TECHNICAL PROPOSALS FOR EACH CONTRACT ITEM WILL BE REVIEWED AND EVALUATED BY AN EVALUATION COMMITTEE WHO WILL NOT HAVE ACCESS TO THE PRICE PROPOSALS. THE COMMITTEE WILL RATE EACH PROPOSAL ON THE BASIS OF ITS TECHNICAL MERIT. TECHNICAL PROPOSALS WILL BE RATED ON THE BASIS OF THE FOLLOWING FACTORS. A POINT SYSTEM WILL BE USED. THE MAXIMUM POSSIBLE POINT VALUES FOR EACH FACTOR ARE:

1. QUALIFICATIONS OF FIRM 50

(REPUTATION, RELEVANT EXPERIENCE)

2. QUALIFICATIONS OF PROPOSED STAFF 50

(QUALIFICATIONS OF LEADERSHIP AND OTHER STAFF PROPOSED)

3. APPROACH AND PERFORMANCE PLAN 50

(COMPREHENSION, CLARITY, METHOD OF PERFORMING THE CONTRACT)

4. DMS - WEIGHTED FEATURES 850

TOTALS 1,000

NOTE: THE DMS WILL BE RATED ON THE BASIS OF A METHODOLOGY OUTLINE IN FINAL REPORT, AUER. - 1834-TR-2, INFORM DATA MANAGEMENT SYSTEM STUDY. THE FEATURES TO BE RATED ARE INCLUDED IN SECTION 6 OF THE SPECIFICATIONS. PROSPECTIVE OFFERORS WHO WISH TO REVIEW THE ABOVE REPORT MAY ARRANGE TO DO SO BY CONTACTING MR. PHILLIP HAUG, ROOM 4226, SOUTH AGRICULTURE BUILDING, TELEPHONE NO. (202) 388-5309.

AFTER EVALUATION OF THE TECHNICAL PROPOSALS IS COMPLETED, THE COMMITTEE WILL EVALUATE PRICE QUOTATIONS FOR THOSE PROPOSALS MEETING THE DMS REQUIREMENTS. PROPOSALS REMAINING WITHIN A COMPETITIVE RANGE CONSIDERING PRICE, COST EFFECTIVENESS, AND OTHER FACTORS SHALL THEN BE GIVEN, WITH THE COMMITTEE'S RECOMMENDATIONS, TO THE BOARD OF CONTRACT AWARDS FOR CONSIDERATION FOR AWARD.

AS CLAUSE 10 INDICATES, A BOARD OF CONTRACT AWARDS WAS ESTABLISHED TO REVIEW THE PROPOSALS RECEIVED, WEIGH RELATIVE MERITS OF EACH PROPOSAL AND SELECT THE CONTRACTOR. IN ADDITION, AN EVALUATION TEAM COMPOSED OF FOUR SENIOR COMPUTER SYSTEMS ANALYSTS AND TWO SENIOR FORESTERS EXPERIENCED IN MANAGEMENT ANALYSIS WAS SELECTED TO ASSIST THE BOARD.

SIX OF THE SEVEN PROPOSALS RECEIVED WERE DETERMINED RESPONSIVE. COPIES OF THE TECHNICAL PORTION OF THE SIX PROPOSALS WERE GIVEN TO EACH MEMBER OF THE EVALUATION TEAM. IN ACCORDANCE WITH THE RFP, THE PRICE PORTION OF THE PROPOSALS WAS WITHHELD PENDING COMPLETION OF THE TECHNICAL EVALUATION. TECHNICAL RATINGS WERE ESTABLISHED INDEPENDENTLY BY EACH TEAM MEMBER FOR EACH PROPOSAL; THEREAFTER, A TEAM SCORE FOR EACH FEATURE AND THE SYSTEM AS A WHOLE WAS DEVELOPED. UPON COMPLETION OF THE TECHNICAL EVALUATION, THE PROPOSAL PRICES WERE INTRODUCED AND THE PROPOSALS WERE FURTHER RATED BY USING PROPOSAL PRICES AS THE ONLY COST. AT THIS POINT IT WAS DETERMINED THAT PMI, TRW AND UNIVERSITY COMPUTING COMPANY (UCC) WERE THE ONLY OFFERORS WITHIN A COMPETITIVE RANGE.

IN ACCORDANCE WITH SECTION 5.2 OF THE RFP, THE THREE FIRMS STILL IN CONTENTION WERE REQUESTED TO SUBMIT A MACHINE RUN SOLUTION TO THE BENCHMARK PROBLEM STATED IN SECTION 7 OF THE SOLICITATION. AT THE SAME TIME THE DESIGN DOCUMENTATION REQUIRED BY SECTION 5.1.3 WAS REVIEWED, DISCUSSED WITH ALL THREE FIRMS, AND EVALUATED. IN ADDITION, DMS USERS REFERENCED IN THE RESPECTIVE PROPOSALS AS REQUIRED BY SECTION 5.3 WERE CONTACTED AND RESPONSES EVALUATED. AFTER THESE PROCEDURES WERE COMPLETED, IT WAS DETERMINED THAT PMI DID NOT MEET ONE OF THE MANDATORY SPECIFICATION REQUIREMENTS; IT WAS NOTIFIED OF THE DEFICIENCY AND AMENDED ITS TECHNICAL PROPOSAL ACCORDINGLY. THEREAFTER, THE THREE PROPOSALS WERE SCORED AGAIN AND, WITH THE ADDITIONAL INFORMATION GAINED AS TO THE OPERATION AND MAINTENANCE OF EACH SYSTEM, A 5-YEAR COST EFFECTIVENESS EVALUATION WAS MADE.

THE RESULT OF THIS EVALUATION WAS REPORTED TO THE BOARD OF CONTRACT AWARDS, AS FOLLOWS:

TRW UCC PMI

TOTAL OPERATION AND $1,735,000 $2,400,000 $3,375,000

MAINTENANCE

PROPOSAL PRICE 145,600 156,400 70,000

$1,880,600 $2,556,400 $3,445,000

SYSTEM SCORE COST 103.9 95.1 86.3

EFFECTIVENESS (SCORE .052 .037 .025

DIVIDED BY COST)

AS A RESULT OF THIS EVALUATION, THE EVALUATION TEAM RECOMMENDED ON MAY 24, 1971, THAT AWARD BE MADE TO TRW. THE BOARD OF CONTRACT AWARDS MET ON MAY 25, 1971, AND IN A MEMORANDUM DATED JUNE 1, 1971, ADVISED THE CONTRACTING OFFICER OF ITS CONCURRENCE IN THE EVALUATION TEAM RECOMMENDATION. THE MEMORANDUM STATES THAT THE SELECTION WAS "BASED ON THE COST EFFECTIVENESS EVALUATION PREPARED BY THE COMMITTEE WHICH REVEALS THAT THE TRW SYSTEM WILL RESULT IN THE LOWEST OVERALL COST TO THE GOVERNMENT OVER THE EXPECTED LIFE OF THE SYSTEM." THEREFORE, AWARD WAS MADE TO TRW ON JUNE 3, 1971, WITH PERFORMANCE TO BE COMPLETED IN 20 WEEKS AS PROPOSED BY TRW IN ACCORDANCE WITH AMENDMENT 1 TO THE RFP, WHICH PERMITTED DEVIATIONS FROM THE 90-DAY DELIVERY REQUESTED.

AGAINST THE FOREGOING FACTUAL BACKGROUND, PMI HAS RAISED A NUMBER OF MATERIAL CONTENTIONS, EACH OF WHICH, TOGETHER WITH THE SPECIFIC FACTS NECESSARY FOR THEIR DISPOSITION, IS TREATED BELOW.

PMI CONTENDS THAT THE FOREST SERVICE FAILED TO FOLLOW THE EVALUATION CRITERIA DISCLOSED BY THE RFP AND USED MATERIAL CRITERIA NOT DISCLOSED BY THE RFP. WE HAVE, AS PMI POINTS OUT, RECOGNIZED THAT AS A MATTER OF SOUND PROCUREMENT POLICY, SOLICITATIONS SHOULD APPRISE OFFERORS OF ALL APPLICABLE EVALUATION CRITERIA AND OF THE RELATIVE IMPORTANCE OF THOSE CRITERIA. CF. 50 COMP. GEN. 59 (1970); 49 ID. 229 (1969). INSOFAR AS THE EVALUATION OF THE TECHNICAL ASPECTS OF THE DMS'S OFFERED IS CONCERNED, WE AGREE, AS PMI URGES, THAT RELIANCE SOLELY ON THE WEIGHTED FEATURES LIST INCLUDED IN THE RFP WOULD NOT APPRISE OFFERORS OF THE RELATIVE IMPORTANCE OF THE TECHNICAL EVALUATION CRITERIA, SINCE THE VALUES ASSIGNED IN THE WEIGHTED FEATURES LIST WERE SUBJECT TO FURTHER WEIGHTING IN ACCORDANCE WITH A MARKING SCALE PRESCRIBED IN APPENDIX "E" OF THE AUERBACH REPORT AND REFERENCED IN SECTION IV OF THE REPORT.

WHILE WE BELIEVE THAT FOR THE SAKE OF CLARITY AND TO AVOID ANY QUESTION, ALL PERTINENT PORTIONS OF THE AUERBACH REPORT SHOULD ALSO HAVE BEEN PHYSICALLY INCLUDED IN THE RFP, THERE IS, IN OUR OPINION, MERIT IN THE FOREST SERVICE'S POSITION THAT CLAUSE 10 OF THE SUPPLEMENTAL INSTRUCTIONS AND CONDITIONS, QUOTED ABOVE, INCORPORATED THE AUERBACH REPORT INTO THE RFP BY REFERENCE. WE NOTE THAT THE OFFERORS' ATTENTION WAS DRAWN TO THE NOTE IN CLAUSE 10 AT THE PREBID CONFERENCE ON MARCH 2, 1971, WHICH WAS ATTENDED BY PMI, AND THAT A NUMBER OF OFFERORS OTHER THAN PMI REVIEWED OR REQUESTED COPIES OF THE REPORT. INDEED, WE DO NOT SEE HOW AN OFFEROR COULD PRUDENTLY IGNORE THE NOTE THAT THE "DMS WILL BE RATED ON THE BASIS OF A METHODOLOGY OUTLINE IN FINAL REPORT, AUER. - 1834-TR-2, INFORM DATA MANAGEMENT SYSTEM STUDY." WE BELIEVE, HOWEVER, THAT THE APPROPRIATE TIME FOR QUESTIONING THE ADEQUACY OF THE DISCLOSURE OF THE EVALUATION CRITERIA AND THEIR IMPORTANCE IS PRIOR TO THE SUBMISSION OF A PROPOSAL. REACHING THIS CONCLUSION, WE DO NOT CONSIDER PMI'S ADVICE IN ITS LETTER OF JULY 22, 1971, THAT IT DECIDED TO FOREGO READING THE AUERBACH REPORT ON THE BASIS OF THE ADVICE OF A FOREST SERVICE EMPLOYEE, OTHER THAN THE CONTRACTING OFFICER, OR MR. HAUG, THAT THE MAIN SUBSTANCE OF THE REPORT WAS IN THE RFP AND THAT READING IT WOULD NOT ADD MATERIALLY TO PMI'S UNDERSTANDING OF THE REQUIREMENT TO BE DECISIVE. THIS RESPONSE, WE NOTE, WAS CONSISTENT WITH THE POSITION THAT THE FOREST SERVICE HAS MAINTAINED BEFORE OUR OFFICE - NAMELY, THE RFP WAS SUFFICIENT TO APPRISE OFFERORS OF THE DMS CAPABILITIES THAT THE FOREST SERVICE REQUIRED.

MOREOVER, WE CANNOT AGREE WITH PMI'S FURTHER CONTENTION THAT THE INCORPORATION OF THE AUERBACH REPORT IS LIMITED TO THE GRADING SCALE TO BE APPLIED IN THE TECHNICAL EVALUATION OF THE DMS AND DOES NOT INCLUDE THE "COST EFFECTIVENESS" EVALUATION REFERRED TO IN THE REPORT AND USED BY THE FOREST SERVICE TO DETERMINE THE MOST ADVANTAGEOUS OFFER. IN OUR VIEW, THE LANGUAGE OF THE NOTE IS SIMPLY TOO BROAD TO SUSTAIN THIS POSITION. FURTHER, CLAUSE 10 ITSELF INDICATES THAT "COST EFFECTIVENESS" WILL BE CONSIDERED:

*** PROPOSALS REMAINING WITHIN A COMPETITIVE RANGE CONSIDERING PRICE, COST EFFECTIVENESS, AND OTHER FACTORS SHALL THEN BE GIVEN, WITH THE COMMITTEE'S RECOMMENDATION, TO THE BOARD OF CONTRACT AWARDS FOR CONSIDERATION FOR AWARD.

TURNING TO THE FOREST SERVICE'S APPLICATION OF THE EVALUATION CRITERIA TO THE PROPOSAL, PMI HAS IN ITS CORRESPONDENCE DRAWN ATTENTION TO A NUMBER OF MATHEMATICAL ERRORS. AS INDICATED IN THE FOREST SERVICE'S SUPPLEMENTAL REPORT, CORRECTION OF THE ERRORS NOTED IN THE EVALUATION AS REPORTED TO THE BOARD OF CONTRACT AWARDS YIELDS THE FOLLOWING RESULT:

TOTAL OPERATION &

MAINTENANCE $1,825,000 $2,400,000 $3,375,000

PROPOSAL PRICE 145,600 156,400 70,000

$1,970,600 $2,556,400 $3,445,000

SYSTEM SCORE 98.2 95.1 87.1

COST EFFECTIVENESS

(SCORE DIVIDED BY COST) .050 .037 .025

ALTHOUGH WE CANNOT AGREE THAT COMPUTATION ERRORS REFLECTED IN THE EVALUATION AND SUBSEQUENTLY EXPLAINED IN THE FOREST SERVICE'S REPORTS ARE SUFFICIENT TO RENDER THE EVALUATION UNRELIABLE, THE ERRORS ARE NEVERTHELESS REGRETTABLE. WE ARE CONCERNED, HOWEVER, WITH THE VALIDITY OF THE "COST EFFECTIVENESS EVALUATION" AND THE IMPACT OF THIS EVALUATION ON THE RELATIVE STANDING OF OFFERORS. THIS CONCERN GOES BEYOND THE MATHEMATICAL ERRORS NOTED BY PMI.

SECTION IV OF THE AUERBACH REPORT STATES THAT THE "EVALUATION" PROCEDURE PRESENTED IS BASED ON ESTABLISHING THE COST EFFECTIVENESS OF EACH CANDIDATE SYSTEM WITH RESPECT TO THE FOREST SERVICE'S PROBLEMS. SECTION 4.3(9) CONTAINS THE FORMULA FOR DETERMINING COST EFFECTIVENESS WHICH WAS EMPLOYED HERE. SECTION 4.3(8) STATES THAT THE "COST OF A SYSTEM CAN BE CONSIDERED AS THE 'PRESENT COST' OF PROCUREMENT, OPERATION, AND MAINTENANCE OVER THE ESTIMATED LIFE OF THE SYSTEM," AND TABLE 4-2 IDENTIFIES FOR CONSIDERATION THE OPERATION AND MAINTENANCE COST ELEMENTS:

OPERATION

COMPUTATION TIME

COMMUNICATIONS

TERMINAL

PROGRAMMING DATA PREPARATION

SUPPLIES

FACILITIES

MAINTENANCE

SYSTEM PROGRAM

OPERATING SYSTEM

HARDWARE

FACILITIES

OF THE OPERATION COST ELEMENTS, THE FOREST SERVICE EVALUATED ONLY COMPUTATION TIME AND PROGRAMMING, WITH THE FOLLOWING RESULTS FOR PMI AND TRW:

PMI TRW

COMPUTATION TIME $450,000 $225,000

PROGRAMMING

1) AUTOMATIC DATA PROCESSING

COSTS 200,000 200,000

2) USER COSTS 2,500,000 1,250,000

TOTAL $3,150,000 $1,675,000

WITH RESPECT TO MAINTENANCE, ONLY SYSTEM PROGRAM AND HARDWARE COSTS WERE ASSESSED:

PMI TRW

SYSTEM PROGRAM $25,000 $50,000

HARDWARE 200,000 100,000

TOTAL $225,000 $150,000

THE FOREGOING OPERATION AND MAINTENANCE COSTS ARE BASED ON A 5-YEAR LIFE CYCLE FOR A DMS, WHICH WAS NOT REVEALED. THE FOREST SERVICE CONSIDERED 5 YEARS TO BE A PRACTICAL MINIMUM SYSTEM LIFE, AND WE ARE NOT PREPARED TO SAY THAT THIS JUDGMENT WAS UNREASONABLE. IN VIEW OF THE RFP'S ADVICE THAT INFORM WILL NOT BE FULLY OPERATIONAL FOR 2 OR 3 YEARS, WE WOULD AGREE WITH THE FOREST SERVICE'S POSITION THAT OFFERORS WERE ON NOTICE THAT THE DMS WOULD HAVE TO BE COST EFFECTIVE FOR AT LEAST SEVERAL YEARS.

WE DO NOT OBJECT TO THE CONSIDERATION OF LIFE-CYCLE COSTS, PER SE, IN DETERMINING THE MOST ADVANTAGEOUS PROPOSAL; INDEED, WE HAVE FAVORED THIS EVALUATION TECHNIQUE, IF REASONABLE. IN ITS LETTER OF JULY 14, 1971, PMI HAS MADE THE FOLLOWING OBSERVATIONS WITH RESPECT TO THE FOREST SERVICE'S ASSESSMENT OF OPERATION AND MAINTENANCE COSTS:

THE KEY FACTOR OF USER COST REPRESENTED SOME $2,500,000 OF THE TOTAL OPERATION AND MAINTENANCE COST OF $3,375,000 CHARGED AGAINST PMI'S PROPOSAL. THIS ELEMENT ALONE CONSTITUTED A COST ADVANTAGE OF $1,250,000 TO TRW.

*** THE OVERWHELMINGLY PREDOMINANT PRACTICE IN GOVERNMENT SOFTWARE PURCHASE IS TO EVALUATE COST ON THE BASIS OF PROCUREMENT COST RATHER THAN THE MORE EPHEMERAL LIFE-CYCLE COST, AND WE HAVE NEVER SEEN A COST EVALUATION INCLUDE A SIMILAR USER COST CRITERION.

*** THE FOREST SERVICE FINAL COST EFFECTIVENESS EVALUATION SHEET LISTING COST ELEMENTS APPARENTLY FOLLOWS TABLE 4-2 OF THE AUERBACH REPORT. HOWEVER, THE EVALUATION SHEET LISTS BOTH "PROGRAMMING-USER" AND "PROGRAMMING-ADP" WHILE TABLE 4-2 LISTS ONLY "PROGRAMMING."

INDUSTRY USAGE WOULD EQUATE "PROGRAMMING" IN THAT CONTEXT TO THE PREPARATION OF COMPUTER PROGRAMS AND WOULD NOT INCLUDE TIME SPENT BY NON- ADP PERSONNEL IN 1000 REMOTE LOCATIONS IN REQUESTING DATA FROM A COMPUTER SYSTEM. THE INCLUSION OF SUCH USER TIME UNDER THE HEADING OF "PROGRAMMING" IS NOT CONSISTENT WITH GENERAL ESTIMATING PRACTICES OR OTHER GOVERNMENT OR COMMERCIAL PROCUREMENTS. THAT THE ORDINARY MEANING OF "PROGRAMMING" WAS THE MEANING EMPLOYED BY THE AUTHORS OF THE AUERBACH REPORT IS PLAIN FROM THE FACT THAT NOWHERE IN THE AUERBACH REPORT IS THE COMPLEX AND DIFFICULT USER COST EVALUATION ATTEMPTED BY THE FOREST SERVICE DISCUSSED, EXPLICITLY RECOMMENDED, OR EVEN DESCRIBED.

THE ESTIMATION OF USER PROGRAMMING COSTS ON AN EXTREMELY CRUDE BASIS 1 HOUR FOR A PMI-USER REQUEST, 1/2 HOUR FOR A TRW REQUEST, AND 3/4 HOUR FOR A UCC REQUEST AS PER EXHIBIT 9 - IS A PROCEDURE OF DOUBTFUL VALIDITY WHEN SUBJECTED TO CRITICAL REVIEW BY KNOWLEDGEABLE ANALYSTS. THESE CRUDE REQUEST TIME FACTORS WERE MULTIPLIED BY 50,000 TO GIVE COST DIFFERENTIALS OF HUGE PROPORTIONS IN COMPARISON TO ANY OTHER COST FACTOR. THE SYSTEM OFFERORS WERE GIVEN NO OPPORTUNITY TO PROVIDE DATA OR TO COMMENT ON THE GOVERNMENT'S FIGURES. INDEED, THEY DID NOT KNOW THAT SUCH COSTS WERE TO BE CONSIDERED PRIOR TO ANNOUNCEMENT OF THE DECISION TO AWARD.

IN CONTRAST, THE *** (AUTOMATIC DATA PROCESSING) COST FACTOR WAS KEPT CONSTANT AND NOT VARIED WITH THE TYPE OF SYSTEM BECAUSE "TWO GOVERNMENT PROGRAMMERS ARE ALL THAT CAN BE DEVOTED TO THIS AREA REGARDLESS OF WHAT THE FUNCTION REQUIRES." THIS TREATMENT IS NOT ONLY ILLOGICAL, CONSIDERING THE VARIABILITY OF OTHER PROGRAMMING COSTS, BUT WORKS TO PENALIZE THOSE SYSTEMS WHICH ARE EASILY MAINTAINED (SUCH AS PMI'S COBOL-BASED SYSTEM) AND TO FAVOR THOSE WHICH ARE DIFFICULT TO MAINTAIN (SUCH AS TRW'S SPECIAL- LANGUAGE SYSTEM).

ANOTHER MAJOR AREA OF CONCERN IS THAT OF "HARDWARE COSTS." IT WAS EXPLAINED AT THE DEBRIEFING THAT PMI WAS ASSESSED A COST OF $200,000 IN THIS AREA FOR STORAGE SPACE ON AN EXTERNAL DRUM DEVICE ON THE U-1108, WHILE TRW WAS ASSESSED ONLY HALF THIS AMOUNT SINCE THEY OFFERED TO COMPRESS THE DATA AFTER PROCESSING, STORE IT ON THE DRUM, AND EXPAND IT AGAIN FOR PROCESSING WHEN BROUGHT BACK.

OFFERORS HAD NO WAY OF KNOWING THAT HARDWARE STORAGE COSTS WOULD BE EVALUATED OR ASSESSED IN THIS MANNER. HAD THIS FACTOR BEEN KNOWN, PMI COULD HAVE EASILY PROPOSED A SIMILAR COMPRESSION TECHNIQUE AS PART OF ITS SYSTEM WITHOUT INCREASING BID COSTS.

IN ADDITION, THE ABILITY OF EVALUATORS TO DETERMINE THAT COMPUTATION TIME ON PMI'S SYSTEM WOULD BE DOUBLE THAT OF TRW IS QUESTIONABLE. SINCE NO RUNNING TIMES WERE REQUESTED ON BENCHMARK TESTS - THE ONLY ACTUAL EXPERIENCE IN COMPUTING FOREST SERVICE DATA ON THE PROPOSED SYSTEMS - THESE ESTIMATES HAD TO BE BASED ON GUESSWORK OR THE EXPERIENCE OF OTHER USERS (WHO HAVE DIFFERENT EQUIPMENT, USE OTHER VERSIONS OF OUR SYSTEM, AND PROCESS THEIR OWN UNIQUE TYPES OF PROBLEMS).

AS A MINIMUM CONSIDERATION THE EVALUATORS, HAVING DERIVED THESE COSTS FROM SUCH SOURCES, SHOULD HAVE VERIFIED THEM WITH THE OFFERORS, THUS TAKING ADVANTAGE OF THE IN-DEPTH EXPERIENCE OF THE SYSTEM SUPPLIERS, AS THE AUERBACH REPORT RECOGNIZES (FOREST SERVICE REPORT, P. 7).

WE ARE IN GENERAL AGREEMENT WITH THE THRUST OF PMI'S COMMENTS - NAMELY, THE OPERATION AND MAINTENANCE COSTS ARE INCAPABLE OF PRECISE ASSESSMENT. WE FURTHER BELIEVE IT FAIR TO OBSERVE THAT THIS UNCERTAINTY IS REFLECTED IN THE STEPS TAKEN BY THE FOREST SERVICE TO VERIFYING ITS EFFECTIVENESS EVALUATION. IN THIS REGARD, PRIOR TO AWARD, THE VALIDITY OF THE COMPUTATION WAS TESTED IN THREE WAYS. TWO OF THE TESTS CONSIDERED THE EFFECT OF THE MAJOR COST ELEMENT OF THE SYSTEM, THE USER PROGRAMMING COST. FIRST, THE COST OF USING EACH OF THE PROPOSED SYSTEMS WAS ASSUMED TO BE EQUAL TO THAT OF USING THE LEAST COSTLY. THE RESULTS DID NOT ALTER THE RELATIVE STANDINGS. SECOND, THE USER PROGRAMMING COSTS WERE DEDUCTED COMPLETELY. THE RANKINGS REMAINED UNCHANGED. A THIRD TEST WAS APPLIED WHERE THE OPERATION AND MAINTENANCE COSTS FOR ALL SYSTEMS WERE ASSUMED TO BE EQUAL TO THAT OF THE LEAST COSTLY. OUR OWN ANALYSIS OF THIS THIRD TEST INDICATES THAT, ACCEPTING THE FINAL TECHNICAL RATINGS OF THE TWO PROPOSALS AND USING THE PRICES BID, TRW BECOMES MORE COST EFFECTIVE AT THE POINT WHEN TOTAL FUTURE OPERATION AND MAINTENANCE COSTS EXCEED ABOUT $480,000. ASSUMING, AS FOREST SERVICE DID, THAT SUCH COSTS WOULD BE $347,000 ANNUALLY, TRW WOULD BECOME MORE COST EFFECTIVE AFTER 1.38 YEARS. COURSE, IF ANNUAL COSTS SHOULD BE LESS, THIS TIME WOULD BE GREATER.

IN OUR OPINION, THE COST EFFECTIVENESS ANALYSIS IS, AT HEART, AN ATTEMPT TO QUANTIFY A TECHNICAL JUDGMENT THAT A PARTICULAR DMS BETTER SUITS THE FOREST SERVICE'S REQUIREMENTS. THIS IS EVIDENT FROM THE FOLLOWING COMMENTS IN THE FOREST SERVICE'S SUPPLEMENTAL REPORT:

AS NOTED IN OUR EARLIER REPLY, SEVERAL SETS OF COST FIGURES WERE USED TO OBTAIN RANKINGS FOR PROPOSALS. THAT OPERATION AND MAINTENANCE COSTS OVER A SYSTEM LIFE WOULD BE CONSIDERED IS CLEAR IN THE AUERBACH REPORT AND IMPLIED IN THE RFP. COST ESTIMATES FOR EACH OF THE SYSTEMS WERE THE BEST ESTIMATES OF THE EVALUATION TEAM OF TIMES AND COSTS REQUIRED TO MAINTAIN AND OPERATE THE SYSTEMS. ALTHOUGH THE COST ESTIMATES ARE NOT ABSOLUTE, WE ARE CERTAIN THAT THE RELATIONSHIPS ARE CORRECT.

THE ESTIMATING OF OPERATION AND MAINTENANCE COSTS IS BY ITS NATURE A PROJECTION OF ANTICIPATED EXPENSE AND THEREFORE NOT ABSOLUTE. HOWEVER, IT IS FOUNDED UPON THE JUDGMENT OF KNOWLEDGEABLE, EXPERIENCED STAFF, AND IS BELIEVED TO REFLECT THE BEST JUDGMENT OF THOSE BEST QUALIFIED TO KNOW.

SOME OF THE OPERATION AND MAINTENANCE COSTS ARE NOT SHOWN IN THE ANALYSIS. TO SHOW VALUES FOR COST ELEMENTS WHERE THE VALUES WOULD BE THE SAME FOR THE THREE SYSTEMS WOULD NOT CHANGE THE RANKING. ANY UNIFORM COST ADDED TO ALL THREE SYSTEMS WOULD FAVOR THE ONE WITH THE HIGHEST TECHNICAL SCORE.

EVEN IF A COST IS ADDED WHICH IS HIGHER FOR A HIGHER RATED SYSTEM (I.E. $70,000 CONVERSION ADDED AS A COST ADDED TO TRW, ZERO TO PMI). THE RANKINGS COULD NOT POSSIBLY CHANGE WITHOUT THE TOTAL COST OF THE PMI OFFER FALLING BELOW THAT OF TRW, AND IF THEN, SUFFICIENTLY LOWER TO ACCOUNT FOR THE DIFFERENCE IN SYSTEM SCORE. THUS THERE ARE SEVERAL ELEMENTS FOR WHICH COSTS ARE NOT SHOWN.

AS FOR THE ASSIGNMENT OF FUTURE OPERATION AND MAINTENANCE COSTS, THE EXHIBITS SHOW SEVERAL RELATIONSHIPS. FOR EXAMPLE, IT WILL TAKE SIGNIFICANTLY MORE COMPUTER TIME TO RUN A SYSTEM REQUIRING GENERATION OF AN INTERMEDIATE LANGUAGE (COBOL) AND ORIENTED TOWARD SEQUENTIALLY FILE PROCESSING (PMI) THAN ONE RELYING MORE HEAVILY ON MACHINE LEVEL CODING. LIKEWISE, THE SYSTEM PROGRAMMED IN COBOL AND HAVING LESS CAPABILITY (PMI) WILL BE LESS COSTLY TO MAINTAIN. CONTRARY TO PMI'S JULY 14 LETTER, THE MAINTENANCE COST FOR PMI WAS NOT CONSIDERED CONSTANT. PMI WAS ASSIGNED A COST HALF THAT OF ITS COMPETITORS.

FACTORS USED FOR ESTIMATING USER TIME REQUIRED FOR THE VARIOUS SYSTEMS INCLUDED SIMPLICITY OF RETRIEVAL AND FILE MAINTENANCE LANGUAGE, AND, AMOUNT OF OWN CODE REQUIRED TO PRODUCE ANSWERS TO A VARIETY OF INFORMATION REQUESTS. GUIDES FOR ESTIMATING REQUIRED USE TIME WERE OBTAINED FROM THE USERS MANUAL FURNISHED WITH PROPOSALS, EVALUATION OF MACHINE BENCHMARK DOCUMENTATION, AND DISCUSSIONS WITH PRESENT USERS OF EACH OF THE SYSTEMS.

WHILE WE MAY QUESTION THE EXPRESSION OF THE FOREST SERVICE'S JUDGMENT IN TERMS OF PROJECTED OPERATION AND MAINTENANCE COSTS, IT REPRESENTS, IN OUR VIEW, A DETERMINATION THAT THE TRW PROPOSAL WAS MORE ADVANTAGEOUS TO THE GOVERNMENT, NOTWITHSTANDING ITS HIGHER INITIAL ACQUISITION COST - A JUDGMENT WHICH MAY LEGITIMATELY SUSTAIN THE AWARD TO TRW.

WITH RESPECT TO THE CONDUCT OF NEGOTIATIONS, PMI ALLEGES THAT THEY WERE NOT MEANINGFUL. IN REPLY, IT IS THE FOREST SERVICE'S POSITION THAT IT DID NOT CONDUCT NEGOTIATIONS WITH ANY OF THE OFFERORS, BUT MADE AN AWARD ON THE BASIS OF THE MOST FAVORABLE INITIAL PROPOSAL IN ACCORDANCE WITH RFP CLAUSE 10(G) OF STANDARD FORM 33A AND SECTION 1 3.805-1(A)(5) OF THE FEDERAL PROCUREMENT REGULATIONS (FPR), WHICH PROVIDES, IN PERTINENT PART:

PROCUREMENTS IN WHICH IT CAN BE CLEARLY DEMONSTRATED FROM THE EXISTENCE OF ADEQUATE COMPETITION OR ACCURATE PRIOR COST EXPERIENCE WITH THE PRODUCT OR SERVICE THAT ACCEPTANCE OF THE MOST FAVORABLE INITIAL PROPOSAL WITHOUT DISCUSSION WOULD RESULT IN A FAIR AND REASONABLE PRICE: PROVIDED, THAT THE REQUEST FOR PROPOSALS CONTAINS A NOTICE TO ALL OFFERORS OF THE POSSIBILITY THAT AWARD MAY BE MADE WITHOUT DISCUSSION OF PROPOSALS RECEIVED AND, HENCE, THAT PROPOSALS SHOULD BE SUBMITTED INITIALLY ON THE MOST FAVORABLE TERMS, FROM A PRICE AND TECHNICAL STANDPOINT, WHICH THE OFFEROR CAN SUBMIT TO THE GOVERNMENT. ***

IN OUR VIEW, THE APPARENT DISTINCTION DRAWN BY THE FOREST SERVICE IN ITS ADMINISTRATIVE REPORTS BETWEEN "DISCUSSIONS" AND "NEGOTIATIONS" FOR THE PURPOSE OF ACCEPTING THE MOST FAVORABLE INITIAL PROPOSAL IS INAPPOSITE. FPR SEC. 1-3.805-1(A) IMPOSES A REQUIREMENT THAT "WRITTEN OR ORAL DISCUSSIONS SHALL BE CONDUCTED WITH ALL OFFERORS WHO SUBMITTED PROPOSALS WITHIN A COMPETITIVE RANGE"; AND THE EXCEPTION TO THIS REQUIREMENT RECOGNIZED IN FPR SEC. 1-3.805-1(A)(5) EXPRESSLY PROVIDES THAT THE AWARD BE MADE "WITHOUT DISCUSSION." IF "DISCUSSIONS" ARE CONDUCTED WITH ONE OFFEROR, THEN DISCUSSIONS MUST BE CONDUCTED WITH ALL OFFERORS WITHIN THE COMPETITIVE RANGE. 50 COMP. GEN. 202 (1970). AS WE STATED IN 50 COMP. GEN. 117 (1970).

FPR 1-3.805-1 REQUIRES THAT DISCUSSIONS BE CONDUCTED WITH ALL OFFERORS WITHIN A COMPETITIVE RANGE, PRICE AND OTHER FACTORS CONSIDERED. IT IS A WELL-ESTABLISHED PRINCIPLE IN FEDERAL PROCUREMENTS THAT SUCH DISCUSSIONS MUST BE MEANINGFUL AND FURNISH INFORMATION TO ALL OFFERORS WITHIN THE COMPETITIVE RANGE AS TO THE AREAS IN WHICH THEIR PROPOSALS ARE BELIEVED TO BE DEFICIENT SO THAT COMPETITIVE OFFERORS ARE GIVEN AN OPPORTUNITY TO FULLY SATISFY THE GOVERNMENT'S REQUIREMENTS. 47 COMP. GEN. 336 (1967). WHEN NEGOTIATIONS ARE CONDUCTED THE FACT THAT INITIAL PROPOSALS MAY BE RATED AS ACCEPTABLE DOES NOT INVALIDATE THE NECESSITY FOR DISCUSSIONS OF THEIR WEAKNESSES, EXCESSES OR DEFICIENCIES IN ORDER THAT THE CONTRACTING OFFICER MAY OBTAIN THAT CONTRACT WHICH IS MOST ADVANTAGEOUS TO THE GOVERNMENT. WE HAVE STATED THAT DISCUSSIONS OF THIS NATURE SHOULD BE CONDUCTED WHENEVER IT IS ESSENTIAL TO OBTAIN INFORMATION NECESSARY TO EVALUATE A PROPOSAL OR TO ENABLE THE OFFEROR TO UPGRADE THE PROPOSAL. THUS, WHERE AN OFFEROR FAILED TO PASS A BENCHMARK TEST, THAT FACTOR ALONE SHOULD NOT HAVE PRECLUDED DISCUSSIONS TO DETERMINE WHETHER THE PROPOSAL COULD BE IMPROVED. 47 COMP. GEN. 29 (1967). *** FROM THE RECORD, WE ARE SATISFIED THAT THE FOREST SERVICE CONDUCTED "DISCUSSIONS" WITH ALL OFFERORS WITHIN THE COMPETITIVE RANGE. FROM THE STANDPOINT OF THE CONDUCT OF THE NEGOTIATIONS OR DISCUSSIONS (THE TERMS ARE SYNONYMOUS IN OUR VIEW), THE RECORD IS CLEAR THAT ALL OFFERORS WERE SIMILARLY TREATED. GENERALLY, DISCUSSIONS WERE CONFINED TO THE ELIMINATION OF UNCERTAINTIES AS TO THE TECHNICAL ASPECTS OF THE PROPOSALS AND TO A SUBSTANTIATION OF THE CLAIMS MADE BY AN OFFEROR FOR ITS DM8. CONTRARY TO PMI'S CONTENTION IN ITS LETTER OF JULY 14, 1971, THAT WEAKNESSES IN ITS PROPOSAL WERE NEVER DISCUSSED, WE HAVE NO DIFFICULTY IN CONSIDERING PROPOSAL UNCERTAINTIES TO BE WEAKNESSES OR DEFICIENCIES. IF UNCERTAINTIES WERE CORRECTED AS A RESULT OF DISCUSSIONS, WE BELIEVE THE DISCUSSIONS WOULD BE "MEANINGFUL." IN THIS CONNECTION, WE WERE ADVISED IN AN INITIAL POSITION PAPER SUBMITTED BY PMI ON JUNE 15, 1971, THAT:

DURING THE COURSE OF THE VALUATION PERIOD, PMI HAD OCCASION TO MEET WITH THE TECHNICAL COMMITTEE MONITORING THIS SELECTION ON THREE OCCASIONS AND TO SPEAK WITH VARIOUS MEMBERS OF THAT COMMITTEE BY TELEPHONE ON NO LESS THAN TEN SEPARATE OCCASIONS. THE SIGNIFICANCE OF THESE EXCHANGES IS TWOFOLD:

A) GUIDANCE ON TECHNICAL ISSUES OF SUBSTANCE WAS GAINED THROUGH THESE CONVERSATIONS, AND

B) A CONTINUING DIALOGUE REGARDING OUR PERFORMANCE WAS MAINTAINED.

MOREOVER, WE CANNOT CONCLUDE THAT THE FOREST SERVICE'S RESTRICTIVE VIEW OF THE NEGOTIATION PROCESS CONFERRED A COMPETITIVE ADVANTAGE ON ANY PARTICULAR OFFER. WE DO BELIEVE, HOWEVER, THAT THE APPROACH REFLECTED HERE IS INCONSISTENT WITH THE REQUIREMENTS FOR COMPETITIVE NEGOTIATION AND NOT NECESSARILY IN THE BEST INTERESTS OF THE UNITED STATES. WE RECOMMEND THAT CORRECTIVE ACTION BE TAKEN TO ELIMINATE THIS PRACTICE.

NEVERTHELESS, FROM OUR REVIEW OF THE RECORD BEFORE US, WE CANNOT CONCLUDE THAT THE FOREST SERVICE'S AWARD OF A CONTRACT TO TRW WAS "CLEARLY ILLEGAL." 50 COMP. GEN. 390 (1970). ACCORDINGLY, PMI'S REQUEST THAT THE AWARD BE CANCELED MUST BE DENIED.