B-191013, B-191013.2, AUGUST 8, 1980, 59 COMP.GEN. 640

B-191013, B-191013.2, AUGUST 8, 1980, 59 COMP.GEN. 640

CONTRACTS - PROTESTS - PROCEDURES - BID PROTEST PROCEDURES - TIME FOR FILING - INITIAL ADVERSE AGENCY ACTION DATE ONCE AGENCY DENIES PROTEST, FACT THAT PROTESTER BELIEVES AGENCY WILL RECONSIDER PROTEST DOES NOT TOLL TIME FOR FILING A PROTEST TO GENERAL ACCOUNTING OFFICE (GAO) SINCE GAO BID PROTEST PROCEDURES REQUIRE PROTEST TO BE FILED WITHIN 10 WORKING DAYS OF WHEN PROTESTER LEARNS OF INITIAL ADVERSE AGENCY ACTION. CONTRACTS - PROTESTS - TIMELINESS - SOLICITATION IMPROPRIETIES - BENCHMARKING - PROPOSED STRUCTURE V. RESULTS RESULTS OF BENCHMARK DO NOT PROVIDE PROPER BASIS FOR RECONSIDERATION OF PRIOR DECISION DEALING WITH PROPOSED BENCHMARK PROCEDURE SINCE BENCHMARK RESULTS DO NOT CONSTITUTE EVIDENCE WHICH SHOULD HAVE BEEN CONSIDERED. CONTRACTS - PROTESTS - TIMELINESS - SOLICITATION IMPROPRIETIES - BENCHMARKING - PROPOSED PROCEDURE, ETC. PROTEST OF METHODS USED TO COMPUTE COSTS FROM BENCHMARK RESULTS IS UNTIMELY WHERE METHODS USED WERE DEFINED IN REQUEST FOR PROPOSALS BUT PROTEST WAS NOT LODGED BEFORE BENCHMARKING WAS COMPLETED. CONTRACTS - SPECIFICATIONS - TESTS - BENCHMARK - ADEQUACY - LIFE CYCLE COST EVALUATION SINCE RECORD SUGGESTS AGENCY'S BENCHMARK-BASED LIFE-CYCLE COST APPROACH MIGHT NOT HAVE BEEN SUFFICIENTLY ACCURATE TO SUPPORT SELECTION OF AWARDEE'S RATHER THAN PROTESTER'S EQUIPMENT, AND SINCE AGENCY'S NEEDS APPEAR TO HAVE CHANGED, GAO RECOMMENDS THAT AGENCY CONDUCT MARKET SURVEY TO DETERMINE, BEFORE FURTHER CONTRACT OPTIONS ARE EXERCISED, IF RELIANCE ON AWARDEE'S EQUIPMENT IS JUSTIFIED.

MATTER OF: INFORMATION INTERNATIONAL, INC., AUGUST 8, 1980:

INFORMATION INTERNATIONAL, INC. (III) PROTESTS SOCIAL SECURITY ADMINISTRATION (SSA) PROCUREMENT SSA-RFP-78-0001, FOR MULTI-FONT OPTICAL SCANNING EQUIPMENT DESIGNED TO MACHINE "READ" OR "SCAN" W-2, W 2P, AND W-3 FORMS. THE EQUIPMENT WAS LEASED (WITH AN OPTION TO PURCHASE) FROM RECOGNITION EQUIPMENT, INC. (REI), THE LOW EVALUATED OFFEROR. WE SUSTAIN A PORTION OF III'S PROTESTS AND DISMISS OR DENY THE REMAINDER.

THE SAME PROCUREMENT WAS THE SUBJECT OF OUR DECISION IN INFORMATION INTERNATIONAL, INC., B-191013, MAY 31, 1978, 78-1 CPD 406. III ALSO SEEKS RECONSIDERATION OF THAT DECISION, IN WHICH WE DISMISSED AS UNTIMELY TWO OF III'S OBJECTIONS TO THE BENCHMARK THAT WAS REQUIRED AS PART OF PROPOSAL EVALUATION AND FOUND NO ABUSE OF DISCRETION BY SSA WITH RESPECT TO ITS USE OF AN "OFFSET PRINTED TEST DECK" TO BE "READ" DURING THE BENCHMARK.

PROPOSALS WERE EVALUATED BY MEANS OF A LIFE-CYCLE COST METHOD IN WHICH VARIOUS "BID EQUALIZATION FACTORS" WERE ADDED TO DEPRECIATED EQUIPMENT COSTS. AN ESTIMATED COST TO THE GOVERNMENT WAS COMPUTED FOR EACH SYSTEM CONFIGURATION EVALUATED. THE AMOUNT OF EACH OF THE BID EQUALIZATION FACTORS WAS CALCULATED FROM DATA PRODUCED DURING THE BENCHMARK.

THE SYSTEMS PROPOSED BY III AND REI DIFFER IN COMPLEXITY AND SOPHISTICATION. REI OFFERED WHAT IS CALLED A "DIRECT PAPER SYSTEM," WHICH READS THE ORIGINAL DOCUMENTS. III'S SYSTEM READS A MICROFILM COPY AND INCLUDES AN ERROR CORRECTION PROCESS WHICH PERMITS COMPUTER GENERATED IMAGES OF DOUBTFUL CHARACTERS TO BE QUERIED, DISPLAYED AND MANUALLY CORRECTED BY KEYBOARD OPERATORS WORKING IN AN ADJACENT TERMINAL ROOM.

BENCHMARK TIMING DATA WAS USED TO COMPUTE PRODUCTION (THROUGHPUT) RATES WHICH IN TURN WERE USED TO CALCULATE THE AMOUNT OF EACH VENDOR'S EQUIPMENT NEEDED TO MEET SSA'S PROJECTED WORKLOAD, CALCULATION OF REI'S EQUIPMENT REQUIREMENTS WAS RELATIVELY STRAIGHTFORWARD. III, HOWEVER, QUALIFIED PROPOSALS FOR USE OF TWO DIFFERENT MICROFILM CAMERAS (USING 16 MM AND 35 MM FILM) AND SUBMITTED THREE BASIC SCANNING EQUIPMENT CONFIGURATIONS FOR EACH. ALL OF III'S CONFIGURATIONS SHARE A COMMON PROCESS BUT UTILIZE RESOURCES DIFFERENTLY, PROVIDING A RANGE OF POTENTIAL CAPABILITY AT DIFFERING COSTS. PROPOSALS WERE SUBMITTED ON LEASE, FULL PAYMENT LEASE, LEASE WITH OPTION TO PURCHASE AND PURCHASE TERMS. WHILE REI WAS REQUIRED TO OFFER SIX UNITS INITIALLY AND OPTIONS TO FURNISH TWO ADDITIONAL UNITS, III'S LOWEST COST EVALUATED PROPOSAL (35 MM SYSTEMS ON PURCHASE TERMS) OFFERED THREE INITIAL SO-CALLED DUAL SCANNER INSTALLATIONS.

III'S BENCHMARK PERFORMANCE LARGELY OVERCAME ITS SIGNIFICANTLY HIGHER EQUIPMENT COST AND BROUGHT ITS LOWEST EVALUATED PROPOSAL WITHIN APPROXIMATELY $2.55 MILLION (9 PERCENT) OF THE REI PROPOSAL ACCEPTED FOR AWARD.

ESSENTIALLY, III SAYS THAT ITS PROPOSALS SHOULD HAVE BEEN EVALUATED AS LOWEST IN COST, THAT SSA MADE A NUMBER OF ERRORS IN COMPUTING THE EVALUATION FACTORS, THAT SSA IMPROPERLY MADE OTHER "ADJUSTMENTS" TO LIFE- CYCLE COSTS WHICH IMPROPERLY PENALIZED III, AND THAT III EQUIPMENT WOULD HAVE BEEN SHOWN TO HAVE BEEN LEAST COSTLY HAD THE BENCHMARK BEEN REPRESENTATIVE OF SSA'S ACTUAL REQUIREMENTS. III ALSO BELIEVES SSA IMPROPERLY REFUSED TO EVALUATE SEVERAL III LOWER COST ALTERNATE AND "UNSOLICITED" PROPOSALS.

I. RECONSIDERATION

III'S MAJOR OBJECTION IN ITS ORIGINAL PROTEST CONCERNED SSA'S DECISION TO LIMIT BENCHMARK SCANNING TO THE MOST EASILY READ FORMS-- APPROXIMATELY 70 PERCENT OF THE TOTAL. OUR PRIOR DECISION CONCLUDED THAT III KNEW OR SHOULD HAVE KNOWN SHORTLY AFTER DECEMBER 1, 1977, WHEN IT RECEIVED A LETTER OF THAT DATE FROM SSA, THAT SSA HAD REJECTED THE OBJECTIONS WHICH IT HAD RAISED AGAINST THIS "70 PERCENT LIMITATION." BECAUSE III DID NOT FILE ITS COMPLAINT WITH US WITHIN 10 DAYS THEREAFTER, WE VIEWED THE COMPLAINT AS UNTIMELY.

III CONTENDS THAT THE DECEMBER 1 LETTER SHOULD NOT HAVE BEEN REGARDED AS CONTROLLING BECAUSE IT HAD REASON TO BELIEVE THAT SSA WOULD STILL CONSIDER THE MATTER IN CONJUNCTION WITH CONSIDERATION OF III'S OTHER COMPLAINT CONCERNING THE BENCHMARK. III HAS NOW SUBMITTED ADDITIONAL EVIDENCE REGARDING A TELEPHONE CONVERSATION BETWEEN ITS GENERAL COUNSEL AND SSA'S ASSOCIATE COMMISSIONER FOR MANAGEMENT AND ADMINISTRATION, AS WELL AS A LETTER FROM ITS GENERAL COUNSEL TO THE ASSOCIATE COMMISSIONER DATED FEBRUARY 14, 1978. ACCORDING TO III, THIS DEMONSTRATES THAT SSA WAS WILLING TO CONSIDER THE MATTER FURTHER.

SSA DISAGREES. IT ARGUES THAT III'S EVIDENCE IS NOT NEWLY DISCOVERED EVIDENCE WHICH COULD NOT HAVE BEEN CONSIDERED EARLIER AND DENIES THAT THE ASSOCIATE COMMISSIONER AT ANY TIME GAVE III'S COUNSEL REASON TO BELIEVE THAT SSA WAS CONTEMPLATING REOPENING THE 70 PERCENT MATTER FOR RECONSIDERATION.

WE FIND NO MERIT TO III'S POSITION. THE FACT THAT AN AGENCY MAY BE WILLING TO FURTHER CONSIDER A PROTEST WHICH IT FIRST REJECTS DOES NOT TOLL THE TIME BY WHICH A PROTEST MUST BE LODGED WITH THIS OFFICE. HAD SSA REEXAMINED III'S COMPLAINT AFTER DECEMBER 1, 1977, III'S OBJECTION TO THE 70 PERCENT LIMITATION FILED HERE AFTER THAT REEXAMINATION WOULD STILL HAVE BEEN UNTIMELY BECAUSE OUR PROTEST PROCEDURES REQUIRE A PROTEST TO BE FILED HERE WITHIN 10 DAYS OF INITIAL ADVERSE AGENCY ACTION. 4 C.F.R. 20.2(A)(1980).

III ALSO ARGUES THAT ITS PRIOR PROTEST OF THE BENCHMARK IS INEXTRICABLY BOUND TO ISSUES RAISED BY ITS SUBSEQUENT PROTEST AND SHOULD BE CONSIDERED AT THIS TIME BECAUSE THE RESULTS ACHIEVED BY THE BENCHMARK ARE NOW KNOWN.

OUR PRIOR DECISION DEALT WITH A PROTEST FILED BEFORE BENCHMARKING. THE PROTESTER SOUGHT OUR REVIEW OF THE PROPOSED BENCHMARK PROCEDURE. WE WERE NOT ASKED TO REVIEW OTHER ASPECTS OF SSA'S EVALUATION OF PROPOSALS, OR REI'S SELECTION, WHICH HAD NOT THEN BEEN MADE.

WE DO NOT BELIEVE RECONSIDERATION OF OUR DECISION REGARDING III PRE BENCHMARK ASSERTIONS TO BE APPROPRIATE BECAUSE EVEN IF WE AGREED WITH III IN RETROSPECT THAT SSA'S BENCHMARK ASSUMPTIONS WERE FAULTY, WE WOULD NOT AGREE THAT ITS EARLIER COMPLAINTS SHOULD BE CONSIDERED AT THIS POINT SIMPLY BECAUSE THE TEST RESULTS ARE NOW KNOWN. THE RECORD DOES NOT SHOW THAT OUR DECISION DISMISSING TWO OF THE ALLEGATIONS AND DENYING THE VALIDITY OF THE OTHER WAS FOUNDED ON ANY ERROR OF LAW OR ANY MISUNDERSTANDING OF THE FACTS EXISTING DURING THE TIME FRAME WITH WHICH OUR DECISION WAS CONCERNED. SEE, EG., ORDNANCE RESEARCH, INC.-- RECONSIDERATION, B-194043, JUNE 26, 1979, 79-1 CPD 455.

CONSEQUENTLY, III'S REQUEST FOR RECONSIDERATION IS DENIED.

II. THE PROTEST

A. PRELIMINARY MATTERS

AT THE OUTSET, WE DECLINE TO CONSIDER CERTAIN ASPECTS OF THE PROTEST. FIRST, WE DO NOT BELIEVE IT APPROPRIATE TO REVIEW III'S OBJECTIONS TO SSA'S PROPOSED POST-SCANNING ERROR CORRECTIONS PROCESS. SSA'S ELABORATE POST-SCANNING CORRECTION PROCESS CONSISTED OF SEVERAL STEPS INCLUDING MANUALLY CORRECTING DATA. III, HOWEVER, BELIEVES ERRORS EXPERIENCED DURING ACTUAL OPERATIONS OF THE EQUIPMENT SHOULD NOT BE CORRECTED MANUALLY THROUGH A SEPARATE PROCESS LONG AFTER SCANNING, BUT THAT ERRORS DUE TO INDIVIDUAL MISFILMED DOCUMENTS COULD BE MORE EFFECTIVELY CORRECTED BY REFILMING DOCUMENTS CONTAINING ERRORS DURING SUBSEQUENT SCANNING. THE DISPUTE DOES NOT INVOLVE COMPLETELY MISFILMED BATCHES OF DOCUMENTS, WHICH SSA WOULD HAVE REFILMED IN ANY EVENT.

WE VIEW THIS ISSUE AS OUTSIDE THE PURVIEW OF THE BID PROTEST PROCEDURE. IT DOES NOT CONCERN SPECIFICATION REQUIREMENTS OR ANY OTHER ASPECT OF THE PROCUREMENT FOR THIS EQUIPMENT, BUT RATHER THE AGENCY'S PLANS FOR OPERATING THE EQUIPMENT ONCE IT IS IN PLACE. WHILE SSA'S APPLICATION OF ITS CORRECTIONS PROCESS TO THE BENCHMARK IMPACTED ON THE LIFE CYCLE COST EVALUATION, THE VALIDITY OF THE PROCESS ITSELF IS NOT, IN OUR VIEW, A PROPER PART OF THIS PROTEST.

SECOND, WE ALSO VIEW AS INAPPROPRIATE FOR OUR CONSIDERATION III'S BELIEF THAT SSA WAS BIASED AGAINST INNOVATIVE TECHNOLOGY. HOWEVER WISE III MAY BELIEVE SSA SHOULD HAVE BEEN IN SEEKING A MORE TECHNICALLY ADVANCED SYSTEM, WE ARE AWARE OF NO LEGAL REQUIREMENT THAT SSA OBTAIN THE MOST TECHNOLOGICALLY SOPHISTICATED APPROACH AVAILABLE. ON THE CONTRARY, SSA'S PROCUREMENT DECISIONS REFLECT A BELIEF IN THE IMPORTANCE OF PROVEN PERFORMANCE-- A PARTICULARLY LEGITIMATE CONCERN CONSIDERING THE CONSEQUENCES FACING SSA IF DATA ENTRY USING A SCANNING PROCESS PROVED UNSUCCESSFUL. SEE PENTECH DIVISION, HOUDAILLE INDUSTRIES, INC., B-192453, JUNE 18, 1980, 80-1 CPD 427; CF. SYSTEM DEVELOPMENT CORPORATION, 58 COMP.GEN. 475(1979), 79-1 CPD 303.

WE ALSO FIND THAT SOME OF III'S POST-BENCHMARK COMPLAINTS ARE UNTIMELY.

SECTION 20.2(B)(1) OF OUR BID PROTEST PROCEDURES STATES THAT PROTESTS BASED UPON ALLEGED IMPROPRIETIES IN ANY TYPE OF SOLICITATION WHICH ARE APPARENT PRIOR TO ANY CLOSING DATE FOR RECEIPT OF INITIAL OR AMENDED PROPOSALS MUST BE FILED BY THAT DATE. THIS INCLUDES, FOR EXAMPLE, A DATE SET FOR SUBMISSION OF ADDITIONAL TECHNICAL DATA REQUESTED DURING DISCUSSIONS. IN THIS REGARD, BENCHMARKING IS USED FOR PROPOSAL EVALUATION TO PRODUCE "DESCRIPTIVE" DATA WHICH THE GOVERNMENT BELIEVES IS NECESSARY TO ASSESS THE CAPABILITIES AND/OR COST OF EQUIPMENT PROPOSED. SEE, E.G., 48 COMP.GEN. 320(1968); COMPUTER NETWORK CORPORATION, 56 COMP.GEN. 245, 255-256(1977), 77-1 CPD 31. WE BELIEVE, THEREFORE, THAT A DATE SET FOR SUBMISSION OF AN OFFEROR'S BENCHMARK DATA SHOULD BE TREATED AS A CLOSING DATE WITHIN THE MEANING OF SECTION 20.2(B)(1). THUS, PROTESTS CONCERNING AMENDMENTS TO A SOLICITATION WHICH DEFINE HOW THE BENCHMARK AND EVALUATION OF BENCHMARK RESULTS WILL BE HANDLED MUST BE FILED BY THE BENCHMARK SUBMISSION DATE. CF. COMSHARE, INC., B-192927, DECEMBER 5, 1978, 78-2 CPD 387.

IT IS IN THIS LIGHT THAT WE VIEW III'S OBJECTION THAT IT SHOULD HAVE BEEN CHARGED FOR COSTS ASSOCIATED WITH HIRING ONLY ONE RATHER THAN TWO OPERATORS PER SCANNER FOR ITS DUAL SCANNER CONFIGURATION. THE SOLICITATION PROVIDED THAT OFFERORS WOULD BE CHARGED OPERATOR COSTS BASED UPON THE NUMBER OF OPERATORS ACTUALLY USED DURING THE BENCHMARK. SSA WOULD HAVE CHARGED III WITH COSTS ASSOCIATED WITH ONE OPERATOR HAD ONLY ONE BEEN USED FOR PRODUCTION TASKS, BUT III USED TWO, OSTENSIBLY BECAUSE IT DID NOT HAVE SUFFICIENT TIME (FOUR WEEKS FOLLOWING AN AMENDMENT DELETING A MANDATORY MINIMUM TWO OPERATOR REQUIREMENT) TO RETRAIN ITS OPERATORS. III WAS REMINDED OF THIS SOLICITATION RULE DURING THE MICROFILMING PORTION OF THE TEST. CONCEDING, THEREFORE, THAT TWO OPERATORS WERE USED, III NEVERTHELESS ARGUES THAT OTHER BENCHMARK PARAMETERS WERE TO BE DETERMINED BASED ON ACTUAL MEASUREMENTS OF RESOURCES REQUIRED AND THAT OPERATOR REQUIREMENTS SHOULD BE, TOO. IN OUR VIEW, III SHOULD HAVE PROTESTED SSA'S REQUIREMENT BEFORE BENCHMARKING, OR COMPLAINED THEN THAT THE TIME ALLOWED FOR RETRAINING WAS NOT SUFFICIENT. BECAUSE THE OBJECTION WAS RAISED AFTER EVALUATION, IT WILL NOT BE CONSIDERED.

III ALSO COMPLAINS THAT SSA IMPROPERLY COMPUTED CERTAIN SO-CALLED RESIDUAL ERROR RATES BY ARBITRARILY ASSUMING THAT ONLY ONE PERCENT OF MANUALLY CORRECTED DATA WOULD CONTAIN ERRORS WHICH WOULD REQUIRE CORRECTION A SECOND TIME AND BY MAKING ITS CALCULATION BY COUNTING "FIELDS" RATHER THAN INDIVIDUAL CHARACTER ERRORS. ACCORDING TO SSA, THE ONE PERCENT FIGURE BY FIELDS IS THE ONLY DATA AVAILABLE BECAUSE IT HAS NOT KEPT MORE DETAILED STATISTICS IN THE PAST.

THE 1 PERCENT RESIDUAL ERROR RATE WAS INCLUDED IN SSA'S SOLICITATION COST TABLES WHICH INDICATED HOW THE COMPUTATION WOULD BE PERFORMED. THUS, III SHOULD HAVE KNOWN FROM THE RFP THE BASIS ON WHICH THE CALCULATION WOULD BE MADE AND SHOULD HAVE PROTESTED THIS MATTER ALSO PRIOR TO THE BENCHMARK.

B. LIFE CYCLE COST ADJUSTMENTS AND EVALUATION

III RAISES A NUMBER OF OBJECTIONS TO SPECIFIC ADJUSTMENTS WHICH SSA MADE, OR WHICH III BELIEVES SHOULD HAVE BEEN MADE, BUT WERE NOT. III ALSO QUESTIONS VARIOUS OTHER ASPECTS OF THE EVALUATION. COLLECTIVELY, SSA'S ADJUSTMENTS AND OTHER SCORING ASSUMPTIONS HAD A SIGNIFICANT IMPACT ON ITS CALCULATION OF REI'S AND III'S EVALUATED LIFE-CYCLE COST SO THAT III'S QUESTIONS MUST BE ADDRESSED BEFORE ITS BROADER CONCERNS-- ATTACKING THE MEANINGFULNESS OF SSA'S COST EVALUATION AS BASIS FOR SELECTION-- CAN BE CONSIDERED.

A NUMBER OF III'S COMPLAINTS RELATE TO SSA'S EVALUATION OF THE BENCHMARK RESULTS, OR TO ITS REFUSAL AFTER BENCHMARKING TO CONSIDER UNTESTED ALTERNATE APPROACHES.

FOR EXAMPLE, AFTER BENCHMARKING III FIRST PROPOSED ITS SO-CALLED "PAIRED SINGLES" CONFIGURATION, ONE OF THE THREE BASIC EQUIPMENT CONFIGURATIONS III OFFERED. APPARENTLY, III DID NOT CONCEIVE OF THE PAIRED SINGLES CONFIGURATION UNTIL BENCHMARKING HAD BEEN COMPLETED.

SSA HAD PLACED NO LIMIT ON WHAT CONFIGURATION COULD BE BENCHMARKED, AND PRESUMABLY WOULD HAVE ALLOWED THE PAIRED SINGLES CONFIGURATION TO BE TESTED HAD IT BEEN PROPOSED EARLIER. IT REFUSED, HOWEVER, TO CONSIDER THE PAIRED SINGLES APPROACH BECAUSE IT WAS PROPOSED AFTER BENCHMARKING WAS COMPLETED. III MAINTAINS THAT THE PAIRED SINGLES PROPOSAL NEED NOT BE BENCHMARKED BECAUSE ALL OF ITS ELEMENTS WERE TESTED AND BECAUSE THE BENCHMARK FULLY ESTABLISHED THAT THE EQUIPMENT WOULD OPERATE AT LESS THAN 70 PERCENT OF CAPACITY EVEN IN THE PAIRED SINGLES CONFIGURATION.

WE AGREE WITH III THAT SSA ORDINARILY COULD NOT REQUIRE THAT THE PAIRED SINGLES PROPOSALS BE BENCHMARKED WITHOUT REASON, JUST AS A CONTRACTING ACTIVITY CANNOT REQUIRE UNNECESSARY DESCRIPTIVE DATA OR REJECT A PROPOSAL WHICH FAILS TO INCLUDE SUCH DATA. DOMINION ENGINEERING WORKS, LTD., B-186543, OCTOBER 8, 1976, 76-2 CPD 324.

III ADMITS, HOWEVER, THAT THERE IS A QUESTION AS TO WHETHER SHARING RESOURCES AS PROPOSED WITH THE PAIRED SINGLES APPROACH WOULD LIMIT PERFORMANCE. IT ATTEMPTS TO MEET THIS OBJECTION BY PRESENTING A WORST CASE MATHEMATICAL ANALYSIS SHOWING THAT IT IS THE COMPUTATIONAL CAPACITY OF THE PROCESSOR WHICH IS THE CRITICAL LIMITING FACTOR AND THAT ADEQUATE CAPACITY WOULD BE AVAILABLE.

EVEN ASSUMING THE REASONABLENESS OF III'S TECHNICAL ARGUMENT, WE BELIEVE SSA'S REFUSAL TO CONSIDER THE PAIRED SINGLES PROPOSAL WITHOUT BENCHMARKING IS RATIONALLY FOUNDED. THE FINANCIAL RISKS FACING SSA WERE SUBSTANTIAL. IT CHOSE TO USE BENCHMARKING BECAUSE IT WANTED TO BASE AWARD ON PROVEN RATHER THAN THEORETICAL PERFORMANCE. IN THIS REGARD, SSA POINTS OUT THAT DURING THE COURSE OF THIS PROCUREMENT III PROPOSED SEVERAL DIFFERENT APPROACHES AND MADE NUMEROUS PERFORMANCE CLAIMS, SOME OF WHICH WERE NOT BORNE OUT IN PRACTICE. NOT ALL OF III'S PROPOSED APPROACHES QUALIFIED BY PASSING BENCHMARK MINIMUM PERFORMANCE REQUIREMENTS. IN SSA'S VIEW, ANALYTICAL ABSTRACT CALCULATIONS PREDICTING THE BEHAVIOR OF INTERRELATED SYSTEM COMPONENTS MAY NOT NECESSARILY CORRELATE WITH ACTUAL RESULTS, IMPOSING IN EFFECT UNACCEPTABLE RISKS. SSA'S DECISION THEREFORE REFLECTS ITS BEST JUDGMENT; THAT JUDGMENT HAS NOT BEEN SHOWN TO BE ARBITRARY.

MOREOVER, WE NOTE THAT III KNEW BEFORE IT RAN THE BENCHMARK THAT SSA WOULD INSIST THAT EACH SYSTEM PROPOSED BE BENCHMARKED BECAUSE III WAS TOLD SPECIFICALLY THAT SSA WOULD NOT AGREE TO LIMIT BENCHMARKING TO THE DOUBLE SCANNER SYSTEM. NONE OF THE FACTORS III CITES TO SUPPORT ITS CLAIM THAT THE PAIRED SINGLES APPROACH DID NOT OCCUR TO IT EARLIER BECAUSE SSA'S REQUIREMENTS WERE CONSTANTLY CHANGING INVOLVE MATTERS WHICH III WOULD NOT HAVE CONSIDERED BEFORE BENCHMARKING. OBVIOUSLY, III, BY NOT PROPOSING THIS APPROACH PRIOR TO BENCHMARKING, RAN THE RISK OF NOT HAVING THAT APPROACH ACCEPTED.

III ALSO COMPLAINS THAT SSA IMPROPERLY ADJUSTED CERTAIN 0ENCHMARK DATA, OR FAILED TO ADJUST ALL OFFEROR'S DATA EQUALLY, GIVING REI AN UNFAIR ADVANTAGE. FOR EXAMPLE, SSA FOUND THAT IN SOME INSTANCES CHARACTER AND FIELD SUBSTITUTION AND REJECT RATES DID NOT CORRESPOND. REI'S PROVED TO BE ESPECIALLY SENSITIVE TO THE LOCATION OF CHARACTERS WITHIN PRESCRIBED AREAS ON THE TEST FORMS. WHERE CHARACTERS APPEARED OUTSIDE THE BOX ASSIGNED TO A SPECIFIC FIELD, THE EQUIPMENT PLACED THEM IN THE WRONG FIELD. SSA SAYS IT DISREGARDED SUCH ERRORS IF THE CHARACTERS PRINTED ON THE FORM WERE READ CORRECTLY BUT WERE PLACED IN AN INCORRECT FIELD. III BELIEVES, HOWEVER, THAT THE ADJUSTMENT FAVORED REI AND WAS UNFAIR BECAUSE EDIT FUNCTIONS WERE TESTED DURING THE BENCHMARK.

SSA ADMITS THAT REI OBTAINED A LARGER ADJUSTMENT IN SUBSTITUTION RATE THAN DID III BUT ARGUES THAT THIS WAS PROPER IN THE CIRCUMSTANCES, NOTING THAT:

SINCE THESE TYPES OF ERRORS ARE EASILY CORRECTED THROUGH FIELD EDITING AND BECAUSE EDITING OF THE DATA WAS NOT PART OF THE (BENCHMARK) REQUIREMENT, IT WAS DECIDED THAT THE PROPERLY READ BUT DISLOCATED CHARACTERS SHOULD NOT BE COUNTED AS SUBSTITUTIONS. III HAD A SMALLER AMOUNT OF ADJUSTMENT BECAUSE ITS SCANNING PARAMETERS WERE ESTABLISHED DIFFERENTLY THAN REI'S.

IN THIS REGARD, WE SEE NO BASIS FOR PROTEST IF AN AGENCY ADJUSTS BENCHMARK DATA IN EVALUATING IT, PROVIDED THE ADJUSTMENTS MADE ARE REASONABLY RELATED TO THE ANNOUNCED EVALUATION CRITERIA AND PROVIDED THE BASIS ON WHICH BENCHMARKS WERE RUN OR EVALUATION CRITERIA USED ARE NOT ALTERED. WE FIND NOTHING IMPROPER WITH THE ADJUSTMENTS MADE WHICH ARE BASED ON SSA'S BELIEF THAT DISLOCATED CHARACTERS WOULD BE IMPROPERLY COUNTED IF THEY WERE TREATED AS SUBSTITUTION CHARACTERS WHICH APPEARS TO BE OVERALL CONSISTENT WITH THE LIFE-CYCLE COSTING EVALUATION CRITERION. CF. AEL SERVICE CORPORATION, 53 COMP.GEN. 800(1974), 74-1 CPD 217. UNLESS III COULD ESTABLISH, AS IT HAS NOT, THAT (1) IT SET ITS MACHINE PARAMETERS FOR BENCHMARKING IN RELIANCE ON THE INCLUDED EDIT REQUIREMENT; (2) THESE PARAMETERS WERE SET TO MINIMIZE THE KINDS OF ERRORS WHICH SSA LATER CORRECTED; AND (3) III OTHERWISE COULD HAVE ADJUSTED ITS PARAMETERS TO BETTER OPTIMIZE PERFORMANCE, IT HAS NO BASIS FOR COMPLAINT.

FURTHER, III COMPLAINS THAT SSA ARBITRARILY IMPOSED: (1) A 30 SECOND PENALTY FOR JOB SET-UP; AND (2) A 3-PERCENT PENALTY FOR OPERATION OF OUTPUT EDIT SOFTWARE, WHICH WAS TESTED BUT NOT REQUIRED TO BE USED COMPILING BENCHMARK DATA.

III HAS NOT PRESENTED EVIDENCE ESTABLISHING THAT THESE ADJUSTMENTS WERE UNREASONABLE, PER SE. JOB SET-UP TIME WAS NOT MEASURED BY THE BENCHMARK BECAUSE ONLY ONE WORK UNIT (5,000 DOCUMENTS) WAS TESTED. A SIMILAR PENALTY WAS APPLIED IN COMPUTING REI THROUGHPUT. THE EFFECT OF THE PENALTY WAS TO DOWNGRADE THROUGHPUT SOMEWHAT TO AFFORD SSA ASSURANCE THAT THE QUANTITY OF EQUIPMENT ACQUIRED WOULD PROVIDE SOME MARGIN FOR SET-UP TIME. LIKEWISE, THE 3 PERCENT WAS ASSESSED TO ACCOUNT FOR SYSTEM DEGRADATION WHILE EDITING FUNCTIONS WERE BEING PERFORMED. A 10-PERCENT DEGRADATION WAS OBSERVED DURING THE BENCHMARK, BUT SSA AGREED TO THE LOWER FIGURE TO ACCOUNT FOR EXPECTED ENHANCED PERFORMANCE ONCE THE HIGHER LEVEL LANGUAGE USED DURING THE BENCHMARK WAS CONVERTED TO THE DEC-10 MACHINE LANGUAGE WHICH WOULD HAVE BEEN USED IF III HAD RECEIVED AWARD. CONSEQUENTLY, III'S OBJECTIONS TO THESE ADJUSTMENTS ARE WITHOUT MERIT.

WE AGREE WITH III, HOWEVER, THAT SEVERAL OF SSA'S ADJUSTMENTS, AND ITS REFUSAL IN ONE INSTANCE TO CONSIDER AN ADJUSTMENT, WERE INAPPROPRIATE.

FOR EXAMPLE, WE AGREE WITH III THAT SSA IMPROPERLY APPLIED THE 30 SECOND PENALTY TO MAGNETIC TAPE CHANGES BY ARBITRARILY ADDING 30 SECONDS TO THE PROCESSING TIME REQUIRED FOR EACH 5,000 DOCUMENT UNIT. AS III POINTS OUT, A 2,400 FOOT REEL OF MAGNETIC TAPE WILL HOLD THE INFORMATION RECORDED ON 35 BATCHES OF DOCUMENTS, OR APPROXIMATELY 175,000 DOCUMENTS. SSA INSISTS THAT THE REQUIREMENT IS NECESSARY, NEVERTHELESS, BECAUSE THE 5,000 PAGE LIMITATION IS REQUIRED TO MEET SUBSEQUENT OPERATIONAL STEPS WHICH SSA WOULD USE IN ANNUAL REPORTING.

IMPOSING A 5,000 PAGE LIMIT IS NOT THE ONLY WAY INFORMATION IN 5,000 PAGE BLOCKS COULD BE HANDLED. DATA CAN BE "BLOCKED" ELECTRONICALLY IN 5,000 PAGE UNITS EVEN THOUGH MANY 5,000 PAGE UNITS ARE RECORDED ON ONE TAPE, AND WE UNDERSTAND SUCH A PROCEDURE IS POSSIBLE FOR USE ON THE EQUIPMENT SSA PLANNED TO USE, WHICH WOULD SIGNIFICANTLY REDUCE THE NUMBER OF TAPES NEEDED. SSA'S APPROACH APPEARS TO BE BASED ON ITS EXPERIENCE WITH REI TYPE EQUIPMENT. III'S IN-LINE CORRECTION PROCESS, HOWEVER, WOULD HAVE REDUCED SIGNIFICANTLY THE AMOUNT OF POST-SCANNING PROCESSING WHICH MIGHT BE REQUIRED OTHERWISE. SSA'S EVALUATION OF III'S APPROACH ON THE BASIS OF WHAT SSA WOULD REQUIRE BY USE OF REI'S SYSTEM IS UNREASONABLE.

IN ADDITION TO THE 30 SECOND AND 3 PERCENT PENALTIES, SSA ADDED A 90 SECOND ALLOWANCE FOR FILM CHANGES AT THE SCANNER. III AGREES THAT 90 SECONDS IS A REASONABLE TIME FOR THE CHANGES, BUT QUESTIONS THE MANNER IN WHICH IT WAS APPLIED, POINTING OUT THAT ONE SCANNER IN A DUAL SCANNER CONFIGURATION NORMALLY CONTINUES TO OPERATE WHILE FILM IS BEING CHANGED IN THE OTHER.

SSA STATES THAT IT:

* * * IS AWARE OF THE SITUATION CITED BY III * * * . HOWEVER, SSA COULD NOT ACCEPT THE PREMISE THAT THERE WOULD BE NO SITUATIONS IN WHICH BOTH FILMS WOULD NOT HAVE TO BE CHANGED SIMULTANEOUSLY. THE QUALITY OF THE SCANNER INPUT AND THE NUMBER OF FONTS INVOLVED WOULD BE AMONG THE VARIOUS FACTORS WHICH WILL AFFECT THE READ RATE OF THE SCANNERS. REGARDLESS OF ATTEMPTS TAKEN TO PRECLUDE SUCH SIMULTANEOUS CHANGING OF THE FILM IN BOTH SCANNERS IN A DUAL-SCANNER SYSTEM DURING FULL OPERATIONS, THERE WILL BE RANDOM OCCASIONS WHEN BOTH FILM TRANSPORTS RUN OUT OF FILM AT, ESSENTIALLY, THE SAME TIME.

BELIEVING THAT IT WOULD BE SPECULATIVE TO ATTEMPT TO GAUGE THE RATE THAT THIS WOULD OCCUR, SSA NEVERTHELESS ESTIMATES THAT III WOULD REQUIRE ABOUT 1,100 FILM CHANGES PER MONTH USING 35 MM FILM, OR 560 CHANGES IF 16 MM FILM WERE EMPLOYED.

TO THE EXTENT THAT FILM CHANGES ARE STATISTICALLY MEANINGFUL, CHANGES IN BOTH SCANNERS OF A DUAL SCANNER SYSTEM SHOULD BE COUNTED. KNOWING APPROXIMATELY WHAT NUMBER OF FILM CHANGES ARE REQUIRED, SSA COULD HAVE ESTIMATED STATISTICALLY THE LIKELIHOOD THAT TWO FILM CHANGES WOULD BE REQUIRED WITHIN ANY 90 SECOND PERIOD. ALTHOUGH PROBABLY DE MINIMUS, THE EXTENT TO WHICH FILM CHANGES IN A DUAL SCANNER SYSTEM ARE NOT STATISTICALLY INDEPENDENT EVENTS COULD HAVE BEEN EVALUATED BY MEASURING ANY CHANGES IN THROUGHPUT OF ONE SCANNER WHILE THE OTHER WAS NOT OPERATING.

III ALSO BELIEVES THAT SOME KIND OF FACTOR SHOULD HAVE BEEN APPLIED IN EVALUATING REI'S BENCHMARK RESULTS TO ACCOUNT FOR POSSIBLE PAPER JAMS JUST AS III WAS PENALIZED FOR COSTS ASSOCIATED WITH MANUALLY REPROCESSING INDIVIDUALLY MISFILMED DOCUMENTS. SSA VIEWS III'S COMPLAINT AS UNTIMELY. WE DISAGREE. ALTHOUGH III KNEW THAT NUMEROUS ADJUSTMENTS WERE MADE TO ITS SCORES, IT WAS ENTITLED TO ASSUME THAT COMPARABLE APPROPRIATE ADJUSTMENTS WERE BEING MADE TO OTHER PROPOSALS. ACCORDINGLY, WE TREAT III'S COMPLAINT AS UNTIMELY. WE DISAGREE. ALTHOUGH III KNEW THAT NUMEROUS ADJUSTMENTS WERE MADE TO ITS SCORES, IT WAS ENTITLED TO ASSUME THAT COMPARABLE APPROPRIATE ADJUSTMENTS WERE BEING MADE TO OTHER PROPOSALS. ACCORDINGLY, WE TREAT III'S COMPLAINT AS TIMELY SINCE IT WAS FILED WITHIN 10 DAYS OF THE DATE IT KNEW NO SUCH ADJUSTMENTS HAD BEEN MADE.

SSA ARGUES THAT BOTH THE EFFECT OF PAPER JAMS ON REI'S DIRECT PAPER SYSTEMS AND MISFILMED DOCUMENTS ON THE III PROCESS WERE EVALUATED. THEY WERE EVALUATED DIFFERENTLY, SSA ASSERTS, BECAUSE THE PROCESSES DIFFER. REI RAN THE BENCHMARK AS A SINGLE CONTINUOUS PROCESS. PAPER JAMS OR OTHER TECHNICAL PROBLEMS IN ITS SYSTEM WOULD HAVE SHOWED UP DIRECTLY IN REDUCED THROUGHPUT, BECAUSE PROBLEMS WOULD HAVE TO BE CORRECTED AS THEY AROSE. THIS COULD NOT OCCUR WITH THE III PROCESS, SINCE UNTIL THE MICROFILM USED WAS DEVELOPED THERE COULD BE NO DETERMINATION AS TO MISFILMED DOCUMENTS. SINCE SSA COULD NOT MEASURE MISFILMING ERRORS DURING THE III BENCHMARK IT COMPENSATED BY APPLYING AN ADJUSTMENT BASED ON EXPERIENCE WITH OTHER MICROFILM USES.

AS A RESULT, III WAS EVALUATED USING SO-CALLED "REAL WORLD DATA"; REI WAS EVALUATED ON THIS ITEM ON THE BASIS OF ACTUAL BENCHMARK RESULTS. SSA DOES NOT ADDRESS WHETHER THIS WAS EQUITABLE. HOW MUCH OF A DIFFERENCE WOULD HAVE RESULTED FROM USE OF ACTUAL FORMS RECEIVED BY SSA RATHER THAN THE NEATLY STACKED OFFSET PRINTED FORMS ACTUALLY USED IS UNCLEAR BUT IT SEEMS APPARENT THAT THE DIFFERENCE SHOULD HAVE HAD AN EFFECT ON REI'S EVALUATION. THUS WE BELIEVE A "REAL WORLD" ADJUSTMENT SHOULD ALSO HAVE BEEN APPLIED TO REI'S TEST RESULTS.

III RAISES SEVERAL COMPLAINTS WHOSE RELATIONSHIP TO SSA BENCHMARK IS LESS DIRECT, BUT WHICH ARE IMPORTANT NEVERTHELESS IN LAYING A FOUNDATION ON WHICH WE MAY REVIEW SSA'S LIFE-CYCLE COST EVALUATION.

FOR EXAMPLE, III MAINTAINS THAT MICROFILM PURCHASE AND PROCESSING COSTS WERE EVALUATED IMPROPERLY. EXCEPT FOR III'S FULL SERVICE PLAN OFFERING TO SET UP CONTRACTOR-OWNED CONTRACTOR-OPERATED (COCO) PROCESSING FACILITIES, III'S PROPOSALS ASSUMED THAT THE GOVERNMENT WOULD FURNISH FILM AND PROCESSING. III MAINTAINS, HOWEVER, THAT BY CONSIDERING ONLY EASTMAN KODAK FILM AND PROCESSING, SSA OVERESTIMATED III'S LIFE-CYCLE COST BY AT LEAST $500,000. IN III'S VIEW, SSA SHOULD HAVE SURVEYED POTENTIAL FILM AND PROCESSING SUPPLIERS TO DETERMINE PRICING, USING THE FEDERAL SUPPLY SCHEDULE (FSS) AS A STARTING POINT.

SSA STATES THAT IT USED KODAK PRICING BECAUSE III SPECIFIED KODAK FILM IN ITS PROPOSAL AND USED IT DURING THE BENCHMARK. SSA ADMITS THAT III PROPOSED ROCHESTER FILM PRODUCTS AS AN ALTERNATE SOURCE, BUT STATES THAT THE ROCHESTER FILM PROPOSAL WAS RECEIVED AFTER THE CLOSING DATE FOR RECEIPT OF BEST AND FINAL OFFERS.

III ADMITS THAT IT IDENTIFIED TWO TYPES OF KODAK FILM BY NUMBER AND USED KODAK FILM IN THE BENCHMARK. WE BELIEVE THAT BY FAILING TO INDICATE THAT OTHER FILM MIGHT BE ACCEPTABLE, III RAN THE RISK THAT SSA WOULD BELIEVE THE FILM CHOSEN MIGHT BE CRITICAL TO CONTRACT PERFORMANCE AND EVALUATE ON THAT BASIS. CONSEQUENTLY, WE DO NOT OBJECT TO THIS ASPECT OF THE EVALUATION.

III ALSO OBJECTS TO SSA'S REJECTION OF ITS SO-CALLED BEST AND FINAL "ZERO PREVENTIVE MAINTENANCE" PROPOSAL, IN WHICH IT ENTERED "ZERO" FOR SCHEDULED DAILY PREVENTIVE MAINTENANCE TIME. ARGUING THAT THE CONCEPT OF PREVENTIVE MAINTENANCE IS MADE OBSOLETE BY MODERN SEMICONDUCTOR TECHNOLOGY, III MAINTAINS THAT MAINTENANCE SHOULD BE SCHEDULED ONLY AS REQUIRED. III SAYS DAILY SCHEDULED MAINTENANCE DOWN TIME IS UNNECESSARY, BECAUSE MODERN EQUIPMENT, E.G., MEMORY, IS CONSTANTLY MONITORED AND SHOWS GRADUAL DEGENERATION, THEREBY PERMITTING PLANNED REPLACEMENT. IN III'S VIEW, IT IS UNREALISTIC TO DEDUCT ONE OR TWO HOURS FROM AVAILABLE WORKING TIME TO ALLOW TIME FOR MAINTENANCE WHICH WILL NOT BE PERFORMED.

SSA ASSERTS THAT III'S COMPLAINT IS UNTIMELY, AND REJECTS III'S VIEW THAT PREVENTIVE MAINTENANCE TIME AND DOWN TIME ARE CONCEPTUALLY INTERCHANGEABLE. NOTING THAT OFFERORS WERE REQUIRED FROM THE OUTSET TO STATE HOW MUCH TIME WAS TO BE SET ASIDE FOR PREVENTIVE MAINTENANCE, SSA ASSERTS THAT III'S PROTEST ON THIS ISSUE SHOULD HAVE BEEN FILED BEFORE THE CLOSING DATE FOR RECEIPT OF INITIAL PROPOSALS. ARGUING THAT "PRESCHEDULED MAINTENANCE" IS PREVENTIVE MAINTENANCE REGARDLESS OF WHAT III WISHES TO CALL IT, SSA QUESTIONS HOW III COULD ACHIEVE A 92 PERCENT (OR FOR THAT MATTER, A 90 PERCENT) AVAILABILITY RATE WITHOUT SCHEDULING SOME KIND OF MAINTENANCE. III COUNTERS THAT THE ISSUE IS TIMELY AND THAT, MOREOVER, ITS EQUIPMENT ACHIEVES A 97 TO 98 PERCENT AVAILABILITY RATE.

WE DO NOT VIEW III'S COMPLAINT AS UNTIMELY BECAUSE NOTHING IN THE RFP PREVENTED CONSIDERATION OF AN OFFER PROPOSING ZERO TIME FOR PREVENTIVE MAINTENANCE. SSA MAY NOT HAVE MEANT TO ALLOW A ZERO TIME PROPOSAL. DID NOT PRECLUDE ONE, HOWEVER, AND III PROTESTED AS SOON AS IT LEARNED ITS ZERO TIME OFFER HAD BEEN REJECTED. THE RULE SSA RELIES UPON APPLIES ONLY TO DEFECTS WHICH ARE APPARENT ON THE FACE OF A SOLICITATION. 4 C.F.R. 20.2(B)(1).

WITH RESPECT TO THE MERITS OF THE ISSUE, WE POINT OUT THAT THE BURDEN IS ON THE OFFEROR IN SUBMITTING HIS BEST AND FINAL OFFER TO AFFIRMATIVELY DEMONSTRATE ITS MERITS. THE CONTRACTING OFFICER NEED NOT REOPEN NEGOTIATIONS, OR SPECULATE AS TO WHETHER AN UNSUBSTANTIATED PROPOSAL COULD BE SUPPORTED WITH ADEQUATE TECHNICAL DATA, BUT MAY DOWNGRADE OR REJECT THE PROPOSAL AS THE CIRCUMSTANCES WARRANT. HERE, III'S ZERO PREVENTIVE MAINTENANCE PROPOSAL MARKED A SIGNIFICANT DEPARTURE FROM ITS EARLIER APPROACH, AND SSA DID NOT FIND ANYTHING CONVINCING IN III'S PROPOSAL WITH RESPECT TO THE VALIDITY OF THIS APPROACH. WE FIND NO BASIS FOR TAKING EXCEPTION TO SSA'S REJECTION OF THIS APPROACH.

FROM SSA'S ANSWER TO THIS ALLEGATION, HOWEVER, WE HAVE SOME DOUBT REGARDING SSA'S CONSISTENCY IN APPLYING ITS LIFE CYCLE COSTING TECHNIQUE. IN EXPLAINING ITS POSITION, SSA ARGUES THAT DOWNTIME AND TIME ALLOCATED FOR PREVENTIVE MAINTENANCE:

WERE NEVER ENVISIONED AS, NOR SHOULD THEY BE CONSIDERED, INTERCHANGEABLE ELEMENTS. THE 90 PERCENT UP TIME (10 PERCENT POSSIBLE DOWNTIME) REQUIREMENT IS ONLY A THRESHOLD FACTOR. SSA DOES NOT PLAN FOR THE OCR SYSTEM TO BE DOWN FOR 10 PERCENT OF THE TIME, IN ADDITION TO PREVENTIVE MAINTENANCE. THE 90 PERCENT CRITERION IS THE LEVEL BELOW WHICH THE CONTRACTOR MUST PAY PENALTIES.

INTERCHANGEABLE OR NOT, III CORRECTLY NOTES THAT DOWNTIME AND TIME FOR PREVENTIVE MAINTENANCE WERE TREATED AS CUMULATIVE. SSA ADDED THEIR SEPARATE CONTRIBUTION TOGETHER, IN EFFECT, BY DEDUCTING BOTH FROM TOTAL AVAILABLE TIME TO COMPUTE MONTHLY OPERATING TIME AND PRODUCTION FIGURES. MOREOVER, AN OFFEROR NATURALLY IS INDUCED TO TRADE PREVENTIVE MAINTENANCE FOR AVAILABILITY TO MAXIMIZE THE CALCULATED COST EFFECTIVENESS OF ITS EQUIPMENT. WE NOTE THAT REI SUBMITTED AN ALTERNATE PROPOSAL, ALSO REJECTED, OFFERING 75 MINUTES DAILY PREVENTIVE MAINTENANCE (DOWN FROM 2 1/2 HOURS) PLUS 6 HOURS OF WEEKLY (WEEKEND) MAINTENANCE.

WE AGREE WITH SSA THAT UNSCHEDULED DOWNTIME AND TIME DURING WHICH THE EQUIPMENT WILL BE SHUT DOWN FOR SCHEDULED SERVICING ARE DIFFERENT. WE SEE NO BASIS EITHER FOR OBJECTING TO SSA'S USE OF A THRESHOLD FIGURE FOR DETERMINING WHEN LIQUIDATED DAMAGES SHOULD APPLY, OR FOR THAT MATTER, TO ITS USE OF A 90 PERCENT FIGURE FOR COMPUTING THROUGHPUT AND EQUIPMENT REQUIREMENTS, PROVIDED THE CONTRACTING ACTIVITY REASONABLY BELIEVES THAT A 10 PERCENT DOWNTIME RATE WILL BE EXPERIENCED.

OUR CONCERN IS THAT SSA DID NOT BELIEVE THAT A 10 PERCENT DOWNTIME FIGURE WILL OCCUR. IN ITS OWN WORDS, IT DID "NOT PLAN FOR THE * * * SYSTEM TO BE DOWN 10 PERCENT OF THE TIME." BASED ON THE EXPERIENCE OF OTHER USERS OF III EQUIPMENT, SSA SEEMS TO ACCEPT A 92 PERCENT RATE AS ACHIEVABLE. SSA, III SAYS, INCLUDED PLANNED DOWNTIME FOR PREVENTIVE MAINTENANCE IN COMPUTING THE 92 PERCENT FIGURE FROM III EXPERIENCE DATA AT OTHER INSTALLATIONS IN EFFECT DISCOUNTING IT TWICE. CORRECTLY CALCULATED, THE DATA SHOWS THAT 97 TO 98 PERCENT IS ACHIEVABLE. THUS, WE THINK THERE IS SOME QUESTION AS TO THE RELIABILITY OF SSA'S COSTING APPROACH AS IT IMPACTS ON THE OVERALL COST EVALUATION.

AT THIS POINT, WE TURN TO EXAMINE THE IMPACT WHICH ERRORS IN SSA'S ADJUSTMENTS COULD HAVE HAD ON ITS EVALUATION OF THE REI AND III PROPOSALS, AND IN THIS LIGHT, TO EXAMINE III'S COMPLAINT THAT SSA'S METHODOLOGY DID NOT IN FACT PROVIDE A VALID ANALYTICAL MEASURE OF THE RELATIVE COST OF ITS AND REI'S PROBABLE LIFE CYCLE COSTS. OUR REVIEW OF THE ACCURACY OF SSA'S METHODOLOGY WILL FOCUS ON III'S 35 MM PURCHASE PROPOSAL, BECAUSE THAT PROPOSAL WAS EVALUATED AS LOWEST IN COST AND BECAUSE OUR ANALYSIS INDICATES THAT NO EVALUATING ERROR MADE BY SSA WOULD PERMIT ANY OTHER III PROPOSAL TO DISPLACE THE 35 MM PURCHASE AS LEAST COSTLY.

SSA DID NOT ATTEMPT TO DIFFERENTIATE THROUGH THE BENCHMARKING BETWEEN THE VENDORS' ABILITY TO READ A PARTICULAR PORTION OF SSA'S PROJECTED WORKLOAD. IF WE UNDERSTAND ITS INTENDED METHODOLOGY CORRECTLY, IT ASSUMED THAT REI AND III WOULD BE ABLE TO PROCESS THE PROJECTED 70 PERCENT WORKLOAD. CONTRARY APPARENTLY TO III'S VIEW OF WHAT SHOULD HAVE BEEN DONE, SSA INSTEAD ATTEMPTED TO COMPUTE THE COST IMPACT WHICH COULD BE EXPECTED DUE TO RANDOM SOURCES OF ERROR, IN EFFECT INTRODUCING A SOURCE OF "NOISE" (RANDOMLY FORMED CHARACTERS) AND MEASURING SYSTEM RESPONSE TO IT.

REI PRODUCED SIGNIFICANTLY HIGHER FIGURES, AND RECEIVED CORRESPONDINGLY GREATER COST PENALTIES. THE DIFFERENCE IN MAGNITUDE OF THE REI AND III'S SCORES, HOWEVER, IS OF LITTLE IMPORTANCE, BECAUSE THE ACCURACY OF SSA'S COST EVALUATION DEPENDS UPON THE PRECISION WITH WHICH THESE PROJECTED FIGURES WERE MEASURED IN CALCULATING THE COST DIFFERENCE.

IT IS DIFFICULT IN THIS REGARD TO UNDERSTAND HOW THE BENCHMARK RESULTS COULD BEAR ANY RATIONAL RELATIONSHIP TO PROJECTED COMPARATIVE COSTS-- AT LEAST TO WITHIN AN ACCURACY OF BETTER THAN $2.55 MILLION, THE ORIGINAL DIFFERENCE SEPARATING THE III AND REI PROPOSALS. OUR CALCULATIONS INDICATE THAT COSTS WERE EXTREMELY SENSITIVE TO NUMERIC BALANCE FIELD SUBSTITUTION ERRORS WHILE THE COST OF PROCESSING REJECTED BALANCE FIELDS OR REINSTATEMENT ITEMS WAS OF COMPARATIVELY NEGLIGIBLE IMPORTANCE. ONE SINGLE NUMERIC (BALANCE FIELD) SUBSTITUTION "ERROR" HAS A $24,000 COST IMPACT. A SINGLE "ERROR" IN SSA'S CONSTRUCTION OF TEST SET REPRODUCED TEN FOLD WOULD HAVE HAD AN IMPACT OF ALMOST A QUARTER OF A MILLION DOLLARS. MINIMUM OF ONE HUNDRED SEVEN INDIVIDUAL SSA CHARACTER "ERRORS" IN A 5,000 PAGE TEST DECK COULD ACCOUNT FOR THE ENTIRE DIFFERENCE SSA CALCULATED BETWEEN REI AND III LIFE-CYCLE COSTS.

BY ATTEMPTING TO USE A SINGLE TEST DECK TO MEASURE THROUGHPUT AND ACCURACY, RATHER THAN MULTIPLE TEST DECKS AND APPROPRIATELY ADJUSTED WEIGHTING FACTORS, SSA HAD TO ASSURE THAT THE DECK WOULD BE MEANINGFUL FOR TWO QUITE DIFFERENT PURPOSES. TO THE EXTENT THAT PROCESSING TIME MIGHT REFLECT THE DIFFICULTY OF THE MATERIAL READ, THE TEST DECK HAD TO BE REPRESENTATIVE OF THE TOTAL PROJECTED ANNUAL REPORTING WORKLOAD, LEAVING ONLY A VERY SMALL PORTION OF THE BENCHMARK TO HAVE ANY EFFECT ON ACCURACY.

SSA'S METHODOLOGY, HOWEVER, INCLUDED VIRTUALLY NO CHECKS IN THE BENCHMARK PROCESS. APPARENTLY, SSA LACKED ANY QUALITY CONTROLS SAVE VISUAL INSPECTION OF PORTIONS OF THE TEST MATERIALS AND THE STANDARDS IMPOSED ON THE OFFSET PRINTING PROCESS. IT WOULD HAVE US BELIEVE THAT IT NEVERTHELESS COULD DISTINGUISH TEST RESULTS TO AN ACCURACY APPROACHING ONE PART IN TEN THOUSAND, THE ACCURACY REQUIRED TO MEASURE THE IMPACT OF BALANCE FIELD SUBSTITUTION ERRORS WITH THE PRECISION NEEDED TO SUPPORT SSA'S COST JUSTIFICATION FOR SELECTING REI. SSA EVALUATED TEST RESULTS BY COMPARING EACH VENDOR'S DATA WITH WHAT SSA "KNEW" HAD BEEN TYPED ON THE ORIGINAL 500 PAGE TEST SET. SO FAR AS WE ARE AWARE, SSA DID NOT ATTEMPT TO MANUALLY KEYBOARD THE TEST DECK CONTENTS TO DETERMINE WHAT IF ANY MINIMUM (RESIDUAL) ERROR RATE IT HAD INTRODUCED. SSA, MOREOVER, FOUND THAT OFFSET PRINTING PRODUCED CHANGES, ELIMINATING FINE LINES.

FORCED BY III TO ADMIT THAT IT DID NOT BUDGET FOR POST-SCANNING PROCESSING COSTS AT RATES DETERMINED BY THE BENCHMARK, SSA ARGUES THAT ITS REPRESENTATION IN III'S PRIOR PROTEST THAT THE BENCHMARK WOULD PRODUCE MEANINGFUL RESULTS WAS NOT MEANT TO CONVEY THE NOTION THAT THERE WOULD BE AN EXACT CORRESPONDENCE BETWEEN BENCHMARK AND ACTUAL PERFORMANCE-- ONLY THAT THE RESULTS WOULD BE REPRESENTATIVE. IT DID ORDER THE QUANTITY OF EQUIPMENT WHICH ITS BENCHMARK BASED THROUGHPUT CALCULATIONS INDICATED WOULD BE NEEDED.

THIS, HOWEVER, AMOUNTS TO APPLYING A DOUBLE STANDARD-- ONE TO COMPARE ACCURACY AND A SECOND FOR CALCULATING EQUIPMENT REQUIREMENTS. MEANINGFUL RESULTS CONSISTENT WITH A LIFE CYCLE COSTING APPROACH ARE OBTAINED REGARDLESS OF THE SCALE CHOSEN TO MEASURE COSTS. SCORING WILL NOT BE MEANINGFUL OR RATIONALLY FOUNDED, HOWEVER, IF DISPROPORTIONATE WEIGHTS ARE ASSESSED DIFFERENT ELEMENTS MAKING UP THE TOTAL. A RATIONAL RELATIONSHIP IS NOT MAINTAINED WHERE ANY ONE SIGNIFICANT CONTRIBUTION TO TOTAL COSTS-- HERE, OUT-OF-POCKET EQUIPMENT COST-- IS KEYED TO A FIXED UNIT OF MEASURE WHILE OTHER EQUALLY SIGNIFICANT COSTS ARE NOT. NOR IS IT ENOUGH THAT SSA BELIEVES THE ACCURACY PORTION OF THE BENCHMARK PRODUCED RELATIVE PERFORMANCE DATA. III WON THAT ROUND, BUT LOST BECAUSE ITS EQUIPMENT WAS MORE EXPENSIVE THAT REI'S. A WEIGHTING FACTOR ERROR OF TWO IN THE ACCURACY, I.E., IF SSA'S TEST SHOULD HAVE BEEN TWICE AS DIFFICULT, MAKES A $12 MILLION DIFFERENCE ($7 MILLION FOR BALANCING ALONE) IN THE PARTIES' RELATIVE STANDING.

FURTHER, WE QUESTION THE ADEQUACY OF SSA'S THROUGHPUT RESULTS. SSA DID NOT USE THE ENTIRE TEST DECK TO DETERMINE III SCANNER THROUGHPUT RATES. EVALUATION WAS BASED ON A 1,000 PAGE MICROFILM SAMPLE FOR THE SINGLE SCANNER SYSTEM. A 4,000 PAGE SAMPLE (DIVIDED INTO FOUR 1,000 PAGE SUBSAMPLES) WAS USED TO TIME THROUGHPUT FOR THE III DUAL SCANNER SYSTEM. III'S SCANNERS READ THE MATERIAL IN 12 (SINGLE SCANNER) TO 18 (DUAL SCANNER) MINUTES PER 1,000 PAGE SUBSAMPLE. USING SSA'S 435.6 USABLE HOURS PER MONTH ESTIMATE, THE 8 YEAR EVALUATION PERIOD CONSISTS OF 1,672,704 USABLE MINUTES-- A MAGNIFICATION OF MORE THAN ONE HUNDRED THOUSAND TIMES THE (15 MINUTES) AVERAGE SCANNER TEST PERIOD. TIMINGS OF THE FOUR SUBSAMPLES INCLUDED IN THE DUAL SCANNER TEST VARIED IN EXTREME BY MORE THAN ONE-HALF PERCENT. MOREOVER, TO TEST A HYPOTHESIS THAT NO DETERIORATION WOULD BE EXPERIENCED, SOME DISK AND CORE WERE REMOVED, AND ONE 16 MM SINGLE SCANNER CONFIGURATION WAS TESTED, TWICE. ODDLY, PERFORMANCE USING LESS EQUIPMENT IMPROVED BY 2.28 PERCENT.

THE EFFECT OF A VARIATION OF BUT A FEW PERCENT CAN HAVE DISPROPORTIONATE IMPACT WHERE AS HERE OFFEROR'S COST PROPOSALS ARE EVALUATED BASED ON AN INTEGRAL NUMBER OF UNITS, BY ROUNDING ANY CALCULATED FRACTIONAL EQUIPMENT TO THE NEXT WHOLE NUMBER. USE OF A QUANTIFIED EVALUATION PROCEDURE CAN, AND INDEED IN THIS CASE DID, SKEW THE RELATIONSHIP BETWEEN A CHANGE IN SCANNER THROUGHPUT RATES AND EQUIPMENT COSTS.

CALCULATING THE POTENTIAL IMPACT OF SCANNING THROUGHPUT ERROR IS SOMEWHAT COMPLEX BECAUSE III WAS FREE TO PROPOSE ANY COMBINATION OF SINGLE OR DUAL SCANNING SYSTEMS IT WISHED, PROVIDED THAT SSA'S PEAK MONTHLY WORKLOAD FIGURES WERE MET.

RAW AND ADJUSTED THROUGHPUT RATES FOR III'S 35 MM EQUIPMENT ARE AS FOLLOWS IN PAGES PER MINUTE: (TABLE OMITTED)

IN THIS CONNECTION, SSA ADJUSTED III'S THROUGHPUT RATE DOWNWARD BY INCLUDING TIME FOR TAPE AND FILM CHANGES AS WELL AS THE THREE PERCENT EDIT PROGRAM DEGRADATION FACTOR. (THE GAO ADJUSTED FIGURE INCLUDES A 30 SECOND TAPE CHANGE FACTOR ONLY ONCE EVERY 175,000 PAGES AND DOES NOT ASSUME THAT THE DUAL SCANNER STOPS EACH TIME A FILM CHANGE IS NEEDED FOR EITHER SCANNER.)

OUR REVIEW INDICATES THAT SSA'S MINIMUM THROUGHPUT REQUIREMENT WOULD HAVE BEEN MET BY A III PROPOSAL OFFERING LESS EQUIPMENT, IF THE CORRECTED SSA THROUGHPUT RATES ARE TOO LOW BY AS LITTLE AS THREE PERCENT. A COMPARABLE INCREASE IN REI'S COMPUTED THROUGHPUT RATES HAS NO SUCH EFFECT, HOWEVER, BECAUSE REI'S EQUIPMENT REQUIREMENTS ARE ALTERED DOWNWARD ONLY BY A 10 PERCENT VARIATION AND THEN ONLY IN YEARS 1 THROUGH 3.

IF, INSTEAD, SSA'S BENCHMARK PROCEDURE PRODUCED FAULTY THROUGHPUT RATES SO THAT THE CALCULATED THROUGHPUT RATES ARE TOO HIGH, THE DIFFERENCE IS MORE EASILY ABSORBED BY THE EQUIPMENT PROPOSED BY III THAN THAT OFFERED BY REI. A DECREASE OF ONLY 1.5 PERCENT IN REI THROUGHPUT WOULD REQUIRE THAT AN ADDITIONAL MACHINE BE ADDED DURING YEARS 4 AND 5 BECAUSE REI MET SSA'S REQUIREMENT BY FURNISHING ONLY 8 UNITS BECAUSE SSA DID NOT REQUIRE THAT IT FURNISHED DOCUMENT NUMBERING DURING THOSE TWO YEARS. (REI IS REQUIRED TO FURNISH DOCUMENT NUMBERING DURING YEARS 1 THROUGH 3 AND AFTER YEAR 5 WHILE III WAS REQUIRED TO PROPOSE MICROFILMING EQUIPMENT WITH NUMBERING THROUGHPUT THE EVALUATION PERIOD. EVIDENTLY, SSA VIEWED AUTOMATIC NUMBERING AS MERELY A MATTER OF CONVENIENCE.)

C. INCIDENTAL ISSUES

THE PROTESTER HAS STATED SEVERAL COMPLAINTS WHICH WE CONSIDER TO BE ONLY INCIDENTAL TO OUR DECISION.

FOR EXAMPLE, III COMPLAINS THAT SSA REQUIRED THAT MICROFILMING THROUGHPUT RATES BE COMPUTED USING A SO-CALLED "DOCUMENT FLIPPER" BECAUSE THE III TDC (PLANETARY) CAMERAS OTHERWISE WOULD INVERT THE SEQUENCE IN WHICH THE DOCUMENTS ARE ARRANGED. SSA SAYS IT EXPECTS TO HIRE POORLY QUALIFIED TEMPORARY PERSONNEL TO RETRIEVE AND PROCESS DOCUMENTS FOR CORRECTION DURING REINSTATEMENT PROCESSING, AND CONSEQUENTLY, NEEDS TO ASSURE THAT A CONSISTENT DOCUMENT SEQUENCE IS MAINTAINED. THE COST OF THE FLIPPER ITSELF IS INSIGNIFICANT. WE FIND THIS ISSUE TO BE INCONSEQUENTIAL.

THROUGHPUT RATES FOR 35 MM MICROFILMING EQUIPMENT WERE DETERMINED BY RUNNING TWO 500 PAGE SAMPLES. TIMES OF 13 MINUTES, 22 SECONDS AND 11 MINUTES, 42 SECONDS WERE PRODUCED FOR THE III PROPOSED 35 MM CAMERA USING THE DOCUMENT FLIPPER. ALTHOUGH EQUIPMENT REQUIREMENTS WERE BASED ON THE TOTAL TIME REQUIRED FOR THE TWO RUNS, THE 6.6 PERCENT DIFFERENCE BETWEEN EACH OF THEM AND THEIR AVERAGE ALONE IS ENOUGH TO ACCOUNT FOR ONE CAMERA OF THE EIGHT SSA REQUIRED THAT III OFFER TO MEET YEAR 1 THROUGH 3 REQUIREMENTS. SSA'S COMPUTATIONS, IN THIS REGARD, SHOWED THAT ITS INITIAL WORKLOAD COULD BE PROCESSED IF 7.3 CAMERAS, AND SUPPORTING STAFF AND FACILITIES, WERE PROVIDED. THIS 7.3 CAMERA FIGURE, HOWEVER, INCLUDES A FIVE PERCENT DEGRADATION SSA ESTIMATED WOULD RESULT FROM USE OF THE DOCUMENT FLIPPER. ADJUSTING SSA'S FIGURES TO ELIMINATE THIS FACTOR, BUT CONSIDERING THE 6.6 PERCENT SPREAD, SHOWS A REQUIREMENT FOR BETWEEN 6.5 AND 7.4 CAMERAS WITH SLIGHTLY LESS THAN 7 CAMERAS THE MOST LIKELY ESTIMATE. THE YEARS 4 AND 5 FIGURES WOULD BE 9.8 AND 11.4 WITH 10.5 CAMERAS MOST LIKELY. THE EFFECT OF A ONE CAMERA SAVINGS FOR YEARS 1 THROUGH 3 THUS HAS A LIMITED EFFECT ON SSA'S CALCULATION OF III'S EVALUATED COST REDUCING THE DIFFERENCE IN III AND REI PRICING BY LESS THAN $100,000, TO ALMOST INCONSEQUENTIAL PROPORTIONS.

III FURTHER COMPLAINS OF SSA'S REFUSAL TO CONSIDER ITS SO-CALLED "FULL SERVICE PROPOSAL" IN WHICH III OFFERED TO ESTABLISH A COCO FACILITY TO PROCESS ALL SSA MACHINE READABLE FORMS. CONCEDING EVIDENTLY THAT THE FULL SERVICE PROPOSAL COULD NOT BE CONSIDERED UNDER THE RFP, III STATES THAT IT WAS OFFERED AS AN UNSOLICITED PROPOSAL AND THAT SSA'S FAILURE TO CONSIDER IT "IS SIMPLY ANOTHER PIECE OF EVIDENCE THAT SSA'S PURPOSE WAS TO OBTAIN FAMILIAR TECHNOLOGY, NOT TO SAVE COST." ALTHOUGH SSA TENDERS A MULTITUDE OF REASONS SUPPORTING ITS DECISION TO REJECT THIS PROPOSAL, IT IS ENOUGH TO POINT OUT THAT SSA COULD NOT ACCEPT IT BECAUSE IT WAS CLEARLY OUTSIDE THE SCOPE OF THE PROCUREMENT AND BECAUSE THE FEDERAL PROCUREMENT REGULATIONS (FPR) DO NOT ANTICIPATE ACCEPTANCE OF AN UNSOLICITED PROPOSAL TO FURNISH SUPPLIES OR SERVICES WHICH WOULD NORMALLY BE PROCURED BY COMPETITIVE METHODS. FPR SEC. 1-4.901. TO HAVE ACCEPTED THIS PROPOSAL WOULD HAVE AMOUNTED TO A SOLE-SOURCE AWARD WITHOUT JUSTIFICATION.

III ALSO BELIEVES SSA COUNTED EXISTING SSA-OWNED OR LEASED REI EQUIPMENT IN COMPUTING REI EQUIPMENT REQUIREMENTS, BUT NOT IN DETERMINING HOW MUCH EQUIPMENT III WOULD HAVE TO FURNISH. NOTHING IN THE RECORD SUPPORTS III'S CONCERN. AT THE OUTSET OF THE PROCUREMENT SSA OWNED ONE REI SCANNER AND LEASED ANOTHER, USING THEM TO PROCESS QUARTERLY REPORT REQUIREMENTS. THE SECOND MACHINE WAS LEASED IN PART TO MEET EXCESS REQUIREMENTS UNTIL THE CURRENT PROCUREMENT WAS COMPLETED. THE LEASE SINCE HAS BEEN TERMINATED, WHILE THE FIRST REI MACHINE CONTINUES TO BE USED TO MEET OTHER SSA SCANNING REQUIREMENTS. THERE WOULD HAVE BEEN NO PROPER BASIS FOR III TO HAVE OBJECTED, HOWEVER, IF SSA HAD ADVISED OFFERORS THAT EXISTING SURPLUS CAPACITY WOULD BE CONSIDERED OR IF SSA WERE TO USE ANY SURPLUS CAPACITY FOR ANNUAL REPORTING PURPOSES.

FINALLY, III RAISES VARIOUS QUESTIONS REGARDING ACTUAL PERFORMANCE UNDER THE REI CONTRACT. THE POSSIBILITY, OF COURSE, THAT A CONTRACT DOES NOT WORK OUT AS EXPECTED IS NOT MATERIAL TO A BID PROTEST, BECAUSE THE REASONABLENESS OF THE ASSUMPTIONS MADE DURING THE PROCUREMENT PROCESS MUST BE JUDGED BY EXAMINING THE CIRCUMSTANCES AS THEY WERE THEN BELIEVED TO EXIST. PERFORMANCE, MOREOVER, IS ORDINARILY A MATTER OF CONTRACT ADMINISTRATION WHICH IS NOT CONSIDERED BY THIS OFFICE.

D. CONCLUSION AND RECOMMENDATIONS

ALTHOUGH WE BELIEVE THE BENCHMARK METHODOLOGY USED MAY HAVE PROVIDED A REASONABLE BASIS FOR DETERMINING THE COMPETITIVE RANGE, WE AGREE WITH III THAT THE VALIDITY OF THE BENCHMARK AS AN ANALYTICAL TOOL FOR DISTINGUISHING BETWEEN ITS AND REI'S PROPOSALS IS QUESTIONABLE. GREAT PRECISION IS REQUIRED OF THE ACCURACY PORTION OF THE BENCHMARK IF IT IS TO SERVE AS A BASIS FOR DRAWING A RATIONAL DISTINCTION BETWEEN THE III AND REI PROPOSALS BASED ON DIFFERENCES IN INDIRECT COSTS. MOREOVER, BECAUSE INDIRECT COSTS WERE ADDED TO DIRECT (EQUIPMENT AND EQUIPMENT RELATED MANPOWER, SPACE AND FACILITIES) COSTS, IT WOULD NOT HAVE BEEN SUFFICIENT HAD SSA MANAGED, AS IT BELIEVES IT DID, TO MEASURE A "RELATIVE DIFFERENCE" IN III AND REI PERFORMANCE. THE TWO TYPES OF COSTS MUST BE MEASURED USING A STANDARD WHICH PERMITS THEM TO BE COMPARED. SSA'S USE OF VALIDATION METHODS-- SPECIFICATION OF OFF-SET PRINTING STANDARDS COUPLED WITH VISUAL INSPECTION OF BUT A FRACTION OF THE TEST MATERIALS-- APPEAR ON THEIR FACE INADEQUATE TO ASSURE THAT ANYTHING CLOSE TO THE NECESSARY DEGREE OF ACCURACY WAS MAINTAINED. ITS EVIDENT LACK OF CONCERN THAT ALL FACTORS (E.G., ACTUAL ANTICIPATED AVAILABILITY AND THE EFFECT OF 200 RATHER THAN 26 TYPE-FONTS, AS DISCUSSED IN THE SSA TECHNICAL EVALUATION) BE CONSIDERED AND ITS BELIEF THAT RELATIVE ACCURACY DATA WAS SUFFICIENT LEAVE US WITHOUT ANY FOUNDATION FROM WHICH TO CONCLUDE THAT THE COMPARATIVELY CLOSE COST DATA COMPUTED FOR III AND REI NECESSARILY REFLECTS A MEASURED DIFFERENCE IN THE LIFE CYCLE COST OF EITHER THE REI OR III SYSTEM. THE RECORD THUS SUGGESTS TO US THAT THE LIFE CYCLE COST EVALUATION WAS INCONCLUSIVE WITH RESPECT TO MEASURING THE COSTS THE GOVERNMENT COULD REASONABLY ASSUME IT WOULD INCUR.

HOWEVER, WE ARE AWARE OF NO EVIDENCE INDICATING SSA ACTED OTHER THAN IN GOOD FAITH, OR THAT SSA WOULD NOT HAVE AWARDED III A CONTRACT HAD III BEEN ABLE TO ESTABLISH THAT ITS APPROACH WOULD BE MORE COST EFFECTIVE THAN REI'S. WE BELIEVE THIS RESULT OCCURRED BECAUSE THE PROCUREMENT WAS STRUCTURED FOR DIRECT PAPER SYSTEMS USING A BENCHMARK THAT WAS CONCEIVED TO DISCRIMINATE BETWEEN RELATIVELY SIMILAR EQUIPMENT. SSA RESTRUCTURED ITS SOLICITATION, ALLOWING MICROFILM BASED PROCESSES, ONCE IT BECAME CLEAR THAT III MIGHT BE ABLE TO COMPETE. THE BENCHMARK WAS ADAPTED TO PERMIT COMPARISON OF MICROFILM AND DIRECT PAPER SYSTEMS, BUT WAS PRESSED BEYOND ITS LIMITS NOT AS A RESULT OF ANY SSA DESIRE TO PRECLUDE III BUT BECAUSE III'S PERFORMANCE AND HIGHER EQUIPMENT COSTS FOCUSED THE COMPETITION ALONG LINES WHICH THE BENCHMARK METHODOLOGY WAS INCAPABLE OF HANDLING.

IN VIEW OF THE UNCERTAINTY AND DIFFICULTY SSA FACED, AND THE INCONCLUSIVE NATURE OF THE EVALUATION SCORES IT COMPUTED, WE BELIEVE IT WOULD BE APPROPRIATE FOR SSA TO VALIDATE ITS INITIAL PROCUREMENT DECISION. THUS, WE BELIEVE SSA SHOULD CONDUCT A MARKET SURVEY TO DETERMINE WHETHER CONTINUED RELIANCE ON REI EQUIPMENT ACTUALLY SERVES THE GOVERNMENT'S BEST INTEREST, PROVIDING IN CONNECTION WITH THE SURVEY AS OPPORTUNITY FOR FURTHER TESTING, USING STATISTICALLY REPRESENTATIVE SAMPLES SELECTED FROM ACTUAL ANNUAL REPORTING DATA. IN THIS CONNECTION WE NOTE THAT WHILE SSA PROFESSES TO BE SATISFIED WITH THE REI EQUIPMENT SSA'S ANTICIPATED WORKING ENVIRONMENT HAS CHANGED SINCE THE PROCUREMENT WAS CONDUCTED. FOR EXAMPLE, ALTHOUGH SSA HAD MICROFILMED EARNINGS DOCUMENTS FOR MORE THAN 20 YEARS, IT VIEWED MICROFILMING AS AN UNNECESSARY EXPENSE WITH SCANNING AND AS A UNIQUE COST TO BE CHARGED TO III. SINCE MAKING AWARD, SSA HAS ENTERED INTO AN AGREEMENT WITH THE IRS WHICH REQUIRES THAT SSA MICROFILM ALL FORMS FOR THE IRS. WE ALSO UNDERSTAND THAT ORIGINAL DOCUMENTS ARE BEING RETAINED ON SITE UNTIL THE MICROFILM COPY IS DEVELOPED, ALTERING IN PART THE ASSUMPTIONS ON WHICH SSA STATES ITS APPROACH TO POST-SCANNING PROCESSING WAS BASED. SINCE MICROFILMING COSTS ALONE CONTRIBUTED MORE THAN FIVE MILLION DOLLARS TO THE COST OF THE III PROPOSAL, A QUITE DIFFERENT RESULT MIGHT BE ACHIEVED WERE THE PROCUREMENT CONDUCTED TODAY. MOREOVER, A TEST CONDUCTED TODAY SHOULD SUFFER FEW OF THE DIFFICULTIES SSA FACED, BECAUSE SSA KNOWS WHAT ITS EXPERIENCE HAS BEEN AND HAS ACTUAL FORMS FROM WHICH A STATISTICALLY REPRESENTATIVE WORKLOAD SAMPLE COULD BE SELECTED. A MUCH LARGER SAMPLE COULD BE USED, REDUCING THE STATISTICAL SIGNIFICANCE OF ANY ONE INDIVIDUAL ANOMALY.

BY SEPARATE LETTER TO THE SECRETARY OF HEALTH AND HUMAN SERVICES, WE ARE RECOMMENDING SUCH ACTION BE CONSIDERED BEFORE ANY DECISION IS MADE TO EXERCISE FURTHER REI CONTRACT OPTIONS. SEE B & W STAT LABORATORY, INC., B-195391, MARCH 10, 1980, 80-1 CPD 184.