B-192987(1), AUG 28, 1979

B-192987(1): Aug 28, 1979

Additional Materials:

Contact:

Edda Emmanuelli Perez
(202) 512-2853
EmmanuelliPerezE@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

DIGEST: OTHER VENDORS WERE NOT PREJUDICED UNDER TELEPROCESSING SERVICES SELECTION PROCESS WHERE ONE VENDOR WAS PERMITTED TO CONDUCT SECOND BENCHMARK 10 DAYS AFTER DEADLINE FOR SUCH BENCHMARK HAD EXPIRED SINCE AT THAT TIME ALL OTHER VENDORS HAD SUCCESSFULLY PASSED INITIAL BENCHMARK OR FAILED SECOND BENCHMARK AND RECORD SHOWS THAT VENDOR DID NOT "OPTIMIZE" BENCHMARK PROGRAM DURING PERIOD BETWEEN ITS FIRST AND SECOND BENCHMARK AND DELAY WAS. WE HAVE RECEIVED AND CONSIDERED NUMEROUS SUBMISSIONS FROM TYMSHARE. TYMSHARE AND OTHER COMPANIES HAVE ENTERED INTO MULTIPLE AWARD SCHEDULE CONTRACTS (MASCS) UNDER GSA'S TSP. TSP IS THE MANDATORY MEANS BY WHICH FEDERAL AGENCIES ACQUIRE TELEPROCESSING SERVICES FROM THE PRIVATE SECTOR.

B-192987(1), AUG 28, 1979

DIGEST: OTHER VENDORS WERE NOT PREJUDICED UNDER TELEPROCESSING SERVICES SELECTION PROCESS WHERE ONE VENDOR WAS PERMITTED TO CONDUCT SECOND BENCHMARK 10 DAYS AFTER DEADLINE FOR SUCH BENCHMARK HAD EXPIRED SINCE AT THAT TIME ALL OTHER VENDORS HAD SUCCESSFULLY PASSED INITIAL BENCHMARK OR FAILED SECOND BENCHMARK AND RECORD SHOWS THAT VENDOR DID NOT "OPTIMIZE" BENCHMARK PROGRAM DURING PERIOD BETWEEN ITS FIRST AND SECOND BENCHMARK AND DELAY WAS, IN PART, FAULT OF AGENCY.

TYMSHARE, INC.:

TYMSHARE, INC. (TYMSHARE) PROTESTS FURTHER CONSIDERATION AND EVALUATION OF CONTROL DATA CORPORATION (CDC) BY THE UNITED STATES COAST GUARD IN THE SELECTION OF A VENDOR FOR TELEPROCESSING SERVICES UNDER THE GENERAL SERVICES ADMINISTRATION (GSA) TELEPROCESSING SERVICES PROGRAM (TSP). TYMSHARE BASES ITS PROTEST ON THE COAST GUARD'S DECISION TO PERMIT CDC TO PERFORM A SECOND BENCHMARK AFTER THE DEADLINE FOR SUCH TESTS HAD EXPIRED. WE HAVE RECEIVED AND CONSIDERED NUMEROUS SUBMISSIONS FROM TYMSHARE, CDC, THE COAST GUARD AND GSA IN CONNECTION WITH THIS MATTER.

TYMSHARE AND OTHER COMPANIES HAVE ENTERED INTO MULTIPLE AWARD SCHEDULE CONTRACTS (MASCS) UNDER GSA'S TSP. AS PROVIDED IN FEDERAL PROPERTY MANAGEMENT REGULATIONS, TEMPORARY REGULATION E-47, AUGUST 3, 1976, AS AMENDED, TSP IS THE MANDATORY MEANS BY WHICH FEDERAL AGENCIES ACQUIRE TELEPROCESSING SERVICES FROM THE PRIVATE SECTOR.

THE PROCEDURE TO BE FOLLOWED BY EACH AGENCY FOR SELECTING A VENDOR FOR THE TELEPROCESSING SERVICES IS SET FORTH IN DETAIL IN THE MASC. UNDER PARAGRAPH D.9 (BASIS FOR USER SOURCE SELECTION), SELECTION OF A SOURCE IS PRIMARILY BASED ON A DETERMINATION BY THE REQUIRING ACTIVITY OF LOWEST SYSTEM LIFE COST. PARAGRAPH D.10 (USER S-U-CE SELECTION CONSIDERATIONS) PROVIDES FOR THE RUNNING OF ANY NECESSARY BENCHMARK PROGRAM OR SERIES OF PROGRAMS TO DETERMINE WHICH OF THE TELEPROCESSING SCHEDULE CONTRACTORS' SERVICES MEET THE USER'S NEEDS AT THE LOWEST OVERALL COST. THE MASC ALSO REQUIRES THAT POTENTIAL SUBSCRIBERS GIVE EACH CONTRACTOR AT LEAST 20 DAYS WRITTEN NOTICE OF THEIR INTENT TO CONDUCT BENCHMARK TESTS.

PROGRAMS USED IN THE BENCHMARK TESTS ARE TO BE THOSE SELECTED BY THE PROSPECTIVE SUBSCRIBER AS TYPIFYING ITS PLANNED APPLICATION. WHEN APPROVED BY THE PROSPECTIVE SUBSCRIBER, THE CONTRACTOR IS PERMITTED TO MAKE CHANGES IN THE BENCHMARK PROGRAMS. THE CONTRACTOR MUST PROVIDE DOCUMENTATION TO AGENCY EVALUATORS OF ANY CHANGES MADE IN THE PROGRAMS, JOB CONTROL LANGUAGE, AND SIMILAR MATTERS.

THE TELEPROCESSING SERVICES INVOLVED IN THE PRESENT PROCUREMENT ARE FOR THE COAST GUARD'S MARINE SAFETY INFORMATION SYSTEM (MSIS). ON MAY 18, 1978, THE COAST GUARD SOLICITED TYMSHARE AND OTHER SCHEDULE CONTRACTORS TO PARTICIPATE IN THE PROCUREMENT BY SUBMITTING THEIR CURRENT TSP/MASC FOR TECHNICAL EVALUATION. THE NOTICE SET FORTH THE AGENCY'S TECHNICAL REQUIREMENTS, A DESCRIPTION OF THE PROPOSED BENCHMARK AND COST EVALUATION PROCEDURES, AND A "TIMETABLE" OF "MILESTONES" WITH SPECIFIC "CUT-OFF DATES."

THE NOTICE REQUIRED THAT THE AWARD BE MADE TO THE SCHEDULE CONTRACTOR WITH THE "LOWEST TOTAL COST." VENDORS FOUND TO MEET CERTAIN TECHNICAL REQUIREMENTS AFTER EVALUATION OF THEIR CURRENT TSP/MASC WERE REQUIRED TO PROCESS A BENCHMARK PROGRAM AS PART OF THE SELECTION AND EVALUATION PROCEDURES TO DETERMINE THAT CONTRACTOR WHICH WILL SATISFY THE COAST GUARD'S NEEDS AT THE LOWEST COST. THE NOTICE PROVIDED THAT VENDORS WOULD BE GIVEN A SECOND OPPORTUNITY TO BENCHMARK IF THEY FAILED THE INITIAL RUN. NO DEFINITION OF A "FAILURE" WAS SET FORTH. (THE COAST GUARD MAINTAINS THAT THE DETERMINATION OF BENCHMARK SUCCESS OR FAILURE WAS SOLELY ITS RESPONSIBILITY.) THE NOTICE, AS REVISED ON JULY 13, 1978, ESTABLISHED AUGUST 3 THROUGH 14 FOR CONDUCTING THE INITIAL BENCHMARKS AND AUGUST 15 THROUGH 18 FOR ANY BENCHMARK RETRIALS.

THE NOTICE LISTED SEVEN CATEGORIES OF COSTS FOR EVALUATION PURPOSES. TWO COST CATEGORIES - STORAGE AND CONNECT TIME - WERE DERIVED FROM THE COST SCHEDULE IN EACH VENDOR'S MASC. FOUR OTHER COST CATEGORIES - TRAINING, MISCELLANEOUS, DATA COMMUNICATIONS, AND DESIRABLE TECHNICAL COST ASSESSMENTS - WERE EVALUATED IN ACCORDANCE WITH CERTAIN CRITERIA SET FORTH IN THE NOTICE. THE REMAINING AND POTENTIALLY SINGLE LARGEST ELEMENT OF COST WAS THE COMPUTER INPUT/OUTPUT RESOURCES CATEGORY. THE NOTICE PROVIDED THAT THESE RESOURCE USAGE COSTS WOULD BE DETERMINED FROM EACH VENDOR'S BENCHMARK PERFORMANCE:

"EVALUATION OF PARTICIPANT'S RESOURCE USAGE AND COST WILL BE BASED ON ACTUAL DATA OBTAINED FROM THE BENCHMARKS. COST EVALUATION WILL BE MADE ON TWO PREPARATORY TASKS AND SIX TERMINAL SESSIONS, FOR A TOTAL OF EIGHT COST EVALUATION COMPONENTS."

SPECIFICALLY, THE COST EVALUATION FORMULA PROVIDED THAT EACH VENDOR'S COMPUTER INPUT/OUTPUT RESOURCE COST BE DETERMINED BY COMPUTING THE NUMBER OF SYSTEM RESOURCE UNITS (SRUS) WHICH THE VENDOR REQUIRED FOR PARTICULAR FUNCTIONS. THE RAW NUMBER OF SRUS CONSUMED IN EACH FUNCTION WAS MEASURED DURING THE BENCHMARK TESTS. THE NOTICE THEN REQUIRED VENDORS TO ADJUST THEIR RAW SRU SCORES BY VARIOUS UNIFORM FACTORS CONTAINED IN THE NOTICE. ONCE ADJUSTED, THE RESULTING NUMBER OF SRUS WERE MULTIPLIED BY THE VENDOR'S COST PER SRU. THUS, THE COST EVALUATION FORMULA CONSISTS OF THE EVALUATED COST OF PERFORMING THE VARIOUS FUNCTIONS WHICH IS BASED ON THE NUMBER OF SRUS CONSUMED BY EACH FUNCTION DURING THE VENDOR'S BENCHMARK PERFORMANCE. DEPENDING UPON THE FREQUENCY A SPECIFIC FUNCTION IS PERFORMED, EVEN IF ONLY ONE SRU IS ELIMINATED IN PERFORMING A TASK WHICH IS A HEAVILY WEIGHTED COST EVALUATION FACTOR, IT IS POSSIBLE FOR A VENDOR TO SIGNIFICANTLY REDUCE ITS COST EVALUATION.

THUS, IN THIS INSTANCE AND UNDER TSP/MASC GENERALLY, THE BENCHMARK SERVES NOT ONLY TO DETERMINE TECHNICAL ACCEPTABILITY, BUT ALSO TO COMPUTE A SIGNIFICANT ELEMENT OF EACH VENDOR'S EVALUATED COSTS. EACH VENDOR'S PERFORMANCE DURING A BENCHMARK DIRECTLY AFFECTS ITS

STS, AND THEREFORE WHETHER IT IS ENTITLED TO SELECTION AS THE SCHEDULE CONTRACTOR OFFERING THE LOWEST TOTAL SYSTEM COST.

IN LATE JUNE OR EARLY JULY 1978, THE VENDORS WERE PROVIDED THE MSIS BENCHMARK TAPE BY THE COAST GUARD WHICH CONTAINED THE JOB CONTROL LANGUAGE, PROGRAMS, TABLES AND DATA BASES FOR THE BENCHMARK TEST. HOWEVER, THE VENDORS WERE NOT TOLD THE PARTICULAR TASKS THAT THEY WOULD BE REQUIRED TO PERFORM UNTIL THE INITIAL BENCHMARKS WERE CONDUCTED. THE MSIS WAS DEVELOPED ON IBM 370 SERIES EQUIPMENT. SINCE MSIS COULD BE PROGRAMMED TO FUNCTION ON COMPARABLE HARDWARE SYSTEMS, HOWEVER, PROSPECTIVE CONTRACTORS WERE NOT LIMITED TO OFFERING IBM EQUIPMENT. CDC BASED ITS OFFER ON THE USE OF ITS HARDWARE SYSTEM, THE CYBER 170 SERIES, AND WAS THE ONLY VENDOR TO HAVE DONE SO.

THE NOTICES' RULES CONCERNING THE BENCHMARK PROGRAM PROVIDED AS FOLLOWS:

"PARTICIPANTS ARE REQUESTED TO MAKE ONLY MACHINE DEPENDENT CHANGES; NO OPTIMIZATION IS ALLOWED."

VENDORS WERE ALSO REQUIRED TO PREPARE A LISTING AND EXPLANATION OF ALL CHANGES MADE TO THE BENCHMARK PROGRAM FOR COAST GUARD ANALYSIS.

MACHINE DEPENDENT CHANGES TO A BENCHMARK PROGRAM ARE THOSE CHANGES THAT ARE NECESSARY TO RUN THE BENCHMARK ON A VENDOR'S PROPOSED EQUIPMENT. DOES NOT INCLUDE CHANGES IN THE PROGRAM LOGIC, OR METHOD OF PROCESSING FUNCTIONS, WHICH ARE MADE SOLELY TO IMPROVE OR OPTIMIZE THE PROGRAM'S PERFORMANCE. ANY SUCH "OPTIMIZATION" (NONMACHINE DEPENDENT CHANGES) SERVES ONLY TO "ARTIFICALLY" REDUCE THE NUMBER OF SRUS CONSUMED DURING THE PERFORMANCE OF THE BENCHMARK TASKS, AND CORRESPONDINGLY, "ARTIFICIALLY" REDUCE A VENDOR'S EVALUATED COST RELATIVE TO ITS COMPETITORS. STATED SOMEWHAT DIFFERENTLY, WHERE OPTIMIZATION IS PERFORMED BY A CONTRACTOR, THE IDENTICAL BENCHMARK PROGRAM IS NOT BEING RUN BY ALL VENDORS. STANDARDIZATION OF EVALUATION DOES NOT OCCUR AND VENDORS ESSENTIALLY ARE NOT COMPETING ON AN EQUAL BASIS FOR BENCHMARK EVALUATION PURPOSES. WHERE IDENTICAL HARDWARE CONFIGURATIONS ARE PROPOSED BY ALL VENDORS, NO MACHINE DEPENDENT CHANGES ARE REQUIRED, ASSUMING THE BENCHMARK TAPE WAS DEVELOPED FOR THAT PARTICULAR HARDWARE CONFIGURATION. THE GREATER THE DIFFERENCES IN A VENDOR'S PROPOSED EQUIPMENT FROM THE SPECIFIC CONFIGURATION-DEVELOPED BENCHMARK PROGRAM, THE GREATER THE LIKELIHOOD AND MAGNITUDE OF MACHINE DEPENDENT CHANGES REQUIRED.

FOR EXAMPLE, THERE ARE SEVERAL DIFFERENT MODELS EVEN WITHIN THE IBM 370 SERIES. DIFFERENCES IN TYPES OF PROCESSORS, DIFFERENT MAKES AND NUMBER OF TAPE UNITS EMPLOYED, OR DIFFERENCES IN CAPACITY OF DISC DRIVES USED, (NON- IDENTICAL HARDWARE CONFIGURATION WITHIN THE IBM 370 SERIES) CAN REQUIRE MACHINE DEPENDENT CHANGES TO PERFORM A GIVEN BENCHMARK PROGRAM, ALBEIT TO A SUBSTANTIALLY LESSER DEGREE THAN A TOTALLY DIFFERENT HARDWARE SYSTEM, I.E. NON-IBM EQUIPMENT SUCH AS THE CYBER 170 OFFERED BY CDC.

FURTHER, THERE IS NO ABSOLUTE AND PRECISE DEFINITION OF MACHINE DEPENDENT CHANGES. CATEGORIZING INDIVIDUAL CHANGES AS MACHINE OR NONMACHINE DEPENDENT INVOLVES JUDGMENTAL DETERMINATIONS. GSA, IN ITS REPORT TO OUR OFFICE, STATES THAT IN MAKING THESE JUDGMENTAL DECISIONS OF MACHINE AND NONMACHINE DEPENDENCY, THERE IS A "GRAY AREA INCAPABLE OF AN ALL- ENCOMPASSING DEFINITION," I.E., IT IS ESSENTIALLY A MATTER OF JUDGMENT.

AS STATED PREVIOUSLY, THE NOTICE ESTABLISHED A COMPLETION DATE OF AUGUST 14 FOR INITIAL BENCHMARKS AND AUGUST 18 FOR ANY BENCHMARK RETRIALS. THE THREE VENDORS TO ULTIMATELY "PASS" THE BENCHMARK, CDC WAS THE ONLY ONE TO HAVE FAILED THE INITIAL BENCHMARK. IT WAS THEREFORE THE ONLY VENDOR TO BE ALLOWED TO PERFORM A SECOND BENCHMARK ON AUGUST 28, TWO WEEKS AFTER ITS INITIAL FAILURE AND TEN DAYS AFTER THE COMPLETION DATE ESTABLISHED BY THE NOTICE FOR BENCHMARK RETRIALS. THE INITIAL FAILURE WAS BASED ON A DETERMINATION BY THE COAST GUARD THAT PROHIBITED NONMACHINE DEPENDENT CHANGES HAD BEEN INSERTED INTO THE BENCHMARK PROGRAM BY CDC. THE CONTRACTING OFFICER STATES AS FOLLOWS:

"CONTROL DATA CORPORATION WAS FIRST BENCHMARKED ON MONDAY, 14 AUGUST 1978. PENDING A DETAILED ANALYSIS OF THE BENCHMARK PROGRAM LISTINGS BY INTERNATIONAL BUSINESS SERVICES (IBS), (CONSULTANTS HIRED BY THE COAST GUARD TO EVALUATE BENCHMARK RESULTS) CONTROL DATA APPEARED TO THE COAST GUARD AND CDC TO HAVE PERFORMED THE BENCHMARK SUCCESSFULLY. IBS NOTIFIED THE COAST GUARD ON MONDAY, 21 AUGUST 1978, THAT THEIR ANALYSIS SHOWED NONMACHINE DEPENDENT CHANGES HAD BEEN MADE TO THE PROGRAMS. CONTROL DATA WAS THEN ADVISED VERBALLY BY PHONE ON 21 AUGUST 1978, AND FORMALLY BY LETTER ON 22 AUGUST 1978, THAT THEY HAD FAILED THE 14 AUGUST 1978 BENCHMARK AND WOULD BE REQUIRED TO RERUN THE BENCHMARK ON MONDAY, 28 AUGUST 1978.

"PRIOR TO CONDUCTING THE FIRST MSIS BENCHMARK TRIAL, CONTROL DATA CORP. PROVIDED THE COAST GUARD WITH THE REQUIRED PROGRAM LISTINGS, OPERATING INSTRUCTIONS, REVISED MSO USER'S GUIDE, AND A LIST OF MACHINE DEPENDENT CHANGES ON 14 AUGUST 1978. ALL CHANGES, ALTHOUGH NOT IN ALL CASES LITERALLY IDENTIFIED IN THE LIST OF CHANGES, WERE DESCRIBED IN A FUNCTIONAL SENSE. CHANGES THAT WERE NOT LISTED WERE IDENTIFIED AFTER OUR REVIEW OF THE BENCHMARK DOCUMENTATION. COMPLETE DISCOVERY AND ANALYSIS OF ALL CHANGES AND COAST GUARD DETERMINATION THAT UNAUTHORIZED NONMACHINE DEPENDENT CHANGES WERE MADE TO THE PROGRAMS WAS NOT ACCOMPLISHED UNTIL AFTER THE 18 AUGUST 1978 CUT-OFF DATE."

TYMSHARE PROTESTS THE COAST GUARD'S DECISION TO ALLOW CDC TO PERFORM AN UNTIMELY SECOND BENCHMARK AFTER THE DEADLINE FOR SUCH TESTS SET FORTH IN THE NOTICE HAD EXPIRED, WITHOUT HAVING AMENDED THE NOTICE. TYMSHARE CHARACTERIZES THE COAST GUARD'S DECISION AS A "SELECTIVE WAIVER OF SOLICITATION REQUIREMENTS FOR THE SOLE BENEFIT OF CDC." IN THIS CONNECTION, TYMSHARE CONTRASTS THE COAST GUARD'S REFUSAL TO CONSIDER FOR EVALUATION PURPOSES A TECHNICAL AMENDMENT TO TYMSHARE'S MASC WHICH SIGNIFICANTLY REDUCED ITS COST OF SERVING TWO LOCATIONS BECAUSE THE AMENDMENT WAS SUBMITTED AFTER THE NOTICE'S DEADLINE FOR PROPOSING TECHNICAL AMENDMENTS. (THE NOTICE EXPRESSLY INFORMED OFFERORS THAT TECHNICAL AMENDMENTS TO THEIR MASC MUST BE PROPOSED TO GSA BY JUNE 19 "FOR CONSIDERATION IN TECHNICAL EVALUATION.")

THE PROTESTER ARGUES THAT CDC WAS GIVEN A SUBSTANTIAL COMPETITIVE ADVANTAGE OVER ALL OTHER VENDORS SINCE IT WAS THE ONLY VENDOR PERMITTED TO CONDUCT A BENCHMARK AFTER THE DEADLINE FOR RETESTING HAD PASSED. TYMSHARE STATES THAT THE EXTRA TIME GIVEN TO CDC IS PARTICULARLY SIGNIFICANT IN A TSP PROCUREMENT WHERE THE BENCHMARK IS USED NOT ONLY TO DETERMINE WHETHER A VENDOR'S SYSTEM IS TECHNICALLY ACCEPTABLE BUT ALSO TO DETERMINE THE VENDOR'S EVALUATED COST.

THE PROTESTER EXPLAINS THAT A VENDOR CAN REDUCE ITS EVALUATED COSTS IF GIVEN EXTRA TIME TO PREPARE FOR THE BENCHMARK. THE ADDITIONAL TIME PERMITS THE VENDOR TO REDUCE THE NUMBER OF SRUS USED IN EACH FUNCTION AND THEREBY REDUCE ITS EVALUATED COSTS. LIKEWISE, THE VENDOR MAY USE THIS ADDITIONAL TIME TO "FINE TUNE" MACHINE DEPENDENT CHANGES OR MAKE NONMACHINE DEPENDENT CHANGES WHICH ALSO REDUCE EVALUATED COSTS. THE PROTESTER FURTHER NOTES THAT IF, AS IS THE CASE HERE, A VENDOR FAILS ITS FIRST BENCHMARK AND HAS BENCHMARK PROBLEMS AVAILABLE FOR STUDY PRIOR TO THE RETEST, IT CAN MAKE "EVEN MORE SUBSTANTIAL SRU REDUCTIONS." TYMSHARE CONCLUDES "THE RANGE OF WAYS TO REDUCE SRU CONSUMPTION IS LIMITED ONLY BY THE OFFEROR'S EXPERTISE AND AMOUNT OF AVAILABLE TIME."

IN ITS REPORT TO OUR OFFICE, GSA AGREES IN PART WITH TYMSHARE "THAT A VENDOR, ARMED WITH THE DETAILS OF AN INITIAL BENCHMARK AND SUFFICIENT TIME, CAN FIND WAYS OF REDUCING THE SRUS OR OF OPTIMIZING THE BENCHMARK PERFORMANCE TO REDUCE THROUGHPUT." THUS, TYMSHARE ARGUES, AN INITIAL BENCHMARK FAILURE IS REWARDED BY A COMPETITIVE ADVANTAGE POTENTIALLY GIVEN ONLY THAT VENDOR OR VENDORS WHICH ARE AFFORDED THE SECOND BENCHMARK OPPORTUNITY, WHILE THE REMAINING VENDORS ARE "LOCKED IN" TO THE EVALUATED COSTS RESULTING FROM THE INITIAL BENCHMARK CONDUCTED WITHOUT BENEFIT OF DETAILED KNOWLEDGE OF THE BENCHMARK PROBLEMS. TYMSHARE BELIEVES THAT THIS CASE IS DIRECTLY ANALOGOUS TO A LATE PROPOSAL SITUATION AND QUOTES FROM OUR DECISION IN COMPUTER SCIENCES CORPORATION, 57 COMP.GEN. 627 (1978), 78-2 CPD 85 IN WHICH WE STATED ON PAGE 640: "*** THE RATIONALE UNDERLYING STRICT APPLICATION OF THE LATE PROPOSAL (AND LATE BID) RULES IS TO PREVENT EVEN THE SLIGHTEST POSSIBILITY OF ANY OFFEROR GAINING AN UNFAIR COMPETITIVE ADVANTAGE BY BEING ABLE TO MAKE MATERIAL CHANGES IN ITS OFFER AFTER THE CUTOFF DATE AND TIME. ***"

FOR ITS PART, CDC ARGUES THAT IT WAS ENTITLED TO A SECOND BENCHMARK IRRESPECTIVE OF THE CAUSE OF THE INITIAL FAILURE AND CHARACTERIZES THE NOTICE'S PROMISE OF A SECOND BENCHMARK AS AN "UNCONDITIONAL GUARANTEE." FURTHER, CDC ATTRIBUTES THE UNTIMELY SECOND BENCHMARK SOLELY TO COAST GUARD ADMINISTRATIVE DELAYS, POINTING OUT THAT CDC WAS NOT INFORMED OF ITS INITIAL BENCHMARK FAILURE UNTIL AUGUST 21, THREE DAYS AFTER THE COMPLETION DATE ESTABLISHED BY THE NOTICE FOR BENCHMARK RETRIALS. CDC'S DISTRICT MANAGER FOR PROFESSIONAL SERVICES, IN AN AFFIDAVIT FILED WITH OUR OFFICE, STATES AS FOLLOWS:

"THERE ARE MANY DIFFERENCES BETWEEN THE IBM SYSTEM 370 AND THE CDC CYBER 170 SERIES OF COMPUTER SYSTEMS THAT HAD TO BE TAKEN INTO ACCOUNT IN CONVERTING THE IBM-ORIENTED MSIS BENCHMARK TAPE INTO A FORM THAT WOULD BE COMPATIBLE WITH THE CDC EQUIPMENT, INCLUDING NOT ONLY THE DIFFERENCES BETWEEN THE IBM *** 'OPERATING SYSTEM' AND CDC'S 'NETWORK OPERATING SYSTEM.' CDC DID SEEK TO RESOLVE ITS TECHNICAL QUESTIONS RELATING TO THE MSIS. HOWEVER, RESPONSES WERE OFTEN NOT RECEIVED UNTIL A WEEK OR MORE AFTER THE CDC INQUIRY. BECAUSE THE DEADLINE FOR THE INITIAL BENCHMARK WAS APPROACHING, CDC WAS FORCED TO MAKE INDEPENDENT JUDGMENTS AS TO WHETHER THE USCG WOULD ACCEPT CDC'S CONVERSION OF THE SYSTEM ***.

"HAD THERE BEEN NO PROVISION IN THE APPLICABLE PROCUREMENT INSTRUCTIONS FOR A SECOND BENCHMARK, CDC WOULD HAVE APPROACHED THE INITIAL BENCHMARK IN AN ALTOGETHER DIFFERENT FASHION THAN IT DID. CDC WOULD HAVE ENGAGED IN MUCH MORE EXTENSIVE DISCUSSIONS THAN IT DID WITH USCG (COAST GUARD) TECHNICAL REPRESENTATIVES CONCERNING THE TECHNICAL REQUIREMENTS OF THE PROCUREMENT AND WOULD HAVE DETERMINED WITH PRECISION PRIOR TO THE BENCHMARK THOSE CHANGES PROPOSED BY CDC THAT MIGHT HAVE BEEN REGARDED BY THE USCG AS 'NON-MACHINE DEPENDENT' IN NATURE AND THEREFORE IMPERMISSIBLE. HOWEVER, BECAUSE OF THE EXPLICIT PROMISE OF A RETEST IN THE EVENT OF FAILURE, IRRESPECTIVE OF THE BASIS OR BASES THEREFORE, CDC REGARDED EXTENSIVE PREBENCHMARK DISCUSSIONS AS AN UNNECESSARY EXPENDITURE OF TIME AND RESOURCES. INSTEAD, AND BECAUSE THE TIME DELAYS ASSOCIATED WITH A CONTINUAL DIALOGUE WITH THE USCG CONCERNING THE MACHINE DEPENDENCY OF THE MANY CHANGES INVOLVED IN THE CONVERSION EFFORT COULD HAVE JEOPARDIZED THE SCHEDULE FOR INITIAL BENCHMARK, CDC CONVERTED THE MSIS TO CDC'S HARDWARE ON THE BASIS OF INDEPENDENT JUDGMENTAL DETERMINATIONS, ON A CASE BY CASE BASIS, AS TO WHETHER A PARTICULAR CHANGE WAS MACHINE DEPENDENT OR NON-MACHINE DEPENDENT IN NATURE."

CDC STATES THAT THE EFFORT NEEDED TO CONVERT THE MSIS TAPE INVOLVED NUMEROUS JUDGMENTAL DETERMINATIONS AS TO WHETHER A CHANGE WAS MACHINE OR NONMACHINE DEPENDENT AND "IF THESE JUDGMENTS WERE IN ANY WAY ERRONEOUS, CDC SHOULD NOT BE PENALIZED FOR THAT ERROR BECAUSE THE PROCUREMENT PROCEEDED UPON THE ASSUMPTION THAT SUCH ERRORS COULD, OR WOULD OCCUR, AND PROVIDED FOR CORRECTIVE MEASURES." FINALLY, CDC HAS SUBMITTED AFFIDAVITS WHICH STATE THAT IT DID NOT ENGAGE IN ANY OPTIMIZATION DURING THE PERIOD BETWEEN THE FIRST AND SECOND BENCHMARKS AND THAT THE SOLE PURPOSE OF THE SECOND BENCHMARK WAS TO ELIMINATE ANY SYSTEM OPTIMIZATION OR EFFICIENCIES THAT RESULTED FROM "NON-MACHINE DEPENDENT" CHANGES MADE TO THE MSIS TAPE BY CDC DURING THE PROCESS OF ADAPTATION. CDC THEREFORE CONCLUDES THAT SINCE, IN FACT, NO OPTIMIZATION OR FINE TUNING OCCURRED DURING THE TEN-DAY PERIOD BETWEEN THE INITIAL AND SECOND BENCHMARK, TYMSHARE WAS NOT IN ANY WAY PREJUDICED BY THE COAST GUARD'S ALLOWANCE OF A LATE SECOND BENCHMARK.

TYMSHARE ARGUES, HOWEVER, THAT CDC WAS SOLELY RESPONSIBLE FOR THE DELAY IN SCHEDULING THE SECOND BENCHMARK BECAUSE CDC MADE THREE "CRUCIAL" DECISIONS WHICH PRECLUDED THAT FIRM FROM PERFORMING A TIMELY BENCHMARK RETRIAL. FIRST, TYMSHARE ALLEGES, CDC DECIDED NOT TO OBTAIN COMPLETE CLARIFICATION OF THE BENCHMARK RULES BEFORE IT PERFORMED THE FIRST BENCHMARK, I.E., CDC DID NOT REQUEST FROM THE COAST GUARD A COMPREHENSIVE AND PRECISE DETERMINATION AS TO WHAT SYSTEM CHANGES WERE NONMACHINE DEPENDENT - WHICH THE NOTICE PROHIBITED - AND WHAT CHANGES WERE MACHINE DEPENDENT - WHICH THE NOTICE PERMITTED. TYMSHARE POINTS TO A MEETING ON AUGUST 23 BETWEEN CDC AND THE COAST GUARD, AFTER CDC'S INITIAL FAILURE, IN WHICH THE QUESTIONS OF MACHINE AND NONMACHINE DEPENDENCY WERE RESOLVED "IN A SINGLE MEETING." (ACCORDING TO CDC, THE MAJORITY OF CDC CHANGES TO THE MSIS WHICH HAD BEEN INITIALLY DETERMINED BY THE COAST GUARD TO BE NONMACHINE DEPENDENT WERE DETERMINED BY THE COAST GUARD TO BE MACHINE DEPENDENT IN NATURE AT THAT MEETING, INDICATING THAT CDC'S INDEPENDENT DETERMINATIONS OF MACHINE DEPENDENCY WERE BONA FIDE.) SECOND, TYMSHARE ARGUES THAT CDC DID NOT DISCLOSE ALL CHANGES THAT HAD BEEN MADE TO THE BENCHMARK PROGRAM PRIOR TO THE FIRST BENCHMARK IN VIOLATION OF SPECIFIC BENCHMARK RULES. THIRD, TYMSHARE STATES THAT CDC'S SUBMISSIONS TO OUR OFFICE STRONGLY SUGGEST THAT CDC INTENTIONALLY MADE NONMACHINE DEPENDENT CHANGES TO REDUCE ITS COST EVALUATION IN THE FIRST BENCHMARK. ACCORDING TO TYMSHARE:

"THE SOLICITATION CERTAINLY DID NOT PROVIDE A SECOND BENCHMARK SO THAT OFFERORS WOULD HAVE A LICENSE TO MAKE PROHIBITED MANIPULATIONS IN THE FIRST."

FOR THESE REASONS, TYMSHARE CONCLUDES THAT THE COAST GUARD SHOULD ELIMINATE CDC FROM FURTHER CONSIDERATION.

THE RECORD SHOWS THAT CDC'S EVALUATED COST UNDER THE COMPUTER INPUT/OUTPUT RESOURCE CATEGORY INCREASED AS A RESULT OF THE SECOND BENCHMARK BY MORE THAN 13 PERCENT. ITS OVERALL EVALUATED COST INCREASED BY ABOUT FOUR PERCENT.

WE FIND THAT TYMSHARE AND THE OTHER VENDORS WERE NOT PREJUDICED BY THE COAST GUARD'S DECISION TO PERMIT CDC TO PERFORM A SECOND BENCHMARK AFTER THE ESTABLISHED COMPLETION DATE WITHOUT HAVING AMENDED THE NOTICE. FIRST, ALL VENDORS EXCEPT CDC HAD SUCCESSFULLY PASSED THE INITIAL BENCHMARK OR FAILED THE SECOND ATTEMPT BY THE TIME THE NEED FOR A FORMAL AMENDMENT TO THE NOTICE ARGUABLY AROSE. AS CDC POINTS OUT, REQUIRING THE COAST GUARD TO AMEND THE NOTICE UNDER SUCH CIRCUMSTANCES WOULD BE TANTAMOUNT TO REQUIRING A "USELESS ACT", HAVING NO EFFECT ON OTHER PROSPECTIVE CONTRACTORS. THUS, WE BELIEVE THE COAST GUARD'S REFUSAL TO CONSIDER TYMSHARE'S LATE TECHNICAL AMENDMENT TO ITS MASC IS DISTINGUISHABLE SINCE ALL PROSPECTIVE CONTRACTORS WOULD HAVE BEEN POTENTIALLY AFFECTED BY SUCH AGENCY ACTION. SECOND, THE RECORD CLEARLY ESTABLISHES THAT IN FACT CDC DID NOT ENGAGE IN OPTIMIZATION OR OTHER MANIPULATION OF THE BENCHMARK PROGRAMS DURING THE PERIOD BETWEEN ITS FIRST AND SECOND BENCHMARK AND THAT THE DELAY IN NOTIFYING CDC OF ITS FAILURE TO PASS THE FIRST BENCHMARK WAS, AT LEAST IN PART, THE FAULT OF THE COAST GUARD.

HOWEVER, WHILE WE FIND NO BASIS TO SUSTAIN THE PROTEST, WE BELIEVE THAT THE NOTICE AND THE BENCHMARK PROCEDURES EMPLOYED BY THE COAST GUARD SHOULD BE IMPROVED BY IMPOSING ADDITIONAL SAFEGUARDS IN INSTANCES WHERE THE BENCHMARK IS USED TO DETERMINE VENDORS' EVALUATED COSTS. BY LETTER OF TODAY TO THE ADMINISTRATOR OF GENERAL SERVICES, WE ARE SUGGESTING THAT HE CONSIDER REVISIONS TO THE TSP GUIDANCE PROVIDED TO EXECUTIVE AGENCIES.

THE PROTEST IS DENIED.