Skip to main content

B-195982, DECEMBER 12, 1980, 60 COMP.GEN. 113

B-195982 Dec 12, 1980
Jump To:
Skip to Highlights

Highlights

GENERAL ACCOUNTING OFFICE - JURISDICTION - CONTRACTS - BENCHMARK - STANDARD OF REVIEW GENERAL ACCOUNTING OFFICE STANDARD OF REVIEW FOR BENCHMARK IS SAME AS FOR ANY OTHER TECHNICAL EVALUATION PROCEDURE: IF BENCHMARK IS RATIONALLY BASED. ITS USE AS EVALUATION TOOL IS WITHIN DISCRETION OF PROCURING AGENCY. USE OF BENCHMARK AS EVALUATION TOOL IS WITHIN DISCRETION OF PROCURING AGENCY. ARGUES THAT THE PROGRAMS WHICH COMPRISED THE BENCHMARK WHICH ALL OFFERORS WERE REQUIRED TO RUN DID NOT ACCURATELY OR COMPLETELY REPRESENT THE ACTUAL WORK TO BE PERFORMED UNDER THE CONTRACT. THE DETAILS ARE AS FOLLOWS: BACKGROUND THIS PROCUREMENT WAS CONDUCTED BY THE GENERAL SERVICES ADMINISTRATION FOR THE ARMY MILITARY PERSONNEL CENTER IN CONNECTION WITH THE ARMY'S "REQUEST" AND "RETAIN" SYSTEMS.

View Decision

B-195982, DECEMBER 12, 1980, 60 COMP.GEN. 113

CONTRACTS - PROTESTS - PROCEDURES - BID PROTEST PROCEDURES - TIME FOR FILING - CLARIFICATION V. "INITIAL ADVERSE AGENCY" ACTIONS WHEN, AT TIME EXCHANGES OCCURRED, BOTH PROTESTER AND CONTRACTING OFFICER REGARDED SERIES OF LETTERS AND MEETINGS AN OPPORTUNITY TO CLARIFY AGENCY'S REQUIREMENTS, EXCHANGES DO NOT CONSTITUTE PROTEST AND SUBSEQUENT "INITIAL ADVERSE AGENCY ACTION" WHICH WOULD REQUIRE FILING OF PROTEST TO GENERAL ACCOUNTING OFFICE WITHIN 10 DAYS. GENERAL ACCOUNTING OFFICE - JURISDICTION - CONTRACTS - BENCHMARK - STANDARD OF REVIEW GENERAL ACCOUNTING OFFICE STANDARD OF REVIEW FOR BENCHMARK IS SAME AS FOR ANY OTHER TECHNICAL EVALUATION PROCEDURE: IF BENCHMARK IS RATIONALLY BASED, ITS USE AS EVALUATION TOOL IS WITHIN DISCRETION OF PROCURING AGENCY. CONTRACTS - SPECIFICATIONS - TESTS - BENCHMARK - USE AS EVALUATION TOOL - ADMINISTRATIVE DISCRETION WHEN BENCHMARK PROGRAMS APPEAR TO REPRESENT SYSTEM WORKLOAD AND, COMBINED WITH FUNCTIONAL DEMONSTRATION, PROVIDE REASONABLE BASIS FOR IDENTIFYING OFFEROR WITH LOWEST LIFE-CYCLE COST, USE OF BENCHMARK AS EVALUATION TOOL IS WITHIN DISCRETION OF PROCURING AGENCY.

MATTER OF: COMPUTER SCIENCES CORPORATION, DECEMBER 12, 1980:

COMPUTER SCIENCES CORPORATION (CSC) PROTESTS THE AWARD OF A CONTRACT FOR TELEPROCESSING SERVICES TO SUPPORT THE ARMY'S RECRUITING, TRAINING, AND REENLISTMENT PROGRAMS TO BOEING COMPUTER SERVICES COMPANY.

CSC, THE INCUMBENT CONTRACTOR, ARGUES THAT THE PROGRAMS WHICH COMPRISED THE BENCHMARK WHICH ALL OFFERORS WERE REQUIRED TO RUN DID NOT ACCURATELY OR COMPLETELY REPRESENT THE ACTUAL WORK TO BE PERFORMED UNDER THE CONTRACT.

ON ACCOUNTING AND FINANCIAL MANAGEMENT DIVISION HAS PERFORMED A TECHNICAL REVIEW OF THE BENCHMARK. BASED ON THIS REVIEW AND THE RECORD BEFORE US, WE DENY CSC'S PROTEST. THE DETAILS ARE AS FOLLOWS:

BACKGROUND

THIS PROCUREMENT WAS CONDUCTED BY THE GENERAL SERVICES ADMINISTRATION FOR THE ARMY MILITARY PERSONNEL CENTER IN CONNECTION WITH THE ARMY'S "REQUEST" AND "RETAIN" SYSTEMS. THESE SYSTEMS PERMIT RECRUITERS AND CAREER COUNSELORS TO IDENTIFY AND RESERVE TRAINING SPACES FOR NEW RECRUITS AND REENLISTEES, BASED ON THEIR INDIVIDUAL PREFERENCES AND QUALIFICATIONS. ADDITION, ALL ARMY ENLISTED ASSIGNMENTS ARE PROCESSED THROUGH RETAIN.

BOTH REQUEST AND RETAIN WERE INSTALLED DURING THE 1970'S ON CSC'S INFONET SYSTEM, WITH SOFTWARE DEVELOPED BY SYSTEMS AUTOMATION CORPORATION (SAC). THE PROTESTED PROCUREMENT WAS A NEW COMPETITION FOR THE SERVICES WHICH CSC HAS BEEN PROVIDING SINCE THAT TIME.

THE SOLICITATION, NO. CDPA 78-5, WAS PREPARED BY SAC IN CONJUNCTION WITH THE ARMY OVER A PERIOD OF APPROXIMATELY 2 1/2 YEARS, BEGINNING IN MAY 1976. IT INCLUDED A SERIES OF COMPUTER PROGRAMS, DESIGNED TO REPRESENT THE REQUEST AND RETAIN WORKLOAD, WHICH OFFERORS CONVERTED, COMPILED, AND EXECUTED ON THEIR OWN SYSTEMS PRIOR TO SUBMITTING COST AND TECHNICAL PROPOSALS. THOSE WHO FOUND IT NECESSARY TO MAKE CHANGES IN ORDER TO ADAPT THE PROGRAMS, BASED ON THE INFONET SYSTEM, TO OTHER MANUFACTURERS' EQUIPMENT WERE REQUIRED TO SUBMIT ALL SUCH CHANGES TO THE CONTRACTING OFFICER'S TECHNICAL STAFF FOR APPROVAL.

APPROVED OFFERORS THEN COMPLETED A PRE-PROPOSAL BENCHMARK WHICH TESTED THEIR SYSTEMS FOR CERTAIN MANDATORY CAPABILITIES LISTED IN THE SOLICITATION. THEY ALSO RAN THE VARIOUS PROGRAMS 10 TIMES EACH AND AVERAGED THE RESULTS. THESE RESULTS WERE USED TO COMPLETE COST TABLES, CONTAINED IN THE SOLICITATION, WHICH OFFERORS WERE REQUIRED TO SUBMIT TO GSA ALONG WITH DETAILED WRITTEN DESCRIPTIONS OF THEIR EXECUTION OF THE BENCHMARK PROGRAMS AND PRINTOUTS OF THE RESULTS.

AFTER EVALUATING COST AND TECHNICAL PROPOSALS, GSA SCHEDULED A SECOND, GOVERNMENT-WITNESSED BENCHMARK FOR ALL OFFERORS WHO WERE IN THE COMPETITIVE RANGE. ACCORDING TO THE SOLICITATION, THE PRIMARY PURPOSE OF THIS SUPERVISED BENCHMARK WAS TO VALIDATE RESULTS OF THE PRE PROPOSAL RUN. GSA ALSO INDICATED THAT IT INTENDED TO USE IT TO MONITOR COSTS AND PERFORMANCE OF THE SUCCESSFUL CONTRACTOR BY RE-RUNNING THE PROGRAMS AT RANDOM INTERVALS DURING THE LIFE OF THE CONTRACT. IF COSTS WERE MORE THAN FIVE PERCENT OVER THOSE DEVELOPED FROM THE BENCHMARK, THE CONTRACTOR'S MONTHLY INVOICE WAS TO BE ADJUSTED ACCORDING TO A SPECIFIC FORMULA CONTAINED IN THE SOLICITATION.

AWARD WAS TO BE MADE TO THE OFFEROR WHOSE SYSTEM, MEETING ALL MANDATORY REQUIREMENTS, HAD THE LOWEST EVALUATED LIFE-CYCLE COST.

TIMELINESS

A THRESHOLD ISSUE IS WHETHER, AS GSA ARGUES, CSC'S PROTEST IS UNTIMELY. THE AGENCY CONTENDS THAT A SERIES OF LETTERS AND MEETINGS BETWEEN CSC AND THE CONTRACTING OFFICER CONSTITUTED PROTESTS AND THAT "INITIAL ADVERSE AGENCY ACTION" OCCURRED ON OR BEFORE JUNE 14, 1979, WHEN THE CONTRACTING OFFICER INFORMED CSC BY LETTER THAT ITS ASSUMPTIONS REGARDING THE BENCHMARK WERE "ERRONEOUS."

CSC, ON THE OTHER HAND, ASSERTS THAT THESE LETTERS AND MEETINGS WERE PART OF A "CONTINUING PROCESS" BY WHICH IT "ATTEMPTED TO GAIN CLARIFICATION OF THE RFP'S REQUIREMENTS AND GSA'S INTERPRETATION" OF THEM.

OUR BID PROTEST PROCEDURES STATE THAT IF A PROTEST IS FILED WITH THE CONTRACTING AGENCY, ANY SUBSEQUENT PROTEST TO OUR OFFICE MUST BE RECEIVED WITHIN 10 DAYS AFTER THE PROTESTER KNEW OR SHOULD HAVE KNOWN OF "INITIAL ADVERSE AGENCY ACTION." 4 C.F.R. 201(A)(1980). THE CONTRACTING OFFICER IN THIS CASE CHARACTERIZED HIS LETTER OF JUNE 14 AS AN "OPPORTUNITY TO CLARIFY AND CORRECT CSC'S MISCONCEPTIONS," AND IT DOES NOT APPEAR THAN ANY OF THE PARTIES REGARDED IT AS A DENIAL OF A PROTEST AT THAT TIME. THEREFORE DO NOT BELIEVE THIS LETTER CONSTITUTED ADVERSE ACTION BY GSA.

CSC'S PROTEST WAS FILED WITH OUR OFFICE MORE THAN 30 DAYS BEFORE THE AMENDED CLOSING DATE FOR RECEIPT OF INITIAL PROPOSALS, AS REQUIRED BY SECTION 20.2(B) OF OUR PROCEDURES, SUPRA, AS WELL AS BEFORE THE DATE FOR SUBMISSION OF BENCHMARK DATA, AS REQUIRED BY OUR DECISION IN INFORMATION INTERNATIONAL, INC., 59 COMP.GEN. 640(1980), 80-2 CPD 100; ID., SSA-- REQUEST FOR RECONSIDERATION, OCTOBER 7, 1980, 80-2 CPD 246. THEREFORE, EXCEPT FOR AN ISSUE INVOLVING REENTRANT CODE CAPABILITY WHICH WAS NOT RAISED UNTIL AFTER THE CLOSING DATE FOR RECEIPT OF INITIAL PROPOSALS, WE FIND THE PROTEST IS TIMELY.

SUFFICIENCY OF THE BENCHMARK

CSC CONTESTS THE LEGALITY OF THE BENCHMARK AS AN EVALUATION TOOL FROM BOTH A TECHNICAL AND COST STANDPOINT. (THE FIRM HAS STATED THAT A NUMBER OF OTHER GROUNDS OF PROTEST RAISED WITH OUR OFFICE WERE RENDERED "MOOT" BY ACTIONS TAKEN BY GSA.)

A. SIMULTANEOUS ACCESS

SPECIFICALLY, CSC ALLEGES THAT THE BENCHMARK FAILED TO TEST OFFERORS' ABILITY TO HANDLE A LARGE NUMBER OF SIMULTANEOUS USERS. CSC POINTS OUT THAT ACCORDING TO THE SOLICITATION, THE AVERAGE NUMBER OF SIMULTANEOUS USERS OF REQUESTS AND RETAIN WILL VARY FROM 16 TO 80, DEPENDING ON TIME OF DAY, AND MAY BE AS HIGH AS 139. A BENCHMARK FOR A SYSTEM REQUIRING SIMULTANEOUS ACCESS CAPABILITY-- BUT CONCEDEDLY NOT TESTING THAT CAPABILITY-- IS INSUFFICIENT, CSC ARGUES.

ACCORDING TO CSC, THE BENCHMARK COULD NOT PROVIDE AN ACCURATE PICTURE OF A SYSTEM'S "LEVEL OF RESOURCE CONSUMPTION." IN MOST SYSTEMS, CSC CONTENDS, MORE COMPUTER RESOURCES MAY BE CONSUMED AS MORE SIMULTANEOUS USERS COME "ON LINE." ALTHOUGH SUCH IS NOT THE CASE WITH INFONET, CSC STATES, THE ARMY HAS NO WAY OF VERIFYING THIS FROM THE BENCHMARK.

B. FILE ACCESS METHOD

CSC FURTHER COMPLAINS THAT THE BENCHMARK DID NOT REQUIRE OFFERORS TO EXERCISE THEIR "RANDOM ACCESS CAPABILITY," ALTHOUGH THIS IS A REQUIREMENT OF REQUEST AND RETAIN. (RANDOM OR DIRECT ACCESS INDICATES THAT A RECORD CAN BE ENTERED INTO OR OBTAINED FROM THE FILE IN A MANNER WHICH DEPENDS ONLY ON THE LOCATION OF THAT PARTICULAR RECORD, AND NOT ON THE LOCATION OF ALL PREVIOUSLY ENTERED RECORDS.) IF RANDOM ACCESS FOR MULTIPLE SIMULTANEOUS USERS IS NOT PROVIDED, CSC STATES, A SYSTEM'S PERFORMANCE WILL DEGRADE AND COSTS WILL INCREASE AS THE NUMBER OF USERS INCREASES (PRESUMABLY BECAUSE IN SEQUENTIAL ACCESS, MORE RECORDS MUST BE READ, TAKING MORE TIME AND USING MORE COMPUTER RESOURCES).

C. PERFORMANCE DEMONSTRATIONS

CSC ALSO CRITICIZES THE "FUNCTIONAL" OR "PERFORMANCE" DEMONSTRATIONS WHICH GSA USED TO TEST SYSTEM CAPABILITIES NOT INCLUDED IN THE BENCHMARK-- SUCH AS ABILITY TO SUPPORT MULTIPLE SIMULTANEOUS USERS AND METHOD OF FILE ACCESS-- BECAUSE THE DEMONSTRATIONS (1) DID NOT CONSIDER COSTS AND (2) INVOLVED ONLY TWO USERS. CSC CONTENDS THAT AN OFFEROR COULD "PASS" THE DEMONSTRATIONS AND THE GOVERNMENT WOULD STILL HAVE NO IDEA OF WHAT ACTUAL COSTS OF OPERATING THE SYSTEM WOULD BE.

GSA'S RESPONSE

GSA RESPONDS THAT THE BENCHMARK, COMBINED WITH THE PERFORMANCE DEMONSTRATIONS, PROVIDED AN ACCEPTABLE MEANS OF EVALUATING TECHNICAL PROPOSALS AND ESTIMATING LIFE-CYCLE COSTS.

THE ONLY ALTERNATIVE MEANS OF TESTING OFFERORS' ABILITY TO HANDLE MULTIPLE SIMULTANEOUS USERS, GSA STATES, WOULD HAVE BEEN "LOAD" OR "STRESS" TESTING. THIS TYPE OF TESTING REQUIRES EITHER USE OF REMOTE TERMINAL EMULATION (RTE) (A NEW TECHNIQUE IN WHICH A MICROCOMPUTER SIMULATES A LARGE NUMBER OF USERS) OR A LIVE TEST DEMONSTRATION IN WHICH AS MANY AS 139 USERS ACTUALLY WOULD ATTEMPT TO ACCESS THE SYSTEM AT THE SAME TIME. ACCORDING TO GSA, NOT ALL OFFERORS HAVE RTE CAPABILITY, AND THERE IS NO ASSURANCE THAT ALL RTES ARE FUNCTIONALLY EQUIVALENT. A LIVE TEST OF THE TOTAL PROPOSED NETWORK, GSA CONTINUES, WOULD HAVE GIVEN INCUMBENT CSC AN UNFAIR ADVANTAGE, SINCE ITS EQUIPMENT WAS ALREADY IN PLACE; IN ADDITION, SUCH A TEST WOULD HAVE BEEN DIFFICULT TO CONTROL, NON- REPEATABLE, TIME-CONSUMING, AND EXPENSIVE.

GSA THEREFORE ARGUES THAT ITS DECISION TO BENCHMARK "A LOGICAL SUBSET OF THE TELEPROCESSING CAPABILITY" AND USE THE RESULTS TO PROJECT "TOTAL SYSTEM PERFORMANCE THROUGH EXTRAPOLATION" WAS AN APPROPRIATE ONE. (IN OTHER WORDS, GSA BELIEVES THAT THE BENCHMARK PROGRAMS CONSUMED THE SAME AMOUNTS AND TYPES OF COMPUTER RESOURCES AS REQUEST AND RETAIN, AND THAT THE RESULTS, INCLUDING COSTS, OF RUNNING THOSE PROGRAMS COULD BE ACCURATELY PROJECTED AND WERE VALID REGARDLESS OF THE NUMBER OF SIMULTANEOUS USERS.)

GAO ANALYSIS

BENCHMARKING GENERALLY CAN BE DEFINED AS A TEST UNDER CONTROLLED CIRCUMSTANCES, INTENDED TO PRODUCE DESCRIPTIVE DATA WHICH THE GOVERNMENT NEEDS TO EVALUATE PROPOSALS. IT CAN BE USED, AS HERE, BOTH TO ASSESS THE TECHNICAL CAPABILITY AND TO COMPARE THE EXPECTED OPERATING COSTS OF A PROPOSED SYSTEM. SEE GENERALLY COMPUTER NETWORK CORPORATION; TYMSHARE, INC., 56 COMP.GEN. 245(1977), 77-1 CPD 31.

IN DECIDING A PROTEST INVOLVING BENCHMARKING, OUR STANDARD OF REVIEW IS THE SAME AS FOR ANY OTHER EVALUATION PROCEDURE, I.E., THE ESTABLISHMENT OF QUALIFICATION AND TESTING PROCEDURES IS A MATTER WITHIN THE TECHNICAL EXPERTISE OF THE COGNIZANT PROCURING ACTIVITY. WE WILL NOT QUESTION THE USE OF SUCH PROCEDURES UNLESS THEY ARE WITHOUT A REASONABLE BASIS. TYMSHARE, INC., B-190822, SEPTEMBER 5, 1978, 78-2 CPD 167.

THUS, IF A BENCHMARK IS RATIONALLY BASED, ITS USE AS AN EVALUATION TOOL IS WITHIN THE DISCRETION OF THE PROCURING AGENCY. CF. INFORMATION INTERNATIONAL, INC., SUPRA, (IN WHICH WE FOUND THAT THE BENCHMARK METHODOLOGY USED PROVIDED A REASONABLE BASIS FOR DETERMINING THE COMPETITIVE RANGE).

A. SIMULTANEOUS ACCESS

IN DEVELOPING THE BENCHMARK FOR REQUEST AND RETAIN, THE ARMY USED THE CONCEPT OF A "SINGLE COMPOSITE TRANSACTION." IT DETERMINED, BASED ON HISTORICAL DATA AND CURRENT ESTIMATES, THE AVERAGE MIX OF PROGRAMS REQUIRED TO ENLIST ONE PERSON IN THE ARMY AND INCLUDED THESE PROGRAMS IN THE BENCHMARK. THE ARMY THEN USED THE NUMBER OF ENLISTMENTS AND REENLISTMENTS EXPECTED TO OCCUR EACH YEAR FOR THE NEXT 5 YEARS AS THE FACTOR TO PROJECT COSTS OF RUNNING THE BENCHMARK AND TO ESTIMATE LIFE CYCLE COSTS.

THE FIRST QUESTION PRESENTED BY CSC'S PROTEST IS WHETHER THE SINGLE COMPOSITE TRANSACTION DEVELOPED BY THE ARMY WAS VALID. OUR REVIEW INDICATES THAT A RELATIVELY SMALL NUMBER OF FUNCTIONS REPRESENTS THE MAJORITY OF REQUEST AND RETAIN TRANSACTIONS. MOST OF THESE TRANSACTIONS ARE ASSOCIATED WITH ENLISTMENT OR REENLISTMENT AND INVOLVE SIMILAR OPERATIONS, SUCH AS SELECTION OF A SCHOOL OR A DUTY STATION. IN ADDITION TO COMBINING THESE FUNCTIONS TO PRODUCE A "MODEL" TRANSACTION, SAC AND THE ARMY TESTED THE BENCHMARK PROGRAMS AGAINST ACTUAL REQUEST AND RETAIN OPERATIONS, USING "MACHINE LEVEL UTILIZATION MEASUREMENTS" (CENTRAL PROCESSING UNIT TIME, INPUT-OUTPUT COUNTS, MEMORY, AND THE LIKE) TO INSURE REPRESENTATIVENESS.

WE HAVE CAREFULLY EXAMINED THE RECORD, INCLUDING THOSE TEST RESULTS. OUR OPINION, THE BENCHMARK PROGRAMS DO CONSUME AMOUNTS AND TYPES OF SYSTEM RESOURCES EQUIVALENT TO REQUEST AND RETAIN, AND ACCURATELY REPRESENT WORK TO BE PERFORMED UNDER THE CONTRACT. WE THEREFORE FIND NO BASIS TO OBJECT TO THE USE OF THE "MODEL" TRANSACTION.

THE SECOND QUESTION IS WHETHER, WHEN THE SYSTEM MAY BE USED SIMULTANEOUSLY BY NEARLY 140 USERS, THE COST DATA DEVELOPED FROM A SINGLE COMPOSITE TRANSACTION REASONABLY CAN BE PROJECTED TO ESTIMATE LIFE CYCLE COSTS. IN OUR OPINION IT CAN, BECAUSE THE COSTED PORTION OF THE BENCHMARK IS IN FACT A "LINEAR REPRESENTATION" OF THE SYSTEM LOAD. IN OTHER WORDS, THE AVERAGE AMOUNT OF COMPUTER RESOURCES CONSUMED BY A SINGLE USER IN COMPLETING A TRANSACTION APPEARS TO BE THE SAME, REGARDLESS OF THE NUMBER OF USERS. FOR EXAMPLE, THE RESOURCES CONSUMED AND THE COSTS GENERATED WILL BE EQUIVALENT FOR 140 SIMULTANEOUS USERS AND FOR ONE USER MULTIPLIED BY 140.

WE BELIEVE THE BENCHMARK WOULD BE INVALID (1) IF ANY ELEMENT IN AN OFFEROR'S BILLING ALGORITHM (FORMULA) DEPENDED UPON THE NUMBER OF SIMULTANEOUS USERS, OR (2) IF A SIGNIFICANT NUMBER OF "COLLISIONS" OCCURRED WHEN TWO OR MORE USERS TRIED TO ACCESS THE SYSTEM AT THE SAME TIME, REQUIRING COMPUTER RESOURCES TO "REFEREE" THEIR REQUESTS. NEITHER, IN OUR OPINION, IS THE CASE HERE.

THE ARMY INFORMS US THAT IT HAS INSPECTED ALL OFFERORS' ALGORITHMS AND FINDS THAT NONE VARIES IN PROPORTION TO THE NUMBER OF SIMULTANEOUS USERS. AS FOR COLLISIONS, SOME OF THE REQUEST AND RETAIN FILES ARE "SEGMENTED" SO THAT EACH USER-- EACH ARMY ACTIVITY, FOR EXAMPLE-- HAS ITS OWN DATA BASE, SO THAT COLLISIONS ARE IMPOSSIBLE. FOR FILES WHICH ARE SHARED, THE ARMY HAS STRUCTURED THEM SO THAT EITHER A COUNTER IS INCREMENTED OR THE RECORD IS RE-READ BEFORE IT IS UPDATED. WITH THE "COUNTER" APPROACH, THE SYSTEM MERELY KEEPS TRACK OF THE NUMBER OF PLACES BEING RESERVED, SUCH AS SCHOOL SEATS, UNTIL CAPACITY IS REACHED. WITH THE SECOND APPROACH, IF, WHILE ONE RECRUIT IS CONSIDERING AN ASSIGNMENT WHICH THE SYSTEM HAS INDICATED IS AVAILABLE, ANOTHER SELECTS THE SAME ASSIGNMENT, WHEN THE FIRST USER SUBSEQUENTLY ATTEMPTS TO RESERVE IT, HE WILL BE TOLD THAT THE PLACE HAS BEEN TAKEN. WHETHER A RESERVATION IS ACCEPTED OR REJECTED, THE USER IS ADVISED, SO SIMILAR AMOUNTS OF COMPUTER RESOURCES ARE CONSUMED. IN OUR OPINION, THIS TYPE OF COLLISION WILL NOT HAVE A SIGNIFICANT IMPACT ON COST BECAUSE THE USER RECEIVES A MESSAGE IN ANY CASE. MORE IMPORTANTLY, THE REQUEST AND RETAIN DATA BASE IS SO LARGE-- MORE THAN A MILLION RECORDS- THAT VERY FEW COLLISIONS OCCUR. THE RESERVATION FILE ALONE, FOR EXAMPLE, CONTAINS ABOUT 80,000 RECORDS ON WHICH THERE ARE APPROXIMATELY 16,000 TRANSACTIONS A MONTH; THE GREATEST NUMBER OF COLLISIONS REPORTED BY THE ARMY IN A GIVEN MONTH IS 10.

FURTHER, THE GREAT MAJORITY OF USERS ARE LIMITED TO CERTAIN FUNCTIONS FOR WHICH REQUEST AND RETAIN SOFTWARE HAS BEEN PROGRAMMED, AND DO NOT USE ALL POSSIBLE CAPABILITIES OF THE SYSTEM. ONLY PERSONNEL RESPONSIBLE FOR MAINTENANCE AND CONTROL OF THE SYSTEM HAVE UNRESTRICTED ACCESS, WHICH WAS NOT TESTED IN THE BENCHMARK AND WHICH MIGHT INCREASE COSTS.

WE THEREFORE FIND THAT THE SINGLE COMPOSITE TRANSACTION PROVIDED A REASONABLE BASIS FOR IDENTIFYING THE OFFEROR WITH THE LOWEST EVALUATED LIFE CYCLE COSTS, AND THAT THE BENCHMARK WAS SUFFICIENT-- FROM A COST STANDPOINT-- AS AN EVALUATION TOOL, EVEN THOUGH IT DID NOT TEST SIMULTANEOUS ACCESS CAPABILITY.

B. FILE ACCESS METHOD

CSC ALLEGES THAT THE BENCHMARK DID NOT REQUIRE OFFERORS TO DEMONSTRATE THEIR DIRECT ACCESS CAPABILITY. THIS APPEARS TO BE TRUE. HOWEVER, GSA STATES THAT THIS CAPABILITY WAS TESTED DURING THE PERFORMANCE DEMONSTRATION. CSC'S REAL CONCERN APPEARS TO BE THAT, UNLESS DIRECT ACCESSING WAS REQUIRED DURING THE BENCHMARK, COSTS WHICH WERE EVALUATED ON THE BASIS OF THE BENCHMARK WOULD BEAR NO RELATION TO ACTUAL SYSTEM COSTS. IN OPERATION, THE ARMY'S EXTREMELY LARGE DATA BASE WILL REQUIRE USE OF DIRECT ACCESS IN ORDER TO MEET PERFORMANCE SPECIFICATIONS, PARTICULARLY RESPONSE TIME. THUS, CSC CORRECTLY INDICATES THAT UNLESS OFFERORS USE THE SAME ACCESS METHOD DURING THE BENCHMARK AS THEY WILL BE USING DURING REQUEST AND RETAIN OPERATIONS, ANY COMPARISON OF COSTS AMONG OFFERORS WILL BE INVALID.

THE ARMY, HOWEVER, ASSURES US THAT IT HAS MANUALLY REVIEWED BENCHMARK LISTINGS, CONTROL LANGUAGE, AND OUTPUT TO INSURE THAT DIRECT ACCESS WAS USED DURING THE BENCHMARK. WE THEREFORE BELIEVE CSC'S PROTEST ON THIS GROUND IS ACADEMIC. ALTHOUGH THE SOLICITATION DID NOT SPECIFICALLY REQUIRE DIRECT ACCESS TO BE USED IN THE BENCHMARK, FAILURE TO SO SPECIFY HAS NOT PREJUDICED ANY OFFEROR NOR ADVERSELY AFFECTED COST EVALUATION.

C. PERFORMANCE DEMONSTRATIONS

WE BELIEVE IT WAS ESSENTIAL THAT GSA USE THE PERFORMANCE DEMONSTRATIONS TO DETERMINE WHETHER OFFERORS HAD THE TECHNICAL CAPABILITY TO HANDLE SIMULTANEOUS USERS. HOWEVER, IN VIEW OF OUR CONCLUSION THAT THE NUMBER OF SUCH USERS WILL NOT HAVE A SIGNIFICANT IMPACT ON COSTS, WE DO NOT BELIEVE THAT THESE DEMONSTRATIONS HAD TO BE COSTED EXERCISES. IN ADDITION, AS GSA POINTS OUT, OFFERORS HAVE GUARANTEED SIMULTANEOUS ACCESS CAPABILITY, AND WILL BE REQUIRED TO PROVIDE ANY ADDITIONAL RESOURCES NEEDED TO MEET THIS REQUIREMENT AT NO ADDITIONAL COST TO THE GOVERNMENT. WE THEREFORE DO NOT BELIEVE IT WAS ABSOLUTELY NECESSARY TO TEST MORE THAN TWO USERS DURING THESE DEMONSTRATIONS.

IN SHORT, WE FIND THAT THE EVALUATION PROCEDURES USED BY GSA IN THIS CASE WERE WITHIN THE RANGE OF DISCRETION POSSESSED BY A PROCURING AGENCY.

THE PROTEST IS DENIED.

GAO Contacts

Office of Public Affairs