B-188964, NOV 23, 1977

B-188964, NOV 23, 1977

1. PROTEST CONCERNING PROPRIETY OF BENCHMARK REQUIREMENTS, FILED MORE THAN 3 MONTHS AFTER DETAILS OF BENCHMARK REQUIREMENTS WERE FURNISHED TO OFFERORS, AND SCHEDULE FOR DEMONSTRATION ESTABLISHED, IS UNTIMELY AND WILL NOT BE CONSIDERED ON MERITS. 2. AGENCY ACTED REASONABLY IN EXCLUDING OFFEROR FROM COMPETITION FOR DEFICIENCIES IN BENCHMARK DEMONSTRATION. PROTESTER'S CONTENTION THAT DEFICIENCIES WERE TRIVIAL IS NOT SUSTAINED. 3. RECORD DOES NOT SHOW THAT AGENCY WAIVED DEFICIENCY IN SUCCESSFUL OFFEROR'S BENCHMARK DEMONSTRATION. MOREOVER, WHILE AGENCY MAY HAVE WAIVED BENCHMARK REQUIREMENTS FOR THIRD OFFEROR, SUCH ACTION WAS NOT PREJUDICIAL TO PROTESTER SINCE THAT FIRM WAS NOT THE SUCCESSFUL OFFEROR.

INTERACTIVE DATA CORPORATION:

INTERACTIVE DATA CORPORATION (IDC) PROTESTS THE AWARD OF A CONTRACT TO DATA RESOURCES, INC. (DRI) UNDER DEPARTMENT OF THE TREASURY REQUEST FOR PROPOSAL (RFP) OAP-6246. SERVICES TO BE PROVIDED ARE THE MAINTENANCE OF A "COMPREHENSIVE ECONOMIC DATA BANK."

PROPOSALS WERE TO BE EVALUATED AS FOLLOWS:

"C. AWARD CRITERIA

AWARD WILL BE GRANTED TO A QUALIFIED VENDOR WITH THE LOWEST OVERALL COST TO PROCESS DEMONSTRATION PROBLEMS. THE FIRST STEP IN THE EVALUATION PROCESS WILL BE TO DETERMINE WHICH PROPOSALS MEET THE MANDATORY REQUIREMENTS, BASED ON VENDOR DOCUMENTATION. THE SECOND STEP WILL BE BASED ON A DEMONSTRATION TO DETERMINE WHICH PROPOSALS MEET A SELECTED SET OF THE MANDATORY REQUIREMENTS, BASED ON VENDORS ABILITY TO PERFORM A DEMONSTRATION. THE DEMONSTRATION WILL CONSIST OF A 'USER WITNESSED RUNNING OF A GROUP OF REPRESENTATIVE PROGRAMS ON A VENDOR'S PROPOSED COMPUTER SYSTEM TO VALIDATE HARDWARE AND SOFTWARE IN PROCESSING THE WORKLOAD OF THE PROCURING ACTIVITY IN THE ORDER AND LANGUAGE IN WHICH THE ACTUAL WORKLOAD IS LIKELY TO BE PROCESSED BY THE USER.' THE THIRD STEP WILL BE A COST EVALUATION OF THE PROPOSAL, BASED ON THE DEMONSTRATION.

NOTE: A VENDOR WHO DOES NOT MEET THE TOTAL QUALIFICATIONS OF STEP ONE, AS JUDGED BY THE TREASURY EVALUATION COMMITTEE (TEC) WILL NOT PROCEED TO STEP TWO OR STEP THREE. A VENDOR WHO DOES NOT MEET THE TOTAL QUALIFICATIONS OF STEP TWO AS JUDGED BY THE TEC, WILL NOT PROCEED TO STEP THREE.

C.1 RELATIVE IMPORTANCE OF FACTORS:

STEP ONE ABOVE - PRIMARY STEP TWO ABOVE - SECONDARY STEP THREE ABOVE - TERTIARY"

OF THE FOUR FIRMS WHOSE WRITTEN TECHNICAL PROPOSALS WERE DEEMED ACCEPTABLE PURSUANT TO "STEP ONE" ABOVE, TWO FIRMS, INCLUDING THE PROTESTER, WERE DISQUALIFIED FROM PARTICIPATING IN "STEP THREE" FOR FAILURE TO PASS THE BENCHMARK REQUIREMENTS.

IDC'S BENCHMARK FAILURE WAS BASED ON ITS "BOX-JENKINS" COMPUTER PROGRAM'S FAILURE TO CALCULATE CONFIDENCE LIMITS AS WELL AS ITS FAILURE TO PROVIDE THE "DURBIN-H" REGRESSION ANALYSIS STATISTIC.

AS EXPLAINED BY TREASURY:

"THE BOX-JENKINS IS A HIGHLY COMPLEX STATISTICAL FORECASTING TECHNIQUE. ITS PURPOSE IS TWO-FOLD: IT INDICATES THE SINGLE MOST PROBABLE ANSWER TO A PROBLEM BASED ON THE DATA FURNISHED AND IT GIVES A NUMERICAL RANGE OVER WHICH THE ANSWER MIGHT VARY. THE LATTER IS CALLED THE CONFIDENCE LIMIT AND ALLOWS THE DATA USER TO CONCLUDE HOW MUCH CONFIDENCE HE SHOULD PLACE ON THE CORRECTNESS OF THE SINGLE MOST PROBABLE ANSWER. WITHOUT PROVISION FOR THIS CALCULATION, NO DETERMINATION AS TO RELIABILITY CAN BE MADE (AN OMISSION WHICH IS CLEARLY CRITICAL)."

IDC COMPLAINS THAT THE TREASURY ADDED "REQUIREMENTS TO THE SOLICITATION THROUGH THE BENCHMARK" (INCLUDING THE FAILED PORTIONS OF THE DEMONSTRATION) AND THAT THE THREE WEEKS BETWEEN THE ISSUANCE OF THE BENCHMARK AND THE ACTUAL DEMONSTRATION WAS INSUFFICIENT NOTICE TO ENABLE IDC TO FULLY COMPLY; THAT ITS ADMITTED FAILURE TO INCLUDE THE CONFIDENCE LIMIT AS A STANDARD PART OF ITS BOX-JENKINS PROGRAM, AS WELL AS THE OMISSION OF THE DURBIN-H REGRESSION ANALYSIS, REQUIRED ONLY TRIVIAL EFFORT TO CORRECT. IDC ALSO COMPLAINS THAT ITS PROPOSAL WAS EVALUATED DIFFERENTLY FROM OTHER OFFERORS BECAUSE TREASURY "WAS WILLING TO WAIVE (BENCHMARK) DEFICIENCIES OF OTHER VENDORS BUT NOT THOSE OF IDC.

A. BENCHMARK REQUIREMENTS

THE RFP WAS ISSUED ON SEPTEMBER 9, 1976, WITH PROPOSALS DUE ON SEPTEMBER 30, 1976. THE RFP REQUIRED THE BOX-JENKINS SOFTWARE PROGRAM IN A GENERAL WAY, I.E., "PERFORM A BROAD RANGE OF *** FORECASTING METHODS AND STATISTICAL METHODS, INCLUDING *** BOX-JUNKINS ***." THE "DURBIN-H" STATISTIC DOES NOT APPEAR TO HAVE BEEN SPECIFICALLY MENTIONED IN THE RFP.

DETAILED BENCHMARK REQUIREMENTS WERE FURNISHED TO THE OFFERORS IN JANUARY 1977, WITH DEMONSTRATIONS SCHEDULED TO COMMENCE APPROXIMATELY 2 WEEKS THEREAFTER; HOWEVER, THE DEMONSTRATION SCHEDULE WAS SUBSEQUENTLY EXTENDED FOR ONE ADDITIONAL WEEK. IT IS NOT DISPUTED THAT THE DETAILED BENCHMARK INCLUDED THE REQUIREMENTS WHICH IDC FAILED TO DEMONSTRATE AT ITS BENCHMARK TEST, AND THAT IDC DID NOT OBJECT TO OR QUESTION THE DETAILED BENCHMARK AT THE TIME IT WAS ISSUED OR DURING THE DEMONSTRATION HELD IN JANUARY 1977. IN ADDITION, THERE IS NO INDICATION IN THE RECORD THAT IDC CONSIDERED THE 3-WEEKS TIME AVAILABLE FOR PREPARATION OF THE BENCHMARK DEMONSTRATION AS INADEQUATE, SINCE IT DID NOT REQUEST ANY ADDITIONAL TIME FOR PREPARATION.

GAO BID PROTEST PROCEDURES REQUIRE THAT A BID PROTEST BE FILED NOT LATER THAN 10 DAYS AFTER THE BASIS FOR THE PROTEST IS KNOWN OR SHOULD HAVE BEEN KNOWN. 4 C.F.R. 20.2(B)(2). THIS PROTEST WAS FILED ON APRIL 29, 1977, MORE THAN 3 MONTHS AFTER THE DETAILED BENCHMARK WAS FURNISHED TO IDC, AND CONSEQUENTLY ITS PROTEST REGARDING THE PROPRIETY OF THE BENCHMARK AND THE INADEQUACY OF THE TIME ALLOWED FOR PREPARATION OF DEMONSTRATION IS UNTIMELY, AND WILL NOT BE CONSIDERED ON THE MERITS.

B. BENCHMARK TEST DEFICIENCIES

IDC'S BENCHMARK DEFICIENCIES ARE CHARACTERIZED BY TREASURY AS SERIOUS AND NOT TRIVIAL AS SUGGESTED BY THE PROTESTER.

"THE MAIN POINT HERE *** IS THAT INTERACTIVE DATA WAS NOT DISQUALIFIED BECAUSE OF THE LACK OF SOME TRIVIAL CAPABILITIES. THE TWO OMISSIONS WERE SERIOUS, THEY WERE TOTALLY ABSENT AS OPPOSED TO BEING PRESENT BUT NOT WORKING PROPERLY; AND THE CORRECTION OF THESE DEFICIENCIES WERE NOT JUDGED TO BE MINOR."

TREASURY CLAIMS THAT THE PROGRAMMING NECESSARY TO PROVIDE THE DURBIN H STATISTIC COULD PROBABLY BE COMPLETED IN A WEEK IF THE SAME PERSON WHO DEVELOPED THE EXISTING SOFTWARE FOR SIMILAR STATISTICS WERE AVAILABLE, OR 1 MONTH OR LONGER IF THAT PERSON WERE NOT AVAILABLE. SIMILARLY, TREASURY CLAIMS THE PROGRAMMING EFFORT FOR THE BOX-JENKINS CONFIDENCE LIMIT COULD TAKE A MONTH OR MORE DEPENDING ON THE PERSONNEL AVAILABLE TO THE OFFEROR FOR THAT EFFORT.

TREASURY'S ESTIMATE IN THIS REGARD MAY BE SOMEWHAT OVERSTATED, PARTICULARLY IN VIEW OF THE 3-WEEK TIME PERIOD INITIALLY AVAILABLE TO PREPARE THE TOTAL DEMONSTRATION. NONETHELESS, WE ARE UNABLE TO CONCLUDE THAT THE AGENCY ACTED UNREASONABLY IN EXCLUDING IDC FROM THE COMPETITION FOR FAILURE TO SUCCESSFULLY COMPLETE THE BENCHMARK DEMONSTRATION. WE WILL NOT REGARD AN AGENCY'S TECHNICAL EVALUATION AS UNREASONABLE MERELY BECAUSE THERE IS SUBSTANTIAL DISAGREEMENT BETWEEN THE CONTRACTING AGENCY AND THE OFFEROR. FOR A TECHNICAL EVALUATION TO BE DEEMED UNREASONABLE, IT MUST CLEARLY APPEAR FROM THE RECORD THAT THERE IS NO RATIONAL BASIS FOR THAT EVALUATION. JOANELL LABORATORIES, INCORPORATED, 56 COMP.GEN. (1977), 77-1 CPD 51. HERE THE BENCHMARK WAS CONDUCTED IN ACCORDANCE WITH STATED REQUIREMENTS WHICH WERE NOT QUESTIONED BY THE PROTESTER WHEN ISSUED, AND WE ARE NOT PREPARED TO CONCLUDE THAT THE PROTESTER'S DEFICIENCIES WERE TRIVIAL.

CONTRASTED WITH THE IDC DEFICIENCIES WERE THOSE OF DRI AND CYPHERNETICS WHICH, IDC NOTES, WERE WAIVED BY THE AGENCY.

IDC NOTES THAT SECTION 1.1 OF THE BENCHMARK REQUIRED "REPRODUCTION OF A LINE PLOT WITH FLOATING BARS," AND THAT DRI'S BENCHMARK DID NOT PROVIDE THE "FLOATING BARS." THE AGENCY CHARACTERIZED THE DRI FAILURE TO REPRODUCE "FLOATING BARS" AS A TRIVIAL DEFICIENCY.

SECTION 1.1. ENTITLED "GRAPHICS DEMONSTRATION" REQUIRES OFFERORS TO "REPRODUCE THE ATTACHED THREE GRAPHS, REASONABLY APPROXIMATING THE SCALE, CHARACTERS AND OTHER MARKINGS ***."

IN OUR VIEW, SECTION 1.1, SUPRA IS NOT A REQUIREMENT THAT THE GRAPHS BE REPORDUCED AT THE BANCHMARK DEMONSTRATION IN EVERY PARTICULAR DETAIL OF THOSE ATTACHED TO THE BENCHMARK SPECIFICATION. WHAT IS REQUIRED, IN OUR OPINION, IS A REASONABLE APPROXIMATION OF THE MARKINGS ON THE SAMPLE GRAPHS. THEREFORE, WE DO NOT AGREE WITH IDC THAT TREASURY WAIVED A BENCHMARK REQUIREMENT FOR DRI.

IDC ALSO CLAIMS THAT THE FAILURE OF CYPHERNETIC'S ROUTINE FOR COCHRANE- ORCUTT AUTO-CORRELATION ADJUSTMENT PROCEDURE TO CONVERGE TO A SOLUTION WAS WAIVED BY THE TREASURY, THUS FURTHER DEMONSTRATING IDC'S UNEQUAL TREATMENT IN THE BENCHMARK TEST.

ALTHOUGH IT MAY BE THAT CYPHERNETICS SHOULD NOT HAVE BEEN PERMITTED TO CONTINUE TO "STEP THREE" BECAUSE OF ITS BENCHMARK DEFICIENCY, PERMITTING THAT FIRM TO REMAIN IN THE COMPETITION WAS NOT PREJUDICIAL TO IDC, SINCE CYPHERNETICS WAS NOT THE SUCCESSFUL OFFEROR. THE EXCLUSION OF CYPHERNETICS FROM FURTHER PARTICIPATION WOULD NOT HAVE CHANGED THE FINAL RESULT - AWARD TO DRI. COMPU-SERV, B-186164, MAY 9, 1977, 77-1 CPD 327.

THE PROTEST IS DENIED.