Skip to main content

Matter of: Sensis Corporation File: B-265790.2 Date: January 17, 1996

B-265790.2 Jan 17, 1996
Jump To:
Skip to Highlights

Highlights

Protest against determination that protester lacked a mature software development process and that its proposal therefore involved considerable risk is denied where the solicitation provided for a software capability evaluation to ascertain the extent to which an offeror's software process was documented. The software development model incorporated into the solicitation specifically recognized that success on individual projects as a result of highly skilled individuals was no substitute for the existence of an institutionalized. By modifying or replacing several components which have become unreliable. The RFP stated that award would be made "to the offeror who is deemed responsible in accordance with the Federal Acquisition Regulation.

View Decision

Matter of: Sensis Corporation File: B-265790.2 Date: January 17, 1996

Protest against determination that protester lacked a mature software development process and that its proposal therefore involved considerable risk is denied where the solicitation provided for a software capability evaluation to ascertain the extent to which an offeror's software process was documented, standardized, and integrated into a standard software process for the organization such that all projects used an approved, tailored version of the organization's standard software process for developing and maintaining software, and the software development model incorporated into the solicitation specifically recognized that success on individual projects as a result of highly skilled individuals was no substitute for the existence of an institutionalized, formal software development process.

Attorneys

DECISION

Sensis Corporation protests the Department of the Air Force's award of a contract to the Lockheed Martin Company, under request for proposals (RFP) No. F19628-95-R-0018, for a reliability, maintainability and supportability upgrade of the AN/FPS-117 radar systems. Sensis challenges the conduct of discussions and the evaluation of proposals.

We deny the protest.

BACKGROUND

The solicitation contemplated the award of a predominantly fixed-price requirements contract to improve the reliability, maintainability and supportability of 34 AN/FPS-117 solid state, phased array radars (plus four maintenance control systems), located at sites in Alaska, Canada, Iceland, and California, by modifying or replacing several components which have become unreliable, unmaintainable or unsupportable. The RFP stated that award would be made

"to the offeror who is deemed responsible in accordance with the Federal Acquisition Regulation, possesses the technical, management, financial and appropriate facility capabilities necessary to fulfill the requirements of the contract, and whose proposal conforms to the solicitation requirements and is judged, by an integrated assessment of the Evaluation Criteria . . . and the General Considerations to be the most advantageous to the Government."

The technical area was more important than the cost/price area. The solicitation provided that technical proposals would be assigned an adjectival/color rating and would be evaluated for proposal and performance risk. The RFP listed four specific technical evaluation factors: (1) system engineering approach; (2) reliability, maintainability and supportability; (3) software capability evaluation; and (4) management. The first two factors were weighted equally and each was more important than the third factor, which in turn was more important than the fourth factor. In addition, there was provision for a cost realism assessment to consider technical risk and their impact on costs.

Three offerors submitted initial proposals. Only Sensis's and Lockheed Martin's proposals were included in the competitive range; after discussions, they were requested to submit best and final offers (BAFO). BAFOs were evaluated as follows:

Lockheed Martin Sensis

System Engineering Acceptable Acceptable (Green) (Green) Moderate Risk Low Risk

Reliability, Acceptable Exceptional (Blue) Maintainability and (Green) Low Risk Supportability Low Risk

Software Capability Acceptable Acceptable (Green) (Green) High Risk Low Risk

Management Acceptable Acceptable (Green) (Green) Low Risk Low Risk

OVERALL TECHNICAL Acceptable Acceptable (Green) (Green) Moderate Risk Low Risk

EVALUATED COST/PRICE/RISK $24,804,563 $24,204,795 Low Risk Moderate Risk

In its evaluation of BAFOs, the Air Force recognized that Sensis's proposed approach was the more advantageous of the two in the area of reliability, maintainability and supportability. The solicitation provided in this regard that the government would evaluate an offeror's approach to limiting system failure rates; simplifying software maintenance; providing for supportability, fault detection, and isolation; eliminating manual adjustments/calibrations; and supporting unattended operation of the radars. Although the Air Force found that there were no significant differences between the offerors with respect to satisfying the solicitation limitations on system failure rates, Sensis's proposal otherwise offered several strengths. For example, the agency found Sensis's proposal to eliminate manual adjustments of the radar system transceivers to be a superior approach to supporting unattended operation of the radars. Likewise, the agency concluded that Sensis's proposal to use CASE software development tools and the Ada programming language, as well as to significantly reduce the lines of code, represented a superior approach to increasing the supportability of the system. In addition, the Air Force considered it a strength that Sensis's proposed approach would reduce the time required for running the fault detection and isolation test cycle.

The Air Force, however, determined that award to Sensis would involve significantly greater risks than award to Lockheed Martin. Of primary importance in this regard was the evaluation of offerors' software capability. The solicitation provided that the agency would

"evaluate an offeror's software process by reviewing the offeror's Software Process Improvement Plan (SPIP) and by using the Software Engineering Institute (SEI)-developed Software Capability Evaluation (SCE). The government will evaluate the software process capability by investigating the offeror's current strengths, weaknesses, and improvement activities in the Key Process Areas (KPA) through the defined Maturity Level as defined in the SEI Technical Report, Capability Maturity Model for Software. . . . The Government will perform an SCE . . . on each offeror in the competitive range by reviewing current, government selected projects and comparing methods/processes used on these projects with the written proposal/SPIP."

The Software Engineering Institute's Software Capability Evaluation was also incorporated into the solicitation. This document provides for an examination of an organization's projects in terms of key process areas--that is, activities related to decision-making processes, communications processes, and technical support processes--to ascertain the extent to which the organization possesses a mature software development process--which is defined as one which is "standard and consistent because both software engineering and management activities are stable and repeatable." Specifically, in a mature software development process:

"[t]he software process for both management and engineering activities is documented, standardized, and integrated into a standard software process for the organization. All projects use an approved, tailored version of the organization's standard software process for developing and maintaining software."

This approach was based on a determination that "mature processes reduce the risk associated with the planned development" of software, with the result that "the likelihood of success should increase as the software processes mature."

Based on an examination of the most relevant of the projects selected by the offerors in terms of 13 key process areas, and the results of an on-site inspection of the offerors' software organizations, the Air Force determined that while Lockheed Martin generally possessed a mature software organization, Sensis had not developed a formal software development process. Sensis, most of whose software projects had been relatively small, lacked written software development policies and procedures, and instead had used an ad hoc software development strategy, which was dependent on a small team of highly qualified individuals. Although Sensis had employed more formal software processes as a subcontractor on a relatively larger effort to upgrade the AN/TPS-59 radar, the agency noted that this did not reflect the existence of an institutionalized, formal software development process at Sensis, but instead was imposed on Sensis by its prime contractor. Further, while Sensis had proposed a software process improvement plan addressing some process areas, the agency found only limited evidence that this plan was being implemented. As a result, Sensis's software organization was rated as weak, with high or moderate risk, with respect to all 13 key process areas (as well as high risk overall).

Further, the Air Force found that the risk resulting from Sensis's lack of a mature software development process was increased by its proposal to modify and use software it was developing for the AN/TPS-59 radar upgrade for the proposed AN/FPS-117 radar upgrade here since that software had not completed development and, given the differences between the radars, the potential existed for the loss or degradation of current AN/FPS-117 capabilities. The agency concluded that "[t]he probability of completing the [AN/FPS-117] upgrade successfully on-time and on-budget without a documented process is very low," and accordingly assigned Sensis's proposal a high proposal risk with respect to software capability.

In addition, the Air Force determined that the overall proposal risk associated with Sensis's proposal was increased by several other aspects of its proposed approach. While the Air Force believed that Sensis's proposal to modify and use a radar environment simulator in the development and testing phases could offer certain advantages with respect to generating complex, repeatable test scenarios, the agency expressed concern that, in view of Sensis's extensive reliance on the simulator, any delays in modifying it would greatly reduce the chances of meeting the proposed schedule. Likewise, the Air Force expressed concern that Sensis had proposed using a design from the AN/FPS-59 radar upgrade program which had not completed integration and first article testing; the agency believed that there was a considerable potential that problems identified during these tests could delay the AN/FPS-117 upgrade schedule. Further, the agency noted that Sensis's proposed schedule allowed no time for problem resolution/retest after the scheduled government-witnessed testing; based on past experience, the agency expected that government-witnessed testing would reveal problems in the proposed system.

In contrast, the Air Force evaluated Lockheed Martin's proposal as offering a significantly lower overall risk. The agency considered it significant that Lockheed Martin, unlike Sensis, possessed a generally mature software development process. The agency also determined that Lockheed Martin had proposed a low risk approach of primarily utilizing hardware developed on recent versions of the AN/FPS-117 and rehosting existing software. In addition, the source selection authority (SSA) found that while Sensis's only very relevant experience had been as a subcontractor on the AN/TPS-59 radar upgrade program, and it was uncertain whether Sensis possessed the "technical depth and breadth" necessary to perform the AN/FPS-117 upgrade, Lockheed Martin had a substantial, proven track record, having repeatedly performed as a prime contractor on programs similar to the radar upgrade here, and had demonstrated that it had "the breadth and depth of technical talent essential to the timely, essential execution" of the AN/FPS-117 upgrade. Further, the SSA considered the lower risk associated with Lockheed Martin's proposal to be especially significant in view of the facts that accelerated reductions in manning (as a result of severe funding constraints) and an already reduced operational availability of the radars made it imperative that the upgrade be completed as soon as possible. The SSA therefore concluded that the lower risk associated with the Lockheed Martin proposal was worth the slight additional cost (2.5 percent) and outweighed the slight advantages offered by Sensis's proposed technical improvements in reliability, maintainability and supportability. Upon learning of the resulting award to Lockheed Martin, Sensis filed this protest with our Office.

SOFTWARE CAPABILITY EVALUATION

Sensis challenges the Air Force's evaluation of Sensis's software development capability on the basis that it failed to take into account the fact that Sensis had previously utilized standardized software development processes as a subcontractor on the AN/TPS-59 radar upgrade program and was proposing to use personnel who had worked on that program.

The evaluation of technical proposals is primarily the responsibility of the contracting agency since the agency is responsible for defining its needs and the best method of accommodating them and must bear the consequences of any difficulties resulting from a defective evaluation. Therefore, our Office will not engage in an independent evaluation of technical proposals and make an independent determination of their relative merits. Litton Sys., Inc., B-239123, Aug. 7, 1990, 90-2 CPD Para. 114. Rather, we review the agency's evaluation only to ensure that it was reasonable and consistent with applicable statutes and regulations, as well as with the terms of the RFP. Polar Power, Inc., B-257373, Sept. 2, 1994, 94-2 CPD Para. 92.

We find the Air Force's evaluation of Sensis's software capability was reasonable under the evaluation approach set forth in the solicitation. The agency in fact recognized that Sensis had proposed key personnel with extensive skills and relevant experience. Likewise, the agency recognized that Sensis's most relevant experience was as a subcontractor on the program to upgrade the AN/TPS-59 radar, where it had employed more formal software processes. However, as Sensis concedes, Sensis has otherwise relied on an ad hoc approach to software development, and the Air Force placed more emphasis in its evaluation of Sensis's software capability on this lack of a mature software development process as defined by the solicitation than on Sensis's single instance of using a more formal development process. [1] The software capability evaluation document, incorporated into the solicitation, stated that the evaluation was to ascertain the extent to which an offeror's software process "is documented, standardized, and integrated into a standard software process for the organization," such that "[a]ll projects use an approved, tailored version of the organization's standard software process for developing and maintaining software." The incorporated Software Capability Maturity Model for Software specifically recognized that success on individual projects as a result of highly skilled individuals was no substitute for the existence of an institutionalized, formal software development process, using standard and consistent development processes on all projects, as a predictor (but not a guarantee) of success on future software development projects. [2] In these circumstances, we find nothing unreasonable in the agency's determination that the use of more formal software processes on one project at the direction of the prime contractor was not equivalent to the existence of an institutionalized, formal software development process.

As set forth in the software capability evaluation document, the software capability evaluation was premised on the position that "mature processes reduce the risk associated with the planned development" of software, with the result that "the likelihood of success should increase as the software processes mature." In accordance with this document, the Air Force could reasonably determine that Sensis's overwhelmingly ad hoc approach to software development, when considered in conjunction with its limited experience with larger software development projects and its proposal to modify software which was still under development for a different radar system, was significantly more risky than the proposal of Lockheed Martin, an offeror with substantial relevant experience and resources, to use a generally mature software development process to rehost existing software.

OFFERORS' CAPABILITIES

Sensis contends that the Air Force improperly utilized an undisclosed evaluation criterion--the relative business size of Sensis and Lockheed Martin--as a discriminator in the award selection.

The record indicates that the agency's technical evaluation focused not on the relative business size of the offerors as such, but rather on their relative technical and financial capabilities. As noted above, the SSA concluded that while Lockheed Martin had repeatedly performed as a prime contractor on substantial programs similar to the radar upgrade here and had demonstrated that it had "the breadth and depth of technical talent essential to the timely, essential execution" of the AN/FPS-117 upgrade, it was uncertain whether Sensis possessed the necessary "technical depth and breadth." The SSA also took into account the fact that the technical risks associated with the Sensis approach created the potential for cost growth and that Sensis "may find it difficult to absorb such anticipated costs due to its relatively small size."

Further, the solicitation specifically provided for consideration of whether an offeror "possesses the technical, management, financial and appropriate facility capabilities necessary to fulfill the requirements of the contract." In addition, the solicitation provided for a cost realism evaluation; it directed offerors to identify both "the inherent technical, schedule or other risks which may impact cost," and "the source of company funding" to be used in performing the contract. Thus, we believe that it was clearly consistent with the solicitation for the agency to consider as part of the source selection decision the extent to which Sensis possessed the technical and financial resources to perform the contract. [3]

DISCUSSIONS

Sensis contends that the Air Force failed to advise it during discussions of the perceived weakness of Sensis's software development capabilities; as a result, argues the protester, the discussions were not meaningful.

Agencies are required to conduct meaningful discussions with all competitive range offerors. Price Waterhouse, B-254492.2, Feb. 16, 1994, 94-1 CPD Para. 168. In order for discussions to be meaningful, agencies must generally point out weaknesses, excesses, or deficiencies in proposals, unless doing so would result in disclosure of one offeror's technical approach to another offeror or technical leveling. See Lone Star Fleischwaren Im-Export GmbH, B-259588.2, May 25, 1995, 95-1 CPD Para. 263. However, an agency generally need not discuss matters with offerors which, by their nature, generally are not subject to correction through the discussion process. See EcoTek LSI, B-254506.2, Jan. 11, 1994, 94-1 CPD Para. 13; Communications Int'l, Inc., B-246076, Feb. 18, 1992, 92-1 CPD Para. 194.

We find no basis to question the conduct of discussions. The software capability evaluation that was provided for under the solicitation focused on whether an offeror possessed a mature software development process. This matter was largely one of historical fact which was not subject to correction through discussions. Indeed, Sensis does not claim that it could have furnished information showing the existence of a mature software development process as defined by the solicitation; on the contrary, it concedes that it has generally relied on an ad hoc approach to software developments.

COST/TECHNICAL TRADEOFF

Sensis primarily argues that the source selection decision was in error because it allegedly did not take into account the life-cycle cost savings offered by Sensis's evaluated superior approach to supporting unattended operation of the radars and increasing the supportability of the radar system.

Although Sensis claims that it is not arguing that life-cycle costs should be considered as part of the cost/price evaluation, the protester apparently believes that the agency was required to quantify the claimed life-cycle cost savings for use in the cost/technical tradeoff. Thus, we consider its position to be tantamount to arguing that the agency was required to evaluate life-cycle costs as part of the cost/price evaluation. An agency, however, may apply only those evaluation factors specified in the solicitation, and the solicitation here contemplated consideration only of acquisition costs, not life-cycle costs, as part of the cost/price evaluation. See Inner Harbor West Joint Venture, B-249945.3, Mar. 11, 1993, 93-1 CPD Para. 232; Mennen Medical, Inc., B-246764 et al., Apr. 2, 1992, 92-1 CPD Para. 341.

Further, as noted above, the Air Force did consider as part of the technical evaluation (and ultimately the source selection decision) Sensis's superior approach to supporting unattended operation of the radars and increasing the supportability of the system, and as a result, rated Sensis's proposal as exceptional under the reliability, maintainability and supportability evaluation factor. The agency simply determined, however, that the long-term potential advantages offered by Sensis's approach, and Sensis's slightly lower price, were outweighed by the significantly lower risk associated with the Lockheed Martin proposal and the greater likelihood of Lockheed Martin successfully furnishing a compliant and timely upgrade to the AN/FPS-117 radar system. Based on our review of the record, we find this determination to be consistent with the stated evaluation criteria and reasonable. See Information Sys. & Networks Corp., B-258684.2; B-258684.3, Apr. 4, 1995, 95-1 CPD Para. 255.

The protest is denied.

Comptroller General of the United States

1. According to Sensis, "[f]or most of its software development projects, it was neither necessary nor economically efficient to utilize highly formalized software development processes."

2. As explained in the Capability Maturity Model for Software:

"[e]ven in undisciplined organizations, however, some individual software projects produce excellent results. When such projects succeed, it is generally through the heroic methods of a dedicated team, rather than through repeating the proven methods of an organization with a proven software process. In the absence of an organization-wide software process, repeating results depends entirely on having the same individuals available for the next project. Success that depends solely on the availability of specific individuals provides no basis for long-term productivity and quality improvement through an organization."

3. Although Sensis also argues that it possessed sufficient cash and lines of credit to absorb likely potential cost increases, this does not demonstrate that the agency could not reasonably consider the potential for Sensis to incur losses which, by Sensis's own calculations, could amount to nearly 20 percent of its net worth, to be a risk factor.

GAO Contacts

Office of Public Affairs