This is the accessible text file for GAO report number GAO-02-316 
entitled 'DC Courts: Disciplined Processes Critical to Successful 
System Acquisition' which was released on February 28, 2002. 

This text file was formatted by the U.S. General Accounting Office 
(GAO) to be accessible to users with visual impairments, as part of a 
longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the 
printed version. The portable document format (PDF) file is an exact 
electronic replica of the printed version. We welcome your feedback. 
Please E-mail your comments regarding the contents or accessibility 
features of this document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

United States General Accounting Office: 
GAO: 

Report to the Chairman, Subcommittee on the District of Columbia, 
Committee on Appropriations, House of Representatives: 

February 2002: 

DC Courts: 

Disciplined Processes Critical to Successful System Acquisition: 

GA0-02-316: 

United States General Accounting Office: 
Washington, DC 20548: 

February 28, 2002: 

The Honorable Joe Knollenberg: 
Chairman: 
Subcommittee on the District of Columbia: 
Committee on Appropriations: 
House of Representatives: 

Dear Mr. Chairman: 

This letter responds to your August 20, 2001, request that we perform 
an initial review of the District of Columbia Courts' (DC Courts) 
effort to acquire a new information system. Faced with a myriad of 
nonintegrated systems that do not provide the necessary information to 
support its overall mission, the DC Courts is in the process of 
acquiring a replacement system called the Integrated Justice 
Information System (IJIS). This system is expected to address the 
current system's deficiencies and provide DC Courts with the 
information it needs to perform its mission. The District of Columbia 
Appropriation Act, 2001,[Footnote 1] provides that none of the funds 
in the act or in any other act shall be available for the purchases, 
installations, or operation of IJIS until a detailed plan and design 
has been submitted by DC Courts and approved by the House and Senate 
committees on appropriations. DC Courts is in the initial stages of 
the system acquisition effort—developing a request for proposal (RFP) 
to solicit bids from vendors for the design and implementation of the 
new system.[Footnote 2] As required, this detailed plan and design, 
which includes a draft RFP, was submitted to the appropriations 
committees on May 17, 2001, for review. 

Specifically, we assessed whether
1. DC Courts has implemented the disciplined processes for this project
to reduce the risk associated with this effort to acceptable levels; 

2. DC Courts' requirements to acquire the system, as identified in its 
draft RFP, contain the necessary specificity to reduce the risks from 
requirement defects to acceptable levels;[Footnote 3] and; 

3. DC Courts has performed the necessary actions to determine that a 
commercial off-the-shelf system (COTS) would meet its needs. 

Results in Brief: 

DC Courts has not yet implemented the disciplined processes necessary 
to reduce the risks associated with acquiring and managing the HIS 
acquisition effort to acceptable levels. A disciplined software 
development and acquisition process maximizes the likelihood of 
achieving the intended results (performance) within established 
resources (costs) on schedule. DC Courts officials acknowledged that 
they do not yet have the disciplined processes in place to reduce the 
risks from this effort to acceptable levels. However, they also 
acknowledged that disciplined processes were necessary. They further 
noted that even though sufficient funding to fully implement 
disciplined processes would not be available until the system was 
approved and funded, they had already begun to implement some elements 
of a disciplined process. For example, at the time of our review, DC 
Courts was sending several people to be trained and certified in 
project management skills. 

The majority of the DC Courts' requirements, developed for the draft 
RFP, lacked the necessary specificity to ensure that the defects in 
these requirements have been reduced to acceptable levels and that the 
system would meet its users' needs. In addition, the requirements in 
the draft RFP did not directly relate to industry standards and the 
terms "customization" and "modification" were not clearly defined in 
the draft RFP. We also noted that the system requirements were not 
logically grouped. 

DC Courts officials are electing to use the acquisition process to 
identify the cost, schedule, and performance gaps associated with 
their effort. DC Courts officials acknowledged that this approach 
generally increases risk; however, they concluded that the benefit to 
be obtained—accelerating the implementation of a badly needed system—
justifies those risks. At this point in the process, we concur with DC 
Courts' decision to use the acquisition process to identify the cost, 
schedule, and performance gaps in this case because of the following 
mitigating factors. 

* DC Courts officials concluded, based on a qualitative analysis, that 
they do not have unique requirements that would prevent the 
utilization of a COTS product. Furthermore, they recognized that a gap 
analysis must be performed as part of the vendor selection process to 
identify the cost, schedule, and performance impacts of each vendor's 
product before deciding which product to acquire. 

* DC Courts is still in the early stages of the system acquisition 
effort and has not yet reached the critical point of contract award. 
Therefore, a gap analysis can still be performed prior to the system 
acquisition to ensure that a COTS product can cost effectively meet DC 
Courts' needs. 

* The requirements that had been developed lacked the necessary 
specificity to perform a meaningful gap analysis. Therefore, the time 
spent on attempting to perform the gap analysis before issuing the RFP 
may not have been effective. 

As with any effort, alternative approaches need to be analyzed. In 
this case, DC Courts officials believe that the benefits associated 
with the reduced risks of performing a detailed gap analysis before 
the RFP is issued—having a greater assurance that a COTS product would 
meet their needs—were outweighed by the costs associated with delaying 
this effort. 

During the course of our work, DC Courts officials stated their 
commitment to go forward with this project only after the necessary 
actions have been taken to reduce the risks to acceptable levels. We 
are making recommendations to help ensure that DC Courts adequately
(1) adopts and implements disciplined processes to help ensure that 
its systems development effort is successful, (2) defines the 
requirements necessary to develop its new system in its RFP, and (3) 
improves its ability to assess potential solutions. In commenting on a 
draft of our report, DC Courts agreed with our findings and 
recommendations and said that it has begun to implement our 
recommendations to ensure the successful acquisition of IJIS as 
outlined in our report. 

Background: 

The District of Columbia Court Reform and Criminal Procedure Act of 
1970[Footnote 4] (Court Reform Act) transferred jurisdiction over all 
local judicial matters to a unified court system for the District. 
This entity, known as DC Courts, includes: 

* the Superior Court, which is the trial court with general 
jurisdiction over virtually all local legal matters, including 
criminal, civil, juvenile, domestic relations, probate, and small 
claims cases, and; 

* the Court of Appeals, the highest court of the District of Columbia, 
which reviews all appeals from the Superior Court, as well as 
decisions and orders of D.C. government administrative agencies. 

The Court Reform Act provided for the creation of a policy-making body 
for DC Courts, the Joint Committee on Judicial Administration. The 
joint committee, composed of the two chief judges and three associate 
judges, submits DC Courts' annual budget requests and is responsible 
for DC Courts' general personnel policies, accounting and auditing, 
procurement and disbursement, development and coordination of 
statistical and management information systems and reports, and other 
related administrative matters. The joint committee appoints the 
executive officer, who manages the day-to-day administrative and 
financial management of the court system on the committee's behalf. 

The National Capital Revitalization and Self-Government Improvement 
Act of 1997[Footnote 5] (Revitalization Act) changed DC Courts' 
funding process, nonjudicial employee compensation, and functional 
responsibilities. The Revitalization Act provides for direct federal 
funding of DC Courts. The joint committee submits DC Courts' budget 
request to the Congress through the director of the Office of 
Management and Budget (OMB). In addition, some DC Courts activities 
were transferred to the federal government. 

DC Courts' Effort to Acquire New Information System: 

In October 1998, the Superior Court of the District of Columbia 
launched the HIS project to upgrade and enhance its information 
management capabilities and establish a unified, fully integrated 
computer system that would support data collection and exchange for 
all types of cases processed within the Superior Court. IJIS is a 
multiyear initiative with a two-fold purpose: first, to improve data 
collection and exchange within and across DC Courts' multiple 
divisions, which process different types of cases and provide 
essential support services, such as research and development and 
information technology, and second, to improve interagency data 
collection and exchange among the District's criminal justice 
agencies. Currently, DC Courts information management resources are 
divided among 18 different computer systems that have evolved over the 
past 20 years in response to the differing needs of particular court 
divisions. DC Courts' IJIS initiative is part of a District-wide 
effort to improve the data collections systems and infrastructure of 
District criminal justice agencies and enhance data exchange among 
those agencies. 

The District of Columbia Appropriations Act, 2000,[Footnote 6] 
provided that, of the amounts available for operations of DC Courts, 
an amount not to exceed $2.5 million would be used for the design of 
IJIS. Due to the time needed to prepare a detailed requirement 
analysis to guide the system design, no contract was awarded or funds 
spent during fiscal year 2000. As a result, DC Courts officials 
requested the $2.5 million to design the new system in the fiscal year 
2001 capital budget. 

DC Courts has financed an IJIS requirements analysis through a 
$350,000 grant from the U.S. Department of Justice's Edward Byrne 
Memorial State and Local Law Enforcement Assistance Formula Grant 
Program through a subgrant of the D.C. Office of Justice Grants 
Administration. DC Courts subsequently awarded a contract to conduct 
the IJIS requirements analysis, with the following objectives: (1) 
assess the DC Courts' existing technology infrastructure and systems 
and core business processes supported by these systems, (2) determine 
critical information management needed, and (3) recommend technical 
and business process solutions that would cost effectively meet these 
needs. In September 2000, the contractor issued the final report on 
the requirements analysis, which was conducted from January through 
August 2000. The HIS plan and design prepared by DC Courts is based on 
the requirements analysis conducted by the contractor. 

DC Courts provided its detailed plan and design for the IJIS to the 
chairmen of the Senate and House appropriations committees on May 17, 
2001. On August 20, 2001, we were asked to review the DC Courts' draft 
RFP for the design and implementation of the new system, in 
conjunction with the subcommittee's review of the plan and design, 
before DC Courts began the formal solicitation process. 

Scope and Methodology: 

To carry out our objectives, we reviewed and analyzed: 

* DC Courts' plan and design for the IJIS, including the draft RFP for 
the design and implementation of the new system; 

* DC Courts' annual report; 

* industry automation standards published by the National Consortium 
for Court Functional Standards;[Footnote 7] and; 

* reports produced by the contractor related to (1) DC Courts' 
Integrated Justice Information System Requirements Analysis, (2) 
Business Process Descriptions, Data Flow Diagrams and Entity 
Relationship Diagrams, (3) Inventory of Data Elements, and (4) 
Executive Summary (September 2000). 

We discussed the IJIS project with the following: 

* the chief judge, Superior Court of the District of Columbia; 

* a member of the Technology and Automation Committee; 

* the executive director, DC Courts; 

* the clerk of the court; 

* the director of Information Technology, Information Technology (IT) 
Division, Court System; 

* the information system administrator, IT Division, Court System; 

* other DC Courts personnel; and; 

* representatives from the contractor who prepared the requirements. 

We reviewed and analyzed the draft detailed requirements prepared by 
the contractor and DC Courts personnel and compared them to industry 
standards on selected court activities, such as domestic relations and 
civil cases. We also reviewed the Clinger-Cohen Act of 1996;[Footnote 
8] federal policy governing acquisition efforts, including OMB 
guidance; and guidance and best practice literature[Footnote 9] that 
the Software Engineering Institute (SE1),[Footnote 10] the Project 
Management Institute (PHI),[Footnote 11] and the Institute of 
Electrical and Electronics Engineers (IEEE)[Footnote 12] have issued 
on evaluating information technology investment. 

We did not independently verify or audit the cost data we obtained 
from DC Courts officials. 

Our work was conducted from October 2001 through November 2001 in 
accordance with U.S. generally accepted government auditing standards. 
We requested comments on a draft of this report from the Joint 
Committee on Judicial Administration in the District of Columbia. The 
joint committee provided us with written comments that are discussed 
in the "DC Courts Comments" section and are reprinted in appendix I. 

DC Courts Has Not Implemented the Disciplined Processes Necessary to 
Reduce Risks to Acceptable Levels: 

DC Courts has not implemented the disciplined processes necessary to 
reduce risks associated with managing its system acquisition to 
acceptable levels. However, DC Courts recognizes the importance of 
these processes and plans to implement them once funding for the 
project is available. 

Disciplined processes have been shown to reduce the risks associated 
with software development and acquisition efforts to acceptable levels 
and are fundamental to successful systems acquisition. A disciplined 
software development and acquisition process is needed to maximize the 
likelihood of achieving the intended results (performance) within 
established resources (costs) on schedule. Although a "standard 
cookbook" of practices that will guarantee success does not exist, 
several organizations, such as SEI, PMI, and IEEE, and individual 
experts have identified and developed the types of policies, 
procedures, and practices that have been demonstrated to reduce 
development time and enhance effectiveness. 

SEI and others have documented the positive effects of disciplined 
processes and have developed methodologies that can be used to 
determine whether an organization has such processes. For example, SEI 
first developed the Capability Maturity Model (CMM) to determine an 
organization's ability to develop software and has also developed a 
CMM to assess an organization's ability to acquire software. These
methodologies are designed to determine whether an organization has 
implemented the types of disciplined processes that can lead to lower 
defect rates and help avoid the adverse impacts associated with common 
mistakes. Organizations that have focused on improving their processes 
have found, over time, that they are able to reduce their time to 
market by about one-half and reduce the costs from defects by factors 
of 3 to 10.[Footnote 13] 

The key to having a disciplined system development effort is to have 
disciplined processes in multiple areas. For projects such as the one 
being undertaken by the DC Courts, these include, at this stage in the 
acquisition process, project planning, requirements management, project
management, configuration management, risk management, and testing. 
Effective processes should be implemented in each of these areas 
throughout the project life cycle since constant changes occur. Table 1 
provides a brief description of some of the areas that appear critical 
to the DC Courts' effort.[Footnote 14] 

Table 1: Examples of Disciplined Processes: 

Discipline: Project planning; 
Description: 
Project planning is the process used to establish reasonable plans for 
performing and managing the software project. This includes (1) 
developing estimates of the resources needed for the work to be 
performed, (2) establishing the necessary commitments, and (3) 
defining the plan necessary to perform the work. Effective planning is 
needed to identify and resolve problems as soon as possible when it is 
the cheapest to fix them. According to one author,[A] the average 
system design and implementation project includes about 80 percent of 
its time as unplanned rework—fixing mistakes that were made earlier in 
the project. 

Discipline: Risk management; 
Description: 
Risk management is a set of continual activities for identifying, 
analyzing, planning, tracking, and controlling risks. Risk management 
starts with identifying the risks before they become problems. If this 
step is not performed well, the entire risk management process may 
become a useless exercise since a process cannot be managed without 
information about that process. As with the other disciplined 
processes, risk management is designed to eliminate the effects of 
undesirable events at the earliest possible stage to avoid the costly 
consequences of rework. 
After the risks are identified, they need to be analyzed so that they 
can be better understood and decisions can be made about what actions, 
if any, will be taken to address the risks. Basically, this step 
includes such activities as evaluating the impact on the project if a 
risk does occur, determining the probability of the event occurring, 
grouping like risks together, and prioritizing the risk against the 
other risks. Once the risks are analyzed, a risk management plan is 
developed that outlines the information known about the risks and the 
actions, if any, that will be taken to mitigate those risks. Risk 
management is a continual process because the risks and actions 
planned to address those risks need to be monitored to ensure that the 
risks are being properly controlled and that new risks are identified 
as early as possible. If the actions envisioned in the plan are not 
adequate, then additional controls are needed to correct the 
deficiencies identified. 

Discipline: Testing; 
Description: 
Testing is the process of executing a program with the intent of 
finding errors.[B] Disciplined system development activities recognize 
that testing will not find all defects even though well-designed and 
executed testing programs are used. For example, testing performed 
through the system testing phase often catches less than 60 percent of 
a program's defects.[C] Consequently, testing alone cannot be relied 
on to identify all defects. 

[A] Steve McConnell, Software Project Survival Guide (Redmond, Wash.: 
Microsoft Press, 1998). 

[B] Glendford J. Myers, The Art of Software Testing (New York, N.Y.: 
John Wiley and Sons, Inc., 1979). 

[C] See footnote 13. 

[End of table] 

DC Courts officials agreed that they had not yet fully implemented the 
disciplined processes necessary to reduce the risks associated with 
this project to acceptable levels. They said that they recognized that 
the implementation of the disciplined processes was needed not only 
for this project but for other projects as well, and that effective 
implementation would take time, management commitment, and funding. 
They also noted that they had begun the process of identifying the 
needed funding and staffing levels and that they were sending several 
of the project managers to training so that they could become 
certified project managers, which will facilitate their knowledge of 
disciplined processes. 

DC Courts System Requirements as Specified in RFP Are Inadequate: 

Requirements represent the blueprint that system developers and 
program managers use to design, develop, and acquire a system. 
Requirements should be consistent with one another, verifiable, and 
directly traceable to higher-level business or functional 
requirements. It is critical that requirements be carefully defined 
and that they flow directly from the organization's concept of 
operations (how the organization's day-to-day operations are or will 
be carried out to meet mission needs). Improperly defined or 
incomplete requirements have been commonly identified as a root cause 
of system failure and systems that do not meet their costs, schedules, 
or performance goals. Without adequately defined requirements that 
have been properly reviewed and validated, significant risk exists 
that the system will need extensive and costly changes before it will 
meet the DC Courts' needs. 

DC Court system requirements set forth in the draft RFP lacked the 
necessary specificity to ensure that the defects in these requirements 
have been reduced to acceptable levels and that the system would meet 
its users' needs. In addition, the terms "customization" and 
"modification," though used in the RFP, were not clearly defined. 
Also, industry standards that appeared to be related to the DC Courts 
were either not used or were summarized so that the detail provided in 
the standard was omitted from the DC Courts' requirements. We also 
noted that the requirements were not logically organized—related 
requirements were not grouped together. Better grouping of 
requirements would help the DC Courts and potential contractors in 
assessing whether the RFP's requirements adequately describe the 
functionality necessary to conduct court business. Because DC Courts' 
requirements were not specific, were poorly organized, and did not 
directly relate to industry standards, a significant potential exists 
that the DC Courts' proposed system may not meet its business needs or 
may not be delivered on schedule and within budget if corrective 
action is not taken. 

Requirements Management: A Key Process for Quantifying a System's 
Purpose and Function: 

Requirements management is a process that establishes a common 
understanding between the customer and the software project manager 
regarding the customer's business needs that will be addressed by a 
project!' A critical part of this process is to ensure that the 
requirements development portion of the effort documents, at a 
sufficient level of detail, the problems that need to be solved and 
the objectives that need to be achieved. 

Good requirements have several common characteristics:[Footnote 16] 

* They fully describe the software functionality to be delivered. 
Functionality is a defined objective or characteristic action of a 
system or component. For example, a system may have inventory control 
as its primary functionality.[Footnote 17] 

* They provide the source of the requirement. For instance, the 
citation of the statute, regulation, or industry standard should be 
shown so that others can evaluate the applicability of the requirement 
and better understand the impacts of changes in the requirement. 

* They state the requirement in clear terms that allow for 
quantitative evaluation. Specifically, all readers of a requirement 
should arrive at a single, consistent interpretation of it. 

Studies have shown that problems associated with requirements 
definition are key factors in software projects that do not meet their 
cost, schedule, and performance goals. For example, see the following: 

* A study found that getting a requirement right in the first place 
costs 50 to 200 times less than waiting until after the system is 
implemented to get it right.[Footnote 18] 

* A survey of more than 8,000 software projects found that the top 
three reasons that projects were delivered late, over budget, and with 
less functionality than desired all had to do with requirements 
management.[Footnote 19] 

* A study found that the average project experiences about a 25-
percent increase in requirements over its lifetime, which translates 
into at least a 25-percent increase in the schedule.[Footnote 20] 

* A study noted that between 40 and 60 percent of all defects found in 
a software project could be traced back to errors made during the 
requirements development stage.[Footnote 21] 

Requirements Were Not Specific: 

The majority of the requirements in the draft RFP lacked the necessary 
specificity to ensure that the defects in the requirements had been 
reduced to acceptable levels. Acceptable levels refer to the fact that 
any systems acquisition effort, such as that being undertaken by the 
DC Courts, will have some requirements-related defects. However, the 
goal is to reduce the risks and prevent significant requirements 
defects in order to limit the negative impact of these defects on the 
cost, timeliness, and performance of the project. The lack of 
specificity of the requirements contained in the draft RFP indicate 
that a significant number of requirements-related defects are present 
in this document. These requirements-related defects significantly 
increase the likelihood that the resulting system will not meet DC 
Courts' cost, schedule, and performance goals. In addition, the risk 
exists that the DC Courts may have failed to include critical 
requirements because of the lack of specificity. Omission of critical 
requirements will also adversely affect cost, schedule, or performance 
goals. As noted elsewhere, DC Courts officials have agreed with our 
assessment and have started taking actions to address these weaknesses. 

DC Courts' requirements lacked the specific information necessary to 
understand the required functionality that a vendor should provide and 
how to determine quantitatively, through testing or other analysis, 
whether a vendor's product would adequately address the DC Courts' 
needs. For examples, see the following. 

* One requirement stated that the "system must accept e-filing 
documents and case management data in XML[Footnote 22] standard format 
and then update the case management system with the case filing data, 
case filing fees, and store the document." Deficiencies in this 
requirement include (1) the XML "standard" format is not defined, (2) 
the business rules that should be used to compute the filing fees are 
not referenced, and (3) the response time and capacity (for example, 
number and size of documents) was not specified. 

* Another requirement stated that the system must have the "ability to 
link [a] case with other related cases having the same case 
participants or family unit, because the overall concept in the family 
court is one family, one judge." However, the document did not specify 
such items as (1) the individuals that should be considered part of 
the family unit (for example, mother, father, stepmother, aunt, 
sister, nephew, guardian, etc.) or (2) what are considered related 
cases (for example, criminal, civil, probate, etc.). 

* A user requirement stated that the system must have the "ability to 
move quickly between screens. Must have the ability to change screen 
navigation per each individual user requirement." However, the term 
"quickly" was not defined nor was there a description of how users 
should define the screen navigation. 

* Another requirement stated that the system must have the "ability to 
electronically notify users based on user defined rules when [a] user 
defined event has or has not occurred in a timely manner by either FAX 
or e-mail." Ambiguities in this requirement include (1) the rules that 
users can define, (2) the functional method that a user applies to 
define a rule, (3) whether the rules are established by each user or 
whether "corporate" rules will be defined, (4) what is considered 
"timely," and (5) who are considered users (that is, the reason why DC 
Courts personnel receive faxes instead of e-mail from their own 
system). 

Another important aspect of requirements definition includes defining 
key terms. In its RFP, the DC Courts did not clearly define the terms 
"customization" and "modification," nor did it require that the vendor 
communicate the cost, schedule, and performance impacts of 
implementing a customization or modification associated with a given 
requirement. 

The terms "customization" and "modification" may be confusing since a 
basic premise is that COTS solutions should not be customized or 
modified. The terms "customization" and "modification," as used in 
this report, are differentiated from the basic installation processes 
that are required for systems such as HIS. Those processes are 
commonly referred to as installation and configuration. For purposes 
of this report, we are defining customization and modification as 
follows: 

* Customization: The process of setting parameters within the 
application to make it operate in accordance with the entity's 
business rules. Customizations are normally supported by the vendor in 
subsequent upgrades as part of the normal upgrade process. 

* Modification: The process of writing or changing code. Modifications 
are not generally supported by the vendor in subsequent upgrades as 
part of the normal upgrade process. 

A customization in one package may be considered a modification in 
another package. For example, the architecture of one product may 
allow the entity to develop a report tailored to its needs in such a 
manner that, when the system is upgraded, the report will still be 
available without additional development work. However, the same 
report in another system may need to be rewritten when the system is 
upgraded. Defining these terms is important to ensure that the 
benefits associated with any customizations or modifications are cost-
effective and that alternatives are not available. 

Unclear Relationship to Industry Standards: 

In a number of cases the requirements in the DC Courts' draft RFP did 
not directly relate to industry standards published by the consortium. 
The consortium has been tasked with developing guidelines that will 
help state courts more effectively use their financial and staffing 
resources to obtain state-of-the-art computer systems—either through 
in-house development or through procurement from software developers. 
In doing so, the consortium has focused on ways in which the state 
courts can: 

* reduce the time needed to obtain a new computer system, 

* improve work processes, and, 

* reduce staffing requirements. 

Recognizing the state courts' need for functional standards in 
computer systems and the staffing limitations that exist in most state 
courts, the consortium has developed a set of functional standards for 
developing new computer systems. Courts nationwide would use these 
standards to define functional requirements for in-house systems 
development and RFPs for vendor-supplied systems. Each court must 
customize the standards and add terminology, details, and specificity 
based on local and state procedures, policies, and customs. 

DC Courts officials and the contractor told us that the requirements 
in the DC Courts' draft RFP were based on the published standards 
developed by or currently under development by the consortium. 
However, for many of the requirements in the draft RFP, there was no 
direct link, or cross-referencing, to the standards. For example, one 
standard calls for generating a receipt and assigning a case number 
when a case file is received, but the draft RFP requirement does not 
track back to the industry standard and only speaks vaguely of 
notifying users and generating receipts. Table 2 compares one of the 
standards to some of the draft RFP's requirements. 

Table 2: Comparison of Industry Standard and RFP's Requirement: 

Industry standard: 
Generate receipt for or notify appropriate parties that case filing 
was received and accepted, and give them assigned case number (notice, 
including electronic acknowledgment, would apply primarily when a case 
is transferred from another jurisdiction or filed electronically).
Draft RFP requirement: 
Ability to electronically notify users based on user-defined rules 
when user defined event has or has not occurred in a timely manner by 
either FAX or e-mail. 
Ability to automatically generate a receipt upon the filing of a 
notice of appeal stating the case number, date, and time of filing. 
System should not allow a case to proceed until filing fee is collected.
Ability to issue a sequentially numbered receipt for the payment of 
funds on a specific case. 

[End of table] 

We were also unable to identify in the draft RFP items contained in 
the industry standards that applied to the DC Courts. For example, one 
standard required the system to "generate locally defined case title 
or style ( i.e., a short phrase that identifies case and includes 
plaintiff and defendant names) from party names and other 
information." We were not able to readily identify a related 
requirement in the draft RFP. 

Regardless of whether the draft RFP contained all the industry 
standards that were applicable, the real key is that the draft RFP did 
not provide adequate information so that the prospective vendors and 
others could readily map systems built upon these standards to the 
needs of the DC Courts. This lack of vital information could lead to 
adverse impacts as vendors attempt to rework or develop functions that 
were not clearly understood initially. This, in turn, could lead to 
cost increases, schedule delays, and reduced performance. 

Poor Organization Prevents Requirements Validation: 

Requirements should be organized logically to facilitate understanding 
and requirements validation efforts. The organization of the 
requirements in the draft RFP hampers the DC Courts' and potential 
contractors' ability to understand and validate the requirements since 
it is very difficult to determine where all the requirements related 
to a given functionality are documented. Because the requirements were 
not logically organized, future validation efforts to identify missing 
or duplicate requirements may not be effective. 

Requirements validation reduces requirements-related defects by first 
assessing whether the requirement document actually satisfies the 
project's top-level requirements. The verification process includes 
ensuring that the: 

* requirements document describes the intended system behaviors and 
characteristics; 

* requirements are correctly derived from requirements obtained from 
other sources, for example, if the requirements document states that a 
given statute requires a certain function, then an effort is 
undertaken to ensure that the requirement correctly represents the 
specified statute; 

* requirements are complete, consistent, and of high quality; and; 

* requirements provide an adequate basis to proceed to the next stage 
of system development. 

Requirements validation is intended to help ensure that the 
requirements are complete, correct, feasible, necessary, prioritized, 
unambiguous, and verifiable.[Footnote 23] The organization of the 
requirements in the draft RFP will likely hinder such a process. For 
example, see the following: 

* One section, entitled "Case Financial Activity Requirements," 
contained many of the financial-related requirements such as preparing 
a monthly trial balance and balance sheet. However, another section, 
entitled "Reporting," contained the requirement to produce a Statement 
of Revenue and Expenses. Generally, all requirements related to 
financial statement reporting would be contained in the same section. 

* The requirements contained in the "Case Financial Activity 
Requirements" section were not organized in a functional manner. 
Disbursement requirements were followed by adjusting entry 
requirements, which were followed by additional disbursement 
requirements, which were followed by receipt requirements, which were 
followed by additional disbursement requirements. In an appropriately 
organized requirements document, all related requirements would be 
grouped together. 

* Bar coding requirements for court documents were contained in at 
least three different sections rather than all in the same section. 

DC Courts Has a Reasonable Basis to Assume a Commercial Product Will 
Meet Its Needs: 

The DC Courts intends to use the acquisition process to identify any 
potential gaps between its system requirements and the standard 
features available in a given vendor's product instead of performing a 
detailed gap analysis before the RFP is issued. Although this approach 
can increase risk, we believe the DC Courts' decision to use the 
acquisition process to identify the cost, schedule, and performance 
gaps is acceptable in this case because of the following mitigating 
factors. 

* DC Courts officials concluded, based on a qualitative analysis, that 
they do not have unique requirements that would prevent the 
utilization of a COTS product. Furthermore, they recognized that a gap 
analysis must be performed as part of the vendor selection process to 
identify the cost, schedule, and performance impacts of each vendor's 
product before deciding which product to acquire. 

* DC Courts is still in the early stages of the system acquisition 
effort and has not yet reached the critical point of contract award. 
Therefore, a gap analysis can still be performed prior to the system 
acquisition to ensure that a COTS product can cost effectively meet DC 
Courts' needs. 

* As noted previously, the requirements that had been developed lacked 
the necessary specificity to perform a meaningful gap analysis. 
Therefore, the time spent on attempting to perform the gap analysis 
before issuing the RFP may not have been effective. 

As with any effort, alternative approaches need to be analyzed. In 
this case, DC Courts officials believe that the benefits associated 
with the reduced risks of performing a detailed gap analysis before 
the RFP is issued—having a greater assurance that a COTS product would 
meet their needs—were outweighed by the costs associated with delaying 
this effort. 

A critical step in acquiring a new system is determining whether the 
system requirements can be met by using commercially available 
systems, commonly referred to as COTS systems. If COTS systems cannot 
meet the majority of the requirements, then the acquirer will need to 
undertake a system development effort rather than using a COTS 
product. To make this determination, the agency must perform a gap 
analysis that systematically and quantitatively compares and contrasts 
the vendors' products against the agency's requirements based on 
functional, technical, and cost differences. 

Two different approaches can be taken to perform this gap analysis. 
One approach is to perform a gap analysis on the detailed requirements 
to determine whether they need to be modified before issuing the RFP 
for acquiring a system. A second approach is to structure the 
acquisition process so that the vendor identifies the gaps as well as 
the cost, schedule, and performance impacts of addressing those gaps. 
Each approach has its advantages and disadvantages. For example, if a 
detailed gap analysis is performed before the acquisition process 
begins, and it is later determined that a COTS product could have cost 
effectively met the agency's needs, then time and money have been 
wasted performing an analysis that, in effect, will be repeated during 
the acquisition process. On the other hand, if the second approach is 
taken and the gap analysis discloses that a COTS product cannot cost 
effectively meet the agency's needs, then the acquisition process will 
need to be restructured to support a system development effort, which 
translates into schedule delays and additional costs. 

DC Courts officials selected the second approach—using the acquisition 
process to identify their project's cost, schedule, and performance 
gaps. According to these officials, they selected this approach 
because they had: 

* reviewed a number of vendors' products and believed they had a good 
understanding of the general capabilities of the marketplace; 

* hired a contractor that was knowledgeable of the court environment 
to help them understand whether a COTS product would meet their needs; 

* determined that, based on interactions with other courts of similar 
size and workload, the DC Courts can successfully use comparable 
practices; and; 

* already committed to modifying their business processes to reflect 
the selected COTS product's capabilities unless a significant reason, 
such as a legal requirement, dictated otherwise. 

In discussing this issue with DC Courts officials, they acknowledged 
that the approach they were taking increased risks; however, they 
believe that the benefit obtained—accelerating the implementation of a 
badly needed system—justifies those risks. 

Actions Taken by DC Courts to Address Our Concerns: 

In late October 2001, we discussed our findings on the above three 
areas with DC Courts officials. They generally agreed with our 
findings and restated their commitment to only go forward with this 
project when the necessary actions had been taken to reduce the risks 
to acceptable levels. They specifically agreed to perform the 
following actions: 

* Delay the procurement of the new system until the requirements can 
be revised to provide assurance that defects related to the 
requirements are kept to acceptable levels. They have begun the 
process of selecting a contractor that will be responsible for 
developing a new requirements document. They also stated that the 
contractor would be responsible for documenting the requirements so 
that each requirement (1) fully describes the system functionality to 
be delivered, (2) includes the source of the requirement, and (3) is 
stated in unambiguous terms that allow for quantitative evaluation. 

* Develop a plan that can be used to guide DC Courts' effort to 
implement the necessary disciplined processes. This includes 
identifying the needed skills, determining the best approach to 
acquiring the needed skills through training of DC Courts' staff or 
through contractors, and securing adequate funding. 

Conclusions: 

DC Courts' planned actions and those taken thus far to address our 
findings are a positive step forward and, if effectively implemented, 
will help reduce the risks associated with this effort to acceptable 
levels. We are reaffirming these actions in our recommendations. 
Although the DC Courts has a reasonable basis for believing that a 
COTS product will meet its needs, it has not yet defined the 
requirements adequately or implemented the disciplined processes 
necessary to reduce the risks to the project to acceptable levels—that 
is, to reduce the risks associated with disciplined processes and 
prevent significant requirements-related defects in order to limit the 
negative impact on the cost, timeliness, and performance of the 
project. DC Courts has stated its commitment to correcting the 
deficiencies in its requirements as well as performing a gap analysis 
during the preliminary phases of its acquisition project before it 
commits large amounts of resources. If properly implemented, these 
steps should serve to reduce overall project risk and reduce the 
likelihood that extensive and costly corrections will be needed later. 

Recommendations: 

To help ensure that the DC Courts reduces risks associated with its 
systems development and implementation and increase the chances of a 
successful effort, we recommend that the Joint Committee on Judicial 
Administration in the District of Columbia take the following actions: 

* Develop a plan on how it will implement the disciplined processes 
necessary to reduce the risks associated with this effort to 
acceptable levels. This plan should include the processes, such as 
those identified by SEI and IEEE, that will be implemented and the 
resources, such as staffing and funding, needed to implement the 
necessary processes effectively. 

* Develop requirements that contain the necessary specificity to 
reduce requirements-related defects to acceptable levels and add them 
to the draft RFP. The requirements management process used to develop 
and document the requirements should be adequate to ensure that each 
requirement (1) fully describes the functionality to be delivered, (2) 
includes the source of the requirement, and (3) is stated in 
unambiguous terms that allow for quantitative evaluation. 

* Organize the requirements document to facilitate the requirements 
validation process used by disciplined organizations. 

* Ensure that the acquisition process is adequate to (1) clearly 
define the terms customization and modification and (2) ensure that 
vendor responses clearly communicate the cost, schedule, and 
performance impacts of implementing a customization or modification 
associated with a given requirement. 

* Evaluate the cost, schedule, and performance impacts of the 
customizations and modifications identified during the system 
acquisition process and ensure that the benefits are cost-effective 
and that alternatives to customization or modification are not 
available. 

DC Courts Comments: 

In commenting on a draft of our report, DC Courts agreed with our 
findings and recommendations and said that it has begun to implement 
our recommendations to ensure the successful acquisition of IJIS as 
outlined in our report. Specifically, DC Courts said that it is 
implementing the disciplined processes critical to successful systems 
acquisition. DC Courts also noted that it has a project under way to 
institute the necessary disciplined processes for the entire system 
development life cycle (SDLC). DC Courts further noted that it is 
contracting with subject matter experts in both the SDLC disciplines 
and court operations to increase the specificity in the RFP and to 
reduce the risks from requirement-related defects to acceptable 
levels. DC Courts provided additional details on the actions taken to 
address the findings and recommendations included in our report. If 
these actions are successfully implemented, they will address our 
concerns and help ensure the success of the DC Courts' HIS 
acquisition. The DC Courts' comments are reprinted in appendix I. 

We are sending copies of this report to the ranking minority member of 
your subcommittee and to other interested congressional committees. We 
are also sending copies to the Joint Committee on Judicial 
Administration in the District of Columbia, the chief judge of the 
Superior Court of the District of Columbia, and the executive director 
of the District of Columbia Courts. Copies of this report will also be 
made available to others upon request. 

Please contact me at (202) 512-9406 or by e-mail at franzelj@gao.gov 
if you or your staff have any questions concerning this report. Key 
contributors to this report were Linda Elmore, John C. Martin, Meg 
Mills, and Norma Samuel. 

Sincerely yours, 

Signed by: 

Jeanette M. Franzel: 
Acting Director, Financial Management and Assurance: 

[End of section] 

Appendix I: Comments from the District of Columbia Courts: 

District of Columbia Courts: 
Anne B. Wicks: 
Executive Officer: 
500 Indiana Avenue, NW, Room 1500: 
Washington, DC 20001-2131: 
202-879-1700: 
202-879-1829, Fax: 

February 8, 2002: 

Jeannette M. Franzel: 
Director, Financial Management and Assurance: 
United States General Accounting Office: 
Washington, DC 20548: 

Dear Ms. Franzel: 

In accordance with your request of January 25, 2002, I enclose for 
your consideration comments on behalf of the District of Columbia 
Courts to the draft report entitled District of Columbia Courts: 
Disciplined Processes Critical to Successful System Acquisition. 

The District of Columbia Courts have carefully reviewed the GAO report 
findings and are implementing the recommendations to ensure the 
successful acquisition of the Integrated Justice Information System 
(MS). If you have further questions or concerns, please contact me at 
(202) 879-1700. 

Sincerely, 

Signed by: 

Anne B. Wicks: 
Executive Officer: 
D.C. Courts: 

Attachment: 

[End of letter] 

Comments Of The D.C. Courts: 

According to the General Accounting Office (GAO), an initial review 
was conducted of the DC Courts' effort to acquire a new information 
system to determine whether: 

1) DC Courts implemented the disciplined process for this report to 
reduce the risk associated with this effort to acceptable levels; 

2) DC Courts' requirements to acquire the system, as identified in its 
draft RFP, contain the necessary specificity to reduce the risks from 
requirement defects to acceptable levels; and; 

3) DC Courts performed the necessary actions to determine that a 
commercial off-the-shelf system (COTS) would meet its needs. 

Court Summary In Brief: 

1) The DC Courts have begun to implement the disciplined processes 
critical to successful system acquisition as outlined in the draft GAO 
report. Additionally, a project is underway which will institute the 
necessary disciplined processes for the entire system development life 
cycle (SDLC), especially for the acquisition, installation, testing 
and acceptance of a new major system. 

2) The DC Courts are contracting subject matter experts in both the 
SDLC disciplines and court operations, to increase the specificity in 
the RFP and to reduce the risks from requirement defects to acceptable 
levels. State of the art technology software for requirements 
management will be utilized. 

Finding: DC Courts have not implemented the disciplined processes 
necessary to reduce risks to acceptable levels. 

Court Response: As indicated in the GAO report, the Court recognizes 
the need for disciplined processes and plans on implementing such 
processes once funding for the integrated justice information system 
(HIS) project is available. In an effort to reduce the risks to 
acceptable levels, two measures are currently underway: 

* At the time of the GAO review, Court staff from the Information 
Technology Division, Administrative Services Division, and the Budget 
and Finance Division were attending a project management course 
offered through the Boston University Management Certificate Program. 
In December 2001 ten staff received their Project Management 
certificates. Another team of personnel from the Courts are enrolled 
this semester. Additionally, senior court managers will receive 
project management training conducted by the Institute for Court 
Management from February 27 through March 1, 2002, further 
demonstrating the Court's commitment to implementing the disciplined 
processes necessary to reduce risks to acceptable levels. 

* The Information Technology Division began an analysis of software 
tools for use in the discipline of specification management, change 
management, risk management and testing. The Court selected the 
Rational Suite Enterprise Studio as the tool to attain the Capability 
Maturity Model (CMM). Using the Rational Unified Process software the 
Court will begin to move from Level 1 (Ad hoc) to Level 2 — Repeatable 
and Level 3 — Defined maturity levels of the Software Engineering 
Institute's (SEI) Capability Maturity Model (CMM). Also, the Court has 
solicited proposals for consulting assistance to install the software, 
train staff, and develop the necessary implementation procedures. 
Throughout the acquisition and life cycle of the project, the Court 
will use the Rational Unified Process software. 

Finding: DC Courts system requirements as specified in RFP are 
inadequate. 

Court Response: In accordance with this finding, the Court has taken 
immediate steps to adequately define the system requirements so that 
they will contain the necessary specificity and more closely relate to 
the Court's business operations. The National Center for State Courts, 
our original contractor, will assist in the refinement of the systems 
requirement. To mitigate risks, the Court will organize the system 
requirements logically and relate the requirements to industry 
standards. 

Finding: Unclear Relationship to Industry Standards. 

Court Response: The Court will revise the Request for Proposals (RFP) 
utilizing the Case Management Functional Standards produced by the 
National Center for State Courts (NCSC). With the assistance of the 
NCSC, our original contractor, the Court will revise the RFP to 
include more specificity, including direct linkages, or cross-
referencing to industry standards. The tasks that will be conducted to 
achieve industry standards follows: 

I. Obtain consulting assistance to draft a working document (initial 
functional specifications) for users sessions. This would utilize 
information from the current RFP, Volumes 1-4 of the business and 
technology analysis and the current draft and final Case Management 
(CM) Functional Standards produced by NCSC. 

II. Conduct user sessions to review the working document with the 
various court divisions. 

III. Draft a functional requirements document, using the requirements 
specification tools which also assists in the cross referencing of 
specifications to requirements and organizes the information to 
facilitate the requirements validation process. 

IV. Revise RFP using new functional requirements document. 

V. Revise the acquisition process to ensure adequacy of clearly 
defined terms such as customization and modification so that vendor 
responses clearly communicate the cost, schedule, and performance 
impacts of implementing a customization or modification associated 
with a given requirement. 

VI. Conduct a quality review by the NCSC and other consultant 
specialists. 

VII. Review comments by the reviewers to the Draft RFP. 

VIII. Finalize RFP. 

During the vendor response evaluation process, the Court will evaluate 
the cost, schedule, and performance impacts of any customizations or 
modifications identified during the system acquisition process to 
ensure that the benefits are cost effective and that alternatives to 
customization or modification are not available. 

[End of section] 

Footnotes: 

[1] Public Law No. 106-522, 114 Stat. 2440, 2442 (2000). 

[2] An RFP is a formal request for vendors to provide a solution to a 
stated problem, in this case, a system to handle DC Courts' management 
information needs. 

[3] Although all projects of this size can be expected to have some 
requirements-related defects, the goal is to reduce the number of such 
defects so that they do not significantly affect cost, schedule, or 
performance. 

[4] Public Law No. 91-358, 84 Stat. 473 (1970). 

[5] Public Law No. 105-33, Title XI, 111 Stat. 251, 712 (1997). 

[6] Public Law No. 106-113, 113 Stat. 1501, 1502 (1999). 

[7] The consortium is a subgroup of the Conference of State Court 
Administrators and the National Association for Court Management Joint 
Technology Committee. Its goal is to develop functional standards for 
case management information systems for civil, domestic relations, 
criminal, juvenile, probate, and traffic cases. 

[8] Public Law 104-106. The Clinger-Cohen Act requires agencies to 
analyze their missions and, based on the analysis, revise mission-
related and administrative processes, as appropriate, before making 
significant investments in information technology used to support 
those missions. 

[9] U.S. General Accounting Office, Assessing Risks and Returns: A 
Guide for Evaluating Federal Agencies' IT Investment Decision-Making, 
[hyperlink, http://www.gao.gov/products/GAO/AIMD-10.1.13] (Washington, 
D.C., 1997); U.S. General Accounting Office, Executive Guide: Creating 
Value Through World-class Financial Management, [hyperlink, 
http://www.gao.gov/products/GAO/AIMD-00-134] (Washington, D.C.: 2000); 
and U.S. General Accounting Office, Information Technology: An Audit 
Guide for Assessing Acquisition Risks, [hyperlink, 
http://www.gao.gov/products/GAO/IMTEC-8.1.4] (Washington, D.C.: 1992). 

[10] SEI is recognized for its experience in software development and 
acquisition processes. It has also developed methods and models that 
can be used to define disciplined processes and determine whether an 
organization has implemented them. SEI's stated mission is to provide 
leadership in advancing the state of the practice of software 
engineering and to improve the quality of systems that depend on 
software. 

[11] PMI provides global leadership in the development of standards 
for the practice of the project management profession throughout the 
world. PMI's standards document, A Guide to the Project Management 
Body of Knowledge (PMBOK® Guide), is a globally recognized standard 
for managing projects in today's marketplace. The PMBOK® Guide is 
approved as an American National Standard by the American National 
Standards Institute. 

[12] Institute of Electrical and Electronics Engineers, Transactions 
on Software Engineering (IEEE Transactions on Software Engineering, 
volume 14, number 10 1988). 

[13] Steve McConnell, Rapid Development: Taming Wild Software 
Schedules (Redmond, Wash.: Microsoft Press, 1996). 

[14] A list of other processes necessary to acquire or develop systems 
can be obtained from SEI at [hyperlink, http://www.sei.cmu.edu]. 

[15] Carnegie Mellon University Software Engineering Institute, The 
Capability Maturity Model: Guidelines for Improving the Software 
Process (Reading, Mass.: Addison Wesley Longman, Inc., 1995). 

[16] Karl E. Wiegers, Software Requirement: Practical techniques for 
gathering and managing requirements throughout the product development 
cycle (Redmond, Wash.: Microsoft Press, 1999). 

[17] IEEE 610.12-1990. 

[18] Barry W. Boehm and Philip N. Papaccio, Understanding and 
Controlling Software Costs (IEEE Transactions on Software Engineering, 
volume 14, number 10 1988). 

[19] The Standish Group, Charting the Seas of Information Technology 
(Dennis, Mass.: The Standish Group, 1994). 

[20] Capers Jones, Assessment and Control of Software Risks (Englewood 
Cliffs, NJ.: Yourdon Press, 1994). 

[21] Dean Leffingwell, Calculating the Return on Investment from More 
Effective Requirements Management (American Programmer, 1997). 

[22] The Extensible Markup Language (XML) is a flexible, 
nonproprietary set of rules for tagging information so that it can be 
transmitted using Internet protocols and processed by disparate 
computer systems. 

[23] Karl E. Wiegers. 

[End of section] 

GAO’s Mission: 

The General Accounting Office, the investigative arm of Congress, 
exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and 
accountability of the federal government for the American people. GAO 
examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO’s commitment to good government is reflected in its 
core values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO’s Web site [hyperlink, 
http://www.gao.gov] contains abstracts and full text files of current 
reports and testimony and an expanding archive of older products. The 
Web site features a search engine to help you locate documents using 
key words and phrases. You can print these documents in their 
entirety, including charts and other graphics. 

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as “Today’s Reports,” on 
its Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
[hyperlink, http://www.gao.gov] and select “Subscribe to daily E-mail 
alert for newly released products” under the GAO Reports heading. 

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are 
$2 each. A check or money order should be made out to the 
Superintendent of Documents. GAO also accepts VISA and Mastercard. 
Orders for 100 or more copies mailed to a single address are 
discounted 25 percent. Orders should be sent to: 

U.S. General Accounting Office: 441 G Street NW, Room LM: 
Washington, D.C. 20548: 

To order by Phone: 
Voice: (202) 512-6000: 
TDD: (202) 512-2537: 
Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: fraudnet@gao.gov: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Public Affairs: 

Jeff Nelligan, managing director, 
NelliganJ@gao.gov: 
(202) 512-4800: 
U.S. General Accounting Office: 
441 G Street NW, Room 7149:
Washington, D.C. 20548: