This is the accessible text file for GAO report number GAO-06-11 
entitled 'Information Technology: HHS Has Several Investment Management 
Capabilities in Place, but Needs to Address Key Weaknesses' which was 
released on November 28, 2005. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to the Chairman, Committee on Finance, U.S. Senate: 

October 2005: 

Information Technology: 

HHS Has Several Investment Management Capabilities in Place, but Needs 
to Address Key Weaknesses: 

[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-11]: 

GAO Highlights: 

Highlights of GAO-06-11, a report to the Chairman, Committee on 
Finance, U.S. Senate: 

Why GAO Did This Study: 

The Department of Health and Human Services (HHS) is one of the largest 
federal agencies, the nation’s largest health insurer, and the largest 
grant-making agency in the federal government. The department manages 
over 300 programs that serve to improve the health and well-being of 
the American public. To support these programs, the department funds 
numerous information technology (IT); in fiscal year 2006, it plans to 
spend over $5 billion on IT. GAO was asked to evaluate HHS’s processes 
for making IT investment management decisions. Specifically, the 
objectives of this review were to (1) assess the department’s 
capabilities for managing its IT investments and (2) determine what 
plans, if any, the department might have for improving those 
capabilities. 

What GAO Found: 

Judged against the criteria of GAO’s framework for information 
technology investment management (ITIM), which measures the maturity of 
an organization’s investment management processes, HHS has established 
63 percent of the foundational practices that it needs to manage its IT 
investments individually; and 30 percent to manage its investments as a 
portfolio (see table below). Specifically, HHS has implemented 
processes to ensure that projects support business needs and meet 
users’ requirements, established a process for selecting investments, 
and has created portfolio selection criteria. However, weaknesses 
remain in several areas. The department’s senior investment board does 
not regularly review component agencies’ IT investments, leaving close 
to 90 percent of its discretionary investments without an appropriate 
level of executive oversight. In addition, HHS does not evaluate the 
performance of its portfolio on a continuing basis or conduct 
postimplementation reviews. Finally, HHS currently has no structured 
mechanism in place to ensure that the component agencies are defining 
and implementing investment processes that are aligned with those of 
the department. Until HHS establishes the practices it needs to 
effectively manage its IT investments, executives cannot be assured 
that they are appropriately selecting, managing, and evaluating the mix 
of investments that will maximize returns to the organization, taking 
into account the appropriate level of risk. 

HHS has initiated efforts to improve its investment management 
processes, but has not coordinated these and additional efforts that 
would be needed to address the weaknesses we identify in a 
comprehensive plan that defines and prioritizes improvements to the 
investment process. Such a plan is instrumental in helping HHS to 
coordinate and guide its improvement efforts and sustain its commitment 
to the efforts already under way. Without such a plan and procedures 
for implementing it, the department risks being unable to effectively 
establish mature investment management capabilities. As a result, 
executives may not be able to make informed and prudent investment 
decisions in managing HHS’s multibillion-dollar IT budget. 

HHS’s Current IT Investment Management Capabilities: 

Stage 2: Building the investment foundation 
Percentage of key practices executed: 

Stage 2: Building the investment foundation
Instituting the investment board; 
Percentage of key practices executed: 63 

Stage 2: Building the investment foundation 
Meeting business needs;
Percentage of key practices executed: 100. 

Stage 2: Building the investment foundation 
Selecting an investment; 
Percentage of key practices executed: 70. 

Stage 2: Building the investment foundation 
Providing investment oversight; 
Percentage of key practices executed: 0. 

Stage 2: Building the investment foundation 
Capturing investment information; 
Percentage of key practices executed: 83. 

Stage 2: Building the investment foundation 
Overall Percentage of key practices executed: 63.

Stage 3: Developing a complete investment portfolio
Conducting postimplementation reviews; 
Percentage of key practices executed: 0.

Stage 3: Developing a complete investment portfolio
Evaluating the portfolio; 
Percentage of key practices executed: 0.

Stage 3: Developing a complete investment portfolio
Creating the portfolio;
Percentage of key practices executed: 43.

Stage 3: Developing a complete investment portfolio
Defining the portfolio criteria; 
Percentage of key practices executed: 71.

Stage 3: Developing a complete investment portfolio 
Overall Percentage of key practices executed: 30. 

Source: GAO. 

[End of Table]

What GAO Recommends: 

To strengthen HHS’s investment management capability, GAO recommends 
that HHS develop and implement a plan to address the weaknesses 
identified in this report. In written comments on a draft of this 
report, HHS generally agreed with our findings and recommendations and 
stated that it will leverage the report in its continuing efforts to 
improve its investment management processes. 

[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-11]

To view the full product, including the scope and methodology, click on 
the link above. For more information, contact David Powner, 202-512-
9286, pownerd@gao.gov. 

[End of section] 

Contents: 

Letter: 

Results in Brief: 

Background: 

HHS Has Established Many Key Practices for Managing Its Investments, 
but Has Provided Limited Guidance and Oversight to Component Agencies 
Processes: 

HHS Does Not Have a Plan to Coordinate and Guide Improvement Efforts: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments: 

Appendixes: 

Appendix I: Objectives, Scope, and Methodology: 

Appendix II: Comments from the Department of Health and Human Services: 

Appendix III: GAO Contact and Staff Acknowledgments: 

Tables Tables: 

Table 1: Estimated IT Budget of HHS Component Agencies for Fiscal Year 
2006: 

Table 2: Stage 2 Critical Processes--Building the Investment 
Foundation: 

Table 3: Summary of Results for Stage 2 Critical Processes and Key 
Practices: 

Table 4: Instituting the Investment Board: 

Table 5: Meeting Business Needs: 

Table 6: Selecting an Investment: 

Table 7: Providing Investment Oversight: 

Table 8: Capturing Investment Information: 

Table 9: Stage 3 Critical Processes--Developing a Complete Investment 
Portfolio: 

Table 10: Summary of Results for Stage 3 Critical Processes and Key 
Practices: 

Table 11: Defining the Portfolio Criteria: 

Table 12: Creating the Portfolio: 

Table 13: Evaluating the Portfolio: 

Table 14: Conducting Postimplementation Reviews: 

Figures: 

Figure 1: Simplified HHS Organizational Chart: 

Figure 2: HHS Discretionary IT Investments for Fiscal Year 2006: 

Figure 3: Detailed Breakdown of HHS's Investment Management Process: 

Figure 4: The Five ITIM Stages of Maturity with Critical Processes: 

Abbreviations:

CPIC: Capital Planning and Investment Control: 

CIO: Chief Information Officer: 

HHS: Department of Health and Human Services: 

IT: information technology: 

ITIM: information technology investment management framework: 

ITIRB: Information Technology Investment Review Board: 

PMT: Portfolio Management Tool: 

PIR: postimplementation reviews: 

Letter: 

October 28, 2005: 

The Honorable Charles E. Grassley: 
Chairman, Committee on Finance: 
United States Senate: 

Dear Mr. Chairman: 

The Department of Health and Human Services (HHS) is one of the largest 
federal agencies, the nation's largest health insurer, and the largest 
grant-making agency in the federal government. The department manages 
over 300 programs that serve to improve the health and well-being of 
the American public and is comprised of several component agencies 
covering a wide range of activities including conducting and sponsoring 
medical and social science research, guarding against the outbreak of 
infectious diseases, assuring the safety of food and drugs, and 
providing health care services and insurance. It also manages and funds 
a variety of information technology (IT) initiatives ranging from those 
facilitating the payment of claims for Medicare and Medicaid services 
to those supporting health surveillance and communications. In fiscal 
year 2006, the department plans to spend over $5 billion on information 
technology--the third largest IT expenditure in the federal 
budget.[Footnote 1] 

This report is one of two we prepared in response to your request that 
we evaluate HHS's information technology investment management 
capabilities.[Footnote 2] It focuses on HHS's processes for making IT 
investment management decisions and evaluates how well these processes 
compare with the accepted practices presented in our IT investment 
management (ITIM) framework.[Footnote 3] This framework provides a 
method for evaluating and assessing how well an agency is selecting and 
managing its IT resources. As we agreed with your office, our 
objectives were to (1) assess the department's capabilities for 
managing its IT investments and (2) determine any plans the department 
might have for improving those capabilities. To address these 
objectives, we analyzed documents and interviewed agency officials to 
(1) validate and update HHS's self-assessments of key practices in the 
framework and (2) evaluate HHS's plans for improving its capabilities. 
We performed our work from January through September 2005, in 
accordance with generally accepted government auditing standards. 
Appendix I contains details about our objectives, scope, and 
methodology. 

Results in Brief: 

Because of the management attention that has been given to IT 
investment management, HHS has established over half of the 
foundational practices needed to manage its IT investments individually 
and about 30 percent of the key practices needed to effectively manage 
its portfolio of investments. For example, HHS has implemented many of 
the practices required to ensure that (1) projects support business 
needs and meet users' requirements, (2) a well-defined and disciplined 
process is used to select IT investments, (3) investment information is 
captured in a repository for decision makers, and (4) IT portfolio 
selection criteria are developed and maintained. However, critical 
weaknesses remain in several areas. Specifically, HHS lacks: 

* business representation on its senior IT investment review board of 
component agencies to carry out its full scope of responsibilities, 

* an established process for the IT investment board to regularly 
review a defined set of the component agencies' IT investments and 
maintain visibility of other investments, 

* criteria for assessing portfolio performance or regularly reviewing 
the performance of the organization's investment portfolio, and: 

* processes for conducting postimplementation reviews (PIR) of its IT 
investments. 

The department also does not have a structured mechanism in place for 
ensuring that component agencies define and implement investment 
management processes that are aligned with those of the department. 
Until the department fully establishes all foundational and portfolio- 
level practices and establishes a mechanism to ensure that component 
agencies define and implement processes that are aligned with those of 
the department, executives cannot be assured that they are 
appropriately selecting, managing, and evaluating the mix of 
investments that will maximize returns to the organization, taking into 
account the appropriate level of risk. 

HHS has initiated steps to improve its investment management process; 
however, these steps do not fully address the weaknesses we identify in 
this report, nor are they coordinated along with other needed 
improvement efforts into a plan that (1) is based on an assessment of 
strengths and weaknesses; (2) specifies measurable goals, objectives, 
and milestones; (3) specifies needed resources; (4) assigns clear 
responsibility and accountability for accomplishing tasks; and (5) is 
approved by senior management. Without such a plan and procedures for 
implementing it, the department risks being unable to effectively 
establish mature investment management capabilities. As a result, 
executives may not be able to make informed and prudent investment 
decisions in managing the department's annual multibillion-dollar IT 
budget. 

To further strengthen HHS's investment management capability, we are 
recommending that the department develop and implement a plan aimed at 
addressing the weaknesses that we identify in this report. 

In commenting on a draft of this report, HHS generally agreed with our 
findings and recommendations and stated that it will leverage the 
report in its efforts to improve its investment management processes. 
However, it expressed differing perspectives on the inclusion of 
component agency business representation on the investment review and 
the performance of postimplementation reviews. Specifically, the 
department commented that it used a hierarchy of investment reviews 
combined with investment review board members representing mission 
support areas such as Finance, Acquisition, and Human Resources, to 
provide a structure for making the business decisions regarding the 
department's investments. Nevertheless, we reiterate the importance of 
having business representation from component agencies to make these 
decisions. In addition, the department stated that it was performing 
postimplementation reviews in an informal manner through closeout 
reviews of investments that have recently been implemented and annual 
reviews of systems in operations and maintenance. However, neither of 
these reviews currently identify lessons learned or capture benefits 
realized, key elements of postimplementation reviews. 

Background: 

HHS's Mission, Organizational Structure, and Use of IT: 

HHS is the primary organization within the federal government that is 
devoted to protecting the health of Americans. It provides essential 
human services, such as ensuring food and drug safety and assisting 
needy families. HHS administers more grant dollars than all other 
federal agencies combined, providing over $200 billion of the more than 
$350 billion in federal funds that were awarded to states and other 
entities in fiscal year 2002, the most recent year for which these data 
are available. For fiscal year 2005, HHS had a budget of $581 billion 
and a workforce of over 67,000 employees. 

To accomplish its mission, HHS is comprised of 12 component 
agencies[Footnote 4] and several staff offices that cover a wide range 
of activities--including conducting and sponsoring medical and social 
science research, guarding against the outbreak of infectious diseases, 
assuring the safety of food and drugs, and providing health care 
services and insurance. The Office of the Secretary consists of several 
staff divisions and offices, including the Office of the Assistant 
Secretary for Budget, Technology, and Finance. The HHS Office of the 
Chief Information Officer (CIO) is located within this staff office 
(see fig. 1). 

Figure 1: Simplified HHS Organizational Chart: 

[See PDF for image]

[End of figure]

Information technology investments play a critical role in helping HHS 
carry out its diverse mission. According to the President's most recent 
budget, HHS expects to spend about $5 billion in IT in fiscal year 
2006, making the department's IT investment budget the third largest in 
the federal government. As figure 2 illustrates, approximately $3 
billion is designated as grants to states for investments for Medicaid 
programs and other purposes, such as child support enforcement systems. 
Approximately $2 billion is for discretionary investment spending, of 
which 89 percent is used to fund IT investments for component agencies; 
7 percent is invested in HHS enterprisewide initiatives;[Footnote 5] 
and 4 percent is used to fund other initiatives, including Office of 
the Inspector General IT investments. 

Figure 2: HHS Discretionary IT Investments for Fiscal Year 2006 (in 
millions): 

[See PDF for image] 

[End of figure]

Table 1 provides additional information about the component agencies 
and their estimated IT budget for fiscal year 2006. 

Table 1: Estimated IT Budget of HHS Component Agencies for Fiscal Year 
2006: 

Component agency: Centers for Medicare & Medicaid Services;
Mission: To administer the Medicare program and work in partnership 
with the states to administer Medicaid and the State Children's Health 
Insurance Program. The agency also enforces health insurance 
portability standards and is responsible for implementing a number of 
statutory provisions that have been enacted in recent years, including 
the Medicare Prescription Drug, Improvement, and Modernization Act of 
2003;
Estimated budget for FY 2006 (in millions)[A]: $780. 

Component agency: National Institutes of Health;
Mission: To extend healthy life and reduce the burdens of illness and 
disability by pursuing fundamental knowledge about the nature and 
behavior of living systems and the application of that knowledge;
Estimated budget for FY 2006 (in millions)[A]: $479. 

Component agency: Centers for Disease Control and Prevention;
Mission: To promote health and quality of life by preventing and 
controlling disease, injury, and disability;
Estimated budget for FY 2006 (in millions)[A]: $309[B]. 

Component agency: Food and Drug Administration;
Mission: To protect the public health by ensuring the safety, efficacy, 
and security of human and veterinary drugs, biological products, 
medical devices, the nation's food supply, cosmetics, and products that 
emit radiation;
Estimated budget for FY 2006 (in millions)[A]: $194. 

Component agency: Agency for Healthcare Research and Quality;
Mission: To improve the quality, safety, efficiency, and effectiveness 
of health care for all Americans;
Estimated budget for FY 2006 (in millions)[A]: $65. 

Component agency: Indian Health Service;
Mission: To raise the physical, mental, social, and spiritual health of 
American Indians and Alaska Natives;
Estimated budget for FY 2006 (in millions)[A]: $57. 

Component agency: Health Resources and Services Administration;
Mission: To provide national leadership, program resources, and 
services needed to improve access to culturally competent, quality 
health care;
Estimated budget for FY 2006 (in millions)[A]: $51. 

Component agency: Program Support Center;
Mission: To provide a full range of program support services to all 
components of HHS and other federal agencies, primarily in the areas of 
Human Resources, Health Resources, Acquisition Services, Administrative 
Services, and Financial Management;
Estimated budget for FY 2006 (in millions)[A]: $44. 

Component agency: Substance Abuse and Mental Health Services 
Administration;
Mission: To build resilience and facilitate recovery for people with or 
at risk for substance abuse and mental illness;
Estimated budget for FY 2006 (in millions)[A]: $35. 

Component agency: Administration for Children and Families;
Mission: To administer federal programs that promote the economic and 
social well- being of families, children, individuals, and communities;
Estimated budget for FY 2006 (in millions)[A]: $34. 

Component agency: Administration on Aging;
Mission: To promote the dignity and independence of older people, and 
to help society prepare for an aging population by serving as an 
advocate for older people, and by overseeing the development of a 
comprehensive and coordinated system of care that is responsive to the 
needs and preferences of older people and their family caregivers;
Estimated budget for FY 2006 (in millions)[A]: $2. 

Component agency: Agency for Toxic Substances and Disease Registry;
Mission: To provide health information and take public health actions 
in order to prevent harmful exposures and disease related to toxic 
substances;
Estimated budget for FY 2006 (in millions)[A]: $0[B]. 

Component agency: Total;
Mission: [Empty];
Estimated budget for FY 2006 (in millions)[A]: $2.0 billion. 

Source: GAO analysis based on Office of Management and Budget and HHS 
data. 

[A] Office of Management and Budget, Budget of the U.S. Government, 
Fiscal Year 2006, Report on IT Spending for the Federal Government for 
Fiscal Years 2004, 2005, and 2006. We did not verify these data. 

[B] The Agency for Toxic Substances and Disease Registry investments 
are included in the total for Centers for Disease Control and 
Prevention.

[End of table] 

HHS' investments reflect the diversity of the department's missions and 
operating environments. For example, HHS currently has several 
enterprisewide IT initiatives that enable stakeholders to advance the 
causes of better health, safety, and well-being for American people. 
These initiatives include: 

* Unified Financial Management System, a new core financial system, to 
help management monitor budgets, conduct operations, evaluate program 
performance, and make financial and programmatic decisions. As a core 
financial system, it will interface with an estimated 110 other HHS 
information systems.[Footnote 6] 

* The Office of the Assistant Secretary for Public Health Emergency 
Preparedness maintains a command center where it can coordinate the 
response to public health emergencies from one centralized location. 
This center is equipped with satellite teleconferencing capability, 
broadband Internet hookups, and analysis and tracking software. 

In addition, HHS's component agencies have several projects and systems 
that are critical to the effective implementation of HHS's mission, 
including the following: 

* The Food and Drug Administration's Automated Drug Information 
Management System is to be developed as a fully electronic information 
management system that will receive, evaluate, and disseminate 
information about investigational and marketing submissions for human 
drugs and therapeutic biologics. 

* The National Institutes of Health's major IT initiative, the Clinical 
Research Information System, is a comprehensive effort to modernize the 
systems that support clinical care and the agency's collection of 
research data for the intramural clinical research programs. 

* The Centers for Disease Control and Prevention's major IT initiative, 
Public Health Information Network, is a national initiative to 
implement a multiorganizational business and technical architecture for 
public health information systems.[Footnote 7] 

Prior Reviews Identified Weaknesses in HHS's IT Investment Management 
Process: 

In January 2004, we reported[Footnote 8] on a broad view of the 
government's implementation of investment management practices at 26 
major departments and agencies, including HHS. We also reported--and 
HHS acknowledged--that there were serious weaknesses in investment 
management. Notably, the department had not yet established selection 
criteria for project investments or a requirement that investments 
support work processes that have been simplified or redesigned. In 
addition, the department did not have decision-making rules to guide 
oversight of IT investments, review projects at major milestones, or 
systematically track corrective actions. Accordingly, we made several 
recommendations, including that HHS revise its investment management 
policy and require PIRs to address validating benefits and costs. In 
response to our recommendations, the department has been modifying 
several of its investment management policies, including its capital 
planning and investment control guidance and its governance policies. 

More recently, in June 2005, we reported[Footnote 9] that the HHS IT 
Investment Review Board had conducted only budgetary reviews of the 
Centers for Disease Control and Prevention's Public Health Information 
Network and some of its initiatives, until this past February, when HHS 
initiated steps for better monitoring of system development projects. 
We concluded that until management implements a systematic method for 
IT investment reviews, it will have difficulty minimizing risks while 
maximizing returns on these critical public health investments. 

HHS's Approach to Investment Management: 

HHS has several groups and individuals involved in managing both the 
enterprisewide and component agency IT investments.[Footnote 10] They 
are involved from reviewing and approving a proposed IT project, 
through the process of budgeting for it, monitoring it through 
implementation, and evaluating it at its conclusion. The composition, 
roles, and responsibilities of these individuals and groups are 
described below: 

Information Technology Investment Review Board (ITIRB)--Chaired by 
HHS's CIO, this board is responsible for selecting, controlling, and 
evaluating all departmental IT investments. Members include the Deputy 
Assistant Secretary for Budget, Finance, Performance and Planning; the 
Directors for Acquisition Management Policy and Human Resources; and 
the component agency CIOs. The board is supported by an executive 
secretary who is responsible for, among other things, managing the flow 
of IT investment documentation, scheduling meetings, and assisting the 
members in preparing for their meetings. Currently, this board reviews 
all enterprisewide investments and delegates responsibilities for 
component agency investments to each individual component agencies 
investment review boards in accordance with departmental policies and 
procedures. 

CIO Council--Also chaired by the HHS CIO and comprised of component 
agency CIOs, this board advises the HHS ITIRB on the technical 
soundness of all IT investments that require departmental review and 
provides recommendations regarding, among other things, technical 
aspects of affordability, soundness of design, risk, and compliance 
with architectural and security standards. 

Critical Partners--Comprised of departmental officials from various 
functional areas, including enterprise architecture, security and 
privacy, acquisition management, finance, budget, human resources, and 
e-government; this group is responsible for ensuring that most 
investments[Footnote 11] comply with the HHS policy in each of the 
functional areas and for advising the HHS ITIRB and individual IT 
investment managers on issues in their areas of expertise. Each review 
results in a determination whether the investment is approved, 
conditionally approved, or not approved. A not approved result is 
flagged for executive review. 

Business Case Quality Review Team--Comprised of component agency 
officials, this group evaluates the justifications for IT investments-
-both formal business cases and information documented in the 
department's portfolio management tool's Select forms--against the 
criteria used by the Office of Management and Budget's to evaluate 
business cases[Footnote 12] agencies submit to the office as part of 
the formulation of the federal budget[Footnote 13] and provides 
recommendations for improving these justifications. 

Capital Planning and Investment Control (CPIC) Reengineering/Portfolio 
Management Tool (PMT) Implementation Team--Chaired by the Office of the 
CIO officials with representatives from the Critical Partners and the 
Business Case Quality Review Team, this group advises the board on 
issues regarding investment management policies and procedures and the 
implementation of the department's portfolio management tool. 

Investment Managers--Responsible for managing investments in accordance 
with approved cost, schedule, and performance baselines, and for 
maintaining information on project status, control, performance, risk, 
and corrective actions. 

Process for Managing Investments: 

The department has defined a three-phase process for managing 
investments that involves selecting proposed projects and reselecting 
ongoing projects (select phase), controlling ongoing projects through 
development (control phase), and evaluating projects that have been 
deployed (evaluate phase). The department retains direct management of 
HHS enterprisewide IT investments and delegates considerable authority 
for other investments to component agencies. Specifically, the 
department selects ongoing and new component agency investments through 
the process for selecting enterprisewide IT investments described 
below. Controlling and evaluating component agency IT investments are 
delegated to the component agencies, which are required by the 
department to follow a process similar to the one described below. 

Each phase of the process for enterprisewide investments is comprised 
of multiple steps that set out requirements needed for the HHS ITIRB to 
make the decision to move forward with the project. 

The purpose of the select phase is to ensure that HHS chooses the 
projects that best support its mission and applies resources to the 
most important and valuable investments. The select phase is also 
intended to help the department justify budget requests by 
demonstrating sound business cases and project plans. To select 
investments, HHS has established two separate components--investment 
screening for new investment proposals and investment scoring and 
screening for ongoing investments. 

During the new investment screening, the investment manager is expected 
to develop a project prospectus, which identifies a specific business 
need and preliminary, high-level system requirements. A high-level 
determination of resource and schedule requirements is also to be 
conducted as part of the business need identification activities. 
Approval of the project prospectus by the HHS ITIRB signifies that the 
agency agrees that the need is critical enough to proceed to the next 
step in which the business case is developed. During business case 
development, the investment manager is required to develop the business 
case, which establishes the lifecycle cost, schedule, benefits, and 
performance baselines and includes an analysis for each investment to 
identify alternatives that may satisfy the needs of the department. In 
addition, the investment managers sign a document called the 
accountability agreement form to accept responsibility for reporting on 
the project status in achieving performance baselines throughout the 
remaining phases of the investment management process. 

After the project is initially approved by the HHS ITIRB, the business 
cases and Select forms for most IT investments are updated annually as 
part of the budget formulation process. (The Select forms are a 
collection of forms with HHS's portfolio management tool that capture 
investment data to justify funding and ensure adequate project planning 
during the select phase.) The first step within the annual budget 
formulation process requires that all component agencies use the Select 
forms to report the project cost estimates that best represent the 
level of funding required to meet program or business needs. At this 
point, the Critical Partners and the Business Case Quality Review Team 
score and rank the Select forms using the department's portfolio 
management tool[Footnote 14] to create a single HHS portfolio as well 
as component agency portfolios to provide recommendations to the 
component agencies for making final adjustments to their portfolio 
ranking. 

Once the component agencies have made the appropriate changes, the 
Office of the CIO develops prioritized IT portfolios for HHS as a whole 
as well as each component agency to present to the HHS ITIRB. The 
departmental board and CIO Council review and comment on the 
prioritized portfolio and submit it to the Secretary's Budget Council 
for input into their budget deliberations. The Secretary's Budget 
Council then makes recommendations to the Secretary regarding HHS and 
component agencies' budgets. Finally, the department submits its 
approved Secretary's IT budget to the Office of Management and Budget 
for inclusion in the President's Budget. 

Once selected for inclusion in the department's IT portfolio, each 
project is to be managed by an investment manager and reviewed by the 
ITIRB on a quarterly basis throughout the end of development. The board 
performs reviews of projects that deviate from predetermined budget, 
schedule, or performance milestones established in the business case 
and works with the investment managers to develop a correction action 
plan. The ITIRB must also decide whether to continue to fund the 
project; rebaseline the scope, schedule, or budget; or to terminate the 
project. 

Once a project has been fully implemented, the HHS ITIRB is to conduct 
annual reviews of all HHS enterprisewide steady state investments--that 
is, investments in operations and maintenance--to determine whether 
they continue to meet the business needs. In addition, investments that 
have recently completed implementation or a significant phase are to 
undergo PIRs to evaluate actual development events against project 
management plans and to identify lessons learned that can be applied to 
current and future investments. 

Figure 3 illustrates HHS's investment management process phases and 
steps. The highlighted steps represent the activities that the 
department conducts for both enterprisewide and component agency 
investments. 

Figure 3: Detailed Breakdown of HHS's Investment Management Process: 

[See PDF for image]

[End of figure]

ITIM Maturity Framework: 

The ITIM framework is a maturity model composed of five progressive 
stages of maturity that an agency can achieve in its investment 
management capabilities.[Footnote 15] It was developed on the basis of 
our research into the IT investment management practices of leading 
private-and public-sector organizations. In each of the five stages, 
the framework identifies critical processes for making successful IT 
investments. The maturity stages are cumulative; that is, in order to 
attain a higher stage the agency must have institutionalized all of the 
critical processes at the lower stages, in addition to the higher stage 
critical processes. 

The framework can be used to assess the maturity of an agency's 
investment management processes and as a tool for organizational 
improvement. The overriding purpose of the framework is to encourage 
investment processes that increase business value and mission 
performance, reduce risk, and increase accountability and transparency 
in the decision process. We have used the framework in several of our 
evaluations,[Footnote 16] and a number of agencies have adopted it. 
These agencies have used ITIM for purposes ranging from self-assessment 
to redesign of their IT investment management processes. 

ITIM's five maturity stages represent steps toward achieving stable and 
mature processes for managing IT investments. Each stage builds on the 
lower stages; the successful attainment of each stage leads to 
improvement in the organization's ability to manage its investments. 
With the exception of the first stage, each maturity stage is composed 
of "critical processes" that must be implemented and institutionalized 
in order for the organization to achieve that stage. These critical 
processes are further broken down into key practices that describe the 
types of activities that an organization should be performing to 
successfully implement each critical process. It is not unusual for an 
organization to be performing key practices from more than one maturity 
stage at the same time, but efforts to improve investment management 
capabilities should focus on implementing all lower stage practices 
before addressing higher stage practices. 

In the ITIM framework, Stage 2 critical processes lay the foundation 
for sound IT investment processes by helping the agency to attain 
successful, predictable, and repeatable investment control processes at 
the project level. Specifically, Stage 2 encompasses building a sound 
investment management foundation by establishing basic capabilities for 
selecting new IT projects. It also involves developing the capability 
to control projects so that they finish predictably within established 
cost and schedule expectations and the capability to identify potential 
exposures to risk and put in place strategies to mitigate that risk. 
The basic selection processes established in Stage 2 lays the 
foundation for more mature selection capabilities in Stage 3, which 
represents a major step forward in maturity, in which the agency moves 
from project-centric processes to a portfolio approach, evaluating 
potential investments by how well they support the agency's missions, 
strategies, and goals. 

Stage 3 requires that an organization continually assess both proposed 
and ongoing projects as parts of a complete investment portfolio--an 
integrated and competing set of investment options. It focuses on 
establishing a consistent, well-defined perspective on the IT 
investment portfolio and maintaining mature, integrated selection (and 
reselection), control, and evaluation processes, which are to be 
evaluated during PIRs. This portfolio perspective allows decision 
makers to consider the interaction among investments and the 
contributions to organizational mission goals and strategies that could 
be made by alternative portfolio selections, rather than to focus 
exclusively on the balance between the costs and benefits of individual 
investments. 

Stages 4 and 5 require the use of evaluation techniques to continuously 
improve both the investment portfolio and the investment processes in 
order to better achieve strategic outcomes. At Stage 4 maturity, an 
organization has the capacity to conduct IT succession activities and, 
therefore, can plan and implement the deselection of obsolete, high- 
risk, or low-value IT investments. An organization with Stage 5 
maturity conducts proactive monitoring for breakthrough information 
technologies that will enable it to change and improve its business 
performance. Organizations implementing Stages 2 and 3 have in place 
the selection, control, and evaluation processes that are required by 
the Clinger-Cohen Act of 1996.[Footnote 17] Stages 4 and 5 define key 
attributes that are associated with the most capable organizations. 

Figure 4 shows the five ITIM stages of maturity and the critical 
processes associated with each stage. 

Figure 4: The Five ITIM Stages of Maturity with Critical Processes: 

[See PDF for image]

[End of figure]

As defined by the model, each critical process consists of "key 
practices" that must be executed to implement the critical process. 

HHS Has Established Many Key Practices for Managing Its Investments, 
but Has Provided Limited Guidance and Oversight to Component Agencies 
Processes: 

In order to have the capabilities to effectively manage IT investments, 
an agency, at a minimum, should, (1) build an investment foundation by 
putting basic, project-level control and selection practices in place 
(Stage 2 capabilities) and (2) manage its projects as a portfolio of 
investments, treating them as an integrated package of competing 
investment options and pursuing those that best meet the strategic 
goals, objectives, and mission of the agency (Stage 3 capabilities). 
These practices may be executed at various organizational levels of the 
agency, including at the component level. However, overall 
responsibility for their success remains at the department level. 
Therefore, at a minimum, the department should effectively oversee 
component agencies' IT investment management processes. 

HHS has executed 24 of the 38 key practices that the ITIM framework 
requires to build a foundation for IT investment management (Stage 2) 
and 8 of the 27 key practices required to manage investments as a 
portfolio (Stage 3). However, the department has only provided limited 
oversight of component agencies' ITIM processes. Until HHS implements 
and oversees a stable investment management process throughout the 
department, it will lack essential management controls over all of its 
IT investments, and it will be unable to ensure that it is 
appropriately selecting, managing, and evaluating the mix of 
investments that will maximize returns to the organization, taking into 
account the appropriate level of risk. 

HHS Has Established Over Half of the Foundational Practices Needed to 
Manage Its Investments: 

At the ITIM Stage 2 level of maturity, an organization has attained 
repeatable, successful IT project-level investment control processes 
and basic selection processes. Through these processes, the 
organization can identify expectation gaps early and take the 
appropriate steps to address them. According to the ITIM, critical 
processes at Stage 2 include (1) defining IT investment board[Footnote 
18] operations, (2) identifying the business needs for each IT 
investment, (3) developing a basic process for selecting new IT 
proposals and reselecting ongoing investments, (4) developing project- 
level investment control processes, and (5) collecting information 
about existing investments to inform investment management decisions. 
Table 2 describes the purpose of each of these Stage 2 critical 
processes. 

Table 2: Stage 2 Critical Processes--Building the Investment 
Foundation: 

Critical process: Instituting the investment board;
Purpose: To define and establish an appropriate IT investment 
management structure and the processes for selecting, controlling, and 
evaluating IT investments. 

Critical process: Meeting business needs;
Purpose: To ensure that IT projects and systems support the 
organization's business needs and meet users' needs. 

Critical process: Selecting an investment;
Purpose: To ensure that a well-defined and disciplined process is used 
to select new IT proposals and reselect ongoing investments. 

Critical process: Providing investment oversight;
Purpose: To review the progress of IT projects and systems, using 
predefined criteria and checkpoints, in meeting cost, schedule, risk, 
and benefit expectations and to take corrective action when these 
expectations are not being met. 

Critical process: Capturing investment information;
Purpose: To make available to decision makers information to evaluate 
the impacts and opportunities created by proposed (or continuing) IT 
investments. 

Source: GAO. 

[End of table] 

In the federal government, the agency head and the CIO are responsible 
for effectively managing information technology.[Footnote 19] The 
agency head, through the department-level CIO, is responsible for 
providing leadership and oversight for foundational critical processes 
by ensuring that written policies and procedures are established, 
repositories of information are created that support investment 
decision making, resources are allocated, responsibilities are 
assigned, and all the activities are properly carried out where they 
may be most effectively executed. In a large and diverse organization 
such as HHS, it is especially critical that the CIO create this 
structure and framework to ensure that the organization is effectively 
managing its investments at every level. This means that the CIO must 
ensure that component agencies have investment management processes in 
place that adequately support the department's investment management 
process to make certain that funds are being expended on component 
agency investments that will fulfill mission needs. 

Because of the management attention that has been given to IT 
investment management, the department has put in place over half of the 
key practices needed to establish the investment foundation. The 
department has satisfied all of the key practices associated with 
ensuring that projects and systems support organizational needs and 
meet users' needs. It has satisfied most of the key practices 
associated with identifying and collecting investment information, 
selecting new proposals[Footnote 20] and reselecting ongoing 
investments, and instituting the department's investment review board. 

However, because of its limited involvement in overseeing component 
agency investments, the department has not executed any of the key 
practices related to providing investment oversight. 

Table 3 summarizes the status of HHS's critical processes for Stage 2 
and shows how many key practices HHS has executed in managing its IT 
investments. 

Table 3: Summary of Results for Stage 2 Critical Processes and Key 
Practices: 

Critical process: Instituting the investment board;
Key practices executed: 5;
Total required by critical process: 8;
Percentage of key practices executed: 63. 

Critical process: Meeting business needs;
Key practices executed: 7;
Total required by critical process: 7;
Percentage of key practices executed: 100. 

Critical process: Selecting an investment;
Key practices executed: 7;
Total required by critical process: 10;
Percentage of key practices executed: 70. 

Critical process: Providing investment oversight;
Key practices executed: 0;
Total required by critical process: 7;
Percentage of key practices executed: 0. 

Critical process: Capturing investment information;
Key practices executed: 5;
Total required by critical process: 6;
Percentage of key practices executed: 83. 

Critical process: Total;
Key practices executed: 24;
Total required by critical process: 38;
Percentage of key practices executed: 63. 

Source: GAO.

[End of table] 

HHS Has Established an Investment Review Board, but It Is Operating 
without a Comprehensive Process Guide: 

The establishment of decision-making bodies or boards is a key 
component of the IT investment management process. At the Stage 2 level 
of maturity, organizations define one or more boards, provide resources 
to support the boards' operations, and appoint members who have 
expertise in both operational and technical aspects of proposed 
investments. The boards should operate according to a written IT 
investment process guide that is tailored to the organization's unique 
characteristics, thus ensuring that consistent and effective management 
practices are implemented across the organization.[Footnote 21] The 
organization selects board members to ensure that they are 
knowledgeable about policies and procedures for managing investments. 
Organizations at the Stage 2 level of maturity also take steps to 
ensure that executives and line managers support and carry out the 
decisions of the investment board. According to the ITIM, organizations 
should (1) use an investment management guide as an authoritative 
document to initiate and manage investment processes and (2) provide a 
comprehensive foundation for the policies and procedures that are 
developed for all of the other related processes. (The complete list of 
key practices is provided in table 4.) 

The department has executed 5 of the 8 key practices for this critical 
process. The department established an IT investment review board as 
its corporate-level investment board that consists of senior officials, 
including the CIO and the Deputy Assistant Secretaries for Budget, 
Finance, and Performance & Planning. The board is adequately resourced, 
with most support being provided by the Office of the CIO, whose 
responsibilities include developing and modifying the department's 
criteria for selecting, controlling, and evaluating potential and 
existing IT investments. In addition, the CIO Council reviews the 
enterprisewide investments for technical soundness and provides its 
recommendations to the board. The Critical Partners and Business Case 
Quality Review Team provide additional support to the board by 
reviewing and scoring most of their IT investments. 

To ensure that the board's decisions are carried out for enterprisewide 
investments, the ITIRB approves an accountability agreement document 
and business case that identify the benefits, costs, and schedule for 
the approved investments. The board then monitors the investments 
through the end of development. HHS requires the component agencies to 
follow a similar process in accordance with departmental policies and 
procedures. We verified that an accountability agreement document was 
signed and the business case identified performance expectations for 
the two enterprisewide IT investments we reviewed--Public Key 
Infrastructure and Enterprise Architecture initiatives.[Footnote 22] 
Additionally, the board has oversight of the development and 
maintenance of the documented IT investment process through the CPIC 
Reengineering/PMT Implementation Team, who provides investment 
management policy change recommendations to the board for approval. 

Although HHS has implemented these key practices, it does not have a 
comprehensive organization-specific process guide to direct the 
operations of the investment board. While the Information Resources 
Management policy, guidelines, and standard operating procedures 
provide general guidance on the organization's investment management 
process, they do not reflect the current investment management process. 
Moreover, they do not constitute an IT investment process guide because 
they do not sufficiently define the investment process. Specifically, 
the policies and procedures do not include information on the roles of 
the key players such as the CIO Council, Critical Partners, Business 
Case Quality Review Team, or the component agency investment review 
boards. In addition, they do not identify the manner in which 
investment board's processes are to be coordinated with other key 
organizational plans and processes (such as the budget formulation 
process). HHS has recently drafted a revised investment management 
policy addressing many of these weaknesses; however, it has not been 
finalized, and HHS officials could not provide a final issuance date. 
Without a comprehensive investment management process guide, the 
department lacks the assurance that IT investment activities will be 
coordinated and performed in a consistent and cost-effective manner. 

Moreover, while HHS has established an IT investment board, the board 
does not have business representation (that is, mission representation) 
from component agencies. Instead, Chief Information Officers represent 
the component agencies. According to HHS's CIO, the membership of the 
board is adequate for carrying out the investment activities it 
currently performs--primarily focusing on enterprisewide IT 
investments. However, because allocating resources among major IT 
investments may require fundamental trade-offs among a multitude of 
business objectives, portfolio management decisions are essentially 
business decisions, and therefore require sufficient business 
representation on the board. Until the department adjusts its board 
membership to include business representation from component agencies, 
it will not have assurance that it includes those executives who are in 
the best position to make the full range of decisions needed to enable 
the agency to meet its mission most effectively, particularly as it 
begins to execute its full range of responsibility. 

Finally, the HHS ITIRB is not operating according to its assigned 
authority and responsibility. The department's investment management 
policy and the HHS ITIRB's charter state that the board has oversight 
responsibility for both enterprisewide and a defined set of component 
agency IT investments, including projects that are high risk, 
crosscutting, and require review by the Office of Management and 
Budget. However, the board currently oversees only enterprisewide IT 
investments. According to HHS officials, the department has delegated 
authority to the component agencies to conduct investment reviews; 
however, the board does not have a mechanism in place for ensuring that 
component agencies are conducting such reviews in accordance with 
department policies and procedures. Until the board operates according 
to its assigned authority, it cannot ensure that component agency 
investments are properly aligned with the organization's objectives or 
reviewed by the appropriate board. 

Table 4 shows the rating for each key practice required to institute 
the investment board. Each of the "executed" ratings shown below 
represents instances where, on the basis of the evidence provided by 
HHS officials, we concluded that the specific key practices were 
executed by the organization. 

Table 4: Instituting the Investment Board: 

Type of practice: Organizational commitments;
Key practice: 1. An enterprisewide IT investment board composed of 
senior executives from IT and business units is responsible for 
defining and implementing the organization's IT investment governance 
process;
Rating: Not executed;
Summary of evidence: Although HHS has an enterprisewide IT investment 
board that is responsible for defining and implementing the 
organization's IT investment governance process and consists of the 
department's senior executives from IT and other supporting units, 
including the CIO, Deputy Assistant Secretaries for Budget, Finance, 
Performance & Planning, and the component agencies' CIO, the board does 
not have business representation from component agencies. 

Key practice: 2. The organization has a documented IT investment 
process directing each investment board's operations;
Rating: Not executed;
Summary of evidence: Although the Information Resources Management 
policy, guidelines, and standard operating procedures provide general 
guidance on the department's investment management process, these 
policies and procedures do not reflect the department's current 
investment management process. In addition, these documents do not 
constitute an investment management process guide in that they do not 
(1) include information on the roles of key working groups involved in 
the organization's IT investment processes or (2) identify the manner 
in which investment board's processes are to be coordinated with other 
key organizational plans and processes (such as the budget formulation 
process) or component agency investment management processes. HHS is 
currently revising its documented IT investment process to reflect its 
current investment management practices. 

Type of practice: Prerequisites;
Key practice: 1. Adequate resources, including people, funding, and 
tools, are provided for supporting the operations of each IT investment 
board;
Rating: Executed;
Summary of evidence: Adequate resources are provided to support the 
ITIRB's operations. The executive secretariat provides operations 
support such as scheduling meetings and managing the flow of IT 
investment documentation. The CIO Council performs technical reviews of 
enterprisewide IT investments and provides recommendations to the 
ITIRB. The Critical Partners rank and score most IT investments from a 
functional perspective, while the Business Case Quality Review Team 
ranks and scores these investments against the Office of Management and 
Budget Exhibit 300 quality criteria. 

Key practice: 2. The board members understand the organization's IT 
investment management policies and procedures and the tools and 
techniques used in the board's decision-making process;
Rating: Executed;
Summary of evidence: HHS ITIRB members understand the investment 
board's policies and procedures and the tools and techniques used in 
the board's decision-making process. High-level training has been 
provided to members during past board meetings on an informal basis. 

Key practice: 3. Each board's span of authority and responsibility is 
defined to minimize overlaps or gaps among the boards;
Rating: Executed;
Summary of evidence: HHS' investment board, the ITIRB, is responsible 
for defining and implementing the organization's IT investment 
governance process. 

Type of practice: Activities;
Key practice: 1. The enterprisewide investment board has oversight 
responsibilities for the development and maintenance of the 
organization's documented IT investment process;
Rating: Executed;
Summary of evidence: While the HHS ITIRB does not directly oversee the 
development and maintenance of HHS's documented investment process, it 
is involved in this process through the CPIC Reengineering/PMT 
Implementation Team, who provides investment management policy change 
recommendations to the HHS ITIRB for approval. 

Key practice: 2. Each investment board operates in accordance with its 
assigned authority and responsibility;
Rating: Not executed;
Summary of evidence: While, the HHS ITIRB's charter assigns the board 
authority and responsibility for reviewing both the enterprisewide and 
a defined set of component agency IT investments, the board primarily 
focuses on enterprisewide IT investments. 

Key practice: 3. The organization has established management controls 
for ensuring that investment boards' decisions are carried out;
Rating: Executed;
Summary of evidence: HHS ITIRB has established management controls such 
as the accountability agreement document for ensuring that the board's 
decisions regarding the enterprisewide IT investments, which it 
directly reviews, are carried out;
For the two enterprisewide projects we reviewed, we verified that 
management controls were established through the accountability 
agreement document and business cases. 

Source: GAO. 

[End of table] 

HHS Has a Process for Ensuring That Its Investments Support Business 
Needs and Meet Users' Needs: 

Defining business needs for each IT project helps to ensure that 
projects and systems support an organization's business needs and meet 
users' needs. This critical process ensures that an organization's 
business objectives and its IT management strategy are linked. 
According to the ITIM, effectively meeting business needs requires, 
among other things, (1) documenting business needs with stated goals 
and objectives; (2) identifying specific users and other beneficiaries 
of IT projects and systems; (3) providing adequate resources to ensure 
that projects and systems support the organization's business needs and 
meet users' needs; and (4) periodically evaluating the alignment of IT 
projects and systems with the organization's strategic goals and 
objectives. (The complete list of key practices is provided in table 
5.) 

The department has in place all of the key practices for meeting 
business needs. Specifically, HHS has policy and procedures that call 
for business needs to be identified in the business case or the 
portfolio management tool's Select forms for both proposed and ongoing 
enterprisewide and component agency IT projects. Resources devoted to 
ensuring that IT projects and systems support the organization's 
business needs and meet users' needs include the Business Case Quality 
Review Team, the Critical Partners, the portfolio management tool, and 
detailed procedures and associated templates for developing business 
cases. HHS's specific business mission, with stated goals and 
objectives, is defined in the HHS Strategic Plan for fiscal years 2004 
through 2009. 

Further, HHS defines and documents business needs for both proposed and 
ongoing enterprisewide and component agency IT projects, and identifies 
users and other beneficiaries during its selection activities. In 
addition, according to HHS IT officials, end users participate in 
project management throughout the IT project's life cycle. For the four 
projects we reviewed, we verified that business needs and specific 
users and other beneficiaries were identified and documented in the 
business case or in the Select forms within HHS's portfolio management 
tool. In addition, end users are involved in project management 
throughout the life cycle of the enterprisewide investments. For 
example, users of HHS's Public Key Infrastructure and Enterprise 
Architecture initiatives participate in project management through 
integrated project teams, which meet approximately once a month and are 
comprised of representatives from the component agencies. Because the 
department has executed all of the key practices associated with 
identifying business needs, it has increased confidence that its IT 
projects will meet both business needs and users' needs. 

Table 5 shows the rating for each key practice required to meet 
business needs and summarizes the evidence that supports these ratings. 

Table 5: Meeting Business Needs: 

Type of practice: Organizational commitments;
Key practice: 1. The organization has documented policies and 
procedures for identifying IT projects or systems that support the 
organization's ongoing and future business needs;
Rating: Executed;
Summary of evidence: HHS has policies and procedures for ensuring that 
IT projects and systems support the department's ongoing and future 
business needs. 

Type of practice: Prerequisites;
Key practice: 1. The organization has a documented business mission 
with stated goals and objectives;
Rating: Executed;
Summary of evidence: The HHS Strategic Plan for fiscal years 2004 
through 2009 defines the agency's mission goals and objectives. 

Key practice: 2. Adequate resources, including people, funding, and 
tools, are provided for ensuring that IT projects and systems support 
the organization's business needs and meet users' needs;
Rating: Executed;
Summary of evidence: HHS has adequate resources for ensuring that its 
IT projects and systems support the organization's business needs and 
meet users' needs. They include Business Case Quality Review Team, 
Critical Partners, and the portfolio management tool. Also, HHS has 
templates for developing business cases and training manuals on the use 
of the portfolio management tool. 

Type of practice: Activities;
Key practice: 1. The organization defines and documents business needs 
for both proposed and ongoing IT projects and systems;
Rating: Executed;
Summary of evidence: HHS policies and procedures call for business 
needs for enterprisewide and component agency ongoing and proposed IT 
projects and systems to be specified in the business case or Select 
forms;
We verified that business needs were defined and documented within the 
business case or Select forms in the portfolio management tool for the 
four projects we reviewed. 

Key practice: 2. The organization identifies specific users and other 
beneficiaries of IT projects and systems;
Rating: Executed;
Summary of evidence: HHS policy and procedures call for specific users 
and other beneficiaries of both enterprisewide and component agency IT 
projects and systems to be identified in the business case and Select 
forms;
We verified that customers and stakeholders were defined and documented 
within the business case or Select forms in the portfolio management 
tool for the four projects we reviewed. 

Key practice: 3. Users participate in project management throughout an 
IT project's or system's life cycle;
Rating: Executed;
Summary of evidence: According to HHS IT officials, end users 
participate in project management throughout an IT project's or 
system's life cycle;
We verified that users participated in project management throughout 
the life cycle of the two enterprisewide projects we reviewed. 
According to HHS Office of the CIO, user participation in project 
management is not addressed at the department level for the two 
component agency projects we reviewed since it is delegated to the 
component agency. 

Key practice: 4. The investment board periodically evaluates the 
alignment of its IT projects and systems with the organization's 
strategic goals and objectives and takes corrective actions when 
misalignment occurs;
Rating: Executed;
Summary of evidence: The ITIRB evaluates the alignment of both HHS 
enterprisewide and component agency IT systems through the annual 
budget formulation process and takes corrective action when 
misalignment occurs. 

Source: GAO. 

[End of table] 

HHS Is Selecting New Investments and Reselecting Ongoing Investments, 
but Lacks a Fully Documented Process for Doing So: 

Selecting new IT proposals and reselecting ongoing investments require 
a well-defined and disciplined process to provide the agency's 
investment boards, business units, and developers with a common 
understanding of the process and the cost, benefit, schedule, and risk 
criteria that will be used both to select new projects and to reselect 
ongoing projects for continued funding. According to the ITIM, this 
critical process requires, among other things, (1) making funding 
decisions for new proposals according to an established process; (2) 
providing adequate resources for investment selection activities; (3) 
using a defined selection process to select new investments and 
reselect ongoing investments; (4) establishing criteria for analyzing, 
prioritizing, and selecting new IT investments and for reselecting 
ongoing investments; and (5) creating a process for ensuring that the 
criteria change as organizational objectives change. (The complete list 
of key practices is provided in table 6.) 

HHS has executed 7 of the 10 key practices associated with selecting an 
investment. For example, resources devoted to selection activities 
include the Critical Partners, Business Case Quality Review Team, and 
portfolio management tool, which contains several forms for selecting 
IT projects and systems. HHS also has detailed procedures for using its 
portfolio management tool and developing business cases. The criteria 
for analyzing, prioritizing, selecting and reselecting new and ongoing 
investments address the President's Management Agenda, HHS strategic 
goals, and IT strategic goals, value, and risk. They are incorporated 
into the department's portfolio management tool and are reviewed by the 
investment review board and adjusted within the tool annually at the 
beginning of each budget cycle to reflect organizational objectives. 
This year, HHS added additional criteria--a quality score. 

HHS uses its annual budget formulation process to select both 
enterprisewide and component agency proposed and ongoing IT 
investments. We verified that the four projects we reviewed were 
reselected by the department using the annual budget formulation 
process. 

Although HHS has the above strengths, the department has not executed 
any of the practices associated with documenting policies and 
procedures. Specifically, HHS has not fully documented its process for 
selecting new IT proposals and reselecting ongoing IT investments. 
Although a number of documents address investment selection, they are 
not linked to provide decision makers with a clear understanding of the 
selection and reselection processes. In addition, they do not define 
the roles and responsibilities for all key players involved in these 
processes. Moreover, although the HHS Office of the CIO works directly 
with the department's Office of the Budget, HHS does not have policies 
and procedures documenting the integration of funding with the process 
of selecting and reselecting investments. Until the department fully 
documents policies and procedures for selecting new IT proposals and 
reselecting ongoing IT investments, the department will not be 
adequately certain that it is consistently and objectively selecting 
and reselecting investments that best meet the needs and priorities of 
the department. 

Table 6 shows the rating for each key practice required to select an 
investment and summarizes the evidence that supports these ratings. 

Table 6: Selecting an Investment: 

Type of practice: Organizational commitments;
Key practice: 1. The organization has documented policies and 
procedures for selecting new IT proposals;
Rating: Not executed;
Summary of evidence: Although HHS has a number of documents that 
address investment selection, they are not linked to provide decision 
makers with a common understanding of the selection process. In 
addition, these documents do not define the roles and responsibilities 
for each participating unit involved in the project selection process. 

Key practice: 2. The organization has documented policies and 
procedures for reselecting[A] ongoing IT investments;
Rating: Not executed;
Summary of evidence: Although HHS has a number of documents that 
address investment reselection, they are not linked to provide the 
decision makers with a common understanding of the selection process. 
In addition, these documents do not define the roles and 
responsibilities for each participating unit involved in the project 
selection process. 

Key practice: 3. The organization has policies and procedures for 
integrating funding with the process of selecting an investment;
Rating: Not executed;
Summary of evidence: Although the HHS Office of the CIO works directly 
with the department's Office of the Budget, HHS does not have policies 
and procedures documenting the integration of funding with the process 
of selecting and reselecting investments. 

Type of practice: Prerequisites;
Key practice: 1. Adequate resources, including people, funding, and 
tools, are provided for identifying and selecting IT projects and 
systems;
Rating: Executed;
Summary of evidence: Adequate resources are provided for identifying 
and selecting IT projects and systems. They include the Critical 
Partners, Business Case Quality Review Team, and the department's 
portfolio management tool, which contains several forms for selecting 
IT projects and systems. 

Key practice: 2. Criteria for analyzing, prioritizing, and selecting 
new IT investment opportunities have been established;
Rating: Executed;
Summary of evidence: HHS has established criteria for analyzing, 
prioritizing, and selecting enterprisewide and component agency new IT 
investments. The department selects new IT proposals and reselects 
ongoing investments using the same criteria, which are incorporated 
into its portfolio management tool. 

Key practice: 3. Criteria for analyzing, prioritizing, and reselecting 
IT investment opportunities have been established;
Rating: Executed;
Summary of evidence: HHS has established criteria for analyzing, 
prioritizing, and reselecting both enterprisewide and component agency 
IT investments. The department selects new IT proposals and reselects 
ongoing investments using the same criteria, which are incorporated 
into its portfolio management tool. 

Key practice: 4. A mechanism exists to ensure that the criteria 
continue to reflect organizational objectives;
Rating: Executed;
Summary of evidence: The HHS ITIRB reviews and adjusts criteria 
annually at the start of each budget cycle and updates the portfolio 
management tool to reflect HHS's objectives. 

Type of practice: Activities;
Key practice: 1. The organization uses its defined selection process, 
including predefined selection criteria, to select new IT investments;
Rating: Executed;
Summary of evidence: HHS uses its annual budget formulation process to 
select new IT investments;
We verified that the four projects we reviewed were selected using the 
annual budget formulation activities. 

Key practice: 2. The organization uses the defined selection process, 
including predefined selection criteria, to reselect ongoing IT 
investments;
Rating: Executed;
Summary of evidence: HHS uses its annual budget formulation process to 
reselect ongoing IT investments;
We verified that the four projects we reviewed were reselected using 
the annual budget formulation activities. 

Key practice: 3. Executives' funding decisions are aligned with 
selection decisions;
Rating: Executed;
Summary of evidence: The HHS ITIRB makes funding decisions for new and 
ongoing IT investments through the department's budget formulation 
process, which is used to select both enterprisewide and component 
agency investments. 

Source: GAO. 

[A] According to the GAO ITIM framework, reselecting is the periodic 
reconsideration of an investment's continuing value to the organization 
and the decision to continue funding. It is a recurring process that 
continues for as long as a project is receiving funding.

[End of table] 

HHS Does Not Have a Process for Effectively Overseeing Its Component 
Agency IT Investments: 

An organization should effectively oversee its IT projects throughout 
all phases of their life cycles. Its investment board should observe 
each project's performance and progress toward predefined cost and 
schedule expectations as well as each project's anticipated benefits 
and risk exposure. This does not mean that a departmental board, such 
as the ITIRB, should micromanage each project to provide effective 
oversight; rather it means that the departmental board should be 
actively involved in all IT investments and proposals that are high 
cost or high risk or have significant scope and duration and at a 
minimum, should, have a mechanism for maintaining visibility of other 
investments. The board should also employ early warning systems that 
enable it to take corrective actions at the first sign of cost, 
schedule, and performance slippages. According to the ITIM, effective 
project oversight requires, among other things, (1) having written 
policies and procedures for management oversight; (2) developing and 
maintaining an approved management plan for each IT project; (3) making 
up-to-date cost and schedule data for each project available to the 
oversight boards; (4) having regular reviews by each investment board 
of each project's performance against stated expectations; and (5) 
ensuring that corrective actions for each underperforming project are 
documented, agreed to, implemented, and tracked until the desired 
outcome is achieved. (The complete list of key practices is provided in 
table 7.) 

The department has not executed any of the seven key practices 
associated with effective project oversight, primarily because of its 
limited role in overseeing component agency IT investments. 
Specifically, while the department has documented standard operating 
procedures and instructional memorandums for oversight of 
enterprisewide IT investments, they are not comprehensive in that they 
do not specify the board's responsibilities for investment oversight; 
procedural rules for the ITIRB operations and decision making during 
project oversight; or policies and procedures for overseeing component 
agency IT investments. 

The HHS ITIRB is currently performing regular reviews[Footnote 23] of 
enterprisewide IT projects and systems against stated expectations 
through reports that are available to decision makers on the HHS 
Intranet. However, the department is not regularly reviewing component 
agency investments that are high risk, crosscutting, and require review 
by the Office of Management and Budget, although their policy calls for 
it. The board also does not have a mechanism for maintaining visibility 
of other component agency investments. 

The department delegates oversight of these investments to the 
component agencies but believes it is nonetheless effectively 
overseeing component agency investments through (1) reviews of these 
investments as part of the annual Critical Partner and Business Case 
Quality reviews performed during the annual selection process and the 
use of (2) earned value management data.[Footnote 24] Although the 
annual reviews may provide insight into the status of investments, they 
are not frequent enough to allow for timely identification of problems. 
Moreover, while HHS officials told us that staff responsible for 
collecting earned value management data on component agency investments 
share significant concerns about the data with the ITIRB, they did not 
have formal documentation clearly supporting this issue. In addition, 
formal procedures for elevating issues to the board have not been 
developed. In the absence of effective board oversight, HHS executives 
will not have the information they need to determine whether component 
agency projects are being developed on schedule and within budget. In 
addition, the department will run the risk that underperforming 
component agency projects will not be identified in time for corrective 
actions to be taken. 

We verified that HHS provided oversight for the two enterprisewide 
investments, but had delegated oversight activities for the two 
component agency investments we reviewed. 

Table 7 shows the rating for each key practice required to provide 
investment oversight and summarizes the evidence that supports these 
ratings. 

Table 7: Providing Investment Oversight: 

Type of practice: Organizational commitment;
Key practice: 1. The organization has documented policies and 
procedures for management oversight of IT projects and systems;
Rating: Not executed;
Summary of evidence: Although HHS has developed standard operating 
procedures and instructional memorandums for oversight of 
enterprisewide IT projects and systems, they do not (1) specify the HHS 
ITIRB's responsibilities when providing investment oversight within its 
domain or (2) procedural rules for the ITIRB's operations and for 
decision making during project oversight. In addition, HHS does not 
have policies and procedures for management oversight of component 
agency investments. 

Type of practice: Prerequisites;
Key practice: 1. Adequate resources, including people, funding, and 
tools, are provided for IT project oversight;
Rating: Not executed;
Summary of evidence: Although HHS has adequate resources for providing 
oversight for enterprisewide IT investments, the department does not 
have adequate resources for providing oversight for component agency IT 
investments. 

Key practice: 2. IT projects and systems, including those in steady 
state (operations and maintenance), maintain approved project 
management plans that include expected cost and schedule milestones and 
measurable benefit and risk expectations;
Rating: Not executed;
Summary of evidence: HHS's policy calls for an accountability agreement 
document and business case, including cost, benefit, schedule, and risk 
expectations, to be available to the ITIRB after approval of an 
enterprisewide IT projects and systems, but there is no similar 
requirement for component agency IT projects and systems;
We verified that HHS provided oversight for the two enterprisewide 
investments, but had delegated oversight activities for the two 
component agency investments we reviewed. 

Type of practice: Activities;
Key practice: 1. Data on actual performance (including cost, schedule, 
benefit, and risk performance) are provided to the appropriate IT 
investment board;
Rating: Not executed;
Summary of evidence: Data on actual performance of enterprisewide IT 
investments are provided to the HHS ITIRB;
however, the ITIRB does not regularly receive data on actual 
performance of a defined set of component agencies' IT investments and 
maintain visibility of other investments;
We verified that the two enterprisewide projects provide quarterly 
reports to the ITIRB. For the component agency projects we reviewed, 
this activity is delegated to the component agency and is not addressed 
at the department level. 

Key practice: 2. Using verified data, each investment board regularly 
reviews the performance of IT projects and systems against stated 
expectations;
Rating: Not executed;
Summary of evidence: HHS ITIRB quarterly reviews performance of 
enterprisewide IT investments under development and annually reviews 
enterprisewide IT investment in their operational phase of their life 
cycles;
however, the investment board does not have a process for regularly 
reviewing the performance of a defined set of component agency 
investments and maintaining visibility of other investments. 

Key practice: 3. For each underperforming IT project or system, 
appropriate actions are taken to correct or terminate the project or 
system in accordance with defined criteria and the documented policies 
and procedures for management oversight;
Rating: Not executed;
Summary of evidence: The HHS ITIRB takes appropriate actions to correct 
or terminate the enterprisewide IT projects or systems. However, it 
does not take actions to correct or terminate underperforming component 
agency investments because it does not regularly review these 
investments' performance. 

Key practice: 4. The investment board regularly tracks the 
implementation of corrective actions for each underperforming project 
until the actions are completed;
Rating: Not executed;
Summary of evidence: The HHS ITIRB maintains meeting minutes for 
enterprisewide IT investments to ensure that corrective actions are 
implemented and tracked until the desired outcome is achieved. However, 
it does not take actions to correct or terminate underperforming 
component agency investments because it does not regularly review these 
investments' performance. 

Source: GAO. 

[End of table] 

HHS Has a Defined Process for Capturing Investment Information: 

To make good IT investment decisions, an organization must be able to 
acquire pertinent information about each investment and store that 
information in a retrievable format. During this critical process, an 
organization identifies its IT assets and creates a comprehensive 
repository of investment information. This repository provides 
information to investment decision makers to help them evaluate the 
potential impacts and opportunities created by proposed or continuing 
investments. It can provide insights into major IT cost and management 
drivers and trends. The repository can take many forms and need not be 
centrally located, but the collection method should, at a minimum, 
identify each IT investment and its associated components. This 
critical process may be satisfied by the information contained in the 
organization's current enterprise architecture, augmented by additional 
information--such as financial information and information on risk and 
benefits--that the investment board may require to ensure that informed 
decisions are being made. According to the ITIM, effectively managing 
this repository requires, among other things, (1) developing written 
policies and procedures for identifying and collecting the information; 
(2) assigning responsibilities for ensuring that the information being 
collected meets the needs of the investment management process; (3) 
identifying IT projects and systems and collecting relevant information 
to support decisions about them; and (4) making the information easily 
accessible to decision makers and others. (The complete list of key 
practices is provided in table 8.) 

HHS has executed 5 of the 6 key practices for capturing investment 
information. For example, the department has several documents that 
define the policies and procedures for identifying and collecting 
investment information in its repositories and also assign 
responsibility to the HHS CIO for ensuring that the information 
collected during project and systems identification meets the needs of 
the investment management process. HHS maintains a portfolio management 
tool, which serves as the primary repository for identifying and 
collecting information about both department and component agency IT 
projects and systems. The department's portfolio management tool is 
easily accessible to decision makers at both the department and 
component level and the Office of the CIO has provided decision makers 
with various training manuals and guidance memorandums. In addition, 
the department also identifies and collects information about 
enterprisewide IT investments using its Intranet. Further, the 
department recently began collecting earned value information through 
spreadsheets on major HHS IT investments that compares planned and 
actual cost and schedule information. These repositories are easily 
accessible to the board members. 

The key practice HHS has not executed has to do with the captured 
investment information not yet being used by the HHS ITIRB to fully 
support decisions about component agency investments. For example, the 
earned value investment data received from each component agency has 
not been used by the HHS ITIRB for control and evaluation decisions. 
According to agency officials, the department has recently begun 
monitoring the earned value data to identify investments that report 
cost and schedule variances and these officials acknowledge a need to 
formalize the process for doing so. Until HHS's decision makers use the 
information in the repository to fully support the investment 
management process, it will be unable to effectively evaluate the 
impacts and opportunities created by proposed or continuing 
investments. 

Table 8 shows the rating for each key practice required to capture 
investment information and summarizes the evidence that supports these 
ratings. 

Table 8: Capturing Investment Information: 

Type of practice: Organizational commitments;
Key practice: 1. The organization has documented policies and 
procedures for identifying and collecting information about IT projects 
and systems to support the investment management process;
Rating: Executed;
Summary of evidence: The department has documented policies and 
procedures for identifying and collecting information about IT projects 
and systems to support the investment management process. 

Key practice: 2. An official is assigned responsibility for ensuring 
that the information collected during project and systems 
identification meets the needs of the investment management process;
Rating: Executed;
Summary of evidence: The HHS CIO is responsible for ensuring that the 
information collected during project and systems identification meets 
the needs of the investment management process. 

Type of practice: Prerequisite;
Key practice: 1. Adequate resources, including people, funding, and 
tools, are provided for identifying IT projects and systems and 
collecting relevant investment information about them;
Rating: Executed;
Summary of evidence: According to the HHS IT officials, adequate 
resources are provided for identifying IT projects and systems and 
collecting relevant investment information about them. 

Type of practice: Activities;
Key practice: 1. The organization's IT projects and systems are 
identified, and specific information is collected to support decisions 
about them;
Rating: Executed;
Summary of evidence: HHS's portfolio management tool identifies and 
collects information about both department and component agency IT 
projects and systems to support the investment management process as it 
currently exists. The department also identifies and collects relevant 
investment information for the enterprisewide IT investments through 
the HHS Intranet and component agency IT investments through 
spreadsheets that capture earned value data;
We verified that HHS's portfolio management tool identifies and 
contains investment information for the four projects we reviewed. 

Key practice: 2. The information that has been collected is easily 
accessible and understandable to decision makers and others;
Rating: Executed;
Summary of evidence: IT investment decision makers at both the 
department and component agency level have access to HHS's portfolio 
management tool that is used to capture IT project and system 
information. Instructions on the use and navigation through the 
portfolio management system are available to investment management 
decision makers. In addition, the HHS ITIRB can also access the 
enterprisewide IT investment information posted on the HHS Intranet. 

Key practice: 3. The information repository is used by investment 
decision makers and others to support investment management;
Rating: Not executed;
Summary of evidence: While HHS identifies and collects information 
about IT projects and systems to support the investment management 
process, this information has not been used by the HHS ITIRB to fully 
support the control and evaluate decisions for component agency IT 
investments. 

Source: GAO. 

[End of table] 

HHS Has Some of the Capabilities Needed to Manage IT Investments as a 
Portfolio: 

Once an agency has attained Stage 2 maturity, it needs to implement 
critical processes for managing its investments as a portfolio (Stage 
3). An IT investment portfolio is an integrated, agencywide collection 
of investments that are assessed and managed collectively based on 
common criteria. Managing investments as a portfolio is a conscious, 
continuous, and proactive approach to allocating limited resources 
among an organization's competing initiatives in light of the relative 
benefits expected from these investments. Taking an agencywide 
perspective enables an organization to consider its investments 
comprehensively, so that collectively the investments optimally address 
the organization's missions, strategic goals, and objectives. Managing 
IT investments as a portfolio also allows an organization to determine 
its priorities and make decisions about which projects to fund and 
continue to fund based on analyses of the relative organizational value 
and risks of all projects, including projects that are proposed, under 
development, and in operation. Although investments may initially be 
organized into subordinate portfolios--based on, for example, business 
lines or life cycle stages--and managed by subordinate investment 
boards; they should ultimately be aggregated into this enterprise-level 
portfolio. 

According to the ITIM framework, Stage 3 maturity includes (1) defining 
the portfolio criteria, (2) creating the portfolio, (3) evaluating the 
portfolio, and (4) conducting postimplementation reviews. Table 9 
summarizes the purpose of each critical process in Stage 3. 

Table 9: Stage 3 Critical Processes--Developing a Complete Investment 
Portfolio: 

Critical process: Defining the portfolio criteria;
Purpose: To ensure that the organization develops and maintains IT 
portfolio selection criteria that support its mission, organizational 
strategies, and business priorities. 

Critical process: Creating the portfolio;
Purpose: To ensure that IT investments are analyzed according to the 
organization's portfolio selection criteria and to ensure that an 
optimal IT investment portfolio with manageable risks and returns is 
selected and funded. 

Critical process: Evaluating the portfolio;
Purpose: To review the performance of the organization's investment 
portfolio(s) at agreed- upon intervals and to adjust the allocation of 
resources among investments as necessary. 

Critical process: Conducting postimplementation reviews;
Purpose: To compare the results of recently implemented investments 
with the expectations that were set for them and to develop a set of 
lessons learned from these reviews. 

Source: GAO.

[End of table] 

HHS has executed 8 of the 27 key practices required by Stage 3. For 
example, the department's core IT portfolio selection criteria, 
including cost, benefit, schedule, and risk are approved by the HHS 
ITIRB. In addition, the investment board examines the mix of new and 
ongoing investments and their respective data and analyses to select 
investments to fund. However, many key practices still need to be 
executed before HHS can effectively manage its IT investments from a 
portfolio perspective. For example, HHS has not addressed any of the 
key practices related to evaluating the portfolio or conducting PIRs. 
Until HHS fully implements the critical processes associated with 
managing its investments as a complete portfolio, it will not have the 
data it needs to make informed decisions about competing investments. 

Table 10 summarizes the status of HHS's critical processes for Stage 3, 
showing how many associated key practices it has executed. 

Table 10: Summary of Results for Stage 3 Critical Processes and Key 
Practices: 

Critical process: Defining the portfolio criteria;
Key practices executed: 5;
Total required by critical process: 7;
Percentage of key practices executed: 71. 

Critical process: Creating the portfolio;
Key practices executed: 3;
Total required by critical process: 7;
Percentage of key practices executed: 43. 

Critical process: Evaluating the portfolio;
Key practices executed: 0;
Total required by critical process: 7;
Percentage of key practices executed: 0. 

Critical process: Conducting postimplementation reviews;
Key practices executed: 0;
Total required by critical process: 6;
Percentage of key practices executed: 0. 

Critical process: Total;
Key practices executed: 8;
Total required by critical process: 27;
Percentage of key practices executed: 30. 

Source: GAO. 

[End of table] 

Process for Modifying IT Portfolio Selection Criteria Is Not 
Institutionalized: 

To manage IT investments effectively, an organization needs to 
establish rules or "portfolio selection criteria" for determining how 
to allocate scarce funding to existing and proposed investments. Thus, 
developing an IT investment portfolio requires defining appropriate 
cost, benefit, schedule, and risk criteria with which to evaluate 
individual investments in the context of all other investments. To 
ensure that the organization's strategic goals, objectives, and mission 
will be satisfied by its investments, the criteria should have an 
enterprisewide perspective. Further, if an organization's mission or 
business needs and strategies change, criteria for selecting 
investments should be reexamined and modified as appropriate. Portfolio 
selection criteria should be disseminated throughout the organization 
to ensure that decisions concerning investments are made in a 
consistent manner and that this critical process is institutionalized. 
To achieve this result, project management personnel and others should 
be aware of the criteria and address the criteria in funding 
submissions for projects. Resources required for this critical process 
typically include the time and attention of executives involved in the 
process, adequate funding, and supporting tools. (The complete list of 
key practices is provided in table 11.) 

The department has executed 5 of the 7 key practices for this critical 
process. For example, responsibility has been assigned to the HHS Lead 
Capital Planner for managing the development and modification of the IT 
portfolio selection criteria, and adequate resources have been 
committed for portfolio selection activities, including the Critical 
Partners, portfolio management tool project manager, and the Office of 
the CIO staff. Moreover, the project management personal and other 
stakeholders are aware of the portfolio selection criteria that are 
embedded into the department's portfolio management tool and also 
contained within policies and procedures. 

Finally, the HHS ITIRB approves the core IT selection criteria, 
including cost, benefit, schedule, and risk criteria, based on the 
organization's mission, goals, strategies, and priorities. Beginning in 
fiscal year 2004, HHS began scoring and ranking approximately 80 
percent of its IT investments against alignment, value, and risk 
criteria in order to determine a priority score, which is the sum of 
alignment, value, and risk criteria scores, weighted for relative 
importance. Similarly, for the fiscal year 2007 budget formulation 
process, HHS began collecting investment information on the business 
case quality, Critical Partner reviews, and cost and schedule variance 
to determine a quality score, which is the sum of the business case 
quality, Critical Partner reviews, and cost and schedule variance 
scores, weighted for relative importance. The HHS ITIRB evaluates and 
annually adjusts its portfolio selection criteria within the portfolio 
management tool. 

Despite these important steps in defining portfolio selection criteria, 
weaknesses remain. The department has not developed policies or 
procedures for modifying the portfolio selection criteria to reflect 
changes to HHS mission, goals, strategies, and priorities. In addition, 
the HHS ITIRB began reviewing the IT portfolio selection criteria this 
year. However, the process for modifying portfolio selection criteria 
is not institutionalized because the process to do so was only used 
once and there are no documented policies and procedures to ensure that 
it will be used again. Until HHS defines and implements the practices 
required for defining the portfolio criteria definition, it will not 
have the tool it needs to select investments that support its mission, 
organizational strategies, and business priorities. 

Table 11 shows the rating for each key practice required to define 
portfolio selection criteria and summarizes the evidence that supports 
these ratings. 

Table 11: Defining the Portfolio Criteria: 

Type of practice: Organizational commitments;
Key practice: 1. The organization has documented policies and 
procedures for creating and modifying IT portfolio selection criteria;
Rating: Not executed;
Summary of evidence: While HHS has policies and procedures for creating 
IT portfolio selection criteria, the department lacks policies and 
procedures for modifying the portfolio selection criteria. 

Key practice: 2. Responsibility is assigned to an individual or group 
for managing the development and modification of the IT portfolio 
selection criteria;
Rating: Executed;
Summary of evidence: The HHS Lead Capital Planner is responsible for 
managing the development and modification of the IT portfolio selection 
criteria. 

Type of practice: Prerequisites;
Key practice: 1. Adequate resources, including people, funding, and 
tools, have been committed for portfolio selection criteria activities;
Rating: Executed;
Summary of evidence: Adequate resources have been committed for 
portfolio selection criteria activities. They include the Critical 
Partners, portfolio management tool project manager, and the Office of 
the CIO staff. 

Key practice: 2. A working group has been designated to be responsible 
for developing and modifying the IT portfolio selection criteria;
Rating: Executed;
Summary of evidence: The CPIC Reengineering/PMT Implementation Team 
conducts weekly teleconferences with HHS component agencies to 
coordinate investment management issues, including the development and 
modification of IT portfolio selection criteria. According to HHS IT 
officials, this group will evolve into the Policy Advisory Board, 
which, among other things, will formalize the IT portfolio selection 
criteria activities. 

Type of practice: Activities;
Key practice: 1. The enterprisewide investment board approves the core 
IT portfolio selection criteria, including CBSR criteria, based on the 
organization's mission, goals, strategies, and priorities;
Rating: Executed;
Summary of evidence: The HHS ITIRB approves the core IT portfolio 
selection criteria, including cost, benefit, schedule, and risk 
criteria, based on the organization's mission, goals, strategies, and 
priorities. 

Key practice: 2. Project management personnel and other stakeholders 
are aware of the portfolio selection criteria;
Rating: Executed;
Summary of evidence: Project management personnel and other 
stakeholders are aware of the portfolio selection criteria, which are 
embedded into HHS's portfolio management tool and contained in policies 
and procedures. 

Key practice: 3. The enterprisewide investment board regularly reviews 
the IT portfolio selection criteria, using cumulative experience and 
event-driven data, and modifies the criteria as appropriate;
Rating: Not executed;
Summary of evidence: The HHS ITIRB began reviewing the IT portfolio 
selection criteria this year. However, the process for modifying the 
portfolio selection criteria is not institutionalized because it was 
only used once and there are no documented policies and procedures to 
ensure that it will be used again. 

Source: GAO.

[End of table] 

Process for Creating a Portfolio Is Not Documented: 

At Stage 3, organizations create a portfolio of IT investments to 
ensure that IT investments are analyzed according to the organization's 
portfolio selection criteria and to ensure that an optimal IT 
investment portfolio with manageable risks and returns is selected and 
funded. According to ITIM, creating the portfolio requires 
organizations to, among other things, document policies and procedures 
for analyzing, selecting, and maintaining the portfolio; provide 
adequate resources, including people, funding, and tools for creating 
the portfolio; and capture the information used to select, control, and 
evaluate the portfolio and maintain it for future reference. In 
creating the portfolio, the investment board must also (1) examine the 
mix of new and ongoing investments, and their respective data and 
analyses and select investments for funding and (2) approve or modify 
the performance expectations for the IT investments they have selected. 
(The complete list of key practices is provided in table 12.) 

HHS has executed 3 of the 7 key practices associated with creating the 
portfolio. Beginning in fiscal year 2004, the department began to 
create a portfolio by using its portfolio management tool to collect 
cost, benefit, schedule, risk, strategic alignment, and enterprise 
architecture information on investments accounting for 80 percent of 
the dollar value of the HHS IT investment portfolio. Each component 
agency's IT portfolio is displayed in priority order along with where 
each investment falls within the overall IT portfolio. Further, 
according to HHS IT officials, the agency has adequate resources for 
portfolio selection activities, including the Critical Partners, the 
portfolio management tool project manager, and the Office of the CIO 
staff. These officials also stated that HHS ITIRB members are also 
knowledgeable about the process of creating a portfolio. 

Nevertheless, HHS has a number of significant weaknesses in the way it 
creates a portfolio. First, it does not have policies and procedures 
that sufficiently address this critical process. Although the 
department has policies and procedures for creating IT portfolio 
selection criteria, they lack policies and procedures for using these 
criteria to analyze, select, and maintain the investment portfolio. 
Second, even though the HHS ITIRB has quarterly reviews to compare 
project and system performance with expectations for enterprisewide IT 
investments, the board is not provided with information comparing the 
performance of component agency investments against expectations. In 
addition, the board approves or modifies the performance expectations 
for the enterprisewide IT investments it has selected, but does not 
regularly approve or modify the performance expectations for component 
agency IT investments or ensure that this is done. Moreover, as 
previously mentioned, investment information has not been used to fully 
support control and evaluate decisions for component agency 
investments. Unless HHS defines and implements the practices for 
creating a comprehensive portfolio of IT investments, it will not be 
able to determine whether it has selected the mix of investments that 
best meets its needs considering resource and funding constraints. 

Table 12 shows the rating for each key practice required to create a 
portfolio and summarizes the evidence that supports these ratings. 

Table 12: Creating the Portfolio: 

Type of practice: Organizational commitment;
Key practice: 1. The organization has documented policies and 
procedures for analyzing, selecting, and maintaining the investment 
portfolio;
Rating: Not executed;
Summary of evidence: While HHS has policies and procedures for creating 
IT portfolio selection criteria, the department lacks policies and 
procedures for using these criteria to analyze, select, and maintain 
the investment portfolio. 

Type of practice: Prerequisites;
Key practice: 1. Adequate resources, including people, funding, and 
tools, are provided for the process of creating the portfolio;
Rating: Executed;
Summary of evidence: According to HHS IT officials, adequate resources 
have been committed for portfolio selection criteria activities. They 
include the Critical Partners, portfolio management tool project 
manager, and Office of the CIO staff. 

Key practice: 2. Board members are knowledgeable about the process of 
creating a portfolio;
Rating: Executed;
Summary of evidence: HHS ITIRB members are knowledgeable about the 
process of creating a portfolio;
they have now gone through the process twice. 

Key practice: 3. The investment board is provided with information 
comparing project and system performance with expectations;
Rating: Not executed;
Summary of evidence: While the investment board is provided with 
information comparing HHS enterprisewide project and system performance 
with expectations, it is not provided with information comparing the 
performance of component agency investments against expectations. 

Type of practice: Activities;
Key practice: 1. Each IT investment board examines the mix of new and 
ongoing investments and their respective data and analyses and selects 
investments for funding;
Rating: Executed;
Summary of evidence: The ITIRB examines a mix of new and ongoing 
investments through the department's portfolio management tool, which 
is used to analyze, prioritize, and select investments for funding. 

Key practice: 2. Each investment board approves or modifies the 
performance expectations for its selected IT investments;
Rating: Not executed;
Summary of evidence: While the HHS ITIRB approves the performance 
expectations for its enterprisewide IT investments, it does not have a 
similar process for approving the performance expectations for 
component agency IT investments or ensuring that this is done. 

Key practice: 3. Information used to select, control, and evaluate the 
portfolio is captured and maintained for future reference;
Rating: Not executed;
Summary of evidence: Although HHS is capturing investment information, 
the information is not yet used to fully support control and evaluate 
decisions about component agency investments. 

Source: GAO. 

[End of table] 

Criteria for Portfolio Performance Evaluations Are Not Yet Developed or 
Regularly Modified: 

This critical process builds upon the Stage 2 critical process, 
Providing Investment Oversight, by adding the elements of portfolio 
performance to an organization's investment control capacity. Compared 
with less mature organizations, Stage 3 organizations will have the 
foundation they need to control the risks faced by each investment and 
to deliver benefits that are linked to mission performance. In 
addition, a Stage 3 organization will have the benefit of performance 
data generated by Stage 2 processes. Executive-level oversight of risk 
management outcomes and incremental benefit accumulation provides the 
organization with increased assurance that each IT investment will 
achieve the desired results. (The complete list of key practices is 
provided in table 13.) 

HHS has not executed any of the seven key practices for evaluating a 
portfolio. It has yet to develop policies and procedures that address 
performance oversight from a portfolio perspective. Moreover, while the 
department annually reviews its portfolio as part of its selection 
process, it does not evaluate the investment portfolio on a continuing 
basis to assess its performance. Finally, the results of Providing 
Investment Oversight reviews from Stage 2 are important to this 
critical process. However, as previously mentioned, while the HHS ITIRB 
has oversight of enterprisewide investments, it does not regularly 
review a defined set of component agencies' investments and maintain 
visibility of other investments. Although the department's portfolio 
management tool has the ability to summarize performance metrics for 
each investment and quickly understand the status of each investment 
and any potential emerging problem area, the tool is currently only 
being used on an ad hoc basis to make portfolio oversight decisions. 
Defining and implementing processes to evaluate the performance of its 
entire portfolio would provide HHS with greater assurance that it is 
controlling the risks and achieving the benefits associated with the 
mix of investments it has selected. 

Table 13 shows the rating for each key practice required to evaluate 
the portfolio and summarizes the evidence that supports these ratings. 

Table 13: Evaluating the Portfolio: 

Type of practice: Organizational commitment;
Key practice: 1. The organization has documented policies and 
procedures for reviewing, evaluating, and improving the performance of 
its portfolio(s);
Rating: Not executed;
Summary of evidence: HHS does not have policies and procedures for 
reviewing, evaluating, and improving the performance of its portfolio. 

Type of practice: Prerequisites;
Key practice: 1. Adequate resources, including people, funding, and 
tools have been provided for reviewing the investment portfolio and its 
projects;
Rating: Not executed;
Summary of evidence: Although HHS annually reviews its portfolio as 
part of its selection process, it does not evaluate the performance on 
a continuing basis. 

Key practice: 2. Board members are familiar with the process for 
evaluating and improving the portfolio's performance;
Rating: Not executed;
Summary of evidence: Although HHS annually reviews its portfolio as 
part of its selection process, it does not evaluate the performance on 
a continuing basis. 

Key practice: 3. Results of relevant Providing Investment Oversight 
reviews from Stage 2 are provided to the investment board;
Rating: Not executed;
Summary of evidence: While the HHS ITIRB has oversight of 
enterprisewide investments, it does not effectively oversee its 
component agency IT investments. 

Key practice: 4. Criteria for assessing portfolio performance are 
developed, reviewed, and modified at regular intervals to reflect 
current performance expectations;
Rating: Not executed;
Summary of evidence: HHS does not have criteria for assessing portfolio 
performance. 

Type of practice: Activities;
Key practice: 1. IT portfolio performance measurement data are defined 
and collected consistent with portfolio performance criteria;
Rating: Not executed;
Summary of evidence: HHS does not have criteria for assessing portfolio 
performance. 

Key practice: 2. Adjustments to the IT investment portfolio are 
executed in response to actual portfolio performance;
Rating: Not executed;
Summary of evidence: Although HHS annually reviews its portfolio as 
part of its selection process, it does not evaluate the performance on 
a continuing basis. 

Source: GAO.

[End of table] 

Process for Conducting Postimplementation Reviews Is Not Defined: 

The purpose of a PIR is to evaluate an investment after it has 
completed development (that is, after its transition from the 
implementation phase to the operations and maintenance phase) in order 
to validate actual investment results. This review is conducted to (1) 
examine differences between estimated and actual investment costs and 
benefits and possible ramifications for unplanned funding needs in the 
future and (2) extract "lessons learned" about the investment selection 
and control processes that can be used as the basis for management 
improvements. Similarly, PIRs should be conducted for investment 
projects that were terminated before completion, to readily identify 
potential management and process improvements. (The complete list of 
key practices is provided in table 14.) 

HHS has not executed the six key practices for conducting PIRs. 
Although its policy calls for postimplementation reviews of IT 
investments that have recently completed implementation of the entire 
investment or a significant phase of the investment, the department 
does not have specific procedures for conducting such reviews, 
including specifying who conducts and participates in the PIR, what 
information is presented in a PIR, or how results are to be 
disseminated to decision makers. To date, HHS has conducted closeout 
reviews of two enterprisewide investments following their 
implementation;
however, while these reports do cover investment cost expectations, 
they cannot be considered PIRs because the reports do not address 
general conclusions, lessons learned, or schedule deviations. Unless 
PIRs are conducted on a regular basis, HHS will not be able to 
effectively evaluate the results of its IT investments to determine 
whether continuation, modification, or termination of an IT investment 
would be necessary in order to meet stated HHS mission objectives. 

Table 14 shows the rating for each key practice required to conduct 
PIRs and summarizes the evidence that supports these ratings. 

Table 14: Conducting Postimplementation Reviews: 

Type of practice: Organizational commitment;
Key practice: 1. The organization has documented policies and 
procedures for conducting PIRs;
Rating: Not executed;
Summary of evidence: Although, HHS has policy for conducting PIRs, the 
department does not have associated procedures for conducting such 
reviews. 

Type of practice: Prerequisites;
Key practice: 1. Adequate resources, including people, funding, and 
tools, have been provided for conducting PIRs;
Rating: Not executed;
Summary of evidence: HHS is not conducting PIRs. 

Key practice: 2. Individuals assigned to the investment board to 
conduct PIRs should be familiar with both the policies and the 
procedures for conducting such reviews;
Rating: Not executed;
Summary of evidence: HHS is not conducting PIRs. 

Type of practice: Activities;
Key practice: 1. The investment board identifies which projects will 
have a PIR conducted;
Rating: Not executed;
Summary of evidence: HHS is not conducting PIRs. 

Key practice: 2. Quantitative and qualitative investment data are 
collected, evaluated for reliability, and analyzed during the PIRs;
Rating: Not executed;
Summary of evidence: HHS is not conducting PIRs. 

Key practice: 3. Lessons learned and recommendations for improving the 
investment process are developed during the PIR, documented, and then 
distributed to all stakeholders;
Rating: Not executed;
Summary of evidence: HHS is not conducting PIRs. 

Source: GAO. 

[End of table] 

HHS Has Provided Limited Guidance to and Oversight of Component 
Agencies' Investment Management Processes: 

The ability of a department-level CIO to effectively oversee IT 
investment management processes throughout the agency depends on the 
existence of appropriate management structures with adequate 
authorities and sufficient guidance. Under the Clinger-Cohen Act of 
1996, the CIO of each agency is responsible for effectively managing 
all of the agency's IT resources. To comply with the act, HHS 
designates its CIO to be responsible for ensuring that the component 
agencies are defining and implementing effective investment management 
processes that are appropriately aligned with the department's 
processes. 

Although each component agency has staff responsible for gathering, 
maintaining, and analyzing IT investment information, the HHS Office of 
the CIO has the responsibility to define and implement overall HHS IT 
investment management practices, and monitor component agency 
investment management practices to ensure a cohesive departmental 
process and the capability exists to carry out the process. In 
accordance with this, the department's investment management policies 
and guidelines state that the component agencies are to establish and 
manage investment management processes and governance structures that 
are aligned with the department's policies and procedures. However, as 
mentioned in previous sections, the department's investment management 
policies and procedures have several weaknesses. For example, HHS does 
not have a set of documented procedures that provide decision makers 
with a clear understanding of the selection and reselection process. 

Moreover, HHS currently has no structured mechanism in place to ensure 
that the component agencies are adhering to the department's policies 
and procedures. According to HHS officials, the CIO has the authority 
to audit a component agencies IT investment management process. 
However, they were unable to provide us evidence of having performed 
any such audits. These officials also stated that the department's 
portfolio management tool is another method that will enable HHS to 
oversee component-level investment management processes. However, since 
not all component agencies are using the portfolio management tool to 
individually make select, control, and evaluate decisions, its 
usefulness in this regard is limited. Until the department develops a 
mechanism for ensuring that component agencies define and implement 
investment management processes that align with those of the 
department, it is running the risk that effective processes are being 
institutionalized at both the department and the component agency 
level. In addition, the department will be unable to ensure that it is 
optimizing its investments in IT and effectively assessing and managing 
the risks of these investments. 

HHS Does Not Have a Plan to Coordinate and Guide Improvement Efforts: 

HHS has initiated several efforts to improve its investment management 
process. Specifically, it has drafted a revised investment management 
guide that addresses the weaknesses with current guidance that we 
identify in this report. In addition, in February 2005, HHS 
incorporated capabilities into its portfolio management tool to enhance 
performance of control and evaluate functions. Specifically, the tool 
now has the capabilities to produce (1) scorecards to provide data for 
each investment in a portfolio, allowing cross investment comparisons 
on data elements collected; (2) investor maps to provide a graphical 
depiction of a portfolio in terms of up to six data categories, with 
the ability to show target and actual values; and (3) a workbook module 
to track the identification and resolution of issues that may arise 
regarding the management of an investment or set of investments. 

Although HHS has initiated these efforts, they only fully address 2 of 
the 14 Stage 2 key practices the department did not execute. 

* The draft investment management guidance, when finalized, will 
address weaknesses associated with one of the key practices for 
instituting the investment board by reflecting the current management 
process, including information on the roles of key working groups 
involved in the organization's IT investment processes, and identifying 
the manner in which investments board's processes are to be coordinated 
with other key organizational plans and processes. The guidance will 
also address the integration of the funding and selection processes, a 
key practice the department has not executed that is associated with 
selecting an investment. 

* The enhanced portfolio management tool capabilities will enhance the 
department's ability to oversee investments' performance and position 
the board to perform portfolio evaluation activities, but they will not 
fully address any of the weaknesses we identify. 

HHS has not coordinated these and additional efforts that would address 
the weaknesses we identify in this report in a comprehensive plan that 
(1) specifies measurable goals, objectives, and milestones; (2) 
specifies needed resources; (3) assigns clear responsibility and 
accountability for accomplishing tasks; and (4) is approved by senior 
management. We have previously reported that such a plan is 
instrumental in helping agencies coordinate and guide improvement 
efforts. Until HHS develops a plan that would allow for the systematic 
prioritization, sequencing, and evaluation of improvement efforts, the 
agency risks not being able to effectively establish the mature 
investment management processes that result in greater certainty about 
the outcomes of future IT investments. 

Conclusions: 

Because of the attention that has been given to investment management, 
HHS has established several of the practices needed to effectively 
manage its investments. These practices have strengthened the 
department's basic capabilities for selecting and controlling projects 
and begun to equip the department with the capabilities it needs to 
make informed decisions about competing investments. However, several 
significant weaknesses remain in the foundational practices needed to 
manage individual investments, the portfolio-level investments needed 
to manage investments as a collection, and in the level of guidance and 
oversight provided to component agency investment management processes. 
These weaknesses hamper the department's ability to ensure that it is 
managing the mix of investments that will maximize returns to the 
organization, taking into account the appropriate level of risk. 

Critical to HHS's success, going forward will be the development of an 
implementation plan that (1) is based on an assessment of strengths and 
weaknesses; (2) specifies measurable goals, objectives, and milestones; 
(3) specifies needed resources; (4) assigns clear responsibility and 
accountability for accomplishing tasks; and (5) is approved by senior 
management. Although the department has initiated improvement efforts, 
it has not developed a comprehensive plan to guide these and other 
efforts needed to improve its investment management process. Without 
such a plan and procedures for implementing it, it is unlikely that the 
department will effectively establish mature investment management 
capability. As a result, HHS will continue to be challenged in its 
ability to make informed and prudent investment decisions in managing 
its annual multibillion-dollar IT budget. 

Recommendations for Executive Action: 

To strengthen HHS's investment management capability and address the 
weaknesses discussed in this report, we recommend that the Secretary of 
the Department of Health and Human Services direct the Chief 
Information Officer to develop and implement a plan for improving the 
department's IT investment management processes. The plan should 
address the weaknesses described in this report, beginning with those 
we identified in our Stage 2 analysis and continuing with those we 
identified in our Stage 3 analysis. The plan should, at a minimum, 
provide for accomplishing the following: 

In Stage 2: 

* Develop comprehensive guidance and additional supporting guidance 
that defines and describes the complete investment management process, 
unifies existing processes enterprisewide, reflects changes in 
processes as they occur; define the operations and decision-making 
processes of the HHS investment review board and other management 
entities, such as the component agencies, involved in managing IT 
investments. 

* Ensure that HHS's investment review board's membership includes 
business representation of its component agencies as it begins to 
execute its full range of responsibilities. 

* Develop well-defined and disciplined written procedures that outline 
the process for selecting new IT proposals, reselecting ongoing IT 
investments, and integrating funding with the process of selecting an 
investment. 

* Establish a process for the investment board to regularly review and 
track the performance of a defined set of component agency IT systems 
against expectations, and take corrective actions when these 
expectations are not being met; and establish a mechanism for 
maintaining visibility into other investments. 

In Stage 3: 

* Develop and implement policies and procedures for modifying IT 
portfolio selection criteria. 

* Develop policies and procedures for using the portfolio selection 
criteria to create its portfolio. 

* Develop, review, and modify criteria for assessing portfolio 
performance at regular intervals to reflect current performance 
expectations. 

* Define and implement processes for carrying out PIRs for all IT 
investments. 

We also recommend that the HHS Secretary direct the CIO to ensure that 
the plan draws together ongoing efforts and additional efforts that are 
needed to address the weaknesses identified in this report. The plan 
should also (1) specify measurable goals, objectives, and milestones; 
(2) specify needed resources; (3) assign clear responsibility and 
accountability for accomplishing tasks; and (4) be approved by senior 
management. 

Finally, to improve the department oversight of its component agency 
investment management process, we are recommending that the HHS 
Secretary direct the HHS CIO to establish a mechanism for ensuring 
component agencies define and implement investment management processes 
that are aligned with those of the department. 

Agency Comments: 

The Department of Health and Human Services's Inspector General 
provided written comments on a draft of this report (reprinted in app. 
II). In these comments, HHS generally agreed with our findings and 
recommendations and stated that the report represented a fair 
assessment of the department's progress in IT investment management. 
The department added that it will leverage the report in its efforts to 
improve its investment management processes. 

HHS expressed differing perspectives on the inclusion of component 
agency business representation on the investment review board and the 
performance of postimplementation reviews. Specifically, regarding 
business representation on the board, the department commented that it 
used a hierarchy of investment reviews (with the first review occurring 
at the component agency) combined with ITIRB members representing 
mission support areas, such as Finance, Acquisition, and Human 
Resources, to provide a structure for making the business decisions 
regarding the department's investments. We disagree with the department 
that this arrangement provides an adequate structure for managing the 
department's investments. Because allocating resources among major IT 
investments may require fundamental trade-offs among a multitude of 
business objectives, portfolio management decisions are essentially 
business decisions, and therefore require sufficient business 
representation on the board. CIOs and executives responsible for 
mission-support functions do not constitute sufficient business 
representation because, by virtue of their responsibilities, they are 
not in the best position to make business decisions. Portfolio 
management decisions are better made by executives with business line 
decision-making authority. 

Regarding PIRs, HHS commented that it was currently informally 
performing them by conducting closeout reviews of recently implemented 
investments and annual reviews of systems in operations and 
maintenance. PIRs are conducted to determine whether cost, benefit, 
schedule, and risk expectations that were set for investments were 
achieved and develop lessons learned about the investment selection and 
control processes that can be used as the basis for management 
improvements. However, neither the closeout reviews, nor the reviews of 
systems in operations and maintenance, are addressing all these 
elements. Specifically, as we stated in our report, the closeout 
reviews do not address schedule deviations, determine whether the 
benefits were achieved, or identify lessons learned. In addition, the 
reviews of projects in operations and maintenance do not capture the 
benefits realized or identify lessons learned. 

Commenting on departmental-level oversight of component agency 
investments, HHS stated that it agrees with our recommendation to 
improve its oversight of component agency investments. It stated that 
it would use a number of mechanisms to do this, including performing 
audits to ensure alignment of component agency's processes with those 
of the department, using earned value management data to identify 
potential performance problems with most investments, and directly 
reviewing investments determined to be of high priority. We agree with 
HHS that these steps would help address some of the weaknesses in 
project oversight that we identify in this report. 

As agreed with your office, unless you publicly announce its contents 
earlier, we plan no further distribution of this report until 30 days 
from the date of this report. At that time, we will send copies to 
other interested congressional committees, the Secretary of Health and 
Human Services, and other interested parties. We will also make copies 
available to others upon request. In addition, the report will be 
available at no charge on the GAO Web site at [Hyperlink, 
http://www.gao.gov]. 

Should you or your offices have questions on matters discussed in this 
report, please contact me at (202) 512-9286 or [Hyperlink, 
pownerd@gao.gov]. Contact points for our Offices of Congressional 
Relations and Public Affairs may be found on the last page of this 
report. GAO staff who made major contributions to this report are 
listed in appendix III. 

Sincerely yours, 

Signed by:
David A. Powner: 
Director, Information Technology Management Issues: 

[End of section] 

Appendixes: 

Appendix I: Objectives, Scope, and Methodology: 

The objectives of our review were to (1) assess the Department of 
Health and Human Services's capabilities for managing its IT 
investments and (2) determine any plans HHS might have for improving 
those capabilities. 

To address our first objective, we reviewed the results of the 
department's self-assessment of Stages 2 and 3 practices using our ITIM 
framework and validated and updated the results of the self-assessment 
through document reviews and interviews with officials. We reviewed 
written policies, procedures, and guidance and other documentation 
providing evidence of executed practices, including HHS's Capital 
Planning and Investment Control Policy and Guidelines, standard 
operating procedures, portfolio management tool training manuals, and 
various instructional memorandums. We also reviewed the HHS ITIRB 
meeting materials, including quarterly status reports, meeting minutes, 
and records of decisions. We did not assess progress in establishing 
the capabilities found in Stages 4 and 5 because the department 
acknowledged that it had not executed any of the key practices in 
higher maturity stages. In addition, we conducted interviews with 
officials from the Office of the CIO, whose main responsibility is to 
oversee and ensure that HHS's IT investment management process is 
implemented and followed to determine the level of oversight and 
guidance the department is providing to its component agencies. We also 
interviewed the Centers for Medicare & Medicaid's Director for 
Investment Tracking and Assessment to determine the level of investment 
management guidance and oversight that is provided by the department. 

As part of our analysis, we selected two HHS enterprisewide and two 
component agency IT projects as case studies to verify that the 
critical processes and key practices were being applied. The projects 
selected (1) are recognized as major systems, (2) were in different 
life cycle phases, (3) represent a mix of headquarters and component 
agency investments, (4) support different functional areas, and (5) 
required different levels of funding. The four projects are described 
below: 

* HHS Public Key Infrastructure--This project supports digital 
signatures and other public key-enabled security services; it is 
intended to be the underlying architecture to support secure 
transmissions of electronic communication, such as encrypted email, by 
linking a digital key to a specific person, and issues and manages 
digital certificates. The intent of the project is to provide an 
identity proofing process that is both fast and certificate authority 
neutral. It is an agencywide strategic initiative that provides 
security services. The project is a major enterprisewide investment and 
is in the operations and maintenance phase. The project has a planned 
completion date of July 2011 and is estimated to spend $7.7 million for 
fiscal year 2006. 

* HHS Enterprise Architecture Initiative--This initiative is to provide 
the overall framework for planning and managing the technology- 
supported information assets of HHS and give the department the ability 
to identify data and process redundancies and inefficiencies in its 
information systems. The program's objectives focus on development of 
operational policies and support that enable identification, analysis 
and ongoing management of the business, and information and related 
technology architectures. It is to provide leadership, direction, and 
support to HHS's component agencies in planning and implementing 
information systems to support required business processes. As of 
fiscal year 2005, the initiative is a major enterprisewide program 
investment and is estimated to spend $15.0 million for fiscal year 
2006. 

* National Institutes of Health's Electronic Research Administration-- 
This initiative is the National Institutes of Health's infrastructure 
for conducting interactive electronic transactions for the receipt, 
review, monitoring, and administration of grant awards to biomedical 
investigators worldwide. It is to provide the technology capabilities 
for the agency to efficiently and effectively perform grants 
administration functions. The system is to provide end-to-end support 
of the grants administration process, including receipt of 
applications, review and selection of grantees, financial and progress 
reporting, issuance of final reports and grant dole-out, invention 
reporting, and interface with accounting systems. It is a major 
component agency investment and is expected to have a useful life of 13 
years. The project is estimated to spend $42.1 million for fiscal year 
2006. 

* Food and Drug Administration's Mission Accomplishment and Regulatory 
Compliance Services--This program is a comprehensive redesign and 
reengineering of core mission-critical systems at the agency, including 
the Field Accomplishments and Compliance Tracking System and the 
Operation and Administration Support System. The first of these systems 
is to support the investigation, tracking of compliance, and laboratory 
operations related to domestic operations under the agency's purview; 
the second is to primarily support the review and decision-making 
process of products imported into the United States. Both are legacy 
systems that execute on client-server platforms; while currently 
viable, the current systems cannot address many of the business needs 
due to the exponential growth in functionality on a rigid platform that 
was not designed to support the extent of change that has been 
required. The Mission Accomplishment and Regulatory Compliance Services 
is a major component agency investment and is expected to move to 
production in September 2007 and have a useful life of 10 years. The 
project is estimated to spend $10.2 million for fiscal year 2006. 

For these projects, we reviewed project management documentation, such 
as business cases, status reports, and meeting minutes. We also 
interviewed officials from the Office of the CIO for the two component 
agency investments and the project managers for the two HHS 
enterprisewide projects. 

We compared the evidence collected from our document reviews and 
interviews to the key practices in ITIM. We rated the key practices as 
"executed" on the basis of whether the agency demonstrated (by 
providing evidence of performance) that it had met the criteria of the 
key practice. A key practice was rated as "not executed" when we found 
insufficient evidence of a practice during the review or when we 
determined that there were significant weaknesses in HHS's execution of 
the key practice. In addition, HHS was provided the opportunity to 
produce evidence for key practices rated as "not executed." 

To address our second objective, we obtained and evaluated documents 
showing what management actions had been taken and what initiatives had 
been planned by the agency. This documentation included the Policy 
Advisory Board charter, draft investment management policies and 
procedures, as well as procedures and guidance for control and evaluate 
functionalities within HHS's portfolio management tool. We also 
interviewed officials from the Office of the CIO to determine efforts 
undertaken to improve IT investment management processes. 

We conducted our work at HHS headquarters in Washington, D.C., from 
January through September 2005, in accordance with generally accepted 
government auditing standards. 

[End of section] 

Appendix II Comments from the Department of Health and Human Services: 

Department Of Health & Human Services: 
Office of Inspector General: 

Washington, D.C. 20201: 

October 4, 2005: 

Mr. David A. Powner: 
Director: 
Information Technology Management Issues: U.S. Government 
Accountability Office: Washington, DC 20548: 

Dear Mr. Powner: 

Enclosed are the Department's comments on the U.S. Government 
Accountability Office's (GAO's) draft report entitled, "INFORMATION 
TECHNOLOGY: HHS Has Several Investment Management Capabilities in 
Place, but Needs to Address Key Weaknesses" (GAO-06-11). These comments 
represent the tentative position of the Department and are subject to 
reevaluation when the final version of this report is received. 

The Department appreciates the opportunity to comment on this draft 
report before its publication. 

Sincerely, 

Signed by:

Daniel R. Levinson: 
Inspector General: 

Enclosure: 

The Office of Inspector General (OIG) is transmitting the Department's 
response to this draft city' as the Department's designated focal point 
and coordinator for U.S. Government Accountability Office reports. OIG 
has not conducted an assessment of these comments and therefore 
expresses no opinion: 

COMMENTS OF THE U.S. DEPARTMENT OF HEALTH AND HUMAN SERVICES ON THE 
U.S. GOVERNMENT ACCOUNTABILITY OFFICE'S DRAFT REPORT ENTITLED, 
"INFORMATION TECHNOLOGY: HHS HAS SEVERAL INVESTMENT MANAGEMENT 
CAPABILITIES IN PLACE, BUT NEEDS TO ADDRESS KEY WEAKNESSES" (GAO-06- 
11): 

The Department of Health and Human Services (HHS) appreciates GAO's 
efforts to independently assess the Department's Information Technology 
(IT) investment management capabilities and the opportunity to respond 
to your draft report. 

The GAO report acknowledges the management attention and significant 
improvements HHS has made in this area over the past two years. During 
that time, HHS has implemented an IT portfolio management tool (PMT) 
and begun reengineering its Capital Planning and Investment Control 
(CPIC) processes. Particular strides have been made in integrating the 
CPIC process with budget formulation and prioritizing the Department's 
IT investments in terms of strategic alignment, value, risk, and 
performance during fiscal years (FY) 2006 and 2007 budget cycles. 

HHS has taken what is essentially a rapid prototype development 
approach to improving its IT investment management. We have focused on 
changing actual practices and leveraging the information sharing and 
analytical capabilities available through the PMT. These efforts were 
applied in the FY 2006 budget process and lessons learned were applied 
in the FY 2007 budget cycle. We have deliberately postponed formal 
documentation of the process until some experience was gained in using 
the process. 

Many of GAO's recommendations to HHS center on providing that 
documentation. 

Now that we have had the benefit of two years' experience with improved 
processes, HHS intends to issue policies and procedures in the near 
term. We agree with GAO that better documentation of evolving policies 
and procedures will help to institutionalize the processes and better 
ensure consistent optimal decisionmaking regarding IT investments. 
GAO's assessment will be helpful to HHS in preparing the documentation 
and focusing our efforts as the Department continues to improve its IT 
investment management processes. 

Although the Department is in agreement with the majority of GAO's 
findings and recommendations, we offer some differing perspectives in 
the following areas: 

* Inclusion of Operating Division (OPDIV) business representation on 
the Department-level IT Investment Review Board (ITIRB). HHS believes 
that the intent of this recommendation is to ensure that subject matter 
expertise is available in the targeted areas of investment that come 
before the ITIRB to provide perspective on the efficacy of the approach 
being proposed in that investment and to further ensure that the 
subject approach will have a reasonable opportunity to produce the 
benefits for which the investment is being made. HHS concurs with the 
intent of this recommendation but has chosen to pursue the intended 
result using a different approach due to the size and diversity of 
business/mission activities for which the agencies that compose HHS 
hold responsibility. HHS has hundreds of business/mission programs with 
an extremely diverse mix. To have subject matter expertise in each and 
every business/mission area that is the responsibility of HHS would 
make the Department level ITIRB so large as to become unmanageable and 
ineffective. No substantial discussion would be so relevant to the 
entire group that any level of detail for a particular investment could 
be understood, nor would the majority of the group understand how their 
business/mission related to the investment being discussed. To achieve 
relevancy of discussion, alignment to business goals and objectives, 
and understanding of impact and relationship to supporting investments 
requires subject matter expertise that is conversant in the subject at 
hand. The HHS approach of establishing a hierarchy of reviews allows 
the first level of review to occur in the agency that has direct 
responsibility for the success of that investment in support of the 
business/mission for which they themselves are the owners. This allows 
for a number of subject matter experts that have a vested interest in 
the outcomes being pursued by a particular investment and who fully 
understand the impact of a particular approach to evaluate its efficacy 
at a detailed level. As major investments move up to the Departmental 
ITIRB, business decisions regarding the mix of investments to be made 
in support of particular goals and objectives can be made with an 
assurance that the efficacy of the approach has been validated by 
subject matter experts. The Departmental ITIRB reserves the authority 
to call these experts before them to answer any questions. This allows 
the Departmental ITIRB to evaluate whether it is a good business 
decision to make an investment based on its relative value to the 
Department, which is why the Departmental ITIRB is composed of, in 
addition to each Chief Information Officer of the agencies that make up 
HHS, the Departmental executives for Finance, Acquisition, Human 
Resources, Budget, etc. HHS fully agrees with the recommendation that 
the HHS CIO should provide periodic reports on IT investment portfolio 
priorities and performance to senior Department executives, to include 
the OPDIV heads, and will work to that end. 

Department-level review and tracking of the performance of a defined 
set of OPDIV IT systems. The Department agrees with GAO's 
recommendation that the Department should provide improved oversight 
over OPDIV IT investment management processes. HHS further agrees that 
the Department should review any high risk or under performing OPDIV- 
specific IT investments. If an appropriate and aligned OPDIV process is 
established and adequate audits are in place to ensure continued 
compliance, then the Department should be able to generally rely on 
that process to provide adequate oversight to OPDIV-level investments. 
HHS collects and analyzes earned value data on all HHS Departmental and 
OPDIV major and tactical IT investments. The Department ITIRB will use 
that information to identify potential performance problems in OPDIV IT 
investments. HHS intends to manage OPDIV-level investments by 
exception. Typically, specific investments that are of sufficiently 
high priority or that have performance problems that place them on a 
Departmental "watch list" will be elevated for detailed Department-
level review. This would allow for the set of OPDIV IT systems under 
review at the Departmental level to evolve as corrective actions are 
successful and project performance improves. Using this approach the 
Departmental ITIRB can focus its attention where it is needed most. 

Define and implement processes for carrying out Post Implementation 
Review (PIRs). HHS concurs with the recommendation to better document 
the policies and procedures regarding PIRs and the evaluation of steady-
State IT investments. However, HHS believes that the implication that 
HHS does not perform those functions now is incomplete. Although the 
process is less formal than it should be, closeout reviews of recently 
implemented investments are conducted by the HHS ITIRB to identify 
lessons learned for application to future investments. Each steady-
State investment is also required to provide an annual report to the 
ITIRB discussing its ability to meet continuing or evolving business 
needs, the ability and need for technology upgrades or enhanced 
functionality, cost/benefit analysis, and a number of other aspects 
that are appropriate for managing ongoing investments. HHS does agree 
that there is opportunity for much improvement in this area, which has 
always been the HHS plan, but feels that the Department is already 
doing some of this activity in an informal manner. 

Overall, HHS finds the GAO's report on HHS IT Investment Management 
capabilities to represent a fair assessment of the Department's 
progress in this area. HHS will leverage this report in the 
Department's continuing efforts to improve IT investment management. 

[End of section] 

Appendix III: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

David A. Powner (202) 512-9286, [Hyperlink, pownerd@gao.gov]

Staff Acknowledgments: 

In addition to the person named above, Neil Doherty, Joanne Fiorino, 
Sabine Paul, Nik Rapelje, Niti Tandon, and Amos Tevelow made key 
contributions to this report. 

(310451): 

FOOTNOTES 

[1] Office of Management and Budget, Budget of the U.S. Government, 
Fiscal Year 2006, Report on IT Spending for the Federal Government for 
Fiscal Years 2004, 2005, and 2006. We did not verify these data. 

[2] Our second report, GAO, Information Technology: Centers for 
Medicare & Medicaid Services Needs to Establish Critical Investment 
Management Capabilities, GAO-06-12 (Washington, D.C.: Oct. 28, 2005), 
addresses (1) the agency's capabilities for managing its IT 
investments, (2) determining any plans the agency might have for 
improving these capabilities, and (3) examining the agency's process 
for approving and monitoring the state Medicaid management systems it 
funds. 

[3] GAO, Information Technology Investment Management: A Framework for 
Assessing and Improving Process Maturity, GAO-04-394G (Washington, 
D.C.: March 2004). 

[4] HHS refers to its component agencies as operating divisions. 

[5] Enterprisewide initiatives are mission-support and administrative 
systems that are used by all component agencies. 

[6] GAO, Financial Management Systems: Lack of Disciplined Processes 
Puts Implementation of HHS' Financial System at Risk, GAO-04-1008 
(Washington, D.C.: Sept. 23, 2004). 

[7] GAO, Information Technology: Federal Agencies Face Challenge in 
Implementing Initiatives to Improve Public Health Infrastructure, GAO- 
05-308 (Washington, D.C.: June 10, 2005). 

[8] GAO, Information Technology Management: Governmentwide Strategic 
Planning, Performance, Measurement, and Investment Management Can Be 
Further Improved, GAO-04-49 (Washington, D.C.: Jan. 12, 2004). 

[9] GAO-05-308. 

[10] We did not evaluate HHS administrative processes for managing IT 
grants to states because according to officials, both the department 
and component agencies CIOs are not directly involved in the approval 
or oversight of those IT investments. 

[11] According to HHS IT officials, for the fiscal year 2006 budget 
formulation, the business cases and Select forms were updated for 
investments that represented 80 percent of the entire HHS IT portfolio 
dollar value. The remaining 20 percent are nonmajor investments 
requesting less than $4.5 million in fiscal year 2006. 

[12] These business cases are generally referred to as "exhibit 300s." 

[13] The Office of Management and Budget evaluates the business cases 
against the following 10 criteria: acquisition strategy, project 
(investment) management, enterprise architecture, alternatives 
analysis, risk management, performance goals, security and privacy, 
performance-based management system, life-cycle costs formulation, and 
support the President's Management Agenda. 

[14] The department's portfolio management tool was implemented in May 
2004 and has not been used yet to support the entire investment 
management process. 

[15] GAO-04-394G. 

[16] GAO, Information Technology: DLA Needs to Strengthen Its 
Investment Management Capability, GAO-02-314 (Washington, D.C.: Mar. 
15, 2002); GAO, United States Postal Service: Opportunities to 
Strengthen IT Investment Management Capabilities, GAO-03-3 (Washington, 
D.C.: Oct. 15, 2002); GAO, Information Technology: Departmental 
Leadership Crucial to Success of Investment Reforms at Interior, GAO-03-
1028 (Washington, D.C.: Sept. 12, 2003); GAO, Bureau of Land 
Management: Plan Needed to Sustain Progress in Establishing IT 
Investment Management Capabilities, GAO-03-1025 (Washington, D.C.: 
Sept. 12, 2003); and GAO, Information Technology: FAA Has Many 
Investment Management Capabilities in Place, but More Oversight of 
Operational Systems Is Needed, GAO-04-822 (Washington, D.C.: Aug. 20, 
2004). 

[17] The Clinger-Cohen Act of 1996, 40 U.S.C. §§ 11311-11313. 

[18] An IT investment board is a decision-making body, made up of 
senior program, financial, and information managers, that is 
responsible for making decisions about IT projects and systems on the 
basis of comparisons and trade-offs among competing projects, with an 
emphasis on meeting mission goals. 

[19] 40 U.S.C. § 11312(b)(1). 

[20] According to the ITIM, "new" proposals include both (1) previously 
submitted IT proposals that were not originally selected for funding 
and (2) IT proposals that have never been submitted. 

[21] According to the ITIM, a process is a sequence of steps performed 
for a given purpose, and a process guide is a document that 
specifically defines the manner in which the general IT investment 
guidance will be implemented within the organization. 

[22] We reviewed two enterprisewide projects--HHS Public Key 
Infrastructure and HHS Enterprise Architecture initiative, and two 
component agency projects--National Institutes of Health's Electronic 
Research Administration and Food and Drug Administration's Mission 
Accomplishment and Regulatory Compliance Services. The projects are 
described in appendix I. 

[23] HHS conducts quarterly reviews on its enterprisewide investments 
during the period of development and annual reviews of its steady state 
enterprisewide investments, that is, those systems that have completed 
development and become operational. 

[24] Earned value management is a project management tool that 
integrates the investment scope of work with schedule and cost elements 
for investment planning and control. This method compares the value of 
work accomplished during a given period with that of the work expected 
in the period. Differences in expectations are measured in both cost 
and schedule variances. 

GAO's Mission: 

The Government Accountability Office, the investigative arm of 
Congress, exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO's commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics. 

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as "Today's Reports," on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order 
GAO Products" heading. 

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office 

441 G Street NW, Room LM 

Washington, D.C. 20548: 

To order by Phone: 

Voice: (202) 512-6000: 

TDD: (202) 512-2537: 

Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm 

E-mail: fraudnet@gao.gov 

Automated answering system: (800) 424-5454 or (202) 512-7470: 

Public Affairs: 

Jeff Nelligan, managing director, 

NelliganJ@gao.gov 

(202) 512-4800 

U.S. Government Accountability Office, 

441 G Street NW, Room 7149 

Washington, D.C. 20548: