This is the accessible text file for GAO report number GAO-07-691 
entitled 'Business Modernization: NASA Must Consider Agencywide Needs 
to Reap the Full Benefits of Its Enterprise Management System 
Modernization Effort' which was released on August 20, 2007. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to Congressional Requesters: 

United States Government Accountability Office: 

GAO: 

July 2007: 

Business Modernization: 

NASA Must Consider Agencywide Needs to Reap the Full Benefits of Its 
Enterprise Management System Modernization Effort: 

NASA's IEMP Processes: 

GAO-07-691: 

GAO Highlights: 

Highlights of GAO-07-691, a report to congressional requesters 

Why GAO Did This Study: 

Since 1990, GAO has designated the National Aeronautics and Space 
Administration’s (NASA) contract management as an area of high risk in 
part because it lacked modern systems to provide accurate and reliable 
information on contract spending. In April 2000, NASA began a system 
modernization effort, known as the Integrated Enterprise Management 
Program (IEMP). When GAO last reported on the status of IEMP in 
September 2005, NASA had begun to implement disciplined processes 
needed to manage IEMP, but had yet to implement other best practices 
such as adopting business processes that improve information on 
contract spending. This GAO report addresses (1) actions taken by NASA 
to effectively implement the disciplined processes needed to manage 
IEMP and (2) the extent to which NASA has considered the strategic 
issues associated with developing a concept of operations and defining 
standard business processes. GAO interviewed NASA officials and 
obtained and analyzed documentation relevant to the issues. 

What GAO Found: 

Since GAO last reported on NASA’s IEMP efforts, NASA implemented its 
IEMP contract management module and upgraded the software used for its 
core financial module. NASA has also taken steps to improve its 
processes for managing IEMP—including implementing improved 
requirements management and testing processes, enhancing its 
performance metrics related to tracking system defects, and developing 
an IEMP risk mitigation strategy. Further, NASA has developed 
quantitative entry and exit criteria for moving from one phase of an 
IEMP project to another—a recognized industry best practice. However, 
NASA has not yet addressed weaknesses in the areas of requirements 
development and project scheduling, which ultimately caused the agency 
to assume a greater risk that it would not identify significant system 
defects prior to implementation of the core financial upgrade. Despite 
these difficulties, NASA financial managers have stated that the core 
financial upgrade is now functioning as expected for most transactions. 
As of the end of GAO’s audit work in May 2007, NASA was working to 
correct a number of system errors, including posting errors for certain 
types of transactions. Because NASA was still working to stabilize the 
system, GAO was unable to determine the significance of these 
weaknesses. 

Further, NASA has not yet fully considered higher-level strategic 
issues associated with developing an agencywide concept of operations 
and defining standard business processes. With a planned investment of 
over $800 million for IEMP, NASA must immediately and effectively 
address these strategic building blocks if IEMP is to successfully 
address long-standing management challenges—including overseeing 
contractor performance and properly accounting for NASA’s property, 
plant, and equipment. 

• NASA officials stated that they have begun developing a concept of 
operations to describe how all of its business processes should be 
carried out. According to NASA officials, they expect to complete the 
concept of operations by the summer of 2008. Ideally, a concept of 
operations should be completed before system development begins so that 
it can serve as a foundation for system planning and requirements 
development. Nonetheless, while NASA’s IEMP efforts are already well 
under way, the completion of such a document remains essential for 
guiding the development of the remaining IEMP modules as well as any 
future upgrades. 

• As part of developing a concept of operations, NASA should also 
define standard business processes that are supported by its IEMP 
software. NASA needs to ensure that its business processes and the 
information that flows from those processes support the enterprise’s 
needs. Efforts that primarily focus on the parochial needs of a 
specific organizational unit, such as accounting, do not provide 
reasonable assurance that NASA’s agencywide management information 
needs are addressed. 

What GAO Recommends: 

GAO recommends five new actions directed at improving the processes 
used to manage IEMP, developing a concept of operations, and defining 
standard business processes. NASA concurred with all five 
recommendations and described steps it is taking to improve its 
enterprise management system modernization efforts. 

[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-691]. 

To view the full product, including the scope and methodology, click on 
the link above. For more information, contact McCoy Williams at (202) 
512-9095 or Keith Rhodes at (202) 512-6412. 

Contents: 

Letter: 

Results in Brief: 

Background: 

Progress Made in Developing Disciplined Project Management Processes, 
but Some Problems Remain: 

NASA Has Not Yet Fully Considered an Enterprise View of Its Operations 
and Processes: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Scope and Methodology: 

Appendix II: Comments from the National Aeronautics and Space 
Administration: 

Appendix III: GAO Contacts and Staff Acknowledgments: 

Figures: 

Figure 1: Current NASA Processes for Oversight of Large, Complex 
Contracts and for Asset Accounting: 

Figure 2: Example of How NASA Could Follow an Enterprise Process Using 
IEMP for Program Management and External Reporting: 

United States Government Accountability Office: 

Washington, DC 20548: 

July 20, 2007: 

The Honorable Bart Gordon: 
Chairman: 
Committee on Science and Technology: 
House of Representatives: 

The Honorable Todd R. Platts: 
House of Representatives: 

As we and others have reported in the past, the National Aeronautics 
and Space Administration (NASA) has fundamental problems with its 
financial management operations that undermine its ability to 
effectively manage its major programs and to report externally on its 
financial operations. Since 1990, we have designated NASA's contract 
management as an area of high risk, in large part because NASA has 
lacked a modern financial management system to provide accurate and 
reliable information on contract spending.[Footnote 1] In April 2000, 
NASA began a program expected to address many of its financial and 
management challenges. This program, now known as the Integrated 
Enterprise Management Program (IEMP),[Footnote 2] has been focused on 
implementing a new integrated financial management system. 
Specifically, NASA has invested in an enterprise resource planning 
(ERP) solution[Footnote 3]--a business system that is intended to meet 
the information needs of both internal and external customers and to 
promote standardization and integration of business processes and 
systems across the agency. NASA plans to complete IEMP by 2009 for a 
total cost of over $800 million. 

In April and November 2003--3 years into the IEMP implementation effort 
and with significant investment already made in the program--we issued 
a series of four reports[Footnote 4] that detailed weaknesses in NASA's 
acquisition and implementation strategy for IEMP. Specifically, we 
reported that NASA had not followed key best practices for acquiring 
and implementing IEMP and, therefore, was at risk of making a 
substantial investment in a financial management system that would fall 
far short of its stated goal of providing meaningful, reliable, and 
timely information to support effective day-to-day program management 
and external financial reporting. 

NASA is not alone in its struggle to successfully implement an 
integrated financial management system. Billions of dollars have been 
spent governmentwide to modernize financial management systems that 
have often exceeded budgeted cost, resulted in delays in delivery 
dates, and did not provide the anticipated functionality when 
implemented. In our previous report[Footnote 5] on government financial 
management systems failures, we provided our views on actions that can 
be taken to help improve the management and control of agency financial 
management system modernization efforts. Based on industry best 
practices, we identified three key practices, or building blocks, that 
are needed to ensure a solid foundation for agencies' successful system 
implementation efforts: (1) developing a concept of operations that 
would define how an agency will carry out its day-to-day operations in 
order to meet mission needs; (2) defining standard business processes 
that result in streamlined operations, rather than simply automating 
old ways of doing business; and (3) effectively implementing the 
disciplined processes necessary to manage the project. 

When we last reported on NASA's IEMP effort, in September 
2005,[Footnote 6] NASA had begun to implement a number of 
recommendations from our earlier reports, including taking steps toward 
implementing the disciplined processes necessary to manage IEMP--one of 
the key building blocks discussed above that is needed to ensure a 
solid foundation for agencies' system implementation efforts. For 
example, we reported that NASA had implemented new requirements 
management and testing processes and had developed metrics to evaluate 
the effectiveness of its system implementation processes. However, at 
that time, the agency had not implemented our recommendation to 
properly define and document system requirements for already-deployed 
IEMP modules, including the core financial module. This was important 
not only because it would affect the way the core financial module 
functions but also because it would affect NASA's ability to implement 
future upgrades and other modules expected to interface with the core 
financial module. In addition, we reported that additional enhancements 
could be made in the area of regression testing and performance 
metrics. Finally, NASA had yet to reengineer its business processes so 
that the commercial off-the-shelf (COTS) software products it had 
selected for IEMP could support these processes. 

Since we last reported, in September 2005, on NASA's efforts to 
implement IEMP, NASA implemented its IEMP contract management module 
and upgraded the version of the COTS software that is used for the core 
financial module of IEMP--which was expected to enhance the 
functionality of the core financial module. Because of your continued 
interest in NASA financial management, you asked us to provide periodic 
updates on the status of NASA's financial management improvement 
efforts--including its effort to implement IEMP. Specifically, this 
report addresses (1) actions taken by NASA to effectively implement the 
disciplined processes necessary to manage IEMP and (2) the extent to 
which NASA has considered the higher-level strategic issues associated 
with developing an agencywide concept of operations and defining 
standard business processes--two key building blocks critical to NASA's 
ability to successfully implementing its planned financial management 
system. 

To achieve these objectives, we interviewed the appropriate NASA 
officials and obtained and analyzed documentation supporting the 
process improvements that were cited by NASA. We performed our work 
from January 2006 through June 2007 in accordance with U.S. generally 
accepted government auditing standards. Details on our scope and 
methodology are included in appendix I. 

Results in Brief: 

Since September 2005, when we last reported on NASA IEMP implementation 
efforts, NASA has implemented some of the disciplined processes needed 
to manage IEMP. Specifically, since 2005, NASA has improved its 
requirements management and testing processes, enhanced its performance 
metrics program related to tracking system defects, and developed an 
IEMP risk management strategy--as we previously recommended. In 
addition, NASA has developed quantitative entry and exit criteria for 
moving from one phase of an IEMP project to another--a recognized 
industry best practice. However, weaknesses in the areas of 
requirements development and project scheduling offset some of the 
benefits associated with NASA's improved requirements management and 
testing processes, causing NASA to compress the testing phase of its 
core financial upgrade implementation and assume a greater risk that it 
would not identify significant system defects prior to implementation. 

According to NASA officials, NASA's ability to complete testing for the 
core financial upgrade within the planned implementation time frames 
was not so much due to the use of disciplined processes as it was the 
result of the extraordinary effort put forth by NASA's project 
implementation team. Despite the implementation difficulties, NASA 
financial managers have indicated that the core financial upgrade is 
now functioning as expected for most transactions. As of the end of 
March 2007, the upgrade was in a "stabilization" phase as NASA 
continued to work on correcting a number of system errors, including 
posting errors for certain types of transactions. Because the upgrade 
was still quite new and NASA was continuing to stabilize the system, we 
were unable to determine whether these weaknesses were significant. 

Although NASA has taken action to improve its processes for managing 
the implementation of individual IEMP projects, NASA has not yet fully 
considered the higher-level strategic issues associated with developing 
an agencywide concept of operations and defining standard business 
processes that are supported by its software--two key building blocks 
critical to the successful implementation of an integrated system such 
as IEMP. With a planned investment of over $800 million, completion of 
these strategic building blocks will be critical if IEMP is expected to 
address long-standing management challenges, including overseeing 
contractor performance and properly accounting for its property, plant, 
and equipment (PP&E). 

NASA officials indicated that they have undertaken a critical first 
step--they have begun developing a concept of operations to describe 
how all of its business processes should be carried out. According to 
NASA officials, they expect to complete the agency's concept of 
operations by the summer of 2008. Ideally, a concept of operations 
should be completed before system development begins so that it can 
serve as a foundation for system planning and requirements development. 
Although NASA's IEMP development effort began in April 2000, the 
completion of such a document, even at this late stage in NASA's IEMP 
effort, would be beneficial for the development of the remaining IEMP 
modules as well as any future upgrades to the core financial module. 
For NASA, an effective concept of operations would describe, at a high 
level, (1) how all of the various elements of NASA's business systems 
relate to each other and (2) how information flows among these systems. 
A concept of operations would also provide a useful tool to explain how 
business systems at the agency can operate cohesively. It would be 
geared to a NASA-wide solution rather than individual stovepiped 
efforts. Further, it would provide a road map that can be used to (1) 
measure progress and (2) focus future efforts. 

As part of an agencywide concept of operations, to best leverage its 
investment in IEMP, NASA should also analyze its current business 
processes and determine how these processes can be made more efficient 
and effective. Specifically, it will be important to define standard 
business processes supported by its IEMP software that result in 
streamlined operations rather than simply automating the old ways of 
doing business. To best leverage its investment in IEMP, NASA needs to 
ensure that the business processes supported by this system are 
developed and implemented to support the enterprise's needs rather than 
primarily focusing on the parochial needs of a specific organizational 
entity. For example, system efforts targeted at addressing accounting 
or external financial reporting needs do not provide reasonable 
assurance that the needs of the program or mission managers are 
addressed. With an ERP solution, one source of data is used for 
multiple purposes and processes should be designed to ensure that the 
data obtained and recorded meet the needs of the enterprise. NASA can 
take advantage of the efficiencies inherent in its ERP solution by 
allowing the data needed for external financial reporting to be 
produced as a by-product of the processes it uses to manage its 
mission. 

We are making five recommendations aimed at improving NASA's processes 
for managing IEMP as well as addressing the higher-level strategic 
issues associated with developing a concept of operations and defining 
standard business processes supported by NASA's IEMP software. In 
written comments on a draft of this report, NASA agreed with all five 
of our recommendations and described the steps that it is taking to 
improve its enterprise management system modernization efforts. NASA's 
comments are discussed further in the Agency Comments and Our 
Evaluation section and are reprinted in appendix II. 

Background: 

For more than a decade, we have identified weak contract management and 
the lack of reliable financial and performance information as posing 
significant challenges to NASA's ability to effectively run its largest 
and most costly programs. While NASA has made some progress in 
addressing its contract management weaknesses through improved 
management controls and evaluation of its procurement activities, NASA 
has struggled to implement a modern, integrated financial management 
system. NASA made two efforts in the past to improve its financial 
management processes and develop a supporting system intended to 
produce the kind of accurate and reliable information needed to manage 
its projects and programs and produce timely, reliable financial 
information for external reporting purposes. However, both of these 
efforts were eventually abandoned after a total of 12 years and a 
reported $180 million in spending. 

IEMP Implementation Status: 

In April 2000, NASA began its third attempt at modernizing its 
financial management processes and systems. With its current financial 
management system effort, known as IEMP, NASA has invested in an ERP 
solution that is intended to meet the information needs of both 
internal and external customers and to promote standardization and 
integration of business processes and systems across the agency. NASA 
plans to complete IEMP by 2009 for a total cost of over $800 million. 

As of March 2007, NASA had deployed the following nine IEMP functional 
components: core financial, Travel Manager, ERASMUS,[Footnote 7] resume 
management, position description management, budget formulation, Agency 
Labor Distribution System, Project Management Information Improvement, 
and contract management.[Footnote 8] Early in fiscal year 2007, NASA 
also implemented an updated version of the core financial software, 
which includes several critical enhancements to the previous core 
financial software.[Footnote 9] According to NASA, the core financial 
upgrade provided the opportunity for it to leverage the best practices 
inherent in the new version and allowed it to redesign or enhance 
business processes. NASA updated its core financial system in order to 
improve compliance with Federal Financial Management System 
Requirements, Federal Accounting Standards, and the Federal Financial 
Management Improvement Act, and to respond to GAO recommendations. 
According to NASA, the software upgrade has enabled it to implement 
critical process changes related to financial tracking and reporting, 
support the goal of achieving financial management integrity, and 
provide better project management information. NASA claims that the 
updated software has also provided other enhancements, which should 
contribute to NASA's goals of achieving a clean audit opinion and 
achieving a "Green" rating on the President's Management Agenda 
scorecard for "improved financial performance." Other IEMP modules that 
NASA plans to implement in the future include aircraft management and 
asset management. 

Prior Reporting on IEMP: 

As discussed previously, we issued a series of four reports in April 
and November 2003 that detailed weaknesses in NASA's acquisition and 
implementation strategy for IEMP in general and the core financial 
module in particular. The core financial module, which utilizes SAP 
software and is considered the backbone of IEMP, was implemented in 
June 2003. Because NASA did not follow key best practices or 
disciplined processes for acquiring and implementing IEMP, we reported 
that NASA had made a substantial investment in a financial management 
system that fell far short of its stated goal of providing meaningful, 
reliable, and timely information to support effective day-to-day 
program management and external financial reporting. We noted problems 
in the areas of requirements development, requirements management, 
testing, performance metrics, risk management, and business process 
reengineering. 

* Neither program managers nor cost estimators were involved in the 
process of defining requirements for the core financial module. As a 
result, the module was not designed to maintain the level of detailed 
cost information needed by program managers to perform contract 
oversight and by cost estimators to develop reliable cost estimates. 

* The requirements management methodology and tools used to implement 
the core financial module did not result in requirements that were 
consistent, verifiable, and traceable or that contained enough 
specificity to minimize requirement-related defects. Because NASA had 
not effectively implemented disciplined requirements management 
processes,[Footnote 10] we reported that it had increased the risk that 
it would not be able to effectively identify and manage the detailed 
system requirements necessary to properly acquire, implement, and test 
the core financial module. 

* NASA's ability to effectively test the core financial module was 
limited because of the lack of complete and specific requirements. 
Industry best practices, as well as NASA's own system planning 
documents, indicated that detailed system requirements should be 
documented to serve as the basis for effective system testing.[Footnote 
11] Because the link between these two key processes was not 
maintained, NASA had little assurance that all requirements were 
properly tested. 

* NASA also did not effectively capture the type of metrics that could 
have helped the agency understand the effectiveness of its IEMP 
management processes. For example, NASA did not employ metrics to help 
it identify and quantify weaknesses in its requirements management 
processes. Because of its lack of performance metrics, NASA was unable 
to understand (1) its capabilities to manage IEMP projects; (2) how its 
process problems affected cost, schedule, and performance objectives; 
and (3) the corrective actions needed to reduce the risks associated 
with the problems identified. 

* NASA did not consistently identify known and potential risks for the 
core financial module. Risk management processes are needed to ensure 
that a project's risk is kept at an acceptable level by taking actions 
to mitigate risk before it endangers the project's success. 

* NASA did not use the implementation of IEMP to fundamentally change 
the way it did business. Instead of reengineering its business 
processes, NASA automated many of its existing ineffective business 
processes. First, NASA did not design the system to accommodate the 
information needed to adequately oversee its contracts and programs and 
to prepare credible cost estimates. Second, NASA did not reengineer its 
contractor cost reporting processes and therefore, did not always 
obtain sufficient contract cost information needed by program managers 
to oversee contracts and needed by financial managers for external 
financial reporting. 

When we last reported on NASA's IEMP effort, in September 
2005,[Footnote 12] NASA had begun to implement a number of 
recommendations from our earlier reports--including steps toward 
implementing the disciplined processes necessary to manage IEMP. For 
example, we reported that NASA had engaged program managers to identify 
program management needs, implemented new requirements management and 
testing processes, and developed metrics to evaluate the effectiveness 
of its system implementation processes. However, at that time, the 
agency had not implemented several of our other recommendations, 
including the following: 

* Properly define and document system requirements for already-deployed 
IFMP modules, including the core financial module. This is important 
not only because it would affect the way the core financial module 
functions but also because it would affect NASA's ability to implement 
future upgrades and other modules expected to interface with the core 
financial module. 

* Enhance regression testing processes and performance metrics. 

* Develop a risk mitigation plan. 

* Reengineer its business processes so that the commercial off-the- 
shelf software products selected for IEMP could support these 
processes. 

At the time of our last report, NASA was making plans to reengineer 
some of its business processes. However, because the agency was in the 
very early planning stage of implementing this recommendation, the 
details for how NASA would accomplish its objectives were still vague. 
Overall, our September 2005 report concluded that it was not possible 
to assess whether NASA's plans would accomplish its stated goal of 
enhancing the core financial module to provide better project 
management information for decision-making purposes. 

Progress Made in Developing Disciplined Project Management Processes, 
but Some Problems Remain: 

Since September 2005, when we last reported on NASA IEMP implementation 
efforts, NASA has implemented some of the disciplined processes needed 
to manage IEMP. Specifically, NASA has, as we previously recommended, 
implemented more effective requirements management and testing 
processes, improved its performance metrics program related to tracking 
system defects, and developed an IEMP risk management strategy. In 
addition, NASA has developed quantitative entry and exit criteria for 
moving from one phase of an IEMP project to another--a recognized 
industry best practice. However, weaknesses in the areas of 
requirements development and project scheduling have undermined some of 
the progress made in other key areas. As a result, NASA struggled to 
complete required systems testing and deliver the agency's core 
financial upgrade. Ultimately, through the heroic efforts of the core 
financial upgrade team, NASA delivered the upgrade within about 2 weeks 
of the October 30, 2006, planned completion date. According to NASA 
officials, the system is functioning as expected for most transactions. 
However, until the end of March 2007, the upgrade was in a 
"stabilization" phase as NASA worked on correcting a number of system 
errors, including posting errors for certain types of transactions. 
Because the upgrade was still quite new and NASA was continuing to 
stabilize the system, we were unable to determine the significance of 
these weaknesses. 

Improvements Made to NASA Requirements Management and Testing 
Processes: 

Since our September 2005 report, NASA has used its new requirements 
management process--which documents sufficiently detailed requirements 
that are traceable from the highest (most general) level to the lowest 
(most detailed) level in NASA's requirements management system--for 
both the core financial upgrade and the contract management module. For 
example, we selected several requirements for both the core financial 
module and the contract management module and validated that the 
requirements management process (1) clearly linked related requirements 
consistent with industry standards and (2) contained the information 
necessary to understand how each requirement should be implemented and 
tested in a quantitative manner. 

Because NASA developed and is now using a disciplined requirements 
management process, it has the quantitative information necessary to 
support disciplined testing processes. NASA's disciplined testing 
processes include (1) documentation of the scenarios that need to be 
tested to obtain adequate test coverage, (2) requirements that are 
traced to the test cases to ensure that all requirements are tested, 
(3) instructions and other guidance for the testers, and (4) an 
effective regression testing program.[Footnote 13] Although NASA had 
disciplined requirements management and testing processes in place for 
the implementation of both the contract management module and the core 
financial upgrade, difficulties related to requirements development and 
project scheduling, discussed later, forced NASA to compress the 
testing phase of its core financial upgrade implementation. As a 
result, according to NASA officials, completion of testing for the core 
financial upgrade required an extraordinary effort on the part of 
NASA's implementation team. 

NASA Has Implemented an Effective Metrics Program and Risk Management 
Strategy: 

Since we last reported, in September 2005, NASA has also enhanced its 
metrics measurement program, which is used to evaluate the 
effectiveness of its project management processes by identifying the 
causes of process defects. Understanding the cause of a defect is 
critical to evaluating the effectiveness of an organization's project 
management processes, such as requirements management and testing. For 
example, if a significant number of defects are caused by inadequate 
requirements definition, then the organization knows that corrective 
actions are needed to improve the requirements definition process. When 
we last reported, NASA had made progress in this important area by 
collecting information on the causes of system defects it identified in 
its regression testing efforts but was not collecting similar 
information on defects identified by users and lacked a formal process 
for fully analyzing the data related to system defects by identifying 
the trends associated with them. Since that time, NASA has developed 
additional metrics to track and analyze such things as the number of 
changes made to requirements while a system is under development. In 
addition, NASA has developed processes for tracking and analyzing 
defects identified by IEMP users. For example, since implementation of 
the core financial upgrade, NASA has maintained spreadsheets showing 
specific information on each service request submitted by users, 
including the type of defect involved and the status of the request. 

Finally, NASA has also developed a comprehensive risk management 
strategy. Specifically, NASA now has an IEMP Risk Management Plan that 
outlines the standard processes and techniques for identifying, 
analyzing, planning, tracking, and controlling risks as well as 
defining the roles and responsibilities for each level of project risk 
management. In applying these techniques to the core financial upgrade, 
NASA officials documented the risks that they identified for the 
project, as well as their mitigation strategies, likelihood, 
consequence, and criticality. According to NASA officials, their risk 
management process worked well and was one of the key reasons for the 
success of the core financial upgrade. For example, using the metrics 
information discussed previously, NASA officials said they were able to 
assess the risks of changing requirements late in the project and then 
mitigate those risks by performing additional testing. 

NASA Uses a Recognized Industry Best Practice to Move from One Project 
Phase to the Next: 

In addition to the disciplined processes discussed above, NASA has also 
taken action to establish the use of quantitative entry and exit 
criteria to move from one phase of an IEMP project to another. The use 
of such criteria is considered an industry best practice. Entry 
criteria are the minimum essential items considered necessary to enter 
into a given project phase, while exit criteria are the minimum 
essential items necessary to consider a given project phase 
successfully completed. For example, the NASA entry criterion for 
moving into the regression testing phase requires that all remaining 
significant defects from the integration testing phase be resolved and 
successfully retested before regression testing can begin. NASA 
demonstrated application of this criterion when it implemented the 
contract management module. About 3 weeks before the scheduled start 
date of regression testing, the project had not yet successfully 
completed all test scenarios, and several significant defects had not 
been fully resolved. In addition, a series of critical corrections from 
the software vendor had not yet been delivered, and the project team 
agreed that there would not be adequate time to test the corrections 
prior to beginning the scheduled regression testing. Consequently, the 
team decided to push back the scheduled date for the contract 
management module to begin operating. 

For the core financial upgrade, NASA officials said that they used 
entry and exit criteria as one of the management tools to determine 
whether the project should move forward. However, rather than adopt a 
"hard stop" approach when criteria were not met, they used the criteria 
to make sure that all appropriate factors were considered before moving 
forward, including the risks of not meeting certain criteria. Any 
instances in which the project team thought exceptions to the criteria 
were warranted were ultimately reviewed and decided on by higher levels 
of NASA management, which helped ensure that such decisions were 
adequately considered. 

Improved Requirements Development and Project Scheduling Needed: 

Weaknesses in the areas of requirements development and project 
scheduling offset some of the benefits associated with NASA's improved 
requirements management and testing processes--causing NASA to assume a 
greater risk that it would not identify significant system defects 
prior to implementation. Weaknesses in requirements development and 
project scheduling processes resulted in NASA having to compress the 
testing phase of its core financial upgrade implementation. As a 
result, according to NASA officials, NASA's ability to complete testing 
for the core financial upgrade within the planned implementation time 
frames ultimately depended on the extraordinary effort put forth by 
NASA's project implementation team. 

Because of weaknesses in NASA's requirements development process, it 
did not have reasonable assurance that it identified all appropriate 
requirements for the core financial upgrade when the project began. 
Consequently, NASA continued making changes to the requirements very 
late in the project's development, resulting in increased risks, 
delays, and a compressed testing schedule. Improperly defined or 
incomplete requirements have commonly been identified as a root cause 
of system failure. Although NASA made a concerted effort, as part of 
its core financial system upgrade, to involve program managers and 
other key stakeholders in the requirements development process, it did 
not follow standard industry practices for identifying and documenting 
user requirements. 

According to the Software Engineering Institute (SEI),[Footnote 14] to 
help ensure that critical requirements are identified, an organization 
should have a well-documented, disciplined requirements development 
process that, among other things, (1) defines how customer needs will 
be elicited, developed, and validated; (2) specifies how to identify 
and ensure involvement of relevant stakeholders; and (3) ensures that 
people involved in the requirements development process are adequately 
trained in such topics as requirements definition and analysis. In 
addition, it is critical that requirements flow from an organization's 
business requirements or its concept of operations. However, as 
discussed later, NASA has not yet completed a concept of operations. 

In developing its core financial upgrade requirements, NASA established 
a task force, consisting of both financial and program managers, whose 
primary objective was to "review, assess, and document Program/Project 
Management requirements as they relate to financial management." In 
addition, other groups of program managers were asked to review the 
requirements and provide input to the task force. However, according to 
NASA officials, they have not yet documented and institutionalized 
requirements development procedures as recommended by SEI. Lacking 
documentation, NASA cannot ensure that appropriate procedures are 
followed and that all appropriate stakeholders are included in the 
process so that all requirements are identified. Moreover, the 
requirements that were addressed by the task force and user groups were 
at a very high or general level and therefore, lacked a level of 
specificity that is needed to ensure that users' needs are met. 

Because it did not have a well-documented, disciplined requirements 
development process in place to provide reasonable assurance that all 
requirements had been identified, NASA delayed finalizing the system's 
expected functionality until April 2006--about 6 months before the 
upgrade was expected to be implemented--and continued to change some 
requirements for several months after that. Delays in finalizing the 
requirements contributed to delayed testing and a compressed testing 
schedule. To meet the planned October 30, 2006, implementation date, 
the three rounds of system testing for the core financial upgrade were 
scheduled to occur from mid-June through September 22, with less than a 
week between each round. A less compressed schedule could have allowed 
more time between the testing cycles to perform necessary actions, such 
as additional development work and testing to adequately address the 
defects that had been identified. This, in turn, could have reduced the 
risk that significant system defects would not be detected prior to 
implementation. 

One key to developing a realistic project schedule is determining the 
sequence of activities, which requires identifying and documenting the 
dependencies among the various project activities. For example, testing 
activities cannot be completed before the software being tested is 
developed, and software should not be developed until requirements have 
been defined. However, NASA did not document the dependencies among the 
detailed project tasks for the core financial upgrade and therefore, 
did not have reasonable assurance that the project schedule established 
at the start of the project was realistic. According to NASA officials, 
they recognized this risk and adopted several processes to identify and 
mitigate the weakness, such as having knowledgeable project officials 
review the schedule and holding weekly status meetings to determine 
whether the tasks were on schedule. 

While the techniques used by NASA to constantly evaluate and adjust the 
schedule are considered best practices and allowed NASA to gain 
confidence in the schedule as the core financial upgrade project 
progressed, they were not sufficient to ensure that the original 
schedule was reasonable because they relied on ad hoc processes rather 
than a formal task dependency analysis. If NASA had also identified the 
task dependencies for the core financial upgrade, it would likely not 
have had to rely on extraordinary efforts to complete the project. 
Rather, project management would have been in a better position to 
assess the difficulty in meeting the planned schedule and to take 
further steps to reduce this risk, such as scaling back some aspects of 
the project or adding more resources to the project. 

According to NASA officials, through the heroic efforts of IEMP staff-
-their knowledge and experience with past projects and a considerable 
amount of overtime invested--the core financial project team was able 
to complete testing and other work within about 2 weeks of the planned 
implementation date. Although NASA has made significant improvements in 
its project management processes, NASA management recognizes that 
weaknesses in its requirements development and project scheduling 
processes have undermined some of the progress made. Despite the 
implementation difficulties, NASA financial managers have indicated 
that the core financial upgrade is now functioning as expected for most 
transactions. Through the end of March 2007, the upgrade was in a 
"stabilization" phase as NASA continued to work on correcting a number 
of system errors, including posting errors for certain types of 
transactions. Because NASA was continuing to stabilize the system 
during most of our audit period, we were unable to determine the 
significance of these weaknesses. 

NASA Has Not Yet Fully Considered an Enterprise View of Its Operations 
and Processes: 

Although NASA has significantly improved its processes for implementing 
IEMP projects, these improvements are directed at implementing the 
desired functionality for an individual project. NASA has not yet fully 
considered the higher-level strategic issues that affect how useful 
IEMP will be in addressing long-standing management challenges-- 
including problems associated with stovepiped systems and parochial 
interests of individual NASA components as well as problems in 
overseeing contractor performance and properly accounting for its 
property, plant, and equipment. NASA envisions IEMP to be a leading- 
edge business system[Footnote 15] that will provide management 
information needed for mission success, meet the needs of internal and 
external customers, and promote standardization and integration of 
business processes and systems across NASA. To achieve this vision, it 
is critical that NASA develop an agencywide concept of operations and 
adopt standard business processes that are supported by its software. 

Concept of Operations Would Provide an Important Foundation for IEMP: 

NASA officials stated that they have undertaken a critical first step 
to achieving their vision for IEMP--they have begun developing a 
concept of operations to describe how all of its business processes 
should be carried out. NASA created a framework for developing a 
concept of operations in fiscal year 2006 and plans to complete it by 
the summer of 2008, according to NASA officials. Ideally, a concept of 
operations should be completed before system development begins so that 
it can serve as a foundation for system planning and requirements 
development. Nonetheless, the completion of such a document even at 
this late stage in NASA's IEMP effort would be beneficial for the 
development of the remaining IEMP modules as well as any future 
upgrades to the core financial module. In addition, once a concept of 
operations is complete, NASA could reassess the modules that are 
already implemented and determine whether and how they might need to be 
modified to best meet its agencywide needs. 

A concept of operations defines how an organization's day-to-day 
operations are (or will be) carried out to meet mission needs. The 
concept of operations includes high-level descriptions of information 
systems, their interrelationships, and information flows. It also 
describes the operations that must be performed, who must perform them, 
and where and how the operations will be carried out. Further, it 
provides the foundation on which requirements definitions and the rest 
of the systems planning process are built. Normally, a concept of 
operations document is one of the first documents to be produced during 
a disciplined development effort and flows from both the vision 
statement and the enterprise architecture.[Footnote 16] According to 
Institute of Electrical and Electronics Engineers (IEEE) 
standards,[Footnote 17] a concept of operations is a user-oriented 
document that describes the characteristics of a proposed system from 
the users' viewpoint. The key elements that should be included in a 
concept of operations are major system components, interfaces to 
external systems, and performance characteristics such as speed and 
volume. 

For NASA, an effective concept of operations would describe, at a high 
level, (1) how all of the various elements of NASA's business systems 
relate to each other and (2) how information flows among these systems. 
Further, a concept of operations would provide a useful tool to explain 
how business systems at the agency can operate cohesively. It would be 
geared to a NASA-wide solution rather than individual stovepiped 
efforts.[Footnote 18] Further, it would provide a road map that can be 
used to (1) measure progress and (2) focus future efforts. While NASA's 
enterprise architecture efforts, when fully completed, can be used to 
help understand the relationships between the various systems, a 
concept of operations document presents these items from the users' 
viewpoint in nontechnical terms. Such a document would be invaluable in 
getting various stakeholders, including those in the programs and 
administrative activities, to understand how the business systems are 
expected to operate cohesively and how they fit into "the big picture." 

Adopting Enterprise Business Processes Would Help NASA Transform the 
Way It Does Business: 

As part of an agencywide concept of operations, to best leverage its 
investment in IEMP, NASA should also analyze the agency's current 
business processes and determine how these processes can be made more 
efficient and effective. Specifically, NASA needs to ensure that the 
business processes supported by this system are developed and 
implemented to support the enterprise's needs rather than primarily 
focusing on the needs of a specific organizational entity. For example, 
system efforts targeted only at addressing accounting or external 
financial reporting needs--as was done during the initial 
implementation of the core financial module--do not provide reasonable 
assurance that the needs of the mission managers or other support 
organizations are addressed as well. Our review identified an important 
opportunity for NASA to leverage its investment in IEMP by using the 
system's inherent business processes to meet the enterprise's needs. 

Agencies such as NASA that invest in ERP solutions to meet their 
enterprise needs often face difficulty in shifting from the stovepiped 
processes of the past to the enterprise processes that underlie the ERP 
concept. According to technical experts,[Footnote 19] a key benefit of 
an effective ERP system is that the system provides the entire entity 
consistent data regardless of which entity component generates a 
request or for what purpose; the system maintains data based on the 
concept of "one truth." In other words, in non-ERP environments, one 
system may have one amount for an agency's obligations while another 
system has another amount for the same obligations. While either of 
these systems may be the "official system," actions and plans may be 
based on information in the other system. In order for all of an 
organization's actions and plans to be consistent, the same information 
needs to be available and used by all segments of that organization. 
Under the ERP concept, it does not matter whether an individual is in 
budget, accounting, procurement, or any other organizational component; 
the answer to the question of "how much money has been obligated and 
how much is still available" is consistent. 

One example of an opportunity for NASA to use enterprise processes to 
accomplish multiple needs is in the area of program oversight and 
accounting for PP&E. NASA typically spends about 85 percent of its 
budget procuring goods and services from its contractors each year. 
Therefore, much of the cost information NASA needs to oversee its 
programs and compile its external financial reports resides with its 
contractors. For its larger contracts,[Footnote 20] NASA generally 
obtains cost data from monthly contractor financial management reports, 
commonly referred to as NASA Form 533s. NASA Form 533 captures planned 
and actual contract costs and, according to NASA officials, is used for 
budgeting, monitoring contract costs, and controlling program 
resources. The Office of the Chief Financial Officer (OCFO) also uses 
NASA Form 533 to capture the costs reported on the agency's financial 
statements. However, NASA Form 533 does not contain information related 
to the status of work performed on a contract. Therefore, for all major 
acquisitions[Footnote 21] and for development or production contracts 
and subcontracts valued at $20 million or more, in addition to NASA 
Form 533s, NASA's contractors are also required to provide monthly 
contract cost performance reports. Each of these reports is treated as 
a stovepiped activity; that is, they provide cost information for a 
given contract in two different formats and are used by different 
organizations and for different purposes within NASA. 

For those contracts for which NASA receives contract cost performance 
reports in addition to Form 533s, program managers use the cost 
performance reports to monitor contract performance, while the OCFO 
uses NASA Form 533 to accrue costs that, among other things, are 
reported on the agency's financial statements. Although NASA Form 533 
and the cost performance report reflect cost data pertaining to the 
same contract, the level of detail provided in each report may vary 
considerably depending on the contractor cost reporting requirements 
negotiated as part of the contract. For example, the cost data required 
by program managers to manage major acquisitions are often more 
detailed than those required by the OCFO. In addition, because neither 
the cost performance report nor NASA Form 533 contains the information 
needed by the OCFO to properly account for equipment and other property 
acquired from contractors, NASA also relies on periodic, summary-level 
information provided by its contractors to report property amounts on 
its financial statements. 

When NASA initially implemented its IEMP core financial module in June 
2003, it did not adequately consider program managers' needs and did 
not design the system to accommodate the more detailed cost data 
contained in contractor cost performance reports. Since that time, NASA 
has redesigned the coding structure embedded in the core financial 
module to be more consistent with the work breakdown structure (WBS) 
coding used by program managers. However, NASA continues to use cost 
data from NASA Form 533--generally reported by contract line 
items[Footnote 22]--to populate the core financial module. As a result, 
as shown in figure 1, NASA uses a complex, NASA-specific process to 
allocate the costs reported on NASA Form 533 to the WBS codes in IEMP 
based on available funding. 

Figure 1: Current NASA Processes for Oversight of Large, Complex 
Contracts and for Asset Accounting: 

[See PDF for image] 

Source: GAO.

[End of figure] 

In a very simplified example,[Footnote 23] if NASA received a Form 533 
showing $1,000 of cost incurred for a particular contract line item and 
two WBS codes pertained to that line item, NASA would allocate the 
costs to those two WBS codes. Assuming WBS 1 had more funding available 
than WBS 2, NASA might allocate $600 to WBS 1 and $400 to WBS 2. 
However, the contract cost performance report might show that the 
actual costs were $500 for WBS 1 and $500 for WBS 2. Although this 
allocation process reorganizes cost data reported on NASA Form 533 into 
the same reporting structure that is used by program managers, it still 
results in different costs, maintained in different systems, used for 
different purposes. Accordingly, these separate processes do not result 
in the "one truth" that is provided when an ERP view is taken. 

Further, this dual reporting approach has not addressed one of NASA's 
long-standing financial reporting weaknesses: reporting on its PP&E. 
For example, NASA's processes do not allow the agency to identify 
capital costs--that is, those that should be recorded as assets--as 
they are incurred. Instead, as we recently reported,[Footnote 24] the 
agency performs a retrospective review of transactions entered into its 
property system to determine which costs should be capitalized. This 
subsequent review is labor-intensive and error-prone, and therefore 
increases the risk that not all related costs will be properly captured 
and capitalized. 

Figure 2 provides an example of how NASA could use IEMP to implement an 
enterprise process that (1) provides the necessary data for the 
enterprise operations and (2) reduces the burden on NASA and contractor 
officials. 

Figure 2: Example of How NASA Could Follow an Enterprise Process Using 
IEMP for Program Management and External Reporting: 

[See PDF for image] 

Source: GAO.

[End of figure] 

As shown in figure 2, if NASA received only one monthly report 
containing contract cost data reported in sufficient detail for both 
program management and financial reporting purposes, then it could 
record these costs directly in IEMP without first going through an 
allocation process as it does now. All individuals and components 
throughout NASA could then use the same cost data that reside within 
IEMP for a given contract; IEMP could provide different arrays of cost 
information based on each user's needs, but all cost information for a 
given contract would come from one source. For example, as shown in 
figure 2, the program manager could use the cost data from IEMP along 
with other supplemental contractor performance information, such as 
labor hours used, to see if the project is meeting expectations. In 
addition, if discrete WBS codes were established to identify the costs 
associated with the acquisition of property, then IEMP could 
automatically capitalize those costs and financial managers could 
readily determine how much cost has been recorded for 
property.[Footnote 25] The key is that under the enterprise process 
concept, single data entry is used for multiple purposes. Since the 
enterprise view provides "one truth," an adequate audit trail over the 
data used to report property can be maintained simply by reviewing the 
cost reports that were provided by the contractors. Thus, NASA can take 
advantage of the efficiencies inherent in an ERP solution by allowing 
the data needed for external financial reporting to be produced as a by-
product of the processes it uses to manage its mission. 

Conclusions: 

NASA has made significant strides in developing and implementing more 
disciplined processes for supporting its IEMP efforts since our last 
report in 2005. NASA has recognized the need for the disciplined 
processes necessary to reduce risks to acceptable levels, as evidenced 
by its implementation of several of our recommendations. More 
importantly, NASA officials recognize that improving system 
implementation processes is a continuous effort and that certain 
processes--particularly requirements development and project 
scheduling--may need more attention. However, the real key to realizing 
NASA's IEMP vision is for NASA's management to develop an overarching 
strategy for managing its agencywide management system development 
effort. We are encouraged that NASA has begun to develop a concept of 
operations. As part of the development of this document, it will be 
critical for NASA to define (1) the agency's business processes and 
information needs and (2) the types of systems that will be used to 
carry out these processes and produce the necessary information. 
Another critical factor in developing a concept of operations will be 
analyzing the agency's current business processes and determining how 
these processes can be made more efficient and effective. For example, 
NASA can take advantage of the efficiencies inherent in the solution it 
has selected by utilizing an enterprise view to produce the data needed 
for external financial reporting as a by-product of the processes it 
uses to manage its mission. Unless NASA devotes immediate, focused 
attention to taking these critical strategic planning steps, it will 
continue to face the risk that its planned $800 million investment in 
IEMP will not achieve the transformational changes necessary to provide 
NASA with the information needed to make well-informed business 
decisions and to effectively manage its operations. 

Recommendations for Executive Action: 

To help ensure that disciplined processes are effectively implemented 
for future IEMP modules, upgrades, or other business systems, we 
recommend that the NASA Administrator direct the IEMP Program Director 
to take the following two actions. 

* Establish requirements development policies and procedures regarding 
(1) how customer needs will be elicited, developed, and validated; (2) 
how to identify and ensure the involvement of relevant stakeholders; 
and (3) required training in such topics as requirements definition and 
analysis to be provided to people involved in the requirements process. 

* Develop policies and procedures that require project schedules to 
include the identification and documentation of dependencies among 
various project tasks. 

To help ensure that future IEMP projects are designed to carry out 
NASA's mission in an efficient manner that meets the needs of all 
users, we recommend that the NASA Administrator establish as a high 
priority the completion of a concept of operations that addresses 
NASA's business operations for both its mission offices and 
administrative offices (such as financial management and human capital) 
before any new implementation efforts begin. 

Once the concept of operations is complete, we recommend that the NASA 
Administrator review the functionality of previously implemented IEMP 
modules for the purpose of determining whether enhancements or 
modifications are needed to bring them into compliance with the concept 
of operations. 

To help ensure that NASA receives the maximum benefit from its reported 
$800 million investment in IEMP, we recommend that the NASA 
Administrator establish policies and procedures requiring approval to 
establish or maintain business processes that are inconsistent with the 
processes inherent in the COTS solutions selected for IEMP. The reasons 
for any decisions made to not implement the inherent COTS processes 
should be well-documented and approved by the Administrator or his 
designee. At a minimum, approved documentation should address any 
decisions to maintain current contractor cost reporting processes 
rather than revise these processes to facilitate the use of one 
consistent source of cost data. 

Agency Comments and Our Evaluation: 

We received written comments on a draft of this report from NASA, which 
are reprinted in appendix II. NASA agreed with our recommendations and 
described the approach and steps it is taking or plans to take to 
improve its enterprise management system modernization efforts. We are 
encouraged that a number of these steps are already under way, 
including the establishment of an IEMP advisory body representing 
NASA's missions and centers. As NASA progresses in addressing our 
recommendations, it is important that it focuses on the concepts and 
underlying key issues we discussed, such as considering the need to 
reengineer key business processes to support agencywide needs and to 
take full advantage of its ERP solution. We continue to believe that 
careful consideration of all of the building blocks and key issues we 
identified will be integral to the success of NASA's efforts. NASA also 
provided technical comments, which we incorporated as appropriate. 

As agreed with your offices, unless you announce its contents earlier, 
we will not distribute this report further until 30 days from its date. 
At that time, we will send copies to interested congressional 
committees, the NASA Administrator, and the Director of the Office of 
Management and Budget. We will make copies available to others upon 
request. In addition, the report will be available at no charge on the 
GAO Web site at [hyperlink, http://www.gao.gov]. 

If you or your staff have any questions concerning this report, please 
contact McCoy Williams at (202) 512-9095 or williamsm1@gao.gov or Keith 
Rhodes at (202) 512-6412 or rhodesk@gao.gov. Key contributors to this 
report are acknowledged in appendix III. Contact points for our Offices 
of Congressional Relations and Public Affairs may be found on the last 
page of this report. 

Signed by:

McCoy Williams: 
Director: 
Financial Management and Assurance: 

Signed by:

Keith A. Rhodes: 
Chief Technologist: 
Applied Research and Methods: 
Center for Technology and Engineering: 

[End of section] 

Appendix I: Scope and Methodology: 

To determine whether the National Aeronautics and Space Administration 
(NASA) has improved its management processes for implementing the 
Integrated Enterprise Management Program (IEMP), we reviewed project 
management documentation for several IEMP projects, including the core 
financial upgrade and the contract management module. The documentation 
we reviewed for these projects included requirements management 
documents, detailed testing plans, project schedules, risk management 
plans, and metrics documentation. We also interviewed numerous IEMP 
officials, including the IEMP Director, the Director and Assistant 
Director at the IEMP Competency Center, and the Manager of IEMP 
Application Development and Software Assurance. In addition, we 
interviewed the leader of a NASA team that provided an independent 
assessment of the core financial upgrade project to obtain his views of 
IEMP management processes. 

To assess NASA's implementation of disciplined processes, we reviewed 
industry standards and best practices from the Institute of Electrical 
and Electronics Engineers, the Software Engineering Institute, and the 
Project Management Institute. To assess the effectiveness of NASA's 
requirements management processes, we performed a traceability analysis 
of several requirements for both the contract management module and the 
core financial upgrade, which demonstrated that there was traceability 
among the different levels of requirements and with testing 
documentation. To determine whether NASA had adequately and 
systematically determined the information needs of key users of IEMP 
data when developing system requirements, we reviewed documentation of 
NASA's requirements identification effort for the core financial 
upgrade and interviewed a number of program managers and staff who 
worked on various space and science programs at three NASA centers-- 
Marshall Space Flight Center, Johnson Space Center, and Goddard Space 
Flight Center. We also met with officials from the Office of the Chief 
Financial Officer (OCFO), including the Deputy Chief Financial Officer, 
and with officials from the Office of the Chief Engineer to obtain 
their opinions regarding the requirements of the core financial 
upgrade. In addition, we discussed the requirements development 
methodology with IEMP management. 

To determine the results of the implementation of the core financial 
upgrade, we met with both IEMP and OCFO officials. We reviewed data on 
the amount and types of system defects that were identified by users 
during the project's stabilization phase. We also obtained written 
responses to specific questions about the results of the implementation 
from three NASA centers. 

To determine the extent to which NASA has considered the higher-level 
strategic issues associated with developing an enterprisewide concept 
of operations and defining standard business processes, we met with 
senior management from IEMP and the OCFO. In addition, we also 
discussed these issues with senior officials in the Office of the NASA 
Administrator. We also interviewed IEMP officials about NASA's current 
processes for recording contract costs. We also discussed this issue 
with officials from the OCFO, the Office of the Chief Engineer, and the 
Office of Program and Institutional Integration. In addition, we 
obtained documentation of NASA's plans for reengineering processes 
related to the costs of capital assets. We briefed NASA officials on 
the results of our audit, including our findings and their 
implications. On May 25, 2007, we requested comments from NASA and we 
received them on June 21, 2007. NASA also separately provided technical 
comments. Our work was performed from January 2006 through June 2007 in 
accordance with U.S. generally accepted government auditing standards. 

[End of section] 

Appendix II: Comments from the National Aeronautics and Space 
Administration: 

National Aeronautics and: 
Space Administration:
Office of the Administrator: 
Washington, DC 20546-0001:

June 21, 2007: 
Mr. McCoy Williams: 
Director: 
Financial Management and Assurance: 
United States Government Accountability Office: 
Washington, DC 20548:

Dear Mr. Williams: 

Thank you for the opportunity to review and comment on the draft report 
entitled "NASA Must Consider Agencywide Needs to Reap the Full Benefits 
of Its Enterprise Management System Modernization Effort" (GAO-07-691), 
dated June 2007. I appreciate the Government Accountability Office's 
(GAO) interest in NASA's business modernization as the Agency continues 
to improve its business environment, which is also fully consistent 
with my own personal priorities for enhancing the Agency's ability to 
better manage and report on its mission and goals. 

NASA appreciates the GAO noting that "NASA has made significant strides 
in developing and implementing more disciplined processes" since it 
last reported in this area in 2005. Clearly, NASA's Integrated 
Enterprise Management Program (IEMP) has matured in its processes since 
its inception in 2000, which has contributed to its successes in 
planning, developing, implementing, and operating Agency-wide business 
applications – a daunting task for any Federal organization. 
Nevertheless, we also recognize more work needs to be done if NASA is 
to achieve the full benefit of its investments in this area. 

In its draft report, the GAO makes five recommendations aimed at 
improving the processes used to manage IEMP, ensuring that NASA's 
business systems support NASA's missions and maximize the Agency's 
benefit of its investment in its business systems modernization. NASA's 
response to each of these five recommendations is provided below. 

Recommendation 1: Establish requirements development policies and 
procedures regarding (1) how customer needs will be elicited, 
developed, and validated, (2) how to identify and ensure the 
involvement of relevant stakeholders, and (3) required training in such 
topics as requirements definition and analysis to be provided to people 
involved in the requirements process. 

Response: NASA concurs with this recommendation. As noted in the GAO 
report, IEMP has made significant improvements to its requirements 
processes for projects in formulation and implementation. In addition 
to those improvements, the TEMP is now using an iterative software 
development implementation approach that provides for formal product 
reviews at the end of each development iteration, soliciting feedback 
from a wide range of stakeholders. 

This feedback is used to validate requirements, as well as to elicit 
additional or changed requirements early in the project life cycle. The 
new approach provides more opportunity to validate stakeholder and user 
requirements than the previous development methodology. Further, NASA's 
Operations Management Council (OMC) chartered the Management/Business 
Systems Integration Group (MBSIG) as an advisory body to recommend and 
prioritize future IEMP requirements. Membership is comprised of 
representatives from Mission Support Organizations, Mission 
Directorates, and Centers. The focus of the MBSIG is to assess and 
prioritize future IEMP requirements from a strategic and cross 
functional perspective, taking into account the end-to-end business 
processes. Recommendations from the MBSIG are reviewed and approved by 
the OMC. In conjunction with the establishment of the M/BSIG, IEMP has 
established the role of Integration Manager, which is responsible for 
applying systems engineering techniques to the identification, 
prioritization, development, and collation of requirements for future 
business systems implementation. The IEMP Integration Manager and the 
IEMP Competency Center will assess the systems' capabilities to 
efficiently meet the proposed requirements, and recommendations and 
assessments will be vetted by the MBSIG. 

Notwithstanding the above improvements, IEMP recognizes that policies 
and guidance in this area are not cohesively documented. The IEMP 
Integration Manager will document a singular policy framework regarding 
the identification and prioritization of requirements associated both 
with new efforts and with modifications to operational systems. This 
framework will also describe a process to elicit customer needs, 
develop requirements, prioritize those requirements, and obtain cross 
function Agency approval. The target date for completion of this 
document is December 2007. 

Recommendation 2: Develop policies and procedures that require project 
schedules to include the identification and documentation of 
dependencies among various project tasks. 

Response: NASA concurs with this recommendation. As noted in the above 
response, IEMP has adopted a revised software development 
implementation approach that is geared toward providing short-term 
deliverables. The new methodology, which IEMP began using in early 
fiscal year 2007, requires the identification of feature and object 
dependencies at the start of each iteration to ensure that completed 
components can be delivered by the end of the iteration. The use of the 
time-boxed, iterative approach addresses the weakness in the previous 
methodology that attempted to manage the project schedule with long-
term, activity-based items that were not well-suited to the 
identification of object dependencies. In summary, the new development 
approach adopted by IEMP enables more detailed identification and 
documentation of project tasks and dependencies, which meets the GAO's 
recommendation. 

Recommendation 3: Establish as a high priority the completion of a 
concept of operations (ConOps) that addresses NASA's business 
operations for both its mission offices and administrative offices 
(such as financial management and human capital) before any new 
implementation efforts begin. Response: NASA concurs with this 
recommendation. The IEMP Integration Manager is currently establishing 
a plan to develop a ConOps, which will (1) provide a description of 
NASA's business systems characteristics from an operational 
perspective; (2) facilitate understanding of the overall system goals 
with users, managers, and others; (3) form an overall basis for long-
range operations planning and provide guidance for development and/or 
update of subsequent system definition documents, such as the system 
specification and the interface specification, and (4) describe the 
user organization and mission from an integrated user/system point of 
view (per ANSI/AIAA G-043-1992, Guide for the Preparation of 
Operational Concept Documents). Target completion for the ConOps is the 
summer of 2009 due to the need to engage all functional organizations 
owning business systems (and their customers) in the development of 
this important document. 

Recommendation 4: Review the functionality of previously implemented 
IEMP modules for the purpose of determining whether enhancements or 
modifications are needed to bring them into compliance with the concept 
of operations. 

Response: NASA concurs with this recommendation. TEMP is currently 
working with mission projects to identify business system gaps from the 
perspective of the project management community. Data from this 
activity, to be completed in the July 2007 timeframe, will be 
incorporated into the ConOps. Any enhancements or modifications to 
existing IEMP modules will be assessed by the MBSIG following the 
completion of the ConOps. 

Recommendation 5: Establish policies and procedures requiring approval 
to establish or maintain business processes that are inconsistent with 
the processes inherent in the COTS solutions selected for IEMP. 

Response: NASA concurs with this recommendation. NASA recognizes that 
it has, over time, created several complex business processes which 
have resulted in complicated and convoluted software configurations and 
extensions which create operational inefficiencies. As noted in the 
response to Recommendation 1 above, the role of the IEMP Integration 
Manager and the IEMP Competency Center will be to assess the systems' 
capabilities to efficiently meet the proposed requirements. Assessment 
results will be vetted through the MBSIG to ensure cross functional 
agency-wide impacts have been addressed. This assessment process will 
be included within the forthcoming requirements policy framework 
identified in the response to Recommendation 1. Meanwhile, NASA is 
already taking steps to assess its cost collection processes with a 
target to streamline the business processes while continuing to meet 
the information needs of project management, financial management, and 
asset management.

Technical comments to the draft report have been provided to GAO 
separately. 

My point of contact for this matter is Mr. Bobby German, Program 
Director for NASA's Integrated Financial Management Program. He may be 
contacted by e-mail at bobby.german@nasa.gov or by telephone at (202) 
358-2498. 

Sincerely,

Signed by:

Shana Dale:
Deputy Administrator:

[End of section] 

Appendix III: GAO Contacts and Staff Acknowledgments: 

GAO Contacts: 

McCoy Williams, (202) 512-9095 or williamsm1@gao.gov: 

Keith A. Rhodes, (202) 512-6412 or rhodesk@gao.gov: 

Acknowledgments: 

In addition to the contacts named above, staff members who made key 
contributions to this report were Diane Handley, Assistant Director; J. 
Christopher Martin, Senior Level Technologist; Francine DelVecchio; 
Kristi Karls; and Lauren Catchpole. 

FOOTNOTES 

[1] GAO, High-Risk Series: An Update, GAO-07-310 (Washington, D.C.: 
January 2007). 

[2] The effort was formerly known as the Integrated Financial 
Management Program (IFMP). According to NASA, IFMP was renamed to 
reflect the addition of program management and labor distribution. 

[3] An ERP solution is an automated system using commercial off-the- 
shelf software consisting of multiple, integrated functional modules 
that perform a variety of business-related tasks such as payroll, 
general ledger accounting, contract management, and supply chain 
management. 

[4] GAO, Business Modernization: Improvements Needed in Management of 
NASA's Integrated Financial Management Program, GAO-03-507 (Washington, 
D.C.: Apr. 30, 2003); Business Modernization: NASA's Integrated 
Financial Management Program Does Not Fully Address Agency's External 
Reporting Issues, GAO-04-151 (Washington, D.C.: Nov. 21, 2003); 
Information Technology: Architecture Needed to Guide NASA's Financial 
Management Modernization, GAO-04-43 (Washington, D.C.: Nov. 21, 2003); 
and Business Modernization: Disciplined Processes Needed to Better 
Manage NASA's Integrated Financial Management Program, GAO-04-118 
(Washington, D.C.: Nov. 21, 2003). 

[5] GAO, Financial Management Systems: Additional Efforts Needed to 
Address Key Causes of Modernization Failures, GAO-06-184 (Washington, 
D.C.: Mar. 15, 2006). 

[6] GAO, Business Modernization: Some Progress Made toward Implementing 
GAO Recommendations Related to NASA's Integrated Financial Management 
Program, GAO-05-799R (Washington, D.C.: Sept. 9, 2005). 

[7] ERASMUS is an executive management information system that provides 
information on costs, schedule, and risks for all significant NASA 
programs and projects. According to the IEMP Program Director, ERASMUS 
was discontinued in April 2007. 

[8] The contract management module of IEMP is intended to support 
contract and grant writing and administration, and procurement workload 
management. 

[9] Both the previous and updated versions of the core financial 
software are from SAP, a company whose integrated software is used by 
many of the world's largest corporations. 

[10] According to the Software Engineering Institute, requirements 
management is a process that establishes a common understanding between 
the customer and the software project manager regarding the customer's 
business needs that will be addressed by a project. A critical part of 
this process is to ensure that the requirements development portion of 
the effort documents, at a sufficient level of detail, the problems 
that need to be solved and the objectives that need to be achieved. 

[11] Testing is the process of executing a program with the intent of 
finding errors. 

[12] GAO-05-799R. 

[13] Regression testing is the practice of testing changes to a 
software application before it is released to ensure that modifications 
have not caused unintended effects and that the system still complies 
with its specified requirements. 

[14] Mary Beth Chrissis, Mike Konrad, and Sandy Shrum, CMMI: Guidelines 
for Process Integration and Product Improvement, SEI Series in Software 
Engineering (Boston, Mass.: Pearson Education, May 2004). 

[15] A business system is an information system that is used to support 
business activities such as acquisition, financial management, 
logistics, strategic planning and budgeting, installations and 
environment, and human resources management. 

[16] An enterprise architecture is a blueprint that defines, both in 
logical terms (including integrated functions, applications, systems, 
users, work locations, and information needs and flows) and technical 
terms (including hardware, software, data, communications, and 
security), how an organization's information technology systems operate 
today and how they are to operate in the future and provides a road map 
for the transition. 

[17] IEEE Std. 1362-1998. 

[18] For example, as we discuss later, NASA receives two different 
types of cost reports from its major contractors. Even though both 
types of reports pertain to the same costs for a given contract, one 
report is used for financial management while the other is used for 
program management. A concept of operations might describe how NASA 
could use the information from only one report for both purposes. 

[19] Thomas F. Wallace and Michael H. Kremzar, ERP: Making It Happen; 
The Implementers' Guide to Success with Enterprise Resource Planning 
(New York: John Wiley & Sons, Inc., 2001). 

[20] NASA requires its contractors to report monthly accrued costs on 
NASA Form 533 for cost type, price redetermination, and fixed-price 
incentive contracts with a performance period of 1 year or more and a 
contract value of $500,000 to $999,000 or a performance period of less 
than a year but with a contract value of $1 million or more. 

[21] A major acquisition, as defined by OMB Circular A-11, means a 
system or project requiring special management attention because of its 
importance to the mission or function of the agency, a component of the 
agency, or another organization; is for financial management and 
obligates more than $500,000 annually; has significant program or 
policy implications; has high executive visibility; has high 
development, operating, or maintenance costs; or is defined as major by 
the agency's capital planning and investment control process. 

[22] Contract line items are usually consistent with higher levels of 
the WBS, but do not contain the details that are found in the lower 
levels of the WBS. 

[23] This example is for illustrative purposes only; the dollar amounts 
in the example are not based on actual NASA data. 

[24] GAO, Property Management: Lack of Accountability and Weak Internal 
Controls Leave NASA Equipment Vulnerable to Loss, Theft, and Misuse, 
GAO-07-432 (Washington, D.C.: June 25, 2007). 

[25] In its technical comments on a draft of this report, NASA stated 
that it plans to establish unique WBS codes for contractors to use to 
report asset costs on the Form 533. It is too early to determine the 
extent to which these plans might address agencywide information needs. 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "Subscribe to Updates." 

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office 441 G Street NW, Room LM: 
Washington, D.C. 20548: 

To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202) 
512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 

E-mail: fraudnet@gao.gov: 

Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400: 

U.S. Government Accountability Office, 441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Paul Anderson, Managing Director, AndersonP1@gao.gov (202) 512-4800: 

U.S. Government Accountability Office, 441 G Street NW, Room 7149:
Washington, D.C. 20548: