This is the accessible text file for GAO report number GAO-07-38 
entitled 'Small Business Innovation Research: Agencies Need to 
Strengthen Efforts to Improve the Completeness, Consistency, and 
Accuracy of Awards Data' which was released on November 21, 2006.

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov.

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately.

Report to Congressional Committees:

United States Government Accountability Office:

GAO:

October 2006:

Small Business Innovation Research: Agencies Need to Strengthen Efforts 
to Improve the Completeness, Consistency, and Accuracy of Awards Data:

Small Business Innovation Research:

GAO-07-38:

GAO Highlights:

Highlights of GAO-07-38, a report to congressional committees:

Why GAO Did This Study:

The Small Business Innovation Research (SBIR) program was created to 
increase the use of small businesses to meet federal research needs and 
commercialize the results of this research. To monitor the program, the 
Small Business Administration (SBA) requires participating agencies to 
provide, in a standard format, specific data on all SBIR awards they 
make. SBA then compiles these data into a database known as Tech-Net. 
Congress also required SBA to create, by 2001, a restricted and more 
comprehensive database that would provide information for government 
agencies to use in evaluating the program. GAO was asked to identify 
the (1) types of data that agencies report to SBA for inclusion in the 
Tech-Net database, (2) extent to which these data are provided in a 
standard format, (3) extent to which SBA has established the government-
use database, and (4) extent to which SBIR agencies have developed and 
implemented techniques to track commercialization of SBIR projects. GAO 
reviewed 8 of the 11 agencies participating in SBIR.

What GAO Found:

Federal agencies participating in the SBIR program annually submit over 
40 data elements to SBA for each award they make. These data include 
information on the award, such as value and a descriptive abstract; 
information on the recipient, such as name and gender; and information 
about the firm receiving the award, such as number of employees and 
location. Participating agencies submit most of the information 
required by SBA, but they are not consistently providing all required 
data elements, including the number of employees in the firm, and the 
gender and socio-economic status of the award recipient, resulting in 
incomplete sections of the database. Agencies stated that this happens 
because they do not collect all of the information that SBA wants and 
because SBA’s requirements change regularly. 

Some participating agencies are not submitting SBIR award data in the 
standard format required by SBA, and although SBA’s quality assurance 
processes correct most of these problems, they do not correct all of 
them. In 2004 and 2005, about 25 percent of the data provided by five 
participating agencies did not comply with SBA’s format. Formatting 
inconsistencies occur because the template SBA has provided agencies 
for reporting data can be edited. According to SBA, identifying and 
correcting inconsistently formatted data involves considerable 
resources, therefore the agency has focused its quality assurance 
efforts only on key data elements needed to track awards; other fields, 
such as those containing demographic data, are generally not corrected. 
As a result, comprehensive program evaluations may be limited by the 
quality of the data in these fields. SBA officials expect this problem 
to be resolved by fiscal year 2007, when all data will be submitted via 
an Internet interface that will not allow changes in the format. 

SBA is 5 years behind schedule in meeting the congressional mandate to 
implement a restricted government-use database for the SBIR program. 
SBA had planned to meet this requirement by expanding its Tech-Net 
database to include a restricted government-use section. SBA officials 
attributed the delay in meeting the 2001 deadline primarily to 
increased security requirements needed for the database, agency 
management changes, and budgetary constraints. SBA officials expect the 
government-use section of Tech-Net to be operational by October 1, 
2006, when safeguards to protect the proprietary commercialization 
information in the database are in place.

Most agencies GAO reviewed systematically gather data on the 
commercialization success of SBIR-funded projects. Five of these eight 
agencies regularly survey all awardees to gather information on program 
participation, including commercial success, and one agency is about to 
start a similar survey. In contrast, two agencies only gather anecdotal 
success stories from a small sample of SBIR awardees. SBA and agency 
officials generally agree that despite their best efforts, obtaining 
commercialization information from awardees remains a major challenge.

What GAO Recommends:

GAO recommends that SBA and SBIR participating agencies work together 
to improve the quality of the data in SBA’s Tech-Net database. SBA and 
the SBIR participating agencies included in this report generally 
agreed with our recommendation.

[Hyperlink: http://www.gao.gov/cgi-bin/getrpt?GAO-07-38.]

To view the full product, including the scope and methodology, click on 
the link above. For more information, contact Anu K. Mittal at (202) 
512-3841 or mittala@gao.gov.

Contents:

Letter:

Results in Brief:

Background:

Agencies Provide Most of the Required SBIR-Award Data Elements to SBA, 
but Some Data Submitted are Incomplete:

Agencies Generally Comply with SBA's Formatting Guidance, but Key Data 
on SBIR Award Recipients May Be Inconsistent or Inaccurate:

SBA Is Five Years Behind Schedule in Meeting its Obligation to 
Implement a Government-Use SBIR Database:

Tracking Data on the Commercialization Success of SBIR Projects Varies 
Among Agencies and Remains a Challenge:

Conclusions:

Recommendation for Executive Action:

Agency Comments and Our Evaluation:

Appendix I: Objectives, Scope and Methods:

Appendix II: Comments from the Small Business Administration:

GAO Comments:

Appendix III: Comments from the Department of Defense:

Appendix IV: Comments from the National Aeronautics and Space 
Administration:

Appendix V: Comments from the National Institutes of Health:

Appendix VI: GAO Contact and Staff Acknowledgments:

Abbreviations:

USDA: Department of Agriculture:

DOD: Department of Defense:

DOE: Department of Energy:

EPA: Environmental Protection Agency:

FDA: Food and Drug Administration:

NASA: National Aeronautics and Space Administration:

NIST: National Institute of Standards and Technology:

NIH: National Institutes of Health:

NSF: National Science Foundation:

R&D: research and development:

SBA: Small Business Administration:

SBIR: Small Business Innovation Research:

United States Government Accountability Office:

Washington, DC 20548:

October 19, 2006:

The Honorable Sherwood Boehlert: 
Chairman: 
The Honorable Bart Gordon: 
Ranking Minority Member: 
Committee on Science: 
House of Representatives:

The Honorable Vernon J. Ehlers: 
Chairman: 
The Honorable David Wu: 
Ranking Minority Member: 
Subcommittee on Environment, Technology and Standards: 
Committee on Science: 
House of Representatives:

Recognizing the potential of small businesses to be a source of 
significant innovation, the Congress established the Small Business 
Innovation Research (SBIR) program in 1982. From fiscal year 1983 
through fiscal year 2004, federal agencies that participated in the 
SBIR program awarded over $17 billion in grants, contracts, or 
cooperative agreements to over 82,000 projects. The primary goals of 
the SBIR program are to stimulate technological innovation, meet 
federal research and development (R&D) needs, foster participation by 
minority and disadvantaged persons in technological innovation, and 
increase the commercial success (commercialization) of innovation that 
is derived from federally funded R&D.[Footnote 1] Because the Congress 
did not define what constitutes commercial success of federally funded 
R&D or how best to measure it, agencies have used different 
commercialization outcomes for the SBIR program, such as the sale of 
the resulting SBIR-funded product or process, the extent to which SBIR 
firms have received non-SBIR funding, or the creation of new jobs or 
products. The SBIR program is currently scheduled to expire on 
September 30, 2008.

Every federal agency with an R&D budget of $100 million or more is 
required to establish and operate an SBIR program funded by 2.5 percent 
of the agency's budget for research conducted by others, called 
extramural research. Currently, 11 federal agencies participate in the 
SBIR program.[Footnote 2] Each agency manages its own program, 
including targeting research areas, reviewing proposed projects, and 
making research awards through grants, contracts, or cooperative 
agreements. The Small Business Administration (SBA) plays a central 
administrative role by, for example, issuing policy directives to the 
participating federal agencies, collecting data from participating 
agencies on awards and recipients, and reporting program results 
annually to the Congress.

Over the last 24 years, the SBIR program has been reauthorized and 
modified by the Congress at various times. For example, the Small 
Business Research and Development Enhancement Act of 1992 directed SBA 
and participating agencies to, among other things, emphasize the goal 
of increasing commercialization of research results and to improve the 
government's dissemination of program-related data.[Footnote 3] As a 
result, agencies were required to include commercialization potential 
as a criterion for selecting award recipients. During this same period, 
SBA began to develop a publicly available database, known as Tech-Net, 
that contained information on all awards made through the SBIR 
program.[Footnote 4] The Tech-Net database is intended to be, among 
other things, an electronic gateway of technology information and 
resources for researchers, scientists, and government officials about 
federally-funded leading edge technology research. The Small Business 
Innovation Research Program Reauthorization Act of 2000 formalized this 
database by requiring SBA to develop, maintain, and make available to 
the public a searchable, up-to-date, electronic database that contained 
SBIR award information.[Footnote 5] The 2000 reauthorization act also 
required SBA to develop and maintain another restricted government 
database that would contain additional information on commercialization 
not contained in the public Tech-Net database, thereby allowing better 
evaluations of the SBIR program on an ongoing basis. This database was 
to be established by mid-2001 and made available only to government 
agencies and certain other authorized users.

As the program has evolved over time, congressional direction has 
focused, among other things, on the ability of SBIR award recipients to 
commercialize the results of their research, as evidenced by the 
increased emphasis on commercialization in the 1992 and 2000 
reauthorizations of the program. At various points in the life of the 
program, we have reported that the SBIR program has been successful in 
increasing commercialization of research results and that agencies have 
used various methods to measure the commercial success of the projects 
they fund.[Footnote 6] Largely these methods have consisted of surveys 
of award recipients to obtain data on indicators of commercial success 
and soliciting "success stories" voluntarily provided by SBIR award 
recipients. However, we also reported that these methods provided only 
"snapshots" of commercial success and did not allow for a systematic 
demonstration of changes in program commercialization rates over time. 
We reported that the lack of clarity on how much emphasis agencies 
should place on commercialization versus other SBIR program goals had 
also created challenges for assessing the program's results.

In the context of efforts to monitor and evaluate the success of the 
SBIR program, you requested that we identify the (1) types of data that 
participating SBIR agencies are reporting to SBA for inclusion in the 
Tech-Net database; (2) extent to which agencies provide data for the 
Tech-Net database in a standard format to enable program evaluation; 
(3) extent to which SBA has met the mandate to establish, by mid-2001, 
a government-use database that can be used for program evaluation; and 
(4) extent to which participating SBIR agencies have developed and 
implemented techniques to track the commercialization success of SBIR 
projects.

In conducting this study, we reviewed the SBIR-related activities at 8 
of the 11 SBIR participating agencies, which account for over 98 
percent of the total dollars awarded by the program in fiscal year 
2005.[Footnote 7] To determine the types of data these participating 
agencies are reporting to SBA and the extent to which SBA has complied 
with the requirement to establish a government-use database that can be 
used for program evaluation purposes, we compared the provisions in the 
Small Business Innovation Research Program Reauthorization Act of 2000, 
SBA's amended Policy Directive implementing the Act issued in September 
2002, and other guidance with agency SBIR Tech-Net database reports. To 
determine the extent to which data for the Tech-Net database are 
provided in a standard format to enable program evaluation, we compared 
the data from participating agencies with data in SBA's Tech-Net 
database for fiscal years 2004 and 2005, the 2 most recent years for 
which data were available. Our data reliability review focused on SBA's 
data system and internal controls, rather than on the systems and 
internal controls agencies use to create the data provided to SBA. To 
assess the reliability of the data in SBA's Tech-Net database, we 
interviewed SBA officials about the database and reviewed related 
documentation. We determined that the data are sufficiently reliable 
for our purpose, which is to report on SBA's efforts to ensure 
consistency and completeness of the data it receives. We used GAO's 
data reliability guidance to identify key attributes of data quality 
that can facilitate program evaluation. We also interviewed SBA and 
agency officials to determine the extent to which the government-use 
database requirements have been implemented. To determine the extent to 
which participating agencies and SBA have developed and implemented 
techniques to evaluate commercialization success of SBIR projects, we 
reviewed agency documentation and interviewed SBA and agency officials. 
Appendix I contains a detailed discussion of the scope and methodology 
of our review. We conducted our review from April 2006 through 
September 2006 in accordance with generally accepted government 
auditing standards.

Results in Brief:

Federal agencies participating in the SBIR program annually submit to 
SBA a wide range of descriptive information about each award they make. 
Each year SBA requires participating agencies to provide over 40 data 
elements for each SBIR award made. These data include information about 
the award itself, such as the award amount, a descriptive abstract of 
the project, and a unique tracking number; information about the award 
recipient such as the name, gender, and socio-economic status; and 
information about the type of firm that received the award, such as 
number of employees and geographic location. However, we determined 
that although participating agencies are providing most of the 
information requested by SBA, they are not consistently providing the 
full range of required information. For example, two of the eight 
agencies we reviewed had not consistently provided SBA data on the 
gender and socio-economic status of SBIR award recipients, and five did 
not provide data on the number of employees working at the firms that 
received SBIR awards. As a result, certain sections of the database 
needed for comprehensive program evaluation are incomplete. Agencies 
cited a variety of reasons for not providing the data requested by SBA, 
including frequent changes in SBA's data requirements and differences 
in the types of data agencies collect versus the types of data that SBA 
wants to collect.

Some participating agencies are not submitting SBIR award data in the 
standard format required by SBA, and while some of these problems are 
corrected by SBA's quality assurance processes, others are not. Since 
2000, SBA officials have worked with participating agencies to ensure 
greater consistency in how the data required for the Tech-Net database 
are formatted and have developed a reporting template that includes the 
required data fields and instructions for appropriate data entry. 
However, almost a quarter of the data provided by five participating 
agencies in 2004 and 2005 still did not comply with SBA's formatting 
guidance. As a result, SBA officials said that they have to spend a 
considerable amount of time and resources correcting these data, and 
are unable to correct all of them. SBA's current quality assurance 
efforts therefore focus on obtaining complete and accurate data only 
for those fields essential to tracking specific awards, such as the 
tracking numbers and award amounts, and not correcting data in fields 
that contain demographic information about award recipients. The 
primary reason for the inconsistently formatted data is that the 
template SBA has provided for reporting data can be edited, and some 
agencies have done so and introduced different formats. According to 
SBA, by fiscal year 2007, this issue should be resolved because 
agencies will have to submit their data through an Internet interface 
that will no longer allow the agencies to change the format of titles 
and data. In light of the problems we identified with the Tech-Net 
database and the implications for these errors to limit evaluations of 
the SBIR program, we are recommending that SBA work with participating 
agencies to strengthen efforts to improve the quality of the data.

SBA has not met the congressional mandate to develop and implement, by 
June 2001, a government-use database that can be used for evaluating 
the SBIR program. To meet the congressional requirement to develop and 
implement a government-use database, SBA planned to expand the existing 
Tech-Net database and include a restricted government-use section that 
would be accessible only to government agencies and other authorized 
users. This government-use section of the Tech-Net database would rely 
on information already gathered for the Tech-Net database and 
supplemented by information on the commercialization outcomes for SBIR 
awards. However, SBA officials told us that they have been unable to 
meet the requirement to implement a government-use database by 2001 
because of management changes that have occurred at the agency and 
because of budgetary constraints. To date, with the help of two 
contractors, SBA has developed the framework for (1) importing data 
into the government-use section of the database, and (2) an Internet- 
based interface that would allow agencies and award recipients to 
access and enter commercialization information. According to SBA, the 
government-use section of the database will not be fully operational 
until the agency resolves certain outstanding issues, such as making 
the commercialization information secure because this information is 
considered proprietary and confidential. SBA officials expect that the 
government-use section of the database will be operational by October 
1, 2006; however, they also recognize that additional enhancements to 
improve the efficiency of the database may still be needed after the 
system is operational.

Seven of the eight agencies participating in the SBIR program that we 
reviewed are gathering data on the commercialization success of SBIR- 
funded projects; however, the methodological rigor of their methods 
varies significantly. Under the program's authorizing legislation and 
implementation guidance, agencies have been given considerable 
flexibility in tracking the commercial success of their SBIR-funded 
projects. Specifically, five of the eight agencies systematically and 
periodically survey SBIR award recipients to gather a range of 
information on program participation, including commercialization 
experience. For example, DOD annually surveys SBIR award recipients to 
gather data on (1) sales of SBIR-funded research results, (2) 
commitments SBIR recipients have received for additional development 
funding, and (3) whether SBIR-funded research results have been used by 
other federal programs. Two of the eight agencies we reviewed use less 
systematic data gathering efforts and instead focus on gathering 
success stories and conducting periodic follow-up with a small sample 
of SBIR award recipients. For example, USDA periodically contacts a 
small group of award recipients to update and obtain information on 
their commercialization experiences. The remaining agency--NIST--has 
developed a survey that it plans to send to all of its SBIR award 
recipients in 2006. Regardless of the methods used to gather 
commercialization information, SBA and agency officials believe that 
several factors complicate their efforts to obtain this information in 
a standardized and complete manner. For example, these officials told 
us that many years may elapse from the time an SBIR award is granted to 
the time a product or process achieves commercial success and 
maintaining contact with award recipients during this time period is 
often difficult. Moreover, during this time, firms may change their 
names, be purchased by other firms, or start new businesses to pursue 
the project's commercial potential, making it even more difficult for 
the agencies to track these firms and link them to their original SBIR 
awards.

Background:

The Small Business Innovation Development Act of 1982 established the 
SBIR program and identified four program goals: technological 
innovation, commercialization of the research results, the use of small 
businesses to meet agencies' R&D needs, and participation in federal 
R&D by minorities and disadvantaged persons. The legislation provided 
for a competitive three-phased program: phase I to determine the 
technical and scientific merit and feasibility of a proposed research 
idea; phase II to further develop the idea, taking into account its 
commercial potential; and phase III to commercialize the resulting 
product or process with private or federal investment but no additional 
SBIR funding.

Under the SBIR program provisions, federal agencies that have external 
R&D budgets of $100 million or more are required to use 2.5 percent of 
these budgets to establish and operate an SBIR program. SBA oversees 
and coordinates the efforts of the eleven agencies currently 
participating in the program. In this capacity, SBA coordinates the 
participating agencies' schedules to announce opportunities for firms 
to apply for awards, called a solicitation, and provides access to 
these solicitations through its Web site. As part of its oversight 
effort, SBA collects SBIR data from the participating agencies, 
aggregates the data, and uses the data to, among other things, monitor 
the program and report annually to the Congress.

In reauthorizing the SBIR program in 1992, the Congress stated its 
intention to expand and improve the program by emphasizing its goal of 
increasing private sector commercialization, increasing participation 
in federal R&D by small businesses, and improving the government's 
dissemination of program-related information. One of the new provisions 
under the 1992 legislation requires agencies, when evaluating phase II 
proposals, to consider their commercial potential, including the 
recipients' experiences commercializing the results from previous SBIR 
awards, commitments accompanying the proposals for developmental 
funding from sources other than the SBIR program, and other factors.

The SBIR program was again reauthorized in 2000 by the Small Business 
Innovation Research Program Reauthorization Act of 2000. The 2000 
legislation directed SBA and participating agencies to, among other 
things, expand the scope of publicly available information on specific 
awards and to annually report data on their SBIR programs to SBA. The 
act required that SBIR phase II award recipients be requested to 
voluntarily provide information to the agencies describing the outputs 
and outcomes of their SBIR award. The act also required SBA to 
establish, by mid-2001, a searchable and up-to-date electronic database 
available for public use, and a restricted government-use database. To 
accomplish this mandate, SBA envisioned expanding the electronic 
database, known as Tech-Net, that it had developed in the late 1990s, 
into two sections: a public-use section and a restricted government-use 
section. The public-use section of the Tech-Net database would provide 
access to nearly all of the statutorily-required award information for 
SBIR awards gathered by the agencies. The public-use section was 
intended to be an electronic gateway of technology information and 
resources for researchers, scientists, and government officials who are 
seeking information on potential small business partners, contractors, 
or leading edge technology research. The government-use section would 
be solely used for program evaluation purposes accessible only to 
government agencies and other authorized users, and would contain 
commercialization data voluntarily supplied by SBIR recipients upon 
completion of their phase II SBIR funding agreement, such as revenue 
from the sale of new products or services resulting from the research 
undertaken with the award. In addition, applicants for phase II awards 
would be required to update information on the commercialization 
success of any prior SBIR awards they had received. Currently, SBA has 
created and is maintaining the public-use section of the Tech-Net 
database, which is available on the Internet to the general public.

Agencies Provide Most of the Required SBIR-Award Data Elements to SBA, 
but Some Data Submitted are Incomplete:

While federal agencies participating in the SBIR program submit a wide 
range of descriptive award information to SBA annually, these agencies 
are not consistently providing all of the required information. As 
outlined in SBA's guidance, each year SBIR participating agencies are 
required to collect and maintain information from recipients and 
provide it to SBA so that it can be included in the Tech-Net database. 
Specifically, participating agencies are required to provide over 40 
data elements for each SBIR award they make. These data include award- 
specific information, such as the date and amount of the award, an 
abstract of the project funded by the award, and a unique tracking 
number for each award. Participating agencies are also required to 
provide data about the award recipient, such as gender and socio- 
economic status, and information about the type of firms that received 
the awards, such as number of employees and geographic location. Much 
of the data collected by participating agencies are provided by the 
SBIR applicants at the time they apply for an award. Agencies provide 
additional information, such as the grant/contract number and the 
dollar amount of the award, after the award is made.

For the most part, all of the eight agencies we reviewed provided the 
majority of the SBIR award data requested by SBA, including the program 
identification number; company name, address, and contact information; 
award year and amount; a unique tracking number that will stay with the 
award through both phase I and phase II; and the title and abstract of 
the project funded. However, we also determined that some of the 
agencies are not providing the full range of information required by 
SBA. For example, two of the eight agencies we reviewed had not 
provided SBA data on the gender or socio-economic status of SBIR award 
recipients in 2004 and 2005. Similarly, in 2005, five of the eight 
agencies failed to provide data on the number of employees working at 
the firms that received the awards. As a result, SBA does not have 
information on the number of employees at SBIR awardee firms for about 
one-third of all the awards reported by these agencies in 2004 and 2005.

SBA officials acknowledged that agencies do not routinely provide all 
of the information requested because either they do not capture the 
information in their agency databases or they are not requesting the 
information from SBIR applicants. Officials at the participating 
agencies also cited other reasons for the incomplete data they provided 
to SBA. For example, NIH officials stated that for the past several 
years, the SBA Tech-Net annual reporting requirements have changed each 
year. At the end of calendar year 2003, SBA changed the field 
description from "minority" to "socially and economically disadvantaged 
small business." According to NIH officials, because the SBIR 
information is captured by the agency at the time the application is 
submitted/received, there is a lag in time between when application 
data is input into the NIH database and when the agency receives new 
SBA data field requirements. According to these officials, responding 
to SBA's changes in field names presents significant challenges to NIH 
for collecting the data needed to complete the Tech-Net reports, 
especially if the requirements change annually. As a result, these 
officials stated that complying with the changing requirements 
necessitates significant NIH resources and efforts to research and 
identify the information needed for the new data fields, not all of 
which can be provided. In commenting on a draft of this report, SBA 
stated that it has only requested minor clarifications of data 
requirements and has not made frequent changes as stated by the 
agencies. SBA believes that this may have been caused by lack of clear 
communication to the agencies.

The agencies also noted that data for certain Tech-Net fields will be 
absent from their reports to SBA if the data fields do not exist in the 
NIH application or in its awards database. Similarly, USDA officials 
stated that although they try to keep their records as up-to-date as 
possible, problems occur when company or contact information changes 
and the SBIR recipient fails to provide updated information to the 
agency, such as the e-mail address for the central contact person. 
Additionally, like NIH, USDA officials stated that certain information 
requested by SBA is not collected by their agency. For example, as long 
as SBIR applicants have certified that they meet the criterion of being 
a small business (under 500 employees), they do not ask for nor do they 
record information on the number of employees in the firm.

Agencies Generally Comply with SBA's Formatting Guidance, but Key Data 
on SBIR Award Recipients May Be Inconsistent or Inaccurate:

Participating agencies are providing some data that do not comply with 
SBA's formatting guidance, and while some of these inconsistencies are 
corrected by SBA's quality assurance processes, others are not. As a 
result, some data elements in the Tech-Net database may be inconsistent 
or inaccurate thereby, compromising the value of these data for program 
evaluation. SBA's quality assurance efforts focus on obtaining complete 
and accurate data for those fields essential to tracking specific 
awards, such as the tracking number and award amount, rather than on 
those fields that contain demographic information about the award 
recipient. Because the data contained in the public-use section of Tech-
Net will be incorporated into the government-use section of the 
database, inaccurate data in one section of the database will be 
replicated in the other, and these inaccurate data will limit 
evaluations of the SBIR program.

Both SBA and agency officials acknowledge that SBA has worked with 
participating agencies since 2000 to help ensure greater consistency in 
the formatting of the SBIR award data reported by agencies each year. 
To assist agencies in formatting the data and to minimize the number of 
inconsistencies in the data reported to SBA, SBA has taken a number of 
steps to improve the data formatting process. Specifically, SBA 
provided all of the participating agencies specific guidance on its 
requirements for the data, including its preferred program application 
for submitting the data, the length of each data field, and whether 
data should be entered as numbers, characters, or a mix of both. SBA 
has also included discussions of the Tech-Net database as an agenda 
item at its quarterly meetings with SBIR program managers from each of 
the participating agencies. Additionally, SBA developed a reporting 
template for agencies to use that includes the required data fields and 
instructions for appropriate data entry.

Under the current process, participating agencies aggregate all of the 
data provided by SBIR recipients in their award applications with 
additional information on the award amounts and submit the combined 
data to SBA by March 15 of each year. SBA then electronically checks 
the data to locate and reformat inconsistencies, and adds the data to 
the Tech-Net database. SBA officials told us that when they detect 
inconsistencies in data fields essential to tracking a specific award, 
such as the award tracking number, contact information for the 
recipient or principal investigator, or awarding agency, they contact 
the agency to obtain the correct information. However, SBA does not 
currently take steps to ensure that agency-provided data are accurate 
and complete. For example, SBA does not require agencies to submit a 
random sample of applications so that it can compare the data submitted 
by agencies with the original applicant information to ensure that the 
submitted data contain all the relevant application information. 
Instead, SBA relies on the agencies to fully report all the required 
application information on the awards they make. SBA officials told us 
that they believe that over the past two years the quality and 
consistency of the data received from participating agencies had 
greatly improved.

In reviewing the SBIR Tech-Net data that the eight agencies reported to 
SBA in 2004 and 2005, we determined that almost a quarter of the data 
provided by five agencies was incorrectly formatted for one or more 
fields. For example, phone number and award amount fields contained 
both characters and numbers and first and last names of principal 
investigators were combined into a single word that was used as both 
the first and last names. Moreover, we found that agency-provided data 
on gender and socio-economic status for over half of the awards 
reported in 2004 were incorrectly formatted. SBA officials said that 
detecting and correcting some of these formatting errors required a 
considerable amount of time and effort, and that their electronic check 
does not detect all of the errors that we identified.

The formatting inconsistencies that we identified arise primarily 
because the SBA reporting template used by agencies to submit required 
data can be edited. Consequently, agencies can and do edit the 
template. Agencies can change the names of various data fields in SBA's 
template, delete fields altogether, and enter data as numbers, 
characters, or both, regardless of what SBA has specified. One of the 
data format issues identified by SBA was the deletion of fields that 
agencies do not consider relevant or necessary. For example, DOD is the 
only agency that uses the field called "branches" to specify which 
component of DOD, such as the Army, made the award; other agencies have 
deleted this field because it does not relate to the structure of their 
agency. Similarly, we found instances where agencies had entered data 
on the gender or socio-economic status of the recipient and award 
amount in a format that differed from the numerical format specified by 
SBA. For example, one agency entered "Y" and "N" rather than "0" and 
"1." According to SBA officials, by fiscal year 2007, this issue should 
be resolved because agencies will have to submit their data through an 
Internet interface that will contain edit checks and should eliminate 
many of these problems.

We also determined that inconsistencies or inaccuracies can arise in 
certain data fields because SBA interprets the absence of certain data 
elements as a negative entry without confirming the accuracy of such an 
interpretation with the agency. In other words, if an agency did not 
provide information on whether the recipient is a woman or a member of 
a socio-economically disadvantaged group, SBA has entered a "no" into 
the database. SBA stated that they generally do not contact agencies to 
obtain correct information on data elements that are not used to track 
specific awards, such as gender or socio-economic status of the 
recipient. However, this inaccurate data on the award recipients could 
limit efforts to use these fields for comprehensive program evaluation.

Information in the Tech-Net database will be used to populate the 
government-use section of the database that SBA is developing (as 
discussed below) for the purpose of supporting SBIR program 
evaluations. However, SBA has no plans to correct any of the errors or 
inconsistencies in the database that relate to the historical data 
already collected. As a result, the errors in the existing database 
will migrate to the government-use section of the database and we 
believe will compromise the usefulness of the government-use database 
for program evaluation purposes.

SBA Is Five Years Behind Schedule in Meeting its Obligation to 
Implement a Government-Use SBIR Database:

SBA has not met its obligation to implement a restricted government-use 
database that would allow SBIR program evaluation as directed by the 
2000 SBIR reauthorization act. As outlined in the legislation, SBA, in 
consultation with federal agencies participating in the SBIR program, 
was to develop a secure database by June 2001 and maintain it for 
program evaluation purposes by the federal government. SBA planned to 
meet this requirement by expanding the existing Tech-Net database to 
include a restricted government-use section that would be accessible 
only to government agencies and other authorized users. In constructing 
the government-use section of the database, SBA planned to supplement 
existing data already gathered for the public-use section of the Tech- 
Net database with information from SBIR recipients and from 
participating agencies on commercialization outcomes for phase II SBIR 
awards. However, according to SBA officials, the agency has been unable 
to meet this requirement, primarily because of increased security and 
other information technology project requirements, agency management 
changes, and budgetary constraints. SBA's current goal for having the 
government-use section of the Tech-Net database operational is October 
1, 2006. In commenting on a draft of this report, SBA modified this 
date, stating that they anticipate having the government-use section of 
the Tech-Net database operational early in fiscal year 2007.

To date, with the help of two contractors, SBA has developed the 
framework for importing data into the government-use section of the 
Tech-Net database and for an Internet-based interface that would allow 
agencies and award recipients to access the database and enter 
commercialization information. According to SBA officials, as currently 
envisioned, the government-use section of the Tech-Net database will 
include the records of all applicants, including those that did not 
receive SBIR awards. Participating SBIR agencies will be asked to 
provide a unique business identification number, called the Data 
Universal Numbering System or DUNS number, for each award recipient, 
information about SBIR applicants that were not funded, and any 
historical data they have obtained about the commercialization of SBIR 
funded technologies. SBA has developed a standardized electronic 
commercialization questionnaire to gather data for the government-use 
section of the database from applicants and award recipients. 
Information that will be captured in the questionnaire will include the 
number of SBIR awards the company has received, the number of patents 
or copyrights that have resulted from the award, sales revenue realized 
as a result of the SBIR award, and sources of additional investment 
funding. SBA officials told us that the commercialization questionnaire 
will become an integral part of the SBIR application process in the 
future, and any company applying for an SBIR award will be required to 
complete and update the relevant information on phase II SBIR awards 
previously received by the company at the time the application is 
submitted. According to SBA officials, applications will not be 
accepted until this information is completed, and failure to submit the 
information may affect an applicant's ability to receive an award. In 
addition, SBIR award recipients will be requested to voluntarily update 
the commercialization information in the government-use section of the 
Tech-Net database annually for a minimum period of five years following 
the completion of their SBIR-funded project.

Although SBA has developed the majority of the functions needed to 
populate the government-use section of the Tech-Net database with the 
data currently gathered for the public-use section of the database, it 
can not be made operational until certain security issues are resolved. 
For example, because the government-use section of the database will 
contain information from recipients that is considered proprietary and 
confidential, SBA needs to ensure that adequate security measures are 
in place to prevent unauthorized access to the data. This entails the 
successful completion of a series of security and development checks to 
ensure that the database system is operating as designed. While SBA 
officials expect the government-use section of the database to be 
operational by October 1, 2006, they also recognize that additional 
enhancements, such as improving the user-friendliness of the interface 
for online submission of SBIR data by participating agencies and 
recipients, will be needed after the system is made operational. 
According to SBA officials, the agency's priority is to get the 
government-use section of the database up and running before 
considering further improvements to the database.

Tracking Data on the Commercialization Success of SBIR Projects Varies 
Among Agencies and Remains a Challenge:

Seven of the eight participating agencies we reviewed have implemented 
techniques to track the commercialization success of their SBIR-funded 
projects, and the eighth is planning to do so, although the 
methodological rigor of these techniques varies significantly. Under 
the program's authorizing legislation and SBA's implementation 
guidance, agencies have been given considerable flexibility to design, 
monitor, and evaluate the extent to which their SBIR programs have 
achieved commercialization success. For example, while some agencies 
use more systematic approaches to gathering data, such as periodically 
surveying SBIR award recipients, other agencies are less methodical, 
choosing instead to follow up periodically with a relatively small 
sample of SBIR award recipients. Regardless of how they track 
commercialization success, both SBA and agency officials generally 
agree that several factors complicate their efforts to obtain this 
information.

Agencies' Efforts to Track SBIR Commercialization Success Vary:

Of the eight agencies we reviewed, five systematically and periodically 
survey SBIR recipients to gather a variety of data on program 
participation, including the recipients' commercialization experiences. 
Specifically,

* Since 2000, DOD has systematically gathered information 
electronically on the commercialization of phase II awards from all 
phase I and phase II applicants and award recipients and maintains the 
information in a commercialization database. Commercialization outcomes 
that DOD monitors include such measures as (1) sales revenue from new 
products and non-R&D services resulting from the phase II technology; 
(2) additional investment from sources other than the federal SBIR 
program in activities that further the development, commercialization, 
or both of the phase II technology; (3) whether the phase II technology 
has been used in a DOD system or acquisition program, and if so, which 
system or program; (4) the number of patents resulting from the 
contractor's participation in the SBIR program; (5) growth in the 
number of employees at the firm; and (6) whether the firm has completed 
an initial public offering of stock resulting, in part, from the phase 
II project. DOD uses the accumulated data to assign a commercialization 
score to applicants that have received four or more prior SBIR awards 
based on a comparison of their commercialization experience with the 
average experience of other comparable applicants and uses the score to 
help select proposed projects for funding. In addition, recipients of 
phase II awards are required to update the information one year after 
the start of the project, at the completion of the project, and 
subsequently when the recipient submits a new SBIR application to DOD. 
Firms that do not submit a new SBIR application are asked to 
voluntarily provide updates on an annual basis after the completion of 
their phase II project. According to DOD officials, 66 percent of phase 
II award recipients updated their commercialization information when 
they submitted a new application, 11 percent provided the information 
without submitting a new application, and 23 percent did not update 
their information.

* For over 23 years, DOE has conducted an annual survey of SBIR phase 
II recipients, both active and inactive, to track the commercialization 
success of its SBIR-funded projects. The survey requests recipients to 
(1) list all products and services derived from their SBIR projects; 
(2) report on sales, phase III investment related to these products and 
service, or both; and (3) identify which phase II projects contributed 
to the development of the products and services. According to DOE, 
approximately 90 percent of its phase II recipients respond to these 
annual surveys.

* NASA has systematically gathered information on the commercialization 
of SBIR awards through annual surveys of phase II award recipients from 
1997 to 2002. In these surveys, NASA obtained data on various 
commercial outcomes, such as sales to nongovernment markets of the SBIR-
funded research results, procurement of the research results by NASA or 
other federal agencies, cumulative private capital funding, royalty and 
licensing revenue from nongovernment sources based on the SBIR-funded 
research results, creation of new business ventures based on the SBIR-
funded research results, and number of patents and patent applications 
resulting from these awards. According to NASA, approximately 91 
percent of its phase II award recipients responded to these annual 
surveys.

* NIH also surveys SBIR award recipients to gather commercialization 
information. Specifically, in 2002, NIH conducted a comprehensive 
evaluation of its SBIR program. As part of this evaluation, NIH 
surveyed recipients of phase II awards between 1992 and 2001 to obtain 
data relating to the range of SBIR program goals, including 
commercialization of research results. According to NIH officials, the 
2002 survey results formed a baseline that NIH staff could use to 
systematically monitor and evaluate the program. In 2004 and 2005, NIH 
again contacted award recipients to update the information obtained in 
the 2002 survey. Commercialization outcomes tracked by NIH include (1) 
sales realized for a product or service that resulted from the SBIR- 
funded research; (2) status of the Food and Drug Administration (FDA) 
approval process for the SBIR-funded research results; (3) receipt of 
FDA approval for SBIR-funded research results; and (4) receipt of 
additional non-SBIR funding or capital.

* Since about 1998, NSF has collected historical commercialization 
information from all phase II award recipients at the time an SBIR 
application is submitted. This information is used in the proposal 
review process to help select proposed projects for funding. In 2005, 
NSF developed an annual survey of phase II award recipients that will 
be used to gather information three, five, and eight years following 
their awards. Specific outcomes on which NSF will gather data include 
sales revenue based on the SBIR-funded research results, growth of 
overall company sales and employment, receipt of additional non-SBIR 
funding, and patents related to the SBIR funded research.

In contrast, two of the remaining three agencies we reviewed have 
focused their efforts on gathering anecdotal success stories and 
conducting periodic follow-up with a relatively smaller sample of SBIR 
award recipients. For example, over the past 7 years, EPA has contacted 
all phase II award recipients after their projects end to learn about 
their commercial successes. Based on these contacts, officials estimate 
that approximately 25 percent of the phase II projects funded by EPA 
have been commercialized. EPA defines "success" as the receipt of more 
than $300,000 in revenue from sales of the SBIR-funded project, an 
amount greater than the SBIR funds awarded by EPA. Similarly, USDA has 
periodically contacted a sample of about 20 to 25 percent of award 
recipients to obtain information about sales of their SBIR-funded 
research results. USDA last surveyed its phase II award recipients in 
1997. USDA publishes the success stories on its Web site and in an 
agency newsletter. About 2,500 people receive the newsletter and USDA 
makes copies of the success stories available at SBIR conferences. 
According to USDA officials, in the future they hope to gather data 
more systematically and conduct site visits to the SBIR firms. The 
eighth agency we reviewed, NIST, has recently developed a Web-based 
survey that it plans to send in 2006 to all of its SBIR-award 
recipients.

Tracking the Commercialization Success of the Overall SBIR Program 
Remains a Challenge:

Although each of the eight agencies we reviewed has implemented or 
plans to implement a method for gathering commercialization data, 
agency officials identified several factors that complicate their 
efforts. First, agency officials stated that it is difficult to track 
commercialization because it can take years before companies achieve 
commercial success. For example, USDA officials stated that, even over 
the short term, the effort to contact past award recipients consumes 
considerable effort. During this time, companies may move, change 
names, start a new business, or be purchased by other firms, all of 
which make it difficult for the agencies to track and link companies to 
the original SBIR awards.

Second, because the authorizing legislation lacks a clear definition of 
what constitutes "commercialization" success, agencies not only 
differed on the types of commercialization outcomes they captured, but 
also in how they tracked commercial success. SBA officials acknowledged 
that its guidance has provided considerable latitude to agencies on 
this issue in light of the wide range of industries represented by the 
participating agencies. Commercialization outcomes captured by the 
participating agencies included sales revenue based on the SBIR-funded 
research results, receipt of additional non-SBIR funding to further 
develop the research results, marketing activities ongoing or 
completed, and public offering of company stock. However, not all 
agencies are tracking all of these outcomes; therefore, assessing 
overall commercial success of the SBIR program across the various 
agencies remains a challenge.

Third, agency officials stated that SBIR award recipients may be 
reluctant to provide information related to their trade and business 
operations, which they consider proprietary and sensitive. Companies 
are often not willing to provide comprehensive data on their sales and 
particularly the investments they receive due to competitive concerns. 
Finally, agency officials told us that past recipients have no 
incentive to voluntarily complete commercialization surveys and update 
the information on their commercial experience unless they are applying 
for a new SBIR award. As a result, they do not expect that a large 
percentage of recipients will complete the information needed for the 
government-use section of the Tech-Net database. Agency officials 
believe that despite their best efforts, the data needed to conduct 
evaluations of the SBIR program are likely to be incomplete.

Conclusions:

In the last 5 years, SBA has been unable to meet the congressional 
directive to develop a government-use database that would provide 
better information on the SBIR program and allow for program 
evaluation. Although it has established a public-use Tech-Net database 
and has worked with participating agencies to achieve greater 
consistency in the data submitted for the database, the quality of the 
data remains a concern. The steps on which SBA relies to ensure that 
data are complete and accurate are inadequate and it has no plans to 
correct errors or supply missing data associated with the historical 
data already in the database. We believe that unless necessary controls 
are established and implemented to ensure the completeness, 
consistency, and accuracy of the SBIR data reported to SBA by 
participating agencies, the government-use section of the Tech-Net 
database, which depends on the public-use Tech-Net database, will be of 
limited use for program evaluation purposes when it becomes operational.

Recommendation for Executive Action:

We recommend that the Administrator, SBA, and the SBIR participating 
agencies work together to strengthen efforts to ensure that the data 
collected for SBA's Tech-Net database are complete, consistent, and 
accurate.

Agency Comments and Our Evaluation:

We provided SBA and the eight SBIR participating agencies included in 
this review a draft of this report for their review and comment. SBA 
and the eight agencies generally agreed with the report's findings and 
SBA and five of the eight agencies also stated their concurrence with 
the recommendation. Three agencies--EPA, NIST, and NSF--did not 
indicate whether they agreed or disagreed with the recommendation.

In addition, SBA stated that it was concerned that our conclusions did 
not reflect the fact that it plans to have the government-use section 
of the Tech-Net database operational by early fiscal year 2007. We have 
not modified our conclusions because the fact remains that SBA has not 
met the congressional directive to establish a government-use database 
during the last five years. Moreover, throughout this review, SBA 
officials told us that the database would be operational by October 1, 
2006. However, in its official comments, the agency has again modified 
this date to some time early in fiscal year 2007. SBA also provided us 
with technical comments that we have incorporated as appropriate. SBA's 
letter is included in appendix II and the letters that we received from 
DOD, NASA, and NIH are included in appendix III through V.

As agreed with your office, unless you publicly announce the contents 
of this report earlier, we plan no further distribution until 30 days 
from the report date. At that time, we will send copies of this report 
to interested congressional committees; the Administrators of the 
Environmental Protection Agency, National Aeronautics and Space 
Administration, and Small Business Administration; the Directors of the 
National Institutes of Health, National Institute of Standards and 
Technology, and National Science Foundation; the Secretaries of 
Agriculture, Defense, and Energy; and other interested parties. We will 
also make copies available to others upon request. In addition, the 
report will be available at no charge on the GAO Web site at http:// 
www.gao.gov.

If you or your staff have any questions about this report, please 
contact me at (202) 512-3841 or mittala@gao.gov. Contact points for our 
Offices of Congressional Relations and Public Affairs may be found on 
the last page of this report. GAO staff who made contributions to this 
report are listed in appendix VI.

[Signed by]:

Anu K. Mittal: 
Director, Natural Resources and Environment:

[End of section]

Appendix I: Objectives, Scope and Methods:

Our objectives for this review were to identify (1) the types of data 
that participating Small Business Innovation Research (SBIR) program 
agencies are reporting to the Small Business Administration (SBA) for 
inclusion in the Tech-Net database, (2) the extent to which agencies 
provide data for the Tech-Net database in a standard format to enable 
program evaluation, (3) the extent to which SBA has met the mandate to 
establish by early 2001 a government-use database that can be used for 
program evaluation, and (4) the extent to which participating SBIR 
agencies have developed and implemented techniques to track the 
commercialization success of SBIR projects.

In conducting this study, we reviewed SBA and the SBIR-related 
activities of 8 of the 11 SBIR participating agencies--Department of 
Defense (DOD), Department of Energy (DOE), Environmental Protection 
Agency (EPA), National Aeronautics and Space Administration (NASA), 
National Institutes of Health (NIH), National Institute of Standards 
and Technology (NIST), National Science Foundation (NSF), and 
Department of Agriculture (USDA). These eight agencies account for over 
98 percent of the total dollars awarded by the program in fiscal year 
2005. To determine the types of data these participating agencies are 
reporting to SBA and the extent to which SBA has complied with the 
requirement to establish a government-use database that can be used for 
program evaluation purposes, we compared the provisions in the Small 
Business Innovation Research Program Reauthorization Act of 2000, 
relevant legislative histories, SBA's Policy Directive implementing the 
act issued in September 2002, and other guidance with agency SBIR Tech- 
Net database reports. We identified and interviewed SBIR program 
officials at each agency and officials responsible for submitting 
program data to SBA. For these interviews, we used a protocol guide to 
obtain information on program operations, data reporting, data quality, 
and the Tech-Net database, including development of the government-use 
section of the database. We also reviewed documentation from each of 
the eight agencies and SBA regarding data elements and formatting 
instructions.

To determine the extent to which data for the Tech-Net database are 
provided in a standard format, enabling program evaluation, we compared 
data provided to SBA by the eight participating agencies with data in 
SBA's Tech-Net database for fiscal years 2004 and 2005, the 2 most 
recent years for which data were available. The agency data included 
information about the award itself, such as the award amount, a 
descriptive abstract of the project, and a unique tracking number; 
information about the award recipient, such as gender and socio- 
economic status; and information about the type of firm that received 
the award, such as number of employees and geographic location. To 
assess the reliability of the data provided by the agencies, we 
reviewed SBA's data system, internal controls, and related guidance, 
rather than the systems and internal controls participating agencies 
use to create the data provided to SBA. To assess the reliability of 
the data in SBA's Tech-Net database, we interviewed SBA officials about 
the database and reviewed related documentation. We also conducted 
tests of the data themselves. We used analytic software to compare the 
data provided by participating agencies with the data maintained by 
SBA. We focused our review on data fields considered critical by SBA 
officials, such as the awarding agency, the date of award, the award 
recipient, the amount of the award, and the purpose of the award. We 
also reviewed data fields related to SBIR program goals, such as gender 
and socio-economic status of the recipient, and data on the number of 
employees at the recipient firm. We used GAO's data reliability 
guidance to identify key attributes of data quality that facilitate 
program evaluation. These attributes include completeness, accuracy, 
and consistency in format. Finally, we reviewed internal quality 
control procedures. We determined that the data are sufficiently 
reliable for our purpose, which is to report on SBA's efforts to ensure 
consistency and completeness of the data it receives.

To determine the extent to which the government-use database 
requirements have been implemented, we interviewed SBA officials, and 
reviewed related documentation, such as minutes from meetings of SBA 
and SBIR program directors, and overviews of the existing and planned 
data systems. In addition, we attended a demonstration of the proposed 
Internet interface for the government-use section and interviewed the 
current contractor assisting SBA about implementation progress. We also 
reviewed a contractor-prepared analysis of the functional and data 
requirements for the integration of the public-and government-use 
sections of the Tech-Net database. At each of the eight participating 
agencies, we interviewed SBIR officials regarding the extent to which 
SBA had consulted them in the development of the government-use 
database.

To determine the extent to which participating agencies have developed 
and implemented techniques to evaluate commercialization success of 
SBIR projects, we reviewed agencies' documentation about their 
commercialization assistance and monitoring efforts. Specifically, we 
reviewed surveys that agencies had administered to award recipients, 
resulting reports on survey results, and anecdotal descriptions of 
commercialization success. We also reviewed provisions in SBIR 
legislation, relevant legislative histories, and SBA's Policy Directive 
regarding commercialization of SBIR-funded projects, as well as past 
GAO reports. In addition, we interviewed officials at each of the eight 
participating agencies to obtain information on the specific 
commercialization outcomes they monitor, the history of each agency's 
data collection efforts, and the agencies' experience in obtaining such 
information from current and past award recipients.

We conducted our work from April 2006 through September 2006 in 
accordance with generally accepted government auditing standards.

[End of section]

Appendix II: Comments from the Small Business Administration:

Note: GAO comments supplementing those in the report text appear at the 
end of this appendix.

Page numbers in draft report may differ from those in this report.

[See PDF for image]

[End of figure]

U.S. Small Business Administration; 
Washington, DC 20416:

October 6, 2006:

Ms. Anu Mittal:
Director, Natural Resources and Environment: 
U.S. Government Accountability Office: 
441 G St., NW:
Washington, DC 20548:

Dear Ms. Mittal:

Thank you for the opportunity to review and comment on the Government 
Accountability Office (GAO) draft report entitled, "Small Business 
Innovation Research: Agencies Need to Strengthen Efforts to Improve the 
Completeness, Consistency, and Accuracy of Awards Data," GAO-07-38. We 
have reviewed the draft report and have the following comments:

* Footnote 2 of the report is confusing. While it does indicate there 
are eleven SBIR participating agencies, the language of the footnote is 
uneven.

* On page 4, the report mentions that the SBIR Policy Directive (PD) 
was issued in September 2002. This should be reworded to state that the 
amended PD was issued in September 2002, otherwise, someone reading it 
might think 2002 was the first time SBA issued a PD.

* Footnote 7 is confusing - read literally, it looks like the SBA is an 
SBIR participating agency.

* Pages 5 and 11 of the report state that the SBIR participating 
agencies 
have cited a "variety" of reasons for not providing the data requested 
including "frequent" changes in SBA's data requirements. SBA contends 
that there have not been "frequent" changes, but rather minor 
clarifications of data requirements.

For example, NIH cited a change in SBA's reporting requirements as a 
reason for submission of incomplete data. SBA changed the field name 
from "Minority" to "Socially and Economically Disadvantaged" because it 
is the accepted description today. The content of that field was not 
expected to change. Perhaps our communication about that was not clear, 
but it is not a new field.

* Page 16 discusses the SBIR participating agencies' efforts in 
tracking the commercialization success of the SBIR awardee. The report 
discusses the differences in how the agencies define commercialization 
success. The SBA looks to assist in resolving this problem by using the 
Government Database to assess commercialization program-wide.

As a whole, the general findings and recommendations of the report 
appear reasonable. We do, however, take exception to the tone and 
content of the "Conclusions" section of the report. Specifically, we 
are concerned that while the "Conclusion" indicates that SBA has not 
developed the Government-Use section of the TechNet Database in over 
six years, it does not mention that SBA anticipates having this section 
of the database operational early in FY 2007.

Additionally, while we concur with the finding that the quality of the 
data remains a concern, SBA has taken significant steps to help improve 
the quality of data. SBA is working with the agencies to improve the 
controls on the propriety of the data submitted.

In the section labeled "Conclusions," the report states "The steps that 
SBA relies on to ensure that data are complete and accurate are 
inadequate and it has no plans to correct errors or supply missing data 
associated with historical data already in the database." SBA is aware 
of records in the database that are incomplete. However, this can be 
rectified by using a combination of electronic and manual processes to 
gather the missing data elements. The collection of DUNS numbers in the 
database would allow SBA to access information provided by many 
awardees to the Central Contractor Registration (CCR) database and the 
Dynamic Small Business (DSBS) database. DUNS matching would all ow the 
collection of current firm-level information. SBA and the participating 
agencies will have to work together to correct some of the historical 
data. SBA has spoken with one agency to date about working 
cooperatively to provide historical data for older awards and has a 
commitment from that agency. Similar dialog and success can be expected 
with other agencies. 

Should you have questions on these comments, please contact Edsel 
Brown, Assistant Administrator, Office of Technology at (202) 205-7343.

Sincerely, 

[Signed by]:

Arthur Collins:
Acting Associate Administrator: 
Office of Government Contracting:

The following are GAO’s comments on the Small Business Administration’s 
(SBA) letter dated October 6, 2006.

GAO Comments:

1. We revised footnote 2 to more clearly reflect the 11 SBIR 
participating agencies.
2. We revised the text to clarify this statement.
3. We deleted reference to SBA in footnote 7.
4. We added SBA’s position to the report.
5. We added SBA’s position to the report. However, because SBA has not 
met the congressional deadline to develop a government-use database, we 
did not modify our conclusions.

[End of section]

Appendix III: Comments from the Department of Defense:

[See PDF for image]:

[End of image]:

Office of the Under Secretary of Defense
3000 Defense Pentagon
Washington, DC 2031-3000

Acquisition:
Technology:
And Logistics:

Oct 06 2006

Ms. Anu Mittal:
Director, Natural Resources and Environment: 
U.S. Government Accountability Office: 
441 G St., NW:
Washington, DC 20548:

Dear Ms. Mittal:

This is the Department of Defense (DoD) response to GAO Draft Report 
"Small Business Innovation Research: Agencies Need to Strengthen 
Efforts to Improve the Completeness, Consistency, and Accuracy of 
Awards Data," (GAO Code 360677/ GAO-07-38), dated September 20, 2006. 

The GAO recommends that the Administrator, the Small Business 
Administration (SBA) and agencies participating in the SBIR Program 
work together to strengthen efforts to ensure that the data collected 
for SBA's Tech-Net database are complete, consistent, and accurate. The 
Department concurs with the recommendation recognizing that complete, 
high-integrity data is essential for program monitoring, assessment and 
evaluation. 

We appreciate the opportunity to comment on the draft report and look 
forward to working in concert with the SBA and the other participating 
agencies to improve the quality of Tech-Net data.

Sincerely, 

[Signed by]:

Linda B. Oliver for:

Frank M. Ramos:
Director, Small Business Programs:

Enclosure:
As stated:

GAO Draft Report--Dated September 20, 2006:
GAO Code 360677/ GAO-07-38:

"Small Business Innovation Research: Agencies Need to Strengthen 
Efforts to Improve the Completeness, Consistency, and Accuracy of 
Awards Data"

Department of Defense Comments:
To the Recommendation:

Recommendation: The GAO recommends that the Administrator, the Small 
Business Administration (SBA) and agencies participating in the SBIR 
Program work together to strengthen efforts to ensure that the data 
collected for SBA's Tech-Net database are complete, consistent, and 
accurate. 

Response: Concur. Complete, high-integrity data is essential for SBIR 
program monitoring, assessment and evaluation. The Department fully 
supports SBA annual data calls and looks forward to working together 
with the SBA and the other participating agencies to improve the 
quality of Tech-Net data.

[End of section]

Appendix IV: Comments from the National Aeronautics and Space 
Administration:

National Aeronautics and: 
Space Administration:
Headquarters:
Washington, DC 20546-0001:

[See PDF for image]:

[End of image]

October 6, 2006: 

Innovative Partnerships Program Office:

Reply to Attn of:

Ms. Anu K. Mittal:
Director, Natural Resources and Environment: 
United States Government 
Accountability Office Washington, D.C. 20548:

Dear Ms. Mittal:

NASA appreciates the opportunity to comment on the draft Government 
Accountability Office (GAO) report, "Small Business Innovation Research 
Agencies Need to Strengthen Efforts to Improve the Completeness, 
Consistency, and Accuracy of Awards Data" (GAO-07-3 8).

The draft report makes a single recommendation, addressed to all Small 
Business Innovation Research (SBIR) participating agencies, including 
NASA and the Small Business Administration (SBA), which is responsible 
for providing policy direction for the program across all agencies. The 
report recommends "that the Administrator, SBA, and the SBIR 
participating agencies work together to strengthen efforts to ensure 
that the data collected for SBA's Tech-Net database are complete, 
consistent, and accurate.":

NASA concurs with this recommendation. The SBIR program executive and 
program manager responsible for the execution of the program at NASA 
will contact Mr. Edsel M. Brown Jr., Assistant Administrator, Office of 
Technology, U.S. Small Business Administration, to affirm NASA's 
commitment to participate in joint discussions with all Federal 
agencies involved for the purposes of considering GAO's findings and 
recommendation, as well as determining and implementing an appropriate 
course of action.

Thank you for the opportunity to respond to this draft report.

Sincerely, 

[Signed by]:

Jack Yadvish, Acting Director: 
Innovative Partnerships Program Office:

[End of section]

Appendix V: Comments from the National Institutes of Health:

[See PDF for image]:

[End of image]:

Appendix V: Comments from the National Institutes of Health:

Department Of Health & Human Services: 
Public Health Service:

National Institutes of Health: 
Bethesda, Maryland 20892:

Oct 6, 2006:

Ms. Anu K. Mittal:
Director, Natural Resources and Environment: 
U.S. Government Accountability Office: 
441 G Street, N.W.
Washington, D.C. 20548:

Dear Ms. Mittal:

We appreciate the opportunity to review and comment on the U.S. 
Government Accountability Office (GAO) report, Small Business 
Innovation Research: Agencies Need to Strengthen Efforts to Improve the 
Completeness, Consistency, and Accuracy of Awards Data (GAO-07-38).

The National Institutes of Health supports the recommendation that the 
Small Business Administration (SBA) and Small Business Innovation 
Research participating agencies work together to strengthen efforts to 
ensure that the data collected for SBA's Tech-Net database are 
complete, consistent, and accurate. We recognize the importance of the 
collection of program results and look forward to working 
collaboratively with the SBA to improve the quality of the data in the 
Tech-Net database.

Sincerely,

Elias A. Zerhouni, M.D. 
Director:

[End of section]

Appendix VI: GAO Contact and Staff Acknowledgments:

GAO Contact: Anu K. Mittal, (202) 512-3841:

Acknowledgments: In addition to the contact named above, Cheryl 
Williams, Assistant Director; Bernice H. Dawson; Vondalee R. Hunt; and 
Marcus L. Oliver made key contributions to this report. Also 
contributing to this report were Nancy Crothers, Grant Mallie, Gregory 
Marchand, and Rebecca Shea.

[End of section]

FOOTNOTES

[1] Small Business Innovation Development Act of 1982, Pub. L. No. 97-
219 (1982).

[2] The eleven SBIR participating agencies are the departments of 
Agriculture (USDA), Commerce, Defense (DOD), Education, Energy (DOE), 
Health and Human Services, Homeland Security, and Transportation; the 
Environmental Protection Agency (EPA); the National Aeronautics and 
Space Administration (NASA); and the National Science Foundation (NSF).

[3] Pub. L. No. 102-564 (1992).

[4] The Tech-Net database also contains award information on SBA’s 
Small Business Technology Transfer (STTR) program. However, the focus 
of this report is on the SBIR program and not the STTR program. As 
such, any discussion of the Tech-Net database only refers to the 
information in the database related to the SBIR awards.

[5] Pub. L. No. 106-554 (2000).

[6] See GAO, Federal Research: Observations on the Small Business 
Innovation Research Program, GAO-05-861T, June 28, 2005; Federal 
Research: Observations on the Small Business Innovation Program, GAO/T-
RCED-98-218, June 4, 1998; Federal Research: Small Business Innovation 
Research Shows Success but Can be Strengthened, GAO/RCED-92-37, Mar. 
30, 1992; and Federal Research: Assessment of Small Business Innovation 
Research Programs, GAO/RCED-89-39, Jan. 23, 1989.

[7] The SBIR participating agencies included in this review are DOD, 
DOE, EPA, NASA, National Institutes of Health (NIH) within the 
Department of Health and Human Services, National Institute of 
Standards and Technology (NIST) within the Department of Commerce, NSF, 
and USDA.

GAO's Mission:

The Government Accountability Office, the investigative arm of 
Congress, exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO's commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony:

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics.

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as "Today's Reports," on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order 
GAO Products" heading.

Order by Mail or Phone:

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to:

U.S. Government Accountability Office

441 G Street NW, Room LM

Washington, D.C. 20548:

To order by Phone:

Voice: (202) 512-6000:

TDD: (202) 512-2537:

Fax: (202) 512-6061:

To Report Fraud, Waste, and Abuse in Federal Programs:

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470:

Public Affairs:

Jeff Nelligan, managing director,

NelliganJ@gao.gov

(202) 512-4800

U.S. Government Accountability Office,

441 G Street NW, Room 7149

Washington, D.C. 20548: