Small Business Administration:

Model for 7(a) Program Subsidy Had Reasonable Equations, but Inadequate Documentation Hampered External Reviews

GAO-04-9: Published: Mar 31, 2004. Publicly Released: Mar 31, 2004.

Additional Materials:

Contact:

William B. Shear
(202) 512-3000
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

The Small Business Administration (SBA) approved about $8.6 billion in loan guarantees through its 7(a) loan program in fiscal year 2003. SBA must estimate the subsidy cost of this program. Since fiscal year 2003, SBA has been using econometric modeling to estimate the subsidy. This report reviews SBA's estimation methodology and equations, assesses the default and recovery rates the model produced, identifies ways to enhance the estimates' reliability, describes the process for developing the model, and analyzes SBA's data.

From an economics perspective, SBA's econometric equations were reasonable, and its model produced estimated default and recovery rates that were in line with historical experience. However, from an audit perspective, SBA's lack of documentation of the model development process precluded GAO, and others, from independently evaluating the model's development and determining if SBA used a sound and consistently applied method to select and reject model variables. Taking into account economic reasoning and research, SBA's econometric equations for estimating defaults, prepayments, and recoveries were reasonable. SBA's equations used a limited set of variables; equations using other variables could also be reasonable but would produce different estimates. Since an estimate is an approximation, no one estimate can be considered accurate, and reasonable estimates can fall within a range of values. The model's estimated default and recovery rates were in line with recent historical experience. SBA could improve its estimation methodology by periodically checking for and correcting errors and should consider adding more borrower information, such as credit scores. Some errors in the model resulted in understating the estimated program costs. SBA used the expertise of other agencies and a contractor to develop its model and worked closely with the Office of Management and Budget (OMB), which must approve the methodology agencies use to estimate subsidies. OMB officially approved the model in the fall of 2002. SBA did not adequately document its model development process, including alternative variables considered and rejected, to enable external reviewers to assess the process that was used. Further, GAO and two other independent reviewers could not determine whether a bias existed in the model by systematically excluding variables to influence the subsidy rate in a particular direction. Adequate documentation, a key internal control, would enable SBA and other agencies to demonstrate the rationale and basis for key aspects of the model that provide important cost information for budgets, financial statements, and congressional decision makers and facilitate SBA's annual financial statement audit. Current OMB and other guidance is either silent or unclear about the level of documentation necessary for credit subsidy model development. SBA had a process to help ensure data integrity and data consistency in the equations with the loan-level data in its databases. Although errors existed in SBA's data systems, the magnitude and nature of these errors were not likely to significantly affect the subsidy rate.

Recommendations for Executive Action

  1. Status: Closed - Implemented

    Comments: SBA demonstrated that it has documentation that explains the development of its 7(a) subsidy model, the data variables used and their sources, and any changes to the econometric model. The documentation is generated and updated annually as part of its independent verification and validation reviews.

    Recommendation: To demonstrate and explain the rationale and basis for the 7(a) econometric model and all other models developed, the SBA Administrator should prepare and retain adequate documentation of the model development process including a detailed discussion of the alternative variables or combinations of variables that were considered, tested, and rejected, as well as the reasons for rejecting them.

    Agency Affected: Small Business Administration

  2. Status: Closed - Implemented

    Comments: SBA has implemented a process for ensuring that its statistical model for estimating subsidy costs of the 7(a) Loan Guarantee program is free of errors and accounts for changes in the program. Specifically, SBA has procured a contractor to perform an annual independent verification and validation (IV&V) of the model. The IV&V process involves, among other things, reviewing model changes since the prior year's review, validating programming code used to estimate subsidies, and validating SBA's calculations and projection methodology.

    Recommendation: To further enhance the reliability of SBA's subsidy estimates, the SBA Administrator should ensure that the model remains reasonable by establishing a process for periodically evaluating the model to correct any errors and revising it to reflect changes in the 7(a) business loan program or other factors that could affect the subsidy estimate.

    Agency Affected: Small Business Administration

  3. Status: Closed - Implemented

    Comments: SBA has evaluated how to best include borrower-specific information in future subsidy models. As part of this evaluation, SBA determined that it could not use credit scores (a borrower-specific variable GAO suggested in its recommendation) because it was cost prohibitive. However, SBA determined that it can include information on borrower collateral in its future subsidy model. Further, collateral data will also be used to compute loan-to-value ratios for future loan sales. The agency has initiated collection of collateral data.

    Recommendation: To further enhance the reliability of SBA's subsidy estimates, the SBA Administrator should determine how best to include in future subsidy models borrower-specific information, such as credit scores and loan-to-value ratios, to be collected in the new loan monitoring system.

    Agency Affected: Small Business Administration

  4. Status: Closed - Not Implemented

    Comments: It does not appear that OMB made any changes to the Circular A-11 as recommended.

    Recommendation: To facilitate (1) validation of models used to generate credit subsidy estimates, (2) external oversight, and (3) financial statement audits, the Director, OMB, should revise OMB Circular A-11 to require that agencies document the development of their credit subsidy models, including the process followed for selecting modeling methodologies over alternatives, and variables tested and rejected, along with the basis for excluding them.

    Agency Affected: Executive Office of the President: Office of Management and Budget

 

Explore the full database of GAO's Open Recommendations »

Oct 14, 2014

Sep 22, 2014

Sep 16, 2014

Jul 28, 2014

Jul 23, 2014

Jul 22, 2014

Jul 21, 2014

Jun 26, 2014

Jun 20, 2014

Looking for more? Browse all our products here