This is the accessible text file for GAO report number GAO-08-704R 
entitled 'Defense Transportation: DOD Should Ensure that the Final Size 
and Mix of Airlift Force Study Plan Includes Sufficient Detail to Meet 
the Terms of the Law and Inform Decision Makers' which was released on 
April 28, 2008. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

April 28, 2008: 

The Honorable Robert M. Gates: 
The Secretary of Defense: 

General Larry D. Welch, USAF (Ret.): 
President and CEO: 
Institute for Defense Analyses: 

Subject: Defense Transportation: DOD Should Ensure that the Final Size 
and Mix of Airlift Force Study Plan Includes Sufficient Detail to Meet 
the Terms of the Law and Inform Decision Makers: 

Global mobility[Footnote 1] is a key component of U.S. national 
security. Since the end of the Cold War, senior decision makers have 
relied upon Department of Defense (DOD) mobility studies to provide 
insights they need to build and maintain the right mix of mobility 
capabilities. The most recent study, the Mobility Capabilities 
Study,[Footnote 2] identified the mobility support needed for the full 
range of strategic operations in the context of the September 11, 2001 
attacks, the global war on terror, and DOD's evolving global defense 
posture, all in support of the National Military Strategy. According to 
DOD officials, the department plans to issue the next mobility study-- 
the Mobility Capabilities Requirements Study--in the spring of 2009. 
The 2005 mobility study also assessed requirements for two overlapping 
war fights, DOD support to homeland defense, civil support, lesser 
contingency operations, sustainment of forward-deployed forces, and 
national strategic missions. In accomplishing these missions, DOD 
depends on its airlift force. 

The National Defense Authorization Act for Fiscal Year 2008[Footnote 3] 
(hereafter referred to as the Act) mandated a requirements-based study 
on alternatives for the proper size and mix of the airlift force to 
meet the needs of the National Military Strategy to be done by a 
Federally Funded Research and Development Center (FFRDC).[Footnote 4] 
The Act specifically defined what the study plan should include and set 
time frames for the completion of various events. The FFRDC was to 
submit a study plan to the appropriate congressional committees, the 
Secretary of Defense, and the Comptroller General 60 days after the 
enactment of the Act. The Act required us to review the study plan to 
determine if it is complete and objective and whether it has any flaws 
or weaknesses in scope or methodology and report to the Secretary of 
Defense and the FFRDC within 30 days. It also required us to include in 
the report any recommendations that the Comptroller General considers 
appropriate for improvements to the study plan. DOD selected the 
Institute for Defense Analyses (IDA) to accomplish the study and signed 
a task order with IDA outlining the study framework. On March 28, 2008, 
IDA delivered the draft Size and Mix of Airlift Force Study Plan to 
DOD, congressional committees, and us. The draft study plan comprises 
34 pages of bulleted information, graphs, and diagrams. The seven major 
sections are introduction, background, scope, objective, study 
management, staged approach, and schedule. The single objective of the 
study is to address the numerous airlift issues identified in the Act 
and to report to the Secretary of Defense and to the Congress by 
January 10, 2009. We assessed the draft Size and Mix of Airlift Force 
Study Plan that IDA delivered on March 28, 2008, for completeness, but 
we were unable to evaluate objectivity or identify flaws or weaknesses 
in scope or methodology.[Footnote 5] We also commented on another 
ongoing airlift-related study, the Mobility Capabilities Requirements 
Study, because it is related to the scope of the draft Size and Mix of 
Airlift Force Study Plan. 

DOD uses studies to inform decision making, and study plans are 
important because they define what will be accomplished, what 
methodologies and assumptions will be used in a study, and how it will 
be done. DOD has used detailed study plans to address large and complex 
issues, including analyses of alternatives supporting acquisitions and 
execution plans for force management initiatives. Although DOD has not 
published departmentwide guidance, the Army, Air Force, and Navy have 
publications that describe study plans and/or note the use and 
importance of studies in a variety of efforts.[Footnote 6] Service 
publications describe a number of detailed elements that could be 
considered for use in study plans and that characterize successful 
studies. Our prior work shows that a detailed study plan is a critical 
part of a well-executed study, but may not guarantee a fully successful 
study. 

To conduct our evaluation, we reviewed IDA's submission to determine if 
it fulfilled the study plan elements in the Act. We also assessed the 
completeness of the study plan by reviewing the plan and comparing it 
with study plan descriptions in Army, Air Force, and Navy publications 
that identified the key elements that could be included in a study 
plan. We reviewed prior GAO work that discussed study plans. To obtain 
information on the IDA draft study plan, we interviewed officials from 
IDA; officials from the Office of the Under Secretary of Defense for 
Acquisition, Technology and Logistics; and U.S. Transportation Command. 
We conducted this performance audit from March to April 2008 in 
accordance with generally accepted government auditing standards. Those 
standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe that 
the evidence obtained provides a reasonable basis for our findings and 
conclusions based on our audit objectives. 

Results In Brief: 

The draft Size and Mix of Airlift Force Study Plan does not meet the 
terms of the Act and lacks sufficient detail for assessment. We are 
unable to fully assess the draft Size and Mix of Airlift Force Study 
Plan as required in section 1046 of the Act because the plan does not 
contain sufficient detail for us to evaluate its objectivity and its 
scope and methodology. Because the draft study plan did not address all 
of the specified elements in the Act, it is not complete. The draft 
plan did not include specific and explicit references that can be 
traced directly to the Act, such as the assumptions to be included in 
the study plan and assessments to be accomplished. This absence of 
detail also precludes us from evaluating the scope and methodology. 
Moreover, the plan lacked key details expected in such plans, such as 
assumptions and measures of effectiveness. The lack of details 
discussed above precludes us from making any recommendations concerning 
improvements to the study plan. DOD officials stated that because DOD 
selected IDA and issued a task order for the study only shortly before 
the mandated deadline, sufficient time was not available to produce a 
more detailed study plan. Nevertheless, DOD is responsible for ensuring 
the statutorily required elements of the study plan are fulfilled. IDA 
officials told us that IDA plans to submit to the Secretary of Defense 
in June 2008 a final Size and Mix of Airlift Force Study Plan that will 
be more robust. In addition to the independent draft Size and Mix of 
Airlift Force Study Plan, we note that DOD is conducting another study 
that may also inform decision makers on airlift issues. 

Accordingly, we are recommending that the Secretary of Defense direct 
the Under Secretary of Defense for Acquisition, Technology and 
Logistics to ensure that the final Size and Mix of Airlift Force Study 
Plan include (1) sufficient detail to address, at a minimum, elements 
mandated in section 1046 of the National Defense Authorization Act for 
Fiscal Year 2008; and (2) sufficient detail to inform decision makers 
on airlift issues. 

In oral comments on a draft of this report, DOD concurred with our 
recommendation. DOD's comments are discussed in more detail at the end 
of this report. DOD also provided technical comments and we have 
incorporated them where appropriate. 

Background: 

In previous GAO work, we noted the importance of study plans and usage 
of study plans as a best practice.[Footnote 7] Specifically, we found 
that governmental agencies and the private sector rely on detailed 
study plans, or data collection and analysis plans, to guide the 
development of studies and the collection and analysis of data. 
Additionally, we stated that a study plan provides a feedback loop that 
links the outcomes of the study and subsequent analysis to the original 
goals and objectives of the study. We also found that particularly 
large and complex issues may benefit from a study plan. GAO also 
identified a best practice for a study plan process that mirrors 
information found in service publications concerning study plans. 

Service publications describe the role of studies in complex decision 
making in areas such as acquisitions, comparing alternatives, and 
evaluations of force capabilities. Service publications also describe 
elements, such as assumptions and measures of effectiveness (MOEs), 
that may be considered for inclusion in a study plan. The Navy 
publication that discusses studies describes determining assumptions, 
statements related to the study that are taken as true in the absence 
of facts, as a key step in study plan development.[Footnote 8] The Army 
publication that discusses preparation of a study also lists 
assumptions as a part of a study plan.[Footnote 9] The Air Force 
analysis handbook contains a study plan outline that includes 
assumptions as well.[Footnote 10] According to these service 
publications, MOEs are the level of success to be achieved or how well 
tasks are performed. All of the service publications refer to MOEs as 
part of a study plan. In the Army and Navy publications, selection of 
the MOEs is described as perhaps the most crucial part of any analysis. 

The Draft Size and Mix of Airlift Force Study Plan Does Not Meet the 
Terms of the Act and Cannot Be Fully Assessed: 

We are unable to fully assess the draft Size and Mix of Airlift Force 
Study Plan as required in section 1046 of the Act because it does not 
contain sufficient detail for us to evaluate the objectivity or to 
evaluate the scope and methodology. Since the draft study plan did not 
address all of the specified elements in the Act, it is not complete. 
This absence of detail also precludes us from evaluating the scope and 
methodology. In the absence of a complete study plan, the final study 
may not sufficiently inform decision makers concerning the alternatives 
for the size and mix of the airlift force to meet requirements of the 
National Military Strategy. 

The draft study plan is not complete because it did not address all of 
the required elements in the Act. While IDA officials described the 
draft as generally addressing the three main elements cited in the Act, 
we found that it lacks specific and explicit references that can be 
traced directly to the Act. Specifically, the Act identified 11 
assumptions, at a minimum, to be included in the study plan.[Footnote 
11] For example, one required assumption involved the new capability in 
airlift to be provided by the KC(X) tanker aircraft. The draft study 
plan included a 5-bullet slide concerning the assessment of the KC(X) 
as an airlifter, but did not include the required assumptions. 
Additionally, the Act required the study plan to include assumptions 
concerning airlift mobility requirements in support of homeland defense 
and national emergencies. However, the plan only included a number of 
bulleted statements about the homeland defense and national emergencies 
missions and contained no assumptions. Without the required 
assumptions, the draft Size and Mix of Airlift Force Study Plan does 
not satisfy the Act and is not complete. 

The draft study plan also does not contain key details sufficient for 
us to evaluate the scope and methodology. Service publications reflect 
the importance of key steps such as identification of MOEs. According 
to the service publications, perhaps the most crucial part of any 
analysis is the selection of appropriate measures of effectiveness, 
which are central to evaluating alternatives. They enable decision 
makers to compare the results in a study.[Footnote 12] For example, 
MOEs could allow decision makers to compare the results of two analyses 
that measure different airlift force mixes. In the draft study plan, 
mixes of two aircraft (C27 and C-130) are to be compared for their 
fleet effectiveness. In such an example, the time required for an 
aircraft fleet to accomplish a mission, such as moving cargo or people, 
can be measured in terms of time required, tons moved, or miles flown. 
The task order for the draft Size and Mix of Airlift Force Study Plan 
specified the use of MOEs. The draft Size and Mix of Airlift Force 
Study Plan includes a slide concerning effectiveness analyses but, for 
the purposes of our analysis, it does not detail the MOEs or how they 
are to be used. Without critical details such as MOEs, the final study 
may not sufficiently inform decision makers concerning airlift issues. 

In our discussion with IDA officials concerning the draft study plan, 
they acknowledged the importance of the use of assumptions and MOEs in 
study plans. DOD officials stated that DOD selected IDA only shortly 
before the mandated deadline for a draft study plan and added that 
sufficient time was not available to produce a more detailed draft 
study plan. The plan states that its final stage (phase) will ensure 
all issues raised in the Act are addressed. In our discussion with IDA 
officials, they explained that the March 2008 version of the Size and 
Mix of Airlift Force Study Plan is a draft and that the final study 
plan, scheduled for completion in June 2008, will be more robust and 
detailed. We note that while DOD is using IDA to perform the study plan 
pursuant to a task order, DOD is responsible for ensuring that the 
statutorily required elements of the study plan are fulfilled. We also 
note that, pursuant to the Act, we were to provide any recommendations 
that the Comptroller General considered appropriate for improvement to 
the study plan. The lack of details discussed above precludes us from 
making any recommendations concerning improvements to the study plan. 

In the Navy study guide publication, one of the first steps in 
initiating a study is identifying potential uses for anticipated study 
results. While our scope only included the draft Size and Mix of 
Airlift Force Study Plan, we note that DOD plans to publish results 
from both the Size and Mix of Airlift Force Study and the Mobility 
Capabilities Requirements Study in 2009 as decision makers consider 
airlift requirements. We believe that it is possible that decision 
makers may compare the findings and assumptions of the Size and Mix of 
Airlift Force Study with the airlift portion of the Mobility 
Capabilities Requirements Study. If the Mobility Capabilities 
Requirements Study does not consider the objectives and assumptions of 
the Size and Mix of Airlift Force Study, it is unclear as to whether 
the results of the two studies can be compared and therefore the study 
results may not be fully useful to decision makers. 

Conclusion: 

Well-executed airlift and mobility studies that inform decision making 
are important as DOD continues its efforts to sustain, modernize, and 
recapitalize its airlift programs. Detailed study plans are a critical 
part of well-executed studies and should include key details such as 
assumptions and MOEs. Although the draft Size and Mix of Airlift Force 
Study Plan does not fulfill the requirements of the Act, the final 
version may include corrections and refinements. 

Recommendations for Executive Action: 

We recommend that the Secretary of Defense direct the Under Secretary 
of Defense for Acquisition, Technology and Logistics to ensure that the 
final Size and Mix of Airlift Force Study Plan includes sufficient 
detail to address, at a minimum, elements mandated in section 1046 of 
the National Defense Authorization Act for Fiscal Year 2008 and ensure 
that the study plan includes sufficient detail to inform decision 
makers on airlift issues. 

Agency Comments and Our Evaluation: 

In oral comments on a draft of this report, DOD concurred with our 
recommendation that DOD ensure that the final study plan include 
sufficient detail to address, at a minimum, elements mandated in law 
and sufficient detail to inform decision makers on airlift issues. As 
part of the comments, an official from the Office of the Under 
Secretary of Defense for Acquisition, Technology and Logistics 
reiterated that IDA was not given adequate time to create a detailed 
study plan in compliance with established standards. DOD stated the 
intention to continue monitoring IDA's work on the final study plan, 
adding that DOD fully expects the final study plan to comply with 
established DOD standards regarding studies. DOD also provided 
technical comments and we have incorporated them where appropriate. 

We are sending copies of this report to the Senate and House 
Appropriations and Armed Services Committees and other interested 
congressional committees; the Under Secretary of Defense for 
Acquisition, Technology and Logistics; the Commander of U.S. 
Transportation Command; and the Director, Office of Management and 
Budget. In addition, this report will be available at no charge on the 
GAO Web site at [hyperlink, http://www.gao.gov]. 

If you have any questions regarding this report, please contact me at 
(202) 512-8365 or solisw@gao.gov. Contact points for our Offices of 
Congressional Relations and Public Affairs may be found on the last 
page of this report. GAO staff that made contributions to this report 
include Ann Borseth, Ron La Due Lake, Charles Perdue, Karen Thornton, 
Karen Werner, and Steve Woods. 

Signed by: 

William M. Solis, Director: 

Defense Capabilities and Management: 

[End of section] 

Related GAO Products: 

Defense Acquisition: KC-135 Recapitalization Analysis of Alternatives 
Does Not Inform Decision Makers Regarding Cost, Effectiveness, and 
Suitability. GAO-08-69CR. Washington, D.C.: January 8, 2008. 

Defense Acquisitions: Air Force Decision to Include a Passenger and 
Cargo Capability in Its Replacement Refueling Aircraft Was Made without 
Required Analyses. GAO-07-367R. Washington, D.C.: March 6, 2007. 

Defense Transportation: Study Limitations Raise Questions about the 
Adequacy and Completeness of the Mobility Capabilities Study and 
Report. GAO-06-938. Washington, D.C.: September 20, 2006. 

Military Readiness: Navy's Fleet Response Plan Would Benefit from a 
Comprehensive Management Approach and Rigorous Testing. GAO-06-84. 
Washington, D.C.: November 22, 2005. 

[End of section] 

Footnotes: 

[1] The Air Force defines global mobility as the ability to rapidly 
establish an air-bridge and move military capability in support of 
operations anywhere in the world under any conditions. 

[2] The intent of the December 2005 Mobility Capabilities Study (MCS) 
was to identify and quantify the mobility capabilities needed to 
support U.S. strategic objectives into the next decade. The MCS 
determined that the projected mobility capabilities are adequate to 
achieve U.S. objectives with an acceptable level of risk during the 
period from fiscal years 2007 through 2013; that is, the existing U.S. 
inventory of aircraft, ships, prepositioned assets, and other 
capabilities were concluded to be sufficient, in conjunction with host 
nation support. The MCS emphasized that continued investment in the 
mobility system, in line with current departmental priorities and 
planned spending, is required to maintain these capabilities in the 
future. This included, for example, fully funding Army prepositioned 
assets as planned and completing a planned reengineering of the C-5 
aircraft. The MCS report also made recommendations to conduct further 
studies, develop plans and strategies, and improve data collection and 
mobility models. 

[3] Pub. L. No. 110-181, 1046 (2008). 

[4] The Federal Acquisition Regulation sets forth federal policy 
regarding the establishment and use of Federally Funded Research and 
Development Centers. An FFRDC meets some special long-term research or 
development need which cannot be met as effectively by existing in-
house or contractor resources. FFRDCs enable agencies to use private 
sector resources to accomplish tasks that are integral to the mission 
and operation of the sponsoring agency. The FFRDC is required to 
conduct its business in a manner befitting its special relationship 
with the government, to operate in the public interest with objectivity 
and independence, to be free from organizational conflicts of interest, 
and to provide full disclosure of its affairs to the sponsoring agency. 
FFRDCs are operated, managed, and/or administered by either a 
university or consortium of universities, another not-for-profit or 
nonprofit organization, or an industrial firm, as an autonomous 
organization or as an identifiable separate operating unit of a parent 
organization. 

[5] See GAO, Government Auditing Standards: July 2007 Revision, GAO-07- 
731G (Washington, D.C.: July 2007). Government auditing standards 
define scope as the boundaries of a study that are directly tied to the 
study objectives. The scope defines the subject matter that the 
executors will assess. Government auditing standards define methodology 
as describing the nature and extent of procedures for gathering and 
analyzing evidence to address study objectives. Methodology is to 
obtain sufficient, appropriate evidence to address objectives and 
provide reasonable assurance that the evidence is sufficient and 
appropriate to support study findings and conclusions. Methodology 
includes both the nature and extent of procedures used to address 
objectives. 

[6] Studies that might be guided by a study plan include but are not 
limited to cost, benefit, or effectiveness analysis of concepts, plans, 
training, tactics, forces, systems, policies, personnel management 
methods, and policies or programs; cost and operational effectiveness 
analyses (COEA); evaluations of force capabilities, organizational 
structure, administrative policies, procedures, methods, systems, and 
distribution of functions; research and development of data bases, 
models, and methodologies for accomplishing specific studies and 
analyses; analyses of materiel, personnel, logistics, and management 
systems; and studies to establish materiel requirements. 

[7] See GAO, Military Readiness: Navy's Fleet Response Plan Would 
Benefit from a Comprehensive Management Approach and Rigorous Testing, 
GAO-06-84 (Washington, D.C.: Nov. 22, 2005). 

[8] Navy Warfare Development Command, Study Planning and Conduct Guide, 
Mr. Richard C. Rigazio, Operations Research Analyst (August 2007). 

[9] Department of the Army Pamphlet 5-5, Guidance for Army Study 
Sponsors, Sponsor's Study Directors, Study Advisory Groups, and 
Contracting Officer Representatives, Headquarters (Washington, D.C. Nov 
1, 1996). 

[10] Office of Aerospace Studies, Analysis Handbook, A Guide for 
Performing Analysis Studies: For Analysis of Alternatives or Functional 
Solution Analyses (July 2004). 

[11] Pub. L. No. 110-181, 1046 (d)(1) (2008). 

[12] According to the Air Force analysis handbook, MOEs are important 
to a warfighter because they express both system worth (ability to 
contribute to a warfighter's immediate goal) and military worth 
(ability to contribute to high-level goals of winning the war). 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability.  

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "E-mail Updates."  

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to:  

U.S. Government Accountability Office: 
441 G Street NW, Room LM: 
Washington, D.C. 20548:  

To order by Phone: 
Voice: (202) 512-6000: 
TDD: (202) 512-2537: 
Fax: (202) 512-6061:  

To Report Fraud, Waste, and Abuse in Federal Programs:  

Contact:  

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: fraudnet@gao.gov: 
Automated answering system: (800) 424-5454 or (202) 512-7470:  

Congressional Relations:  

Ralph Dawn, Managing Director, dawnr@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548:  

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: