This is the accessible text file for GAO report number GAO-03-1005 
entitled 'Military Recruiting: DOD Needs to Establish Objectives and 
Measures to Better Evaluate Advertising's Effectiveness' which was 
released on September 19, 2003.

This text file was formatted by the U.S. General Accounting Office 
(GAO) to be accessible to users with visual impairments, as part of a 
longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov.

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately.

Report to the Senate and House Committees on Armed Services:

United States General Accounting Office:

GAO:

September 2003:

Military Recruiting:

DOD Needs to Establish Objectives and Measures to Better Evaluate 
Advertising's Effectiveness:

GAO-03-1005:

GAO Highlights:

Highlights of GAO-03-1005, a report to the Senate and House Committees 
on Armed Services 

Why GAO Did This Study:

The Department of Defense (DOD) must convince more than 200,000 people 
each year to join the military. To assist in recruiting, the military 
services advertise on television, on radio, and in print and 
participate in other promotional activities. In the late 1990s, some 
of the services missed their overall recruiting goals. In response, 
DOD added recruiting resources by increasing its advertising, number 
of recruiters, and financial incentives. By fiscal year 2003, DOD’s 
total recruiting budget was approaching $4 billion annually.

At the request of Congress, GAO determined the changes in DOD’s 
advertising programs and funding trends since the late 1990s and 
assessed the adequacy of measures used by DOD to evaluate the 
effectiveness of its advertising. 

What GAO Found:

Since the late 1990s, DOD has revamped its recruiting advertising 
programs and nearly doubled the funding for recruiting advertising. 
The military services have revised many of their advertising campaigns 
and focused on complementing traditional advertising, such as by 
increasing the use of the Internet, and participating in more 
promotional activities, such as sports car racing events. DOD’s total 
advertising funding increased 98 percent in constant dollars from 
fiscal year 1998 through fiscal year 2003—from $299 million to $592 
million. The advertising cost per enlisted recruit has nearly tripled 
and is now almost $1,900. The military services agree that the revised 
strategies and increased investments have energized their advertising 
campaigns and better positioned them to recruit in an increasingly 
competitive marketplace. Today, almost all of the active and reserve 
components are meeting their overall recruiting goals in terms of the 
quality and quantity of new recruits. 

DOD does not have clear program objectives and adequate outcome 
measures to evaluate the effectiveness of its advertising as part of 
its overall recruiting effort. Thus, DOD cannot show that its 
increased advertising efforts have been a key reason for its overall 
recruiting success. Isolating the impact of advertising on recruiting 
efforts is inherently difficult because joining the military is a 
profound life decision. Moreover, DOD has not consistently tracked key 
information, such as public awareness of military recruiting 
advertising and the willingness of young adults to join the military. 
Such data could be used to help evaluate the effectiveness of 
advertising. Without sufficient information on advertising’s 
effectiveness, DOD cannot determine the return on its advertising 
funding or make fact-based choices on how its overall recruiting 
investments should be allocated.

What GAO Recommends:

GAO recommends that DOD set clear, measurable advertising objectives; 
develop outcome measures to evaluate advertising programs’ 
performance; and use these measures to monitor advertising’s 
performance and make choices on recruiting investments.

In its comments on this report, DOD concurred with the recommendations 
and stated that it will develop a DOD advertising strategic framework 
to provide overall direction.

www.gao.gov/cgi-bin/getrpt?GAO-03-1005.

To view the full product, including the scope and methodology, click 
on the link above. For more information, contact Derek B. Stewart at 
(202) 512-5559 or stewartd@gao.gov.

[End of section]

Contents:

Letter:

Results in Brief:

Background:

DOD Has Revised Advertising Programs and Increased Funding:

DOD Does Not Adequately Measure Advertising's Effectiveness:

Conclusions:

Recommendations for Executive Action:

Agency Comments and Our Evaluation:

Appendix I: Scope and Methodology:

Appendix II: Comments from the Department of Defense:

Related GAO Products:

Tables:

Table 1: DOD's Advertising Campaign Slogans, Program Descriptions, and 
Examples of Key Changes:

Table 2: Summary of DOD's Recruiting Advertising Funding:

Figures:

Figure 1: Army-Sponsored Sports Racing Car:

Figure 2: Total DOD Recruiting Advertising Funding for Fiscal Years 
1998 to 2003:

Figure 3: Total Recruiting Investment per Enlisted Recruit for Fiscal 
Years 1990 to 2003:

Figure 4: The Use of Advertising throughout the Recruiting Process:

Abbreviations:

DOD: Department of Defense:

GAO: General Accounting Office:

GPRA: Government Performance and Results Act:

United States General Accounting Office:

Washington, DC 20548:

September 19, 2003:

The Honorable John W. Warner 
Chairman 
The Honorable Carl Levin 
Ranking Minority Member 
Committee on Armed Services 
United States Senate:

The Honorable Duncan L. Hunter 
Chairman 
The Honorable Ike Skelton 
Ranking Minority Member 
Committee on Armed Services 
House of Representatives:

To meet its human capital needs, the Department of Defense (DOD) must 
convince about 200,000 people each year--the majority of them recent 
high school graduates--to join the military. To assist in this 
recruiting effort, the military services advertise on television, on 
radio, in print, and on the Internet; sponsor sports teams; and 
participate in other promotional activities. Such advertising is 
designed primarily to raise awareness of the military as a career 
option and help recruiters meet their goals for new recruits. During 
the exceptionally strong U.S. economy of the late 1990s, most of the 
services missed their overall recruiting goals. In response, DOD put 
additional resources into recruiting by increasing advertising, the 
number of recruiters, and various incentives, such as enlistment 
bonuses. By fiscal year 2003, DOD's total recruiting budget was 
approaching $4 billion annually.

The Senate Committee on Armed Services directed that we examine DOD's 
growing investments in military recruitment advertising.[Footnote 1] As 
agreed with your committees, the objectives of this report were to (1) 
determine the changes in DOD's advertising programs and funding trends 
since the late 1990s and (2) assess the adequacy of the measures used 
by DOD to evaluate the effectiveness of its advertising. In March 2003, 
we provided your committees with an interim briefing that described the 
trends in advertising funding requests since fiscal year 2000 and DOD's 
justifications for those requests.

This report updates the information discussed in the interim briefing 
and provides our analysis of the other issues in your request. To 
determine the changes in DOD's advertising programs and funding trends 
since the late 1990s, we reviewed the changes in the services' 
advertising programs, DOD's and the services' congressional 
justification books, and DOD funding data. To assess the adequacy of 
DOD's outcome measures, we used established management guidance 
provided in the Government Performance and Results Act[Footnote 2] 
(GPRA) and in Office of Management and Budget guidance. Our scope 
included DOD's active duty services, reserve components, and joint 
advertising program. We conducted our review from October 2002 through 
July 2003 in accordance with generally accepted government auditing 
standards. (See appendix I for more detailed information on our scope 
and methodology.):

Results in Brief:

Since the late 1990s, DOD has revamped its recruiting advertising 
programs and nearly doubled the funding devoted to recruiting 
advertising. The active duty military services, except for the Marine 
Corps, substantially revised their advertising campaigns and selected 
new advertising agencies as contractors. Long-time and well-recognized 
advertising slogans such as the Army's "Be All You Can Be" were 
abandoned in favor of campaigns designed to better appeal to today's 
young adults. The military services agree that these revised strategies 
and increased investments have energized their advertising campaigns 
and better positioned them to recruit young adults in an increasingly 
competitive marketplace. Today, almost all of the active and reserve 
components report that they are meeting their overall recruiting goals, 
both in terms of the quality and quantity of recruits. To better reach 
today's young adults, the services have focused on complementing 
traditional advertising by increasing funding for events marketing, 
public relations, and the Internet. The expenditures for paid 
television, which remains the single largest advertising cost, and 
other national media have declined as a percentage of total advertising 
funding. DOD's total advertising funding increased 98 percent from 
fiscal year 1998 through fiscal year 2003--from $299 million to $592 
million.[Footnote 3] Today, DOD is spending $1,900 in advertising per 
enlisted recruit, which is almost three times as much as it did in 
fiscal year 1990. The increases in funding have not been evenly 
distributed across DOD's advertising programs. The size of each 
service's advertising programs varies greatly. The Army has the largest 
advertising programs; its active and reserve components account for 
nearly half of the total advertising funding.

DOD does not have adequate outcome measures to evaluate the 
effectiveness of its advertising as part of its overall recruiting 
effort. Thus, DOD cannot show that its increased advertising efforts 
have been a key reason for its overall recruiting success. Evaluating 
advertising's effectiveness requires that DOD establish clear program 
objectives and outcome measures. DOD has not established such 
objectives and outcome measures for two reasons. First, isolating the 
impact of advertising on recruiting efforts is inherently difficult 
because joining the military is a profound life decision influenced by 
many factors, including the opportunities available in college or in 
the job market. Second, even though DOD has developed recruiting goals 
to ensure that it meets its human capital needs, these goals do not 
directly relate to or measure the effectiveness of advertising. Owing 
to the absence of program objectives and outcome measures, DOD has not 
consistently tracked key information, such as public awareness of 
military recruiting advertising and changes in the willingness of young 
adults to join the military. Such information could be used to help 
evaluate the effectiveness of advertising. In our 2000 report, we 
recommended that DOD and the services assess the relative success of 
their recruiting strategies, including how the services can create the 
most cost-effective mix of recruiters, enlistment bonuses, college 
incentives, advertising, and other recruiting tools.[Footnote 4] 
Although DOD acknowledges the need for such information, current DOD 
guidance does not require the measurement of outcomes or reports on 
advertising's effectiveness. Without sufficient information on 
advertising's effectiveness, DOD cannot determine the return on its 
advertising funding or make fact-based choices on how its overall 
recruiting investments should be allocated.

We are making recommendations to DOD to improve its guidance to better 
evaluate recruiting advertising's effectiveness. We are recommending 
that DOD set clear, measurable advertising objectives for its 
advertising programs and develop outcome measures to evaluate the 
performance of its advertising programs. We are also recommending that 
DOD use these outcome measures to monitor its advertising programs' 
performance and make fact-based choices about advertising funding as 
part of the overall recruiting investment.

In commenting on a draft of this report, DOD concurred with all of our 
recommendations. DOD stated in its comments that it will implement the 
recommendations by developing a DOD advertising strategic framework to 
provide overall direction for its advertising programs and by 
conducting research initiatives intended to advance the measurement of 
the performance of recruiting and advertising.

Background:

Most of the military services' active and reserve components faced 
recruiting difficulties during the strong economic climate of the late 
1990s. As a result, the services stepped up their recruiting to ensure 
that they would have enough recruits to fill their ranks. Recruiting 
efforts focus on three initiatives. First, a "sales force" of more than 
15,000 recruiters, who are mostly located in the United States, recruit 
from the local population. Second, these recruiters have financial and 
other incentives that they can use to convince young adults to consider 
a military career. Such incentives include enlistment bonuses and 
college benefits. Finally, the services use advertising to raise the 
public's awareness of the military and help the sales force of 
recruiters reach the target recruiting population and generate 
potential leads for recruiters. This advertising can include television 
and radio commercials, Internet and printed advertisements, and special 
events.

DOD believes that advertising is increasingly critical to its 
recruiting effort because convincing young adults to join the military 
is becoming more difficult. In 2001, over 70 percent of polled young 
adults said that they probably or definitely would not join the 
military, compared with 57 percent in 1976.[Footnote 5] The number of 
veterans is declining, which means that fewer young adults have 
influencers--a relative, coach, or teacher--who have past military 
experience. Compounding these difficulties, proportionally more high 
school graduates are attending college. Finally, the perception that 
service in the military is arduous--and possibly dangerous--can inhibit 
recruiting efforts. DOD believes that these factors together make the 
military an increasingly harder sell as a career choice and life-style 
option for young adults.

The Office of the Secretary of Defense is responsible for establishing 
policy and providing oversight for the military recruiting and 
advertising programs of the active and reserve components. Within the 
Office of the Secretary of Defense, the Under Secretary for Personnel 
and Readiness is responsible for developing, reviewing, and analyzing 
recruiting policy, plans, and resource levels. The office provides 
policy oversight for advertising programs and coordinates them through 
the Joint Marketing and Advertising Committee. DOD's strategic plan for 
military personnel human resources emphasizes the need to recruit, 
motivate, and retain adequate and diverse numbers of quality 
recruits.[Footnote 6]

DOD's recruiting and advertising programs are not centrally managed. 
All of the active components and some of the reserve components manage 
their separate advertising programs and work closely with their own 
contracted advertising agencies.[Footnote 7] DOD and the services 
believe that this decentralized approach better differentiates between 
the service "brands" (i.e., Army, Navy, Air Force, Marines). The Joint 
Advertising, Market Research, and Studies program, which is funded 
separately by DOD, exists to address common DOD requirements, such as 
conducting market research and obtaining and distributing lists of 
potential leads. The joint program has developed a DOD-wide advertising 
campaign to target the adult influencers of potential recruits, but 
this program had not been fully implemented at the time of our review.

DOD Has Revised Advertising Programs and Increased Funding:

After most of the services experienced recruiting shortfalls in the 
late 1990s, DOD reviewed its advertising programs and identified 
opportunities for improvement. The services, except the Marine Corps, 
substantially revised their advertising campaigns and slogans and 
contracted with new advertising agencies. The services told us that 
their revised campaigns place them in a better position to recruit 
today's young adults. Currently, almost all of the services and reserve 
components are achieving their recruiting goals, and advertising 
funding has almost doubled since fiscal year 1998. The increases in 
funding have not been used to buy more national media, such as 
television commercials. Rather, the funding increases are being 
directed to other types of advertising, such as special events 
marketing and the Internet, that are intended to better reach today's 
young adults. Advertising funding for DOD increased from $299 million 
in fiscal year 1998 to $592 million in fiscal year 2003, an increase of 
98 percent.[Footnote 8]

Military Services Have Revised Their Advertising Campaigns to Better 
Attract Today's Young Adults:

Recruiting shortfalls in the late 1990s led to an examination and 
revision of DOD's advertising programs. The Army, Navy, and Air Force 
missed their recruiting quantity goals, while some of the reserve 
components fell short of both their quantity and quality 
goals.[Footnote 9] Following these recruiting shortfalls, Congress 
asked the Secretary of Defense to review DOD's advertising programs and 
make recommendations for improvements.[Footnote 10] DOD has revamped 
its advertising programs. The active-duty services, except for the 
Marine Corps, substantially revised their advertising campaigns and 
selected new advertising agencies as their contractors. They produced 
new advertising strategies and campaigns, complete with new slogans and 
revised television, print, and radio advertisements, along with new 
brand images defined by distinct logos, colors, and music. The 
services, in conjunction with their advertising agencies, conducted new 
research on young adults--their primary target market. During this 
period, the joint program developed an advertising campaign to target 
influencers of prospective recruits, as recommended in DOD's review.

In addition to their overall campaigns, all of the services have 
specialized campaigns to target diverse segments of the young adult 
population. For instance, the Navy created a Web site, called El Navy, 
which is designed to better communicate with the Hispanic market, and 
the Army has specifically tailored radio advertisements to reach the 
African American market. The services also incorporated a greater 
variety of public relations and promotional activities, such as 
participating in job fairs and sponsoring sports car racing teams, as 
an integral part of their advertising programs. As shown in table 1, 
there are essentially nine advertising programs that are managed 
separately by the military services, reserve components, and the Office 
of the Secretary of Defense.

Table 1: DOD's Advertising Campaign Slogans, Program Descriptions, and 
Examples of Key Changes:

Components: Army, Army Reserve; Current campaign slogans: (year 
established): An Army of One (2001); Program descriptions and examples 
of key changes: * Army and Army Reserve combined programs; * Army 
National Guard program independently managed; * Largest active and 
reserve recruiting mission and advertising budget; * New advertising 
campaigns and contractors; * Advertises in all national media venues; 
* Engages in promotional events, such as sports car racing sponsorship, 
high school sports, and video games; * Initiated on-line recruiting.

Components: Army National Guard; Current campaign slogans: (year 
established): You Can (1997).

Components: Navy; Current campaign slogans: (year established): 
Accelerate Your Life (2001); Program descriptions and examples of key 
changes: * Separate advertising programs for Navy and Naval Reserve 
(reorganized recruiting under one commander); * Second largest active 
duty recruiting mission; * New advertising campaigns and contractors; 
* Advertises in all major media.

Components: Naval Reserve; Current campaign slogans: (year 
established): Stay Strong (2001).

Components: Air Force; Current campaign slogans: (year established): 
Cross into the Blue (2001); Program descriptions and examples of key 
changes: * Three independent advertising programs; * Third largest 
active duty recruiting mission; * New advertising campaigns and 
contractors; * Active Air Force emphasizing promotional activities and 
events, such as traveling recruiting trucks and sports car racing; * 
Initiated a national television campaign for the active Air Force; * 
Increased use of Internet recruiting across the active and reserve 
components.

Components: Air Force Reserve; Current campaign slogans: (year 
established): Above and Beyond (1998).

Components: Air National Guard; Current campaign slogans: (year 
established): Fuel Your Future (1999).

Components: Marine Corps, Marine Corps Reserve; Current campaign 
slogans: (year established): Marines, The Few, The Proud (1986); 
Program descriptions and examples of key changes: * Marine Corps and 
Marine Corps reserve integrated programs; * Smallest recruiting 
mission of the services; * Marine Corps brand image not changed for 30 
years and same advertising contractor for 56 years; * Emphasis on 
television, especially sports programming.

Components: Joint Program; Current campaign slogans: (year 
established): Today's Military--See It for What It Really Is (2003); 
Program descriptions and examples of key changes: * Developed 
advertising campaign to target influencers of prospective recruits that 
includes magazine advertisements, use of Web site, and television 
public service announcements; * New advertising contractor; * 
Conducts market research and studies for DOD's advertising programs; * 
Provides other support for DOD's advertising programs.

Source: DOD.

[End of table]

The active services told us that they are pleased with their new 
advertising campaigns and agencies, and they believe that the revised 
and better-funded campaigns have placed them in a more competitive 
position to recruit young adults. The sluggish U.S. economy has also 
narrowed employment options and is considered to be an important factor 
in easing the recruiting challenge. Today, all of the active services 
are meeting or exceeding their overall recruiting goals. Most of the 
reserve components are also achieving their recruiting goals. As of 
June 2003, the Army National Guard was falling short of its recruiting 
goals because of extensive overseas deployments and the implementation 
of stop loss (restrictions on leaving the military). Army National 
Guard officials stated that they expect to meet their goals by the end 
of fiscal year 2003. Some reserve officers expressed concerns about the 
negative impact of the recent high deployment rates on future 
recruiting. The services, especially the reserve components, continue 
to face challenges in recruiting individuals with some types of 
specific training or skills, such as medical, legal, and construction, 
and they have developed some specialized advertising campaigns targeted 
to recruit them.

Since fiscal year 1998, the services have changed how they allocate 
advertising funding, according to the figures provided by DOD. Grouped 
into three broad categories, advertising funding includes: (1) events 
marketing, public affairs and public relations, Internet, and other; 
(2) national media; and (3) direct mail and miscellaneous recruiting 
support. One of the categories--events marketing, public affairs and 
public relations, Internet, and other--has shown the greatest increase 
as a percentage of the total budget, nearly tripling from around 10 
percent in fiscal year 1998 to 29 percent in fiscal year 2003. This 
increase was used partly to create and produce new advertising 
campaigns and strategies. Service officials told us that event 
marketing and public relations activities provide recruiters with 
greater opportunities to interact with potential recruits and 
supplement their national media campaigns and other methods of 
advertising. One example is the Army's sponsorship of a sports racing 
car. (See fig. 1.) Internet and Web-site recruiting have also increased 
significantly from fiscal year 1998 through fiscal year 2003. All of 
the active military services have increased the amount of advertising 
on the Internet and have used interactive Web sites to complement their 
traditional recruiting and advertising methods.

Figure 1: Army-Sponsored Sports Racing Car:

[See PDF for image]

[End of figure]

The expenditures for the national media category, which includes paid 
television, radio, and magazine advertisements, have remained 
relatively constant. This means that this category's proportion of the 
growing total advertising budgets has actually declined. Specifically, 
expenditures for the national media in fiscal year 1998 were more than 
half of the advertising budget; currently, it represents about 40 
percent. Television advertising--which offers tremendous reach to 
target audiences--dominates this category. Television advertising has 
remained the single largest advertising expenditure: paid television is 
still about a quarter of the total advertising budget for all of the 
military components.

DOD Has Significantly Increased Funding for Advertising:

DOD's advertising funding has nearly doubled in the years since 1998 
and most of these increases occurred in the earlier years. (See fig. 
2.) Total advertising funding for all of the services increased 98 
percent, from $299 million in fiscal year 1998 to $592 million in 
fiscal year 2003.[Footnote 11] The total DOD advertising budget request 
to Congress for fiscal year 2004 was $592.8 million.

Figure 2: Total DOD Recruiting Advertising Funding for Fiscal Years 
1998 to 2003:

[See PDF for image]

Note: The funding amounts were taken from DOD's and the services' 
congressional budget justification books (adjusted to account for 
inflation).

[A] In-year estimate.

[End of figure]

Since fiscal year 1998, DOD's advertising funding, which is included in 
DOD's operation and maintenance appropriations, has increased at a 
significantly higher rate than the total of all of DOD's operation and 
maintenance funding. DOD officials cite media inflation as one reason 
for increased advertising funding. Inflation for some types of media, 
especially for television commercials, has been higher than general 
inflation. However, this is not the reason for all of the increases in 
advertising funding during this period because not all of the 
advertising funding is used for media advertising. For example, only 
about a quarter of advertising funds are currently spent to buy time to 
run television commercials.

Growing advertising costs are only part of a rapidly increasing total 
investment in recruiting. The rising advertising and overall recruiting 
costs can be seen in the investment per enlisted recruit--an important 
bottom-line measure that shows the amount of money spent to enlist each 
recruit. Today, the services are spending almost three times as much on 
advertising per recruit than in fiscal year 1990. We examined data 
collected by DOD from the services, and it showed that the total 
advertising investment per enlisted recruit rose from approximately 
$640 to $1,900 between fiscal year 1990 and fiscal year 2003. As a 
proportion of the total recruiting investment, advertising has 
increased from 8 percent in fiscal year 1990 to 14 percent in fiscal 
year 2003. Bonuses and incentives to enlist have also increased 
substantially during this same period. The total recruiting investment 
per recruit increased almost 65 percent, from approximately $8,100 in 
fiscal year 1990 to $13,300 in fiscal year 2003. Very steep growth 
occurred between fiscal year 1998 and fiscal year 2002. This is shown 
in figure 3.

Figure 3: Total Recruiting Investment per Enlisted Recruit for Fiscal 
Years 1990 to 2003:

[See PDF for image]

[End of figure]

The increases are not evenly distributed across the services' 
advertising programs. (See table 2.) The Army has the largest 
advertising budget, and the Army active and reserve components account 
for nearly half (about $295 million) of the total advertising funding. 
The Marine Corps, at just under $50 million, has the smallest 
advertising budget. The Air Force has experienced the most significant 
increase in funding, in part owing to the creation of its first 
national television campaign. The Navy's advertising funding has also 
increased, but this is primarily due to the addition of costs related 
to the Blue Angels[Footnote 12] and a program to test recruiting kiosks 
at public locations.

DOD's Joint Advertising, Market Research, and Studies Program is 
responsible for (1) providing market research and studies for 
recruiting and (2) developing an advertising campaign to target adult 
influencers, such as parents, coaches, and career counselors. 
Currently, the joint program is conducting market research and studies 
and providing other support for the services' advertising programs, 
such as purchasing lists of high school students and recent graduates 
for use in mailing advertisements. In addition, the program is 
implementing a limited print advertising campaign targeting influencers 
in fiscal year 2003.

The joint advertising campaign has not had consistent funding. Program 
managers told us that the current funding level is insufficient to 
fully implement the influencer advertising campaign they have 
developed. In past years, DOD cut funding for the joint advertising 
program because of concerns that the program office was not adequately 
executing its advertising budget. For fiscal year 2003, Congress 
provided the joint advertising program with less funding than DOD 
requested, and DOD subsequently reallocated part of the remaining joint 
advertising funding to a program that it considered a higher priority.

Table 2: Summary of DOD's Recruiting Advertising Funding:

Constant fiscal year 2003 dollars in millions.

Army; 1998: (actual): $113.7; 2003: (in-year estimate): $196.9; 
Percentage: change: 73.

Army Reserve; 1998: (actual): 17.0; 2003: (in-year estimate): 50.2; 
Percentage: change: 196.

Army National Guard[A]; 1998: (actual): 23.2; 
2003: (in-year estimate): 48.2; Percentage: change: 108.

Navy; 1998: (actual): 75.7; 2003: (in-year 
estimate): 107; Percentage: change: 41.

Naval Reserve; 1998: (actual): 2.4; 2003: 
(in-year estimate): 7.4; Percentage: change: 208.

Air Force; 1998: (actual): 18.3; 2003: 
(in-year estimate): 90.5; Percentage: change: 395.

Air Force Reserve; 1998: (actual): 4.6; 2003: 
(in-year estimate): 13.5; Percentage: change: 193.

Air National Guard[A]; 1998: (actual): 4.4; 
2003: (in-year estimate): 5.8; Percentage: change: 31.

Marine Corps; 1998: (actual): 29.8; 2003: 
(in-year estimate): 46.5; Percentage: change: 56.

Marine Corps Reserve; 1998: (actual): 3.0; 
2003: (in-year estimate): 2.9; Percentage: change: -3.

Joint Program; 1998: (actual): 6.8; 2003: 
(in-year estimate): 22.9[B]; Percentage: change: 237.

Totals; 1998: (actual): $298.9; 2003: (in-year 
estimate): $591.8; Percentage: change: 98.

Source: DOD.

Note: The funding amounts were taken from DOD's and the services' 
congressional budget justification books (adjusted to account for 
inflation).

[A] The advertising funding for the Army National Guard and the Air 
National Guard is for both recruiting and retention. These figures do 
not include the funding for recruiting and retention advertising done 
by the states and territories.

[B] DOD subsequently reallocated part of this funding.

[End of table]

DOD Does Not Adequately Measure Advertising's Effectiveness:

DOD does not have adequate outcome measures to evaluate the 
effectiveness of its advertising as part of its overall recruiting 
effort. Effective program management requires the establishment of 
clear objectives and outcome measures to evaluate the program, and DOD 
has established neither. This has been a long-standing problem for DOD 
primarily because measuring the impact of advertising is inherently 
difficult, especially for a major life decision such as joining the 
military. Owing to the absence of established advertising objectives 
and outcome measures, DOD has not consistently collected and 
disseminated key information that would allow it to better assess 
advertising's contribution to achieving recruiting goals. This 
information would include public awareness of military recruiting 
advertising and the willingness of young adults to join the military. 
Rather, the services report outcome measures that focus on achieving 
overall recruiting goals instead of isolating the specific contribution 
of advertising. Without adequate information and outcome measures, the 
Office of the Secretary of Defense cannot satisfactorily review the 
services' advertising budget justifications nor can it determine the 
return on their advertising dollars as part of their overall recruiting 
investment.

Effective Management Requires Clear Program Objectives and Outcome 
Measures:

The Secretary of Defense is required by law to enhance the 
effectiveness of DOD's recruitment programs through an aggressive 
program of advertising and market research targeted at prospective 
recruits and those who may influence them.[Footnote 13] DOD guidance 
requires the services, by active and reserve components, to report 
their resource inputs--how much they are spending on 
advertising.[Footnote 14] DOD guidance also requires the services to 
report on overall recruiting outcomes[Footnote 15]--their recruit 
quantity and quality. However, the guidance does not require active and 
reserve components to report information specifically about the 
advertising programs' recruiting effectiveness.

Effective program management requires the establishment of clearly 
defined objectives and outcome measures to evaluate programs. The 
Government Performance and Results Act was intended to help federal 
program managers enhance the effectiveness of their programs.[Footnote 
16] It requires agencies to establish strategic plans for program 
activities that include, among other things, a mission statement 
covering major functions and operations, outcome-related goals and 
objectives, and a description of how these goals and objectives are to 
be achieved. GPRA shifted the focus of accountability for federal 
programs from inputs, such as staffing and resource levels, to 
outcomes. This requires agencies to measure the outcomes of their 
programs and to summarize the findings of program evaluations in their 
performance reports. The Office of Management and Budget's guidance 
implementing GPRA requires agencies to establish meaningful program 
objectives and identify outcome measures that compare actual program 
results with established program objectives.[Footnote 17]

Lack of Adequate Outcome Measures Limits DOD's Ability to Effectively 
Allocate Its Recruiting Investments:

DOD does not have adequate information to measure the effectiveness of 
its advertising as part of the overall recruiting effort. Measuring 
advertising's effectiveness has been a long-standing problem, partly 
because it is inherently difficult to measure the impact that 
advertising has on recruiting. DOD has not established advertising 
program objectives and it lacks adequate outcome measures of the impact 
that advertising programs have on recruiting. Outcome measures are used 
to evaluate how closely a program's achievements are aligned with 
program objectives, and to assess whether advertising is achieving its 
intended outcome. DOD currently requires the services and reserve 
components to report on inputs and outcomes related to overall 
recruiting. These measures are important in assessing DOD's overall 
recruiting success; however, they do not assess advertising's 
contribution to the recruiting process.

Measuring Advertising's Effectiveness Is a Long-standing and Difficult 
Problem:

In our 2000 report, we noted that the services do not know which of 
their recruiting initiatives--advertising, recruiters, or bonuses--
work best.[Footnote 18] This prevented DOD from being able to 
effectively allocate its recruiting investment among the multiple 
recruiting resources. We recommended that DOD and the services assess 
the relative success of their recruiting strategies, including how the 
services can create the most cost-effective mix of recruiters, 
enlistment bonuses, college incentives, advertising, and other 
recruiting tools. In comments on that report, DOD stated that it 
intended to develop a joint-service model that would allow trade-off 
analyses to determine the relative cost-effectiveness of the various 
recruiting resources. This has not been done, and the current DOD cost 
performance trade-off model does not support analyses across the 
services' budgets.

Similarly, a 2002 Office of Management and Budget assessment, known as 
the Program Assessment Rating Tool, found that DOD's recruiting program 
had met its goal of enlisting adequate numbers of recruits; however, 
since there were no measures of program efficiency, the overall rating 
for the recruiting program was only "moderately effective." In its 
assessment, the Office of Management and Budget noted the inability of 
the recruiting program to assess the impact of individual resources, 
such as advertising and recruiters. The services continually adjust the 
mix of funding between advertising and other recruiting resources to 
accomplish their program goals. They have generally increased spending 
on advertising, added recruiters, and increased or added bonuses at the 
same time, making it impossible to determine the relative value of each 
of these initiatives. Other studies have reached similar conclusions. 
In 2000, a review of DOD's advertising programs resulted in a 
recommendation that they be evaluated for program 
effectiveness.[Footnote 19] More recently, the National Academy of 
Sciences also cited the need to evaluate advertising's direct influence 
on actual enlistments.[Footnote 20] The academy is now doing additional 
work on evaluating DOD's advertising and recruiting.

The lack of adequate information is partly attributable to the inherent 
difficulty in measuring advertising's affect on recruiting. Measuring 
advertising's effectiveness is a challenge for all businesses, 
according to advertising experts. Private-sector organizations cannot 
attribute increases in sales directly to advertising because there are 
many other factors influencing the sale of products, such as quality, 
price, and the availability of similar products. Many factors impact 
recruiting as well, such as employment and educational opportunities, 
making it especially difficult to isolate and measure the effectiveness 
of advertising. Enlisting in a military service is a profound life 
decision. Typically, an enlistment is at least a 4-year commitment and 
can be the start of a long military career.

Another complicating factor in measuring advertising's effectiveness is 
that it consists of different types and is employed differently 
throughout the recruiting process to attract and enlist potential 
recruits. Figure 4 displays the recruiting process and demonstrates the 
role of advertising while a young adult may be considering enlisting in 
the military.

Figure 4: The Use of Advertising throughout the Recruiting Process:

[See PDF for image]

[End of figure]

As the figure shows, the use of multiple types of advertising at 
various stages in the recruiting continuum makes it difficult to assess 
the effectiveness of specific types of advertising. A single recruit 
may be exposed to some or all of these advertising types. Traditional 
advertising in the national media, such as television and magazines, is 
intended to disseminate information designed to influence consumer 
activity in the marketplace. The services typically use such national 
media to make young people aware of a military service, the career 
options available in a service, and other opportunities the services 
have to offer them. Direct mail, special events, and the services' Web 
sites are utilized to provide more detailed information about the 
services and the opportunities available for persons who enlist. These 
marketing resources give people the opportunity to let a recruiter know 
they are possibly interested in enlisting in a service.

DOD Lacks Guidance Establishing Advertising Programs' Objectives and 
Outcome Measures:

Another contributing factor to the absence of advertising objectives 
and outcome measures is the lack of DOD-wide guidance. Officials from 
the Office of the Secretary of Defense view their role as overseeing 
the decentralized programs managed by the individual services and 
reserve components. They scrutinize the quality and quantity of 
recruits and gather data about the uses of advertising funds. However, 
they told us they were reluctant to be more prescriptive because of a 
concern about appearing to micromanage the successful recruiting 
programs of the active and reserve components. On the basis of our 
work, their sensitivity is warranted. The active and reserve components 
tend to guard their independence, seeking to maintain their "brand" and 
arguing that the current decentralized structure allows them to be more 
responsive to their individual needs. The Office of the Secretary of 
Defense seeks to coordinate the active and reserve components' 
activities through joint committees and to centralize research that can 
be utilized by all.

Defining exactly what to measure may be difficult, but it is not 
impossible. DOD and the services, as well as their contracted 
advertising agencies, generally agree that there are at least two key 
advertising outcomes that should be measured: (1) the awareness of 
recruiting advertising and (2) the willingness or "propensity" to 
consider joining the military. However, this is not clearly stated in 
any program guidance. Current DOD guidance requires only that the 
services provide information on funding for advertising, the quality 
and quantity of recruits, and the allocation of resources to the 
various advertising categories.[Footnote 21] Although this information 
is valuable--in fact, critical--it is not sufficient to evaluate and 
isolate the effectiveness of the services' advertising programs.

DOD's efforts thus far to measure the awareness of recruiting 
advertising and willingness to join the military have met with 
problems. Inconsistent funding for the Joint Advertising, Market 
Research, and Studies program has hampered consistent collection of 
this information. DOD has sponsored an advertising tracking study 
designed to monitor the awareness of individual service campaigns since 
2001. However, officials from the Army, Navy, and Marine Corps told us 
that they do not regularly use the research provided by this study. 
According to program officials, there were numerous problems with the 
advertising tracking study.[Footnote 22] DOD is implementing changes to 
the study that are intended to improve its usefulness to all of the 
active and reserve components. In the absence of reliable and timely 
advertising tracking, the Army implemented its own tracking study, and 
the Air Force is currently planning an experimental study to assess the 
effectiveness of its national television advertising campaign, 
according to program managers. To monitor the willingness to join the 
military, DOD sponsors youth and adult polls, which are designed to 
track changes in attitudes and young adults' aspirations. These polls 
replaced the Youth Attitude Tracking Survey, which had been in place 
for a number of years and provided long-term trend data about the 
propensity of young adults to consider the military. The services 
expressed concern that the current polls ask questions that are 
significantly different from those asked in the prior survey, which 
makes the analysis of trends difficult.

DOD officials also pointed to research indicating that advertising is a 
cost-effective recruiting investment when compared with other 
recruiting initiatives. For example, a report that was done for DOD 
found that it was less expensive to enlist a recruit through increased 
investments in advertising than through increased investments in 
military pay for new recruits in the Army and Navy.[Footnote 23] 
Similarly, a study for DOD analyzed the marginal cost of different 
recruiting initiatives and concluded that, under certain conditions, it 
was more cost-effective to invest additional funds in advertising than 
in military pay for recruits or recruiters.[Footnote 24] DOD officials 
told us that these reports, which used data from the 1980s and early 
1990s, provide the best research available on the topic. However, the 
situation has changed dramatically in recent years. DOD has altered its 
advertising and recruiting strategies and is spending much more on 
advertising. Advertising itself is also changing and is more fragmented 
with an expanding array of television channels and other media. 
Finally, media inflation, which has increased faster than general 
inflation even in the sluggish economy, has lessened buying power.

Conclusions:

Funding devoted to advertising has increased considerably since fiscal 
year 1998. Although the military services are now generally meeting 
their overall recruiting goals, the question of whether the significant 
increases in advertising budgets were a main contributor to the 
services' recruiting successes remains open. During the same period, 
DOD also greatly increased funding for bonuses and other incentives to 
enlist recruits. At the same time, the U.S. economy slowed 
dramatically, narrowing the other employment options available to young 
people. These factors make it difficult to disentangle the effects of 
the internal DOD investments made in recruiting from the changes in the 
external recruiting environment. Even though the effect of advertising 
is inherently difficult to measure, this issue needs to be addressed. 
This is crucial because DOD is now spending nearly $592 million 
annually on recruiting advertising, or about $1,900 per enlisted 
recruit. In addition, the total funding for all of DOD's recruiting 
efforts is now almost $4 billion.

DOD needs better advertising outcome measures to allow it to oversee 
and manage the advertising investment as part of its overall recruiting 
effort. DOD and the services have an understandable focus on the most 
important program outcome--to ensure that the military has enough 
quality recruits to fill its ranks. Judged by this short-term measure, 
the recruiting programs are successful. But now that DOD is meeting its 
recruiting goals, should it reduce advertising funding or continue at 
its current funding levels? DOD believes that continued investments in 
advertising are critical to keeping awareness up in the young adult 
population and combating the declining propensity among today's young 
adults to join the military. However, DOD has neither stated these 
goals clearly in its guidance, nor consistently gathered information to 
ensure that these objectives are being met. Now that it is meeting its 
recruiting goals, DOD needs to turn its attention to program 
effectiveness and efficiency to ensure that the active and reserve 
components are getting the best return on their recruiting and 
advertising investments.

Recommendations for Executive Action:

To improve DOD's ability to adequately measure the impact of its 
advertising programs on its recruiting mission, we recommend that the 
Secretary of Defense direct the Under Secretary of Defense for 
Personnel and Readiness to issue guidance that would (1) set clear, 
measurable objectives for DOD's advertising programs; (2) develop 
outcome measures for each of DOD's advertising programs that clearly 
link advertising program performance with these objectives; and (3) use 
these outcome measures to monitor the advertising programs' performance 
and make fact-based choices about advertising funding as part of the 
overall recruiting investment in the future.

Agency Comments and Our Evaluation:

DOD concurred with all of our recommendations. In commenting on this 
report, DOD stated that the Office of the Under Secretary of Defense 
for Personnel and Readiness, in concert with the services, will develop 
an advertising strategic framework to provide overall direction for 
DOD's advertising programs. The framework, with associated outcome 
measures, would allow the office to monitor advertising results 
regularly and make fact-based decisions at a strategic level. It would 
provide an overarching structure within which each service would 
develop its own advertising program strategy, program objectives, and 
outcome measures. The framework would also direct the activities of the 
DOD joint program to ensure support to the services. DOD also commented 
that current research has not advanced to the point where models exist 
that adequately account for the many factors that affect recruiting as 
well as for the differences in the services. DOD stated that it will 
address this research gap through several initiatives intended to 
advance the measurement of the performance of recruiting and 
advertising. The National Academy of Sciences is currently developing 
an evaluation framework for recruiting and advertising and expects to 
publish a report in early 2004.

DOD's comments are provided in their entirety in appendix II. DOD 
officials also provided technical comments that we have incorporated as 
appropriate.

We are sending copies of this report to interested congressional 
committees; the Secretaries of Defense, the Army, the Navy, and the Air 
Force; and the Commandant of the Marine Corps. We will send copies to 
other interested parties upon request. In addition, the report will be 
available at no charge on the GAO Web site at http://www.gao.gov.

Please contact me at (202) 512-5559 if you or your staffs have any 
questions regarding this report. Key contributors to this report were 
John Pendleton, Lori Atkinson, Nancy Benco, Kurt Burgeson, Alan 
Byroade, Chris Currie, LaTonya Gist, Jim McGaughey, Charles Perdue, 
Barry Shillito, and John Smale.

Derek B. Stewart 
Director, Defense Capabilities and Management:

Signed by Derek B. Stewart: 

[End of section]

Appendix I: Scope and Methodology:

To describe the changes in the Department of Defense's (DOD) 
advertising programs and advertising funding trends since the late 
1990s, we reviewed advertising exhibits in the operation and 
maintenance congressional justification books as well as budget 
information provided by the Office of the Secretary of Defense. Since 
our objective was to look at broad funding trends, we did not reconcile 
these requested amounts with actual obligations or expenditures by the 
active and reserve components. We interviewed active and reserve 
component officials to understand program changes since the late 1990s. 
We obtained recruiting mission goals and actual accessions back to 
fiscal year 1990 from the Office of the Secretary of Defense and the 
services. We obtained information on the quality of accessions of each 
of the active and reserve components back to fiscal year 1990, as well 
as the investment per active enlisted accession back to fiscal year 
1990. We reviewed information from the Defense Human Resources Activity 
and the Joint Marketing and Advertising Committee for discussions of 
advertising programs. The services provided additional information 
regarding the types of advertising media they use.

To assess the adequacy of the measures used by DOD to evaluate the 
effectiveness of advertising, we reviewed information on outcome 
measures used to evaluate the effectiveness of advertising provided by 
each of the active and reserve components; the advertising agencies 
that are their contractors; and the DOD Joint Advertising, Market 
Research, and Studies program. We spoke with the advertising 
contractors to learn what measures of effectiveness they are aware of 
and use. We also reviewed the requirements for establishing program 
objectives and outcome measures in the Government Performance and 
Results Act and in Office of Management and Budget guidance.

We interviewed DOD and advertising officials from each of the active 
and reserve components, as well as representatives from the services' 
advertising agencies. We also reviewed their programs, procedures, and 
oversight activities. These interviews were conducted with officials in 
the Office of the Under Secretary of Defense for Personnel and 
Readiness; Office of the Under Secretary of Defense (Comptroller/Chief 
Financial Officer); Defense Human Resources Activity, Joint 
Advertising, Market Research, and Studies Office; Army Accessions 
Command, Fort Knox, Kentucky; Air Force Recruiting Service, Randolph 
Air Force Base, Texas; Navy Recruiting Command, Millington, Tennessee; 
Marine Corps Recruiting Command, Quantico Marine Corps Base, Virginia; 
Army National Guard Recruiting and Retention Command, Arlington, 
Virginia; Naval Reserve Command, New Orleans, Louisiana; Air Force 
Reserve Command, Robins Air Force Base, Georgia; and the Air National 
Guard Office of Recruiting and Retention, Arlington, Virginia. We also 
interviewed officials at the contracted advertising agencies for the 
joint program, the Army, the Navy, the Marine Corps, and the Air Force. 
We reviewed reports on recruiting and advertising from DOD, the 
Congressional Research Service, the private sector, and others. We 
obtained recruiting advertising budget and funding data for types of 
advertising from the Office of the Secretary of Defense. We reviewed, 
but did not verify, the accuracy of the data provided by DOD.

We conducted our review from October 2002 through July 2003 in 
accordance with generally accepted government auditing standards.

[End of section]

Appendix II: Comments from the Department of Defense:

Note: Page numbers in the draft report may differ from those in this 
report.

PERSONNEL AND READINESS:

OFFICE OF THE UNDER SECRETARY OF DEFENSE:

4000 DEFENSE PENTAGON WASHINGTON, D.C. 20301-4000:

SEP 2 2003:

Mr. Derek B. Stewart:

Director, Defense Capabilities and Management U.S. General Accounting 
Office:

441 G. Street, N.W. Washington, DC 20548:

Dear Mr. Stewart:

This is the Department of Defense (DoD) response to the GAO draft 
report, "MILITARY RECRUITING: DoD Needs to Establish Objectives and 
Measures to Better Evaluate Advertising's Effectiveness," dated August 
1, 2003 (GAO Code 350274/GAO-03-1005). We appreciate the opportunity to 
comment.

The Department concurs with the report's recommendations that the Under 
Secretary of Defense for Personnel and Readiness set clear measurable 
objectives, develop outcome measures, and use those measures to monitor 
advertising 
program performance and make fact-based choices about advertising 
funding. Of course, at the Service (both Active and Reserve component) 
level, objectives and outcome measures already exist.

The Office of the Under Secretary of Defense for Personnel and 
Readiness (OUSD-PR), in concert with the Services, will develop a DoD 
Advertising Strategic Framework. The framework, with associated outcome 
measures, will allow OUSD-PR to monitor advertising results and make 
fact-based decisions at a strategic level for the Department and the 
Joint Advertising, Market Research, and Studies Program. This will 
provide an overarching structure within which each Service will 
continue to develop its own advertising program strategy, program 
objectives, and outcome measures.

Outcome measures based simply upon advertising awareness and 
propensity, as suggested by the report, must be considered as an 
interim measure. There is danger in attempting to measure advertising 
program performance out of context. However, current research has not 
advanced to the point where models exist that adequately account for 
the many factors that impact recruiting as well as for Service 
differences simultaneously. OUSD-PR will address this research gap 
through several initiatives in FY 2004, as well as with follow-on work 
derived from those efforts.

The enclosure contains detailed Departmental comments on each of the 
three recommendations identified by the GAO. The Department appreciates 
the opportunity to comment on the draft report.

Sincerely,

Charles S. Abell 
Principal Deputy:

SIGNED FOR: Charles S. Abell: 

Enclosures: As stated:

GAO DRAFT REPORT - DATED AUGUST 1, 2003 GAO CODE 350274/GAO-03-1005:

"MILITARY RECRUITING: DoD Needs to Establish Objectives and Measures to 
Better Evaluate Advertising Effectiveness":

DEPARTMENT OF DEFENSE COMMENTS TO THE RECOMMENDATIONS:

RECOMMENDATION 1: The GAO recommended that the Secretary of the Defense 
direct the Under Secretary of Defense for Personnel and Readiness to 
issue guidance that would set clear measurable objectives for the 
advertising programs on its recruiting mission. (Page 22/GAO Draft 
Report):

DOD RESPONSE: Concur.

DoD agrees, in principle, with the GAO's interest in evaluative 
criteria and fact-based decisions on advertising funding.

The GAO notes in the body of the report that measuring advertising 
effectiveness is a challenge for all businesses, let alone for DoD with 
its unique set of characteristics (e.g., impact on recruiting of 
employment and educational opportunities, as well as the fact that 
entering military service is a profound life decision, not merely the 
sale of a product). In addition to the special challenges that face DoD 
mentioned in the report, there is considerable difficulty in attempting 
to apply "one size fits all" evaluative standards to DoD advertising. 
The number of new recruits required by each Service (both Active and 
Reserve component) varies, as does the attractiveness of Service career 
offerings, the absolute size of total Service annual recruiting 
investments, and their allocations to recruiters, bonuses, educational 
benefits, and advertising programs. DoD-level objectives must 
necessarily be broad in nature in order to allow each Service to tailor 
its strategic and tactical efforts to its particular needs and 
strengths. Given that the report fails to make mention of the core 
objectives set by each Service's advertising program which were 
provided to the GAO, DoD assumes that the GAO's recommendation is a 
call for broad, over-arching DoD-level objectives.

The Office of the Under Secretary of Defense for Personnel and 
Readiness, in concert with the Services, will develop a DoD Advertising 
Strategic Framework. This framework will be used by DoD to chart the 
overall direction of departmental advertising efforts, to direct the 
activities of the Joint Advertising, Market Research, and Studies 
Program to ensure support to the Services, and to provide a framework 
for the Services to use in constructing their strategy and tactical 
efforts.

RECOMMENDATION 2: The GAO recommended that the Secretary of the Defense 
direct the Under Secretary of Defense for Personnel and Readiness to 
develop outcome measures for each of DoD's advertising programs that 
clearly link advertising program performance with those objectives. 
(Page 22/GAO Draft Report):

DOD RESPONSE: Concur.

DoD-level outcome measures will follow from the DoD Advertising 
Strategic Framework. Of course, at the Service level, outcome measures 
already exist. Some examples include advertising awareness, propensity-
related measures, advertising lead generation, and lead-to-contract 
conversion.

Currently, several tools are available that support development of 
outcome measures for DoD-level objectives. The Joint Advertising, 
Market Research, and Studies Program has conducted an advertising 
tracking study since 2001. As mentioned in the report, the tracking 
study has recently been revised in an attempt to better meet Service 
advertising program needs. In addition, and also mentioned in the 
report, the Joint Advertising, Market Research, and Studies Program 
conducts a Youth Survey that measures propensity. Prior to this quick 
turn-around poll, the Department conducted an annual youth attitude 
tracking study dating back to 1974. While funding issues in the Joint 
Advertising, Market Research, and Studies Program have hampered 
consistent collection of these data, the program is currently 
conducting both studies.

Outcome measures based simply upon advertising awareness and propensity 
must be considered as an interim measure. Given the interaction effect 
that exists between the many factors that impact recruiting, of which 
advertising is only one, there is a danger in attempting to measure 
advertising program performance out of context. Current research has 
not advanced to the point where models exist that adequately account 
for these factors as well as for Service differences simultaneously.

The Office of the Under Secretary of Defense for Personnel and 
Readiness will address this research gap through several initiatives. 
The National Academy of Sciences Committee on the Youth Population and 
Military Recruitment is currently developing an evaluation framework 
for advertising and recruiting. Expected publication date of the report 
is early 2004. In addition, research will be funded in FY 2004 to 
identify data elements necessary to support empirical research on 
advertising effectiveness, with follow-on work planned to conduct 
quantitative analysis to develop advertising effectiveness models that 
would support decision-making at the macro (Office of the Under 
Secretary of Defense for Personnel and Readiness) and the micro 
(Service) levels. Specifically, more research must be undertaken on the 
determinants of propensity in general, and on advertising's 
contribution to propensity - both DoD and Service-specific.

RECOMMENDATION 3: The GAO recommended that the Secretary of the Defense 
direct the Under Secretary of Defense for Personnel and Readiness to 
use the above:

outcome measures to monitor advertising program performance and make 
fact-based choices about advertising funding as part of the overall 
recruiting investment in the future. (Page 22/GAO Draft Report):

DOD RESPONSE: Concur.

The DoD Advertising Strategic Framework, with associated outcome 
measures, will allow the Office of the Under Secretary of Defense for 
Personnel and Readiness to monitor advertising results regularly and 
make fact-based decisions at a strategic level for the Department and 
the Joint Advertising, Market Research, and Studies Program. This will 
provide an oversight framework within which each Service will continue 
to develop its own advertising program strategy, program objectives, 
and outcome measures. Each Service will continue to vary the mix of 
support elements to improve effectiveness based on its particular 
strategy, thus also continuing to be responsible for making those 
decisions, and to be accountable for demonstrating program results--
effectiveness and efficiency. Meetings of the Joint Marketing and 
Advertising Committee will continue to serve as a forum for information 
sharing among the Service advertising programs.

[End of section]

Related GAO Products:

Program Evaluation: Strategies for Assessing How Information 
Dissemination Contributes to Agency Goals. GAO-02-923. Washington, 
D.C.: September 30, 2002.

Military Personnel: Services Need to Assess Efforts to Meet Recruiting 
Goals and Cut Attrition. GAO/NSIAD-00-146. Washington, D.C.: June 23, 
2000.

Military Personnel: First-Term Recruiting and Attrition Continue to 
Require Focused Attention. GAO/T-NSIAD-00-102. Washington, D.C.: 
February 24, 2000.

Military Recruiting: DOD Could Improve Its Recruiter Selection and 
Incentive Systems. GAO/NSIAD-98-58. Washington, D.C.: January 30, 1998.

Military Personnel: High Aggregate Personnel Levels Maintained 
Throughout Drawdown. GAO/NSIAD-95-97. Washington, D.C.: June 2, 1995.

Military Recruiting: More Innovative Approaches Needed. GAO/NSIAD-95-
22. Washington, D.C.: December 22, 1994.

Military Downsizing: Balancing Accessions and Losses Is Key to Shaping 
the Future Force. GAO/NSIAD-93-241. Washington, D.C.: September 30, 
1993.

FOOTNOTES

[1] S. Rep. 107-151, at 300 (2002).

[2] Pub. L. No. 103-62, Aug. 3, 1993.

[3] All dollars are in constant fiscal year 2003 dollars unless 
otherwise indicated. 

[4] U.S. General Accounting Office, Military Personnel: Services Need 
to Assess Efforts to Meet Recruiting Goals and Cut Attrition, GAO/
NSIAD-00-146 (Washington, D.C.: June 23, 2000).

[5] J.G. Bachman, L.D. Johnston, and P.M. O'Malley, Monitoring the 
Future: Questionnaire Responses from the Nation's High School Seniors 
(Ann Arbor, Mich.: Institute for Social Research, 2001).

[6] Office of the Under Secretary of Defense for Personnel and 
Readiness, Military Personnel Human Resources Strategic Plan 
(Washington, D.C.: Aug. 2002).

[7] The Navy and Naval Reserve have separate advertising programs; 
however, their recruiting programs were recently reorganized under one 
commander.

[8] These amounts are in fiscal year 2003 constant dollars using DOD's 
Operation and Maintenance funding deflators. In nominal dollars, DOD's 
advertising funding increased from $271 million in fiscal year 1998 to 
$592 million in fiscal year 2003--an increase of 118 percent.

[9] "Quantity" is the total number of recruits needed in a given fiscal 
year. "Quality" is the achievement of high school diplomas and adequate 
scores on the Armed Forces Qualification Test.

[10] Office of the Under Secretary of Defense (Force Management 
Policy), A New Focus for Military Advertising and Market Research 
(Washington, D.C.: March 2000) and Rand Corporation, A Report on the 
Audit of the Armed Services Recruitment Advertising (Santa Monica, 
Calif.: 2002).

[11] In nominal dollars, DOD's total recruiting advertising funding for 
fiscal years 1998-2002, consecutively, was $271 million, $379 million, 
$450 million, $501 million, and $595 million. 

[12] The Blue Angels, the Navy's flight demonstration team, performs at 
air shows and special events. The recruiting advertising budget funds 
the operation and maintenance costs related to the team.

[13] 10 U.S.C. § 503.

[14] DOD Instruction 1304.8, Military Procurement Resources Report, May 
28, 1991. 

[15] DOD Instruction 7730.56, Monthly Report of Personnel Statistics, 
September 15, 1975.

[16] Pub. L. No. 103-62, Aug. 3, 1993.

[17] Office of Management and Budget, Preparation and Submission of 
Strategic Plans, Annual Performance Plans, and Annual Program 
Performance Reports, Circular No. A-11, Part 6, June 2002.

[18] GAO/NSIAD-00-146.

[19] Office of the Assistant Secretary of Defense (Force Management 
Policy), A New Focus for Military Advertising and Market Research 
(March 2000).

[20] National Research Council, Attitudes, Aptitudes, and Aspirations 
of American Youth: Implications for Military Recruitment (Washington, 
D.C.: The National Academies Press, 2003).

[21] DOD Instruction 1304.8, Military Procurement Resources Report, May 
28, 1991; and DOD Instruction 7730.56, Monthly Report of Personnel 
Statistics, September 15, 1975.

[22] The advertising agency contracted by DOD's joint program also 
identified several problems, including (1) having an extremely large 
sample size, (2) voluminous data but poor analysis, and (3) poor 
training for the services in using the data.

[23] John Warner, Curtis Simon, and Deborah Payne, Enlistment Supply in 
the 1990s: A Study of the Navy College Fund and Other Enlistment 
Incentive Programs, Defense Manpower Data Center, report No. 2000-015 
(April 2001), p. 45.

[24] Jim Dertouzos and Steven Garber, Is Military Advertising 
Effective? An Estimation Methodology and Applications to Recruiting in 
the 1980s and 90s (Santa Monica, Calif.: Rand Corporation, 2003).

GAO's Mission:

The General Accounting Office, the investigative arm of Congress, 
exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO's commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony:

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics.

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as "Today's Reports," on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order 
GAO Products" heading.

Order by Mail or Phone:

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to:

U.S. General Accounting Office

441 G Street NW,

Room LM Washington,

D.C. 20548:

To order by Phone: 	

	Voice: (202) 512-6000:

	TDD: (202) 512-2537:

	Fax: (202) 512-6061:

To Report Fraud, Waste, and Abuse in Federal Programs:

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470:

Public Affairs:

Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S.

General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C.

20548: