This is the accessible text file for GAO report number GAO-06-672 
entitled 'U.S. International Broadcasting: Management of Middle East 
Broadcasting Services Could Be Improved' which was released on 
September 5, 2006. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to the Chairman, Subcommittee on National Security, Emerging 
Threats and International Relations, Committee on Government Reform, 
House of Representatives: 

August 2006: 

U.S. International Broadcasting: 

Management of Middle East Broadcasting Services Could Be Improved: 

GAO-06-762: 

GAO Highlights: 

Highlights of GAO-06-762, a report to the Chairman, Subcommittee on 
National Security, Emerging Threats and International Relations, 
Committee on Government Reform, House of Representatives 

Why GAO Did This Study: 

The Broadcasting Board of Governors’ (BBG) broadcasting services, Radio 
Sawa, and the Alhurra satellite television networks—collectively known 
as the Middle East Broadcasting Networks, Inc. (MBN)—currently aim to 
reach Arabic speakers in 19 countries and areas throughout the Middle 
East. Annual spending for current activities amounts to about $78 
million. 

GAO reviewed MBN’s (1) strategic planning to address competition in the 
Middle Eastern media market, (2) implementation of internal control, 
(3) procedures MBN has developed to ensure compliance with its 
journalistic standards, and (4) performance indicators and whether 
targets have been met. 

What GAO Found: 

MBN faces a number of competitive challenges in carrying out its 
mission of broadcasting in the Middle Eastern media market and has 
taken some steps to address them. However, MBN lacks a comprehensive, 
long-term strategic plan. As MBN emerges from its start-up mode and 
faces future challenges, a long-term strategic plan will be important. 

While MBN has developed financial and administrative controls to manage 
and safeguard its financial resources, it could take additional steps 
to strengthen its system of internal control. For example, the MBN has 
not (1) convened a meeting of its internal control board to formally 
develop its controls and coordinate audits, (2) completed an internal 
control plan, (3) completed a risk assessment to address potential 
risks to its operation, or (4) developed a comprehensive training 
program for its staff. 

MBN has procedures in place to help ensure its programming meets its 
journalistic standards. However, MBN lacks regular editorial training 
and has not fully implemented a comprehensive, regular program review 
process to determine whether its programming complies with those 
standards or with MBN’s mission. While the BBG calls for its 
broadcasters to undergo an annual program review, Radio Sawa has only 
held one such review, and Alhurra has not completed one to date. 

The BBG has developed several performance indicators and targets for 
MBN’s Radio Sawa and Alhurra services, including measures of audience 
size and program credibility. However, it is not clear whether the 
Radio Sawa and Alhurra performance targets have been met because of 
weaknesses in MBN’s survey methodology and documentation. The BBG did 
not take certain steps that could have explained and increased the 
reliability of its estimates, such as by fully documenting its research 
and estimation methods, measuring the level of uncertainty surrounding 
its estimates, disclosing significant limitations, and consistently 
implementing policies and procedures for verifying data. 

Figure: Alhurra's Broadcast Center in Springfield, Virginia: 

[See PDF for Image] 

Source: GAO. 

[End of Figure] 

What GAO Recommends: 

GAO is making several recommendations to the chairman of the BBG to 
improve the efficiency and operations of MBN. These include fully 
implementing strategic planning, internal control, and regular program 
reviews, as well as addressing weaknesses in the conduct and reporting 
of its audience research. 

The BBG generally concurred with our recommendations, although it 
disagreed with some of our analysis. 

[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-762]. 

To view the full product, including the scope and methodology, click on 
the link above. For more information, contact Jess T. Ford at (202) 512-
4128 or fordj@gao.gov. 

[End of Section] 

Contents: 

Letter: 

Results in Brief: 

Background: 

Lack of Long-term, Comprehensive Planning Could Limit MBN's Ability to 
Respond to Market Challenges: 

MBN Has Made Progress in Developing Internal Control, but Additional 
Steps Could Improve Its Control Structure: 

MBN Has Developed Editorial Procedures to Support Its Journalistic 
Standards, but Could Improve Monitoring and Some Quality Control 
Measures: 

BBG Has Developed Several Performance Indicators, Although Weaknesses 
in Audience Surveys Call into Question Whether MBN Audience Size and 
Credibility Targets Have Been Met: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendixes: 

Appendix I: Scope and Methodology: 

Appendix II: Radio Sawa and Alhurra Programming: 

Appendix III: Radio Sawa Program Review: 

Appendix IV: CIBAR Standards: 

Appendix V: Comments from the Broadcasting Board of Governors: 

GAO Comments: 

Appendix VI: GAO Contact and Staff Acknowledgments: 

Tables: 

Table 1: Actions taken by MBN to Address Grant Thornton's Internal 
Control Recommendations: 

Table 2: Inputs to Program Quality Scores for BBG Broadcasting 
Entities: 

Table 3: BBG's Standard Performance Indicators and MBN's Status: 

Table 4: BBG's Reported Audience Size and Credibility Estimates for 
Radio Sawa and Alhurra, Fiscal Year 2005: 

Table 5: Radio Sawa's Weekly News and Features Programming: 

Table 6: Alhurra's Weekly Programming Schedule: 

Table 7: Alhurra-Iraq's Weekly Programming Schedule: 

Figures: 

Figure 1: Organization of the BBG: 

Figure 2: Timeline Showing Creation and Key Dates of MBN: 

Figure 3: Organization of MBN: 

Figure 4: Appropriations to Radio Sawa, Alhurra, Alhurra-Iraq, and 
Alhurra Europe, Fiscal Years 2002-2006: 

Figure 5: Map of MBN Target Countries and Locations of Overseas Offices:

Figure 6: Radio Sawa's FM and AM Broadcast Range: 

Abbreviations: 

BBC: British Broadcasting Corporation: 

BBG: Broadcasting Board of Governors: 

CFO: Chief financial officer: 

CIBAR: Conference of International Broadcasters' Audience Research 
Services: 

GPRA: Government Performance and Results Act of 1993: 

MBN: Middle East Broadcasting Networks, Inc. 

MTN: Middle East Television Network: 

OMB: Office of Management and Budget: 

RFA: Radio Free Asia: 

RFE/RL: Radio Free Europe/Radio Liberty: 

UAE: United Arab Emirates: 

VOA: Voice of America: 

August 4, 2006: 

The Honorable Christopher Shays: 
Chairman: 
Subcommittee on International Security, Emerging Threats and 
International Relations: 
Committee on Government Reform: 
House of Representatives: 

Dear Mr. Chairman: 

The President's 2006 National Security Strategy stipulates that public 
diplomacy, which includes U.S. international broadcasting, is an 
important instrument for engaging and informing foreign audiences, 
creating and maintaining a dialogue with Muslim leaders, and 
confronting misinformation about the United States. Under the direction 
of the Broadcasting Board of Governors (BBG), an independent agency, 
the mission of all U.S. international broadcasting services is to 
promote freedom and democracy and enhance understanding through 
objective journalism. The BBG's priority broadcasting services, Radio 
Sawa and the Alhurra satellite television networks[Footnote 1]--both 
operated since 2005 by the nonprofit grantee Middle East Broadcasting 
Networks, Inc. (MBN)--currently aim to reach Arabic speakers in 19 
countries and areas throughout the Middle East. Spending for MBN's 
broadcasting services has increased in recent years and currently 
amounts to about $78 million annually. In fiscal year 2005, MBN 
received additional funding to support a new Alhurra satellite 
television initiative to reach Arabic speakers in Europe. MBN has 
recently requested funding to enhance its radio and television 
programming content. 

This report assesses (1) the extent to which MBN has used strategic 
planning to address its competitive challenges in the Middle Eastern 
media market, (2) the progress MBN has made in developing financial and 
administrative controls to ensure that the organization operates 
efficiently and complies with its grant agreement, (3) the procedures 
MBN has developed to ensure compliance with its journalistic standards, 
and (4) the extent to which BBG has developed performance indicators 
for MBN's efforts and met its primary performance targets. 

To accomplish our objectives, we reviewed documentation and spoke with 
officials from the BBG and its broadcasting entities, including the 
grantees (MBN, Radio Free Europe/Radio Liberty (RFE/RL), Radio Free 
Asia (RFA)), the Voice of America (VOA), and the International 
Broadcasting Bureau. We also spoke with several outside experts, 
including representatives from the Department of State, National Public 
Radio, the InterMedia research organization, Zogby International, and 
foreign international broadcasters such as the British Broadcasting 
Corporation (BBC) and Deutsche Welle. In addition, we visited MBN's 
offices in Amman, Jordan, and Dubai, United Arab Emirates (UAE). We 
held telephone interviews with MBN's current affairs contractor in 
Beirut, Lebanon, and corresponded with the head of the MBN office in 
Baghdad, Iraq. We conducted our evaluation from August 2005 through May 
2006 in accordance with generally accepted government auditing 
standards. (Appendix I provides a more detailed description of our 
scope and methodology.) 

Results in Brief: 

MBN faces a number of challenges in carrying out its mission of 
broadcasting in the Middle Eastern media market and has taken some 
steps to address them. Although MBN has participated in BBG's long-term 
planning exercises, it has not yet established a formal long-term 
strategic plan. A formal documented strategic plan could help MBN meet 
challenges in its competitive marketplace. MBN's challenges include the 
fact that its Alhurra television station is currently competing in a 
market with over 140 other stations and also faces operational and 
programmatic competitive disparities, such as having fewer overseas 
bureaus than its primary competitors. In addition, its Radio Sawa 
service lacks FM radio coverage in certain markets. MBN has begun a 
number of efforts and proposed initiatives to address some of its 
challenges, such as proposals for Alhurra to increase its hours of news 
coverage and current affairs programming and for Radio Sawa to increase 
the amount of local content it broadcasts to better compete with local 
stations. However, MBN lacks a shared vision of operations for Radio 
Sawa and Alhurra as well as detailed implementation strategies and 
related resource requirements for its proposed goals. According to GAO 
guidance,[Footnote 2] organizations should make decisions in the 
context of a strategic plan. The absence of a long-term plan and 
comprehensive strategic approach for MBN's operations limits 
opportunities for strategically addressing impediments to the 
organization's achieving its goals and objectives and identifying 
opportunities for more efficient operations. 

MBN has developed several financial and administrative controls to 
manage and safeguard its financial resources. For example, in the past 
year, MBN has hired additional financial staff, acquired a new 
financial information system, developed administrative and financial 
policies and procedures, provided some training to staff, and received 
an unqualified opinion on its fiscal year 2005 financial statements. 
However, additional steps could be taken to strengthen MBN's system of 
internal control,[Footnote 3] including (1) establishing an internal 
control board to formally develop controls and coordinate audits, (2) 
preparing an internal control plan, (3) conducting a risk assessment to 
address potential risks to its operation, and (4) developing a 
comprehensive training program for its staff. These steps could help 
MBN more effectively manage its operations and improve its internal 
control structure. 

MBN has procedures in place to help ensure its programming meets its 
journalistic standards and is consistently accurate, objective, 
balanced, and comprehensive. However, MBN has not fully implemented 
some editorial procedures, or a comprehensive, standardized program 
review process of its broadcast operations. To comply with its 
standards, MBN has put in place a variety of pre-and postbroadcast 
procedures, such as editorial meetings and postbroadcast discussions 
that enable producers and editors to review and provide input to Radio 
Sawa and Alhurra broadcasts on a daily basis. However, MBN could more 
fully develop some quality control measures, such as using listener and 
viewer feedback to improve program quality. In addition, although the 
BBG calls for its broadcasting services to undergo a standardized 
annual program review, MBN's Radio Sawa has had only one program 
review, and Alhurra has not had a program review to date. Furthermore, 
unlike reviews of other broadcasting services and contrary to BBG 
guidance, the Radio Sawa program review was less comprehensive in that 
it relied upon several audience panels but did not include input from 
in-house analysts or external expert listeners. The lack of annual, 
comprehensive program reviews hampers BBG's ability to assure its 
audience, Congress, and the BBG that its services are complying with 
its journalistic standards and mission. 

The BBG has developed performance indicators and targets[Footnote 4] 
related to audience size and program credibility for MBN's Radio Sawa 
and Alhurra broadcasting services. However, it is not clear whether the 
Radio Sawa and Alhurra performance targets have actually been met 
because of weaknesses in MBN's methodology and documentation. To 
develop estimates for audience size and credibility, the BBG uses 
results from audience research conducted overseas. Documentation we 
reviewed indicated that the BBG extensively uses nonprobability 
audience survey results that cannot be reliably projected to represent 
a broader population in the region. While it is not always possible to 
expect the BBG to use probability samples in countries where there are 
security risks, the BBG did not take certain steps that could have 
explained and increased the reliability of its estimates, such as fully 
documenting its research methods, measuring the level of uncertainty 
surrounding its estimates, disclosing significant limitations, limiting 
the scope of its projections to areas actually covered by its surveys, 
and developing and consistently implementing policies and procedures 
for verifying data. 

We are making several recommendations to the chairman of the BBG. 
First, we recommend that the BBG take steps to improve the efficiency 
of MBN and its ability to address challenges by ensuring that MBN 
develops a comprehensive long-term strategic plan. In addition, to 
improve controls and performance monitoring, we recommend that the BBG 
require MBN to implement additional elements of internal control, 
establish a regular mechanism for undertaking annual program reviews, 
implement performance indicators consistent with other BBG entities, 
and improve the transparency and documentation used for audience 
research to allow for assessments of confidence in data used for 
performance indicators. 

We received written comments on a draft of this report from the BBG. In 
general, the BBG concurred with our recommendations and said that MBN 
has made significant progress in the 2 years of its operation in 
establishing a sound journalistic organization with financial and 
administrative controls. However, the BBG raised a number of concerns 
about the report's criticisms of the audience research conducted by the 
BBG and its contractors. 

Background: 

Middle East Broadcasting Networks, Inc. (MBN), which includes the 
services Radio Sawa and Alhurra, is an independent nonprofit grantee 
overseen by the Broadcasting Board of Governors (BBG). The BBG is an 
independent, federal agency responsible for overseeing all U.S. 
government-sponsored, nonmilitary, international broadcasting programs. 
The BBG also manages the operations of the International Broadcasting 
Bureau, the Voice of America (VOA), and the Office of Cuba 
Broadcasting, which are all federal entities. It also provides funding 
and oversight to three independent grantees, MBN, Radio Free Europe/ 
Radio Liberty (RFE/RL), and Radio Free Asia (RFA) (see figure 1). 

Figure 1: Organization of the BBG: 

[See PDF for image] 

Source: BBG. 

[End of figure] 

Development of MBN: 

In March 2002, due to concerns about the effectiveness of its outreach 
to Arabic speakers in the Middle East, the BBG replaced VOA's Arabic 
radio service with Radio Sawa, a 24-hour, 7-day a week, Arabic language 
radio station. In April 2003, Congress, at the request of the 
administration, provided $26 million in the fiscal year 2003 Emergency 
Wartime Supplemental Appropriations Act[Footnote 5] to establish a 
nonprofit corporation, the Middle East Television Network (MTN), as a 
grantee of the BBG to launch and operate Alhurra TV. Ten months later, 
on February 14, 2004, Alhurra, MTN's Arabic-language satellite 
television station, was launched and initially started broadcasting 14 
hours a day, expanding to 24 hours a day 2 months later. In November 
2003, the fiscal year 2004 Emergency Supplemental Act for Defense and 
for Reconstruction of Iraq and Afghanistan[Footnote 6] included $40 
million to establish a second 24-hour channel, Alhurra-Iraq. On April 
27, 2004, Alhurra-Iraq was launched. In 2005, MTN was renamed the 
Middle East Broadcasting Networks, Inc. (MBN), and Radio Sawa 
transferred to MBN. See figure 2 for a timeline of these events and 
figure 3 for details on the current organization of MBN. 

Figure 2: Timeline Showing Creation and Key Dates of MBN: 

[See PDF for image] 

Source: GAO analysis of MBN and BBG data. 

[End of figure] 

Figure 3: Organization of MBN: 

[See PDF for image] 

Source: MBN. 

[End of figure] 

MBN Funding: 

Congress appropriated more than $274 million to fund Radio Sawa, 
Alhurra, and Alhurra-Iraq from fiscal year 2002 through fiscal year 
2006.[Footnote 7] Each year MBN's funding level has increased to 
support additional 24-hour television streams and Radio Sawa's 24-hour 
radio programming. MBN's grant in fiscal year 2006 is $78.7 million to 
support Alhurra, Alhurra-Iraq, and Radio Sawa, as well as the launch of 
Alhurra-Europe. See figure 4 for a breakdown of funds by fiscal year. 

Figure 4: Appropriations to Radio Sawa, Alhurra, Alhurra-Iraq, and 
Alhurra Europe, Fiscal Years 2002-2006: 

[See PDF for image] 

Source: MBN and BBG. 

[End of figure] 

MBN's Mission and Scope of Operations: 

The Alhurra networks and Radio Sawa are BBG's priority broadcasting 
services designed to support the BBG's antiterror broadcasting 
initiatives in the Middle East and counter media campaigns used by 
terrorists by providing accurate reporting and analysis of the news and 
by explaining U.S. policies. Although MBN and its Alhurra broadcasting 
services postdate the BBG's current 2002-2007 strategic plan, Radio 
Sawa, in particular, was singled out as an opportunity in the plan to 
target Arabic-speaking youth and provide them with news that is 
objective, comprehensive, fresh, and relevant and to provide a forum 
for reasoned discussion of "hot button" issues and U.S. policies. MBN's 
current mission statement is to broadcast factual, timely, and relevant 
news and information about the Middle East, the United States, and the 
world to people of all ages in order to advance the long-term U.S. 
interests of promoting freedom and democracy and enhancing 
understanding in the Middle East. Radio Sawa and Alhurra aim to be 
among the sources that audiences turn to in the Middle East for news 
and information, to increase the standards of other broadcasters in the 
region, and to offer distinctive and provocative programming 
unavailable on other stations. 

MBN's target audience includes 19 Arabic-speaking countries and 
territories[Footnote 8] in North Africa, the Near East, and the Gulf 
region, which are home to approximately 250 million people. In addition 
to its headquarters in Springfield, Virginia, MBN has several overseas 
offices, including a production center in Dubai that broadcasts Radio 
Sawa's live newscasts during 8 hours each day and produces some opinion 
features for Radio Sawa and current affairs programming for Alhurra 
(see figure 5).[Footnote 9] 

Figure 5: Map of MBN Target Countries and Locations of Overseas 
Offices: 

[See PDF for image] 

Source: MBN and MapArt. 

[End of figure] 

Radio Sawa's Operations: 

Radio Sawa's broadcasts are designed to reach a target audience of 15- 
to 29-year olds in the Middle East with Western and Arab popular music, 
news broadcasts, and specialized programming. Radio Sawa broadcasts 24 
hours of programming every day through a combination of FM, medium wave 
(AM), digital audio satellite, and Internet transmission resources. See 
figure 6 for a map of Radio Sawa's regional reach. 

Figure 6: Radio Sawa's FM and AM Broadcast Range: 

[See PDF for image] 

Source: GAO(data); Map Resources (map).  

[End of figure]  

Radio Sawa has developed seven distinct programming streams, including 
(1) Iraq, (2) Jordan and the West Bank, (3) the Gulf, (4) Egypt and the 
Levant, (5) Morocco, (6) Sudan and Yemen, and (7) Lebanon. All of the 
streams generally feature the same major newscasts, current affairs, 
and policy features; however, the Iraq program differs slightly, and 
the streams all offer differentiated music programs.[Footnote 10] Radio 
Sawa's streams broadcast between 31 and 35 hours of news each week. 
(See appendix II for more on Radio Sawa's programming.) 

Alhurra's Operations: 

The Alhurra satellite television station is designed to reach a broad 
audience in the Middle East by providing news, current affairs, and 
entertainment programming 24 hours a day, 7 days a week. Alhurra-Iraq 
is designed to provide Iraqi citizens with daily newscasts and talk 
shows that specifically address issues in Iraq. Both Alhurra networks 
broadcast between 36 and 43 hours of news and news updates a 
week.[Footnote 11] Alhurra broadcasts on the Arabsat and Nilesat 
satellites, which currently allow it to cover the entire Middle East. 
Alhurra-Iraq also broadcasts through these satellites and a combination 
of terrestrial transmitters made available in Iraq.[Footnote 12] 
Alhurra also expects to start broadcasting to Europe on August 1, 
2006.[Footnote 13] (See appendix II for more information on Alhurra and 
Alhurra-Iraq programming.) 

Lack of Long-term, Comprehensive Planning Could Limit MBN's Ability to 
Respond to Market Challenges: 

MBN faces a variety of challenges to broadcasting in the Middle East, 
including operating in a competitive satellite television broadcast 
market, operational and programming competitive disparities, and lack 
of coverage for Radio Sawa in certain FM markets. MBN has conducted 
some planning efforts and, by using market research and internal 
assessments of its competitors, has undertaken or proposed some 
initiatives to address many of these challenges, such as increasing its 
hours of news coverage and current affairs programming for Alhurra and 
increasing the amount of local content Radio Sawa broadcasts. However, 
MBN has not developed a long-term strategic plan that fully addresses 
its operational and competitive challenges. 

Radio Sawa and Alhurra Face Significant Competitive Challenges: 

MBN faces several significant competitive challenges. These include the 
competitive Middle East satellite television market; operational and 
programming competitive disparities, such as Alhurra's lack of news 
bureaus compared with its competitors; and lack of coverage in certain 
FM radio markets. 

MBN operates in the competitive Middle Eastern satellite television 
market that has over 140 channels. Pan-Arab satellite television 
stations-in particular, the news stations Al Jazeera and Al Arabiya-- 
are currently the primary competitors to Alhurra. According to the 
BBG's research firm Intermedia, Al Jazeera is currently the top 
international broadcaster as a source of news and information for 
audiences in many countries throughout the Middle East. Moreover, 
Alhurra will face new competition from the BBC's entry to the Middle 
Eastern satellite television market in 2007. BBC officials have 
indicated that the new station's overall approach in the region will be 
multimedia in focus, taking advantage of the BBC's more than 60 years 
experience of broadcasting on the radio to the region as well as its 
award-winning Arabic-language news Web site. For Radio Sawa, the 
primary competitive challenge comes from existing local radio stations 
in its broadcast range and the BBC World Service in Arabic, as well as 
from the generally increasing competitiveness of the Middle East radio 
market. 

Alhurra also faces operational and programmatic competitive 
disparities,[Footnote 14] since both Al Jazeera and Al Arabiya are 
estimated to receive significant, although unknown, levels of funding 
from their respective supporting Qatari and Saudi financiers--allowing 
them to develop large networks of correspondents and bureaus throughout 
the Middle East and other parts of the globe. Al Jazeera, in 
particular, has bureaus in over 30 locations across 6 continents, which 
enable it to respond to breaking news events on a timely basis. Alhurra 
and Radio Sawa, by comparison, only have overseas bureaus in Baghdad, 
Dubai, and Amman. In addition, the BBC has a large network of 
correspondents and bureaus around the globe and, unlike Alhurra, has a 
vast in-house library of desirable BBC-produced content, including 
documentaries and current affairs programming, which can be readily 
translated into Arabic. The BBC also has favorable licensing and co- 
production arrangements with many companies. 

One of Radio Sawa's other primary challenges is its lack of broadcast 
coverage in certain countries in the Middle East region. For example, 
Radio Sawa does not have any broadcasting coverage in Tunisia, Libya, 
and Algeria. Moreover, it has faced difficulties expanding its 
transmission to include FM coverage in some countries, such as Egypt, 
Saudi Arabia, Syria, Yemen, and Oman. MBN is attempting to negotiate 
transmission agreements with several of these countries, but still 
faces significant challenges to finalizing agreements. 

MBN Has Taken Some Steps to Address Its Competitive Challenges: 

Since its inception, MBN has conducted some planning exercises to 
address its competitive challenges. These have included developing a 
"2006 Goals and Strategies" overview document to guide operations for 
the current fiscal year, establishing a 2006 annual performance plan as 
part of the Office of Management and Budget's Program Assessment Rating 
Tool process, and participating in the development of the BBG's new 
long-term strategic plan covering fiscal years 2008-2012. In addition, 
MBN conducts ongoing assessments of its competitors and uses various 
types of market research to gain information about its audience and 
media usage patterns in the Middle East. It has used this information 
to make adjustments to its programming within its current budget, and 
also to develop proposals for obtaining additional funding for new 
efforts. BBG and MBN officials have explained that they use audience 
surveys, audience monitoring panels, focus groups, in-depth interviews, 
and Arab television and music station monitoring to inform MBN's 
current efforts and planning, whenever possible. 

For example: 

* As a result of market research performed in August 2004, MBN 
officials identified television viewing patterns and made changes to 
Alhurra program schedules, such as by offering programming appealing to 
women (e.g., current affairs and health and fitness programming) during 
the daytime. 

* Through a review of the current competition in the market, MBN 
officials decided that it was important to increase the number of 
debate programs they broadcast on Alhurra. As a result, MBN created a 
series of "town hall meetings" that allowed journalists and experts to 
discuss issues of regional interest with interaction from a live 
audience. 

* As a result of audience monitoring panels, MBN officials made changes 
to Radio Sawa's program schedule by adding new features on subjects 
such as social and cultural issues. MBN officials also learned of the 
importance of efforts to localize those features, and made changes to 
tailor programs to the interests of audiences of the various Radio Sawa 
streams. 

MBN has also developed program enhancement proposals for Alhurra and 
Radio Sawa as part of the BBG's language review and budget request 
processes. Several proposals were included in the President's fiscal 
year 2007 budget request. For example, the budget request includes a 
proposal to increase Alhurra's newsroom hours to increase on-the-spot 
and breaking news coverage.[Footnote 15] In addition, in fiscal year 
2005, the administration requested additional funds for providing 
satellite Alhurra broadcasts to Europe. According to planning 
documents, many of the Alhurra proposals were designed to reinforce one 
another with the goal of improving Alhurra's credibility, as well as 
building audience size and increasing viewing time of those who already 
tune in.[Footnote 16] MBN also developed one proposal for enhancing 
Radio Sawa's operations that calls for increasing the amount of 
localized news content offered on five regional streams,[Footnote 17] 
which officials say would allow the station to more effectively compete 
with local stations in its broadcast range. MBN's president said that, 
given the increasing level of competitiveness in radio broadcasting in 
the Middle East and expressed audience interest in news about their 
home country, creating more localized content on Radio Sawa streams is 
important. 

MBN Has Conducted Planning Exercises but Lacks a Long-term Strategic 
Plan: 

Strategic planning is a good management practice for all organizations. 
Although MBN has conducted some planning exercises, it lacks a long- 
term strategic plan and a strategic approach that outlines (1) a shared 
vision of operations for Alhurra and Radio Sawa, (2) detailed 
implementation strategies to achieve measurable outcomes for its goals, 
and (3) the competitive challenges it faces and how it plans to address 
its key challenges to broadcasting in the Middle East. 

Strategic planning, including the development of a strategic plan, is a 
good management practice for all organizations. Additionally, risk 
assessment is an integral part of strategic planning. According to GAO 
guidance,[Footnote 18] organizations should make management decisions 
in the context of a strategic plan, with clearly articulated goals and 
objectives that identify resource issues and internal and external 
threats, or challenges, that could impede the organization from 
efficiently and effectively accomplishing its objectives. Additionally, 
Office of Management and Budget (OMB) guidance suggests that strategic 
plans contain, among other things, a statement of the organization's 
long-term goals and objectives; define approaches or strategies to 
achieve goals and objectives; and identify the various resources needed 
and the key factors, risks, or challenges that could significantly 
affect the achievement of the strategic goals. 

MBN has yet to create its own long-term strategic plan. MBN's president 
stated that funding uncertainties and other more pressing 
organizational needs--such as the development of financial and 
administrative policies and procedures--have delayed the development of 
MBN's strategic plan and related planning policies. In addition, he 
commented that MBN did not emphasize planning in its early stages 
because it was focusing on making its networks broadcast-ready. BBG 
officials said another reason for the delays in planning is that the 
BBG and MBN are still learning about the market, especially for 
Alhurra, and are taking a close look at the results of audience 
surveys, focus groups, and in-depth interviews to determine the best 
direction for these initiatives. MBN has stated that, to date, it has 
primarily used the BBG strategic plan for organizational guidance. 
Nevertheless, BBG officials said the BBG also has the expectation that 
all broadcasting entities will develop their own strategic plans, 
particularly to guide funding decisions. 

In the absence of a strategic plan of its own, MBN lacks a 
comprehensive, strategic approach that fully outlines (1) a shared 
vision of operations for Alhurra and Radio Sawa, (2) detailed 
implementation strategies to achieve measurable outcomes related to its 
goals, and (3) the competitive challenges it faces and how it plans to 
address them. 

First, MBN does not have a comprehensive strategic vision for the 
integration of Radio Sawa and Alhurra operations in the organization. 
For example, although MBN's most recent annual performance plan 
contains a goal to "integrate news operations for more effective 
television and radio news-gathering," none of MBN's current plans 
outline specific, shared objectives for Radio Sawa and Alhurra. MBN 
officials told us that several steps toward integration of Alhurra and 
Sawa have occurred to date, such as sharing financial and 
administrative support staff. However, Radio Sawa and overseas bureau 
staff we talked with said that cooperation between Alhurra and Radio 
Sawa is limited, the identities of the stations are separate, and the 
two stations work largely independently of one another. Radio Sawa 
staff noted several areas for further increasing cooperation, including 
more sharing of interviews, sound bites, field correspondents, Web site 
stories, and copy editors. One staff member in the Baghdad bureau said 
that Radio Sawa and Alhurra operations in Iraq are completely 
independent, including separate offices, and only the financial 
activities of both offices are supervised by the same person. In 
addition to gaining more efficiency in operations, a vision for further 
integration of Radio Sawa and Alhurra may help MBN more effectively 
identify opportunities to address its challenges from increasing 
competition in the Middle East. 

Second, MBN has not yet developed detailed implementation and resource 
strategies needed for successful implementation of its goals. For 
example, with regard to MBN's initiative to localize content on Radio 
Sawa, we were not able to identify a plan directing what types of local 
news and features will be considered on the various streams, to what 
degree existing program schedules might be affected, and how required 
resources might be divided among the various streams. Additionally, 
although MBN's most recent annual performance plan states a goal of 
"expanding overseas production of news coverage for radio and 
television," neither that document nor any other plan MBN identified 
provides details or direction for the overseas production of news for 
Alhurra in existing overseas offices. MBN officials have stated that 
uncertainties in the future commitment of resources to Alhurra have 
affected MBN's ability to, for example, plan for and use existing 
overseas offices for Alhurra news. Further, BBG officials have said MBN 
is still learning about the Middle Eastern media market. However, given 
MBN's internal enhancement requests to the BBG to increase the number 
of news bureaus in the region, among others, it should clarify, for 
example, what implementation steps are necessary to maximize the use of 
existing overseas offices.[Footnote 19] 

Third, MBN has not yet comprehensively outlined its challenges or 
developed a strategic approach for how it plans to address its key 
challenges to broadcasting in the Middle East. While MBN is planning to 
expand its broadcast operations into Europe, it has not clearly 
identified how its broadcasts will meet competitive challenges in the 
Middle East. As an example, MBN has not indicated how it will address 
the implications of the upcoming BBC Arabic-language television 
initiative. The BBC could gain a significant audience that potentially 
would interfere with Alhurra's market share, credibility, and use as a 
source of alternative information.[Footnote 20] 

MBN Has Made Progress in Developing Internal Control, but Additional 
Steps Could Improve Its Control Structure: 

MBN was initially limited in developing its internal control structure 
because it was focused on quickly starting up its broadcasting 
operations. In response to an external review of its financial 
operations by Grant Thornton LLP in May 2004, MBN strengthened several 
of its controls, after which it received an unqualified opinion on its 
Fiscal Year 2005 Single Audit.[Footnote 21] However, MBN has not fully 
implemented several of the Grant Thornton review's key recommendations 
related to its control environment,[Footnote 22] including (1) 
establishing an internal control board to formally develop its controls 
and coordinate audits, (2) preparing an internal control plan, (3) 
conducting a risk assessment to address potential risks to its 
operation, and (4) developing a training program for its staff. 

MBN Faced Initial Challenges in Developing Its Internal Control 
Structure: 

Internal control refers to the policies and procedures that help ensure 
the proper management and application of an organization's assets. 
Clear, strong controls can provide some assurance that management 
problems are unlikely to occur or will be addressed if they do occur. 
MBN's internal control is governed by several OMB circulars cited in 
its grant agreement.[Footnote 23] The Comptroller General's Standards 
for Internal Control in the Federal Government[Footnote 24] also 
provides guidance that is available to MBN. 

MBN faced some initial challenges in establishing its internal control 
structure. According to MBN documents and officials, MBN management 
initially focused on establishing broadcasting operations rather than 
the development of internal control policies and procedures, because 
the organization had only several months to plan the launch of its 24- 
hour a day Alhurra television network. As a result, MBN's internal 
control lagged behind. The MBN chief financial officer (CFO) told us 
that problems encountered by MBN in hiring and retaining staff added to 
the delay in developing internal controls. 

Due to concern over the slow development of MBN's internal control 
structure, the BBG commissioned a review by Grant Thornton LLP 
accountants and management consultants, which was completed in spring 
2004, to assess MBN's internal control and make recommendations for 
improvement. The report by Grant Thornton LLP cited numerous findings, 
such as inadequate financial policies and procedures, understaffing, 
and inadequate training, that impeded MBN's ability to successfully 
mitigate risks.[Footnote 25] The report, however, also noted that MBN's 
controls were improving. 

MBN Has Made Progress in Developing Its System of Internal Control, but 
Some Control Elements Could Be Improved: 

Grant Thornton's May 2004 report on MBN's system of internal control 
made recommendations that covered staffing, MBN's financial system, 
training, administrative policies and procedures, developing a decision 
support structure, improving logs and records, and MBN's control 
environment. MBN accepted the review's recommendations and agreed to 
implement them, according to MBN officials, and the BBG concurred with 
these recommendations. Our analysis shows that MBN subsequently 
responded to many of the recommendations, including hiring additional 
financial department staff, developing financial and administrative 
policies and procedures, and completing an annual single audit (see 
table 1). For example, although MBN provided us with copies of its 
Fiscal Year 2003 and Fiscal Year 2004 Single Audits well past the 
deadlines for completing those audits,[Footnote 26] MBN's Fiscal Year 
2005 Single Audit was completed on time, provided an unqualified 
opinion, and showed marked improvement over previous years.[Footnote 
27] 

Table 1: Actions taken by MBN to Address Grant Thornton's Internal 
Control Recommendations: 

Recommendation: Hire additional staff for the financial department; 
Status of implementation: Completed: New vice president for finance and 
administration hired in August 2004. According to MBN, the organization 
now has 16 financial staff, compared to the 6 who were working there 
when the assessment was conducted. 

Recommendation: Acquire suitable financial information system; 
Status of implementation: Completed: MBN acquired a new enterprise 
resource planning system (Deltek-Costpoint) in October 2004 that has 
enhanced its ability to manage its finance department. 

Recommendation: Enhance and expand MBN's financial and administrative 
policies and procedures; Status of implementation: Completed: Developed 
policies and procedures that lay out roles and responsibilities. 

Recommendation: Improve procurement and travel logs and records; 
Status of implementation: Completed: MBN's CFO told us that MBN 
improved its procurement logs and records. In addition, MBN has hired a 
travel analyst who keeps a monthly log of travel orders and vouchers. 

Recommendation: Develop a system for providing timely and focused 
financial information to senior management; 
Status of implementation: Completed: The finance department provides 
MBN high-level management with a detailed financial package on a 
monthly basis.[.]. 

Recommendation: Undertake and complete an annual audit as required by 
OMB Circular No. A-133; 
Status of implementation: Completed: Completed in late May 2006. MBN's 
Fiscal Year 2003 and Fiscal Year 2004 Single Audits under Circular No. 
A-133 were not released within established deadlines. MBN's Fiscal Year 
2005 Single Audit was completed on time. 

Recommendation: Establish an internal control board; 
Status of implementation: Partially Addressed: MBN's internal control 
board is not yet operational. 

Recommendation: Prepare an internal control plan; 
Status of implementation: Not Completed: Not done as of May 2006. 

Recommendation: Conduct an MBN-wide internal control risk assessment; 
Status of implementation: Partially Addressed: Preliminary risk 
assessment did not identify objectives, identify risks, analyze the 
possible effects of the risks, or propose a strategy to mitigate them. 
This document merely contains a list of completed actions addressing 
the issues raised in the Grant Thornton LLP report. 

Recommendation: Provide training on MBN's internal control processes. 
Manage training by establishing an internal control board; 
Status of implementation: Partially Addressed: MBN has provided some 
training on internal control, but has not yet developed a regular 
training schedule and training is not managed by an internal control 
board. 

Source: GAO analysis of information provided by Grant Thornton LLP and 
MBN. 

[End of table] 

Some control elements could be improved in order to better implement 
best management practices based on OMB circulars and GAO internal 
control standards. For example, although MBN is establishing an 
internal control board, the board has not met to establish protocols 
and outline its responsibilities. The organization also has not 
developed an internal control plan. Furthermore, MBN has not 
established a comprehensive process to analyze risks the organization 
faces from internal and external sources. Finally, MBN has provided 
some training on internal control but has not yet developed a regular 
structured training program for its staff.[Footnote 28]  

MBN's Internal Control Board Has Not Met to Establish Protocols to 
Develop and Monitor Its Internal Control Structure: 

The Grant Thornton review recommended that MBN establish an internal 
control board of key managers and officers to determine the internal 
control risks facing MBN, work towards decreasing these risks, and 
oversee MBN's efforts to employ strong controls. Moreover, according to 
GAO guidance,[Footnote 29] organizations should have an audit committee 
or senior management council-similar to an internal control board-that 
reviews the internal audit work and coordinates closely with external 
auditors. 

According to the MBN president, MBN is establishing an internal control 
board consisting of three members, including, as recommended, the MBN 
president, general counsel, and CFO. However, the board has not yet 
formally met to establish protocols and outline responsibilities. In 
addition, according to MBN's general counsel, MBN has not appointed a 
member from the BBG, as was recommended, to serve on the board. MBN's 
executive committee, which examines issues affecting MBN and reports 
back to the BBG, has provided some support on management and 
administrative issues, such as approving the construction and expansion 
of MBN's new facility and providing guidance on hiring high-level 
staff.[Footnote 30] However, the committee has not fulfilled the role 
of an internal control board as previously described. Instead, MBN's 
CFO has largely taken on the sole responsibility of establishing and 
overseeing MBN's controls, reviewing audits, and coordinating with 
external auditors. The MBN president told us that they have not 
convened an internal control board because the organization is too new 
and, therefore, is focused on developing policies and procedures rather 
than mechanisms to review them. MBN is planning to develop an internal 
audit function, implemented by an external firm, to provide assurance 
to MBN management that the organization is operating appropriately. 

MBN Has Not Completed an Internal Control Plan: 

According to the Grant Thornton review, MBN should develop an internal 
control plan to ensure that effective controls are established and 
monitored regularly. Such a plan should identify the roles and 
responsibilities of all individuals whose work affects internal 
control, lay out specific control areas, cover risk assessment and 
mitigation planning, and include monitoring and remediation procedures. 
MBN officials told us in January 2006 that they were in the process of 
developing such a plan and have developed internal control guidelines, 
but as of the end of May 2006, they had not provided us with a 
finalized plan. 

MBN Has Not Established a Comprehensive Process to Analyze Risk: 

The Grant Thornton review called on MBN to conduct a broad risk 
assessment, led by its internal control board, to evaluate and mitigate 
potential obstacles to efficiently and effectively achieving its 
operational objectives. According to Grant Thornton LLP, failure to 
conduct an MBN-wide risk assessment could result in the loss of 
resources and could decrease confidence of the grantor and of Congress, 
which could ultimately compromise MBN's achievement of its mission. 
According to GAO guidance, risks should be identified during both short-
and long-term forecasting and as part of strategic planning. Moreover, 
after conducting a risk assessment, organizations need to develop 
internal control activities to manage or mitigate the risks that have 
been identified. For example, Radio Free Asia identified avian 
influenza and signal jamming from China as two threats to their 
operations. As part of their risk assessment, they considered how to 
address and overcome these issues, such as by broadcasting from 
alternate locations. 

In February 2005, MBN prepared an initial risk assessment that 
identified a list of actions taken to address the issues raised in the 
Grant Thornton LLP report. However, the document did not identify the 
organization's objectives or the risks it faces, nor did it analyze the 
possible effects of the risks or propose a strategy to mitigate them, 
as recommended by GAO guidance. MBN officials told us that they assess 
risk on an ongoing, biweekly basis. However, in taking a short-term 
approach to analyzing risk, MBN lacks a comprehensive basis from which 
to establish a strong internal control structure. Some risks identified 
by MBN include threats to the security of its staff and bureaus in the 
field, particularly in Iraq, and the risk of a terrorist attack on its 
facilities in the United States. There are some risks that MBN has not 
identified or addressed. For example, PriceWaterhouseCoopers, MBN's 
external auditor, noted that MBN has not adequately addressed its risks 
related to information security. Doing so would reduce the risk of 
security incidents and unauthorized system activity, according to the 
auditor. PriceWaterhouseCoopers also found that MBN's lack of a 
business continuity plan or an adequate disaster recovery plan could 
result in slower recovery in the case of such an event, as well as 
significant loss of revenue, inability to meet customer needs and third 
party obligations, and potential noncompliance with legal requirements. 
The internal audit function planned by MBN may at some point take on 
the function of assessing risk, but this body is not yet operational. 

MBN Has Not Developed a Comprehensive Training Program on Internal 
Controls: 

MBN has provided some training on internal controls but has not yet 
developed a regular structured training program for its staff of about 
240, as recommended by the Grant Thornton LLP review and by GAO leading 
practices.[Footnote 31] MBN's CFO and controller attended a seminar on 
grants management in October 2004 and subsequently shared the 
information with the 16 in-house financial staff. MBN also provided 
internal control compliance training to its managers in December 2004, 
and the organization regularly provides training to staff at its 
business manager meetings. However, other U.S. broadcasting entities, 
such as Radio Free Europe/Radio Liberty (RFE/RL) and Radio Free Asia 
(RFA), have more organized, ongoing training programs on internal 
control. The MBN CFO concurred that there is a great need for internal 
training. According to him, the underdeveloped training situation is 
due to a lack of resources, including a lack of funds specifically 
designated for training and limited time to plan or implement training. 
This lack of structured, recurrent internal control training can cause 
problems if staff are unfamiliar with an organization's business 
processes and controls, and can lead to the inefficient or improper use 
of resources. 

MBN Has Developed Editorial Procedures to Support Its Journalistic 
Standards, but Could Improve Monitoring and Some Quality Control 
Measures: 

MBN has established journalistic standards, as well as procedures to 
help ensure that the organization's broadcasts comply with these 
standards. However, it has not fully developed some quality control 
measures, such as the use of listener and viewer feedback. 
Additionally, the BBG has not held regular comprehensive program 
reviews for MBN, thereby making it difficult for MBN to assure its 
audience, Congress, the administration, and the BBG that its controls 
are working and that it is broadcasting quality programming. 

MBN Has Developed Journalistic Standards and Implemented a Wide Variety 
of Pre-and Postbroadcast Editorial Procedures: 

The International Broadcasting Act of 1994[Footnote 32] calls for U.S. 
international broadcasting to be conducted in accordance with the 
highest professional standards of broadcast journalism, including the 
production of news that is consistently reliable, authoritative, 
accurate, and objective. The act also calls for U.S. international 
broadcasting to present a balanced and comprehensive projection of U.S. 
thoughts and institutions, as well as clear and effective presentation 
of U.S. government policies and responsible discussion of those 
policies. MBN's mission statement, which partly draws upon the 
principles and standards contained in the U.S. International 
Broadcasting Act of 1994, calls for MBN to broadcast factual, timely, 
and relevant news and information that promotes freedom and democracy. 

MBN has developed journalistic standards, including a code of ethics, 
as part of its effort to ensure that its news broadcasts are 
consistently accurate, authoritative, objective, balanced, and 
comprehensive. MBN's general counsel said that the code was also 
established to ensure that MBN fully complies with its mission and the 
U.S. International Broadcasting Act of 1994. MBN's journalistic 
standards were drafted by MBN management using input from professional 
journalistic organizations and another grantee. According to BBG 
officials, MBN's standards appear to be as good as those of other U.S. 
international broadcasters. In our analysis, we found that the 
standards cover areas similar to the codes of other broadcasters--such 
as RFE/RL, RFA, Voice of America (VOA), and National Public Radio-- 
focusing on accuracy, impartiality, establishing context, clearly 
distinguishing analysis from reporting, using a tone of moderation and 
respect, avoiding advocacy, and promoting ethical conduct. The 
standards also include guidelines for conducting interviews, as well as 
editing and production requirements. For example, according to MBN 
officials, MBN strives to present opposing views accurately and achieve 
a balance among the guests on its current affairs shows. According to 
MBN officials, when broadcasting about the war in Iraq, they try to 
ensure that the program incorporates both pro-war and antiwar views. 

To help ensure that staff comply with MBN's journalistic standards, 
Radio Sawa and Alhurra have established a number of pre-and 
postbroadcast procedures, which are roughly similar to those of other 
U.S. international broadcasting entities. Examples of MBN editorial 
procedures include daily editorial meetings, the use of two or more 
sources to support a news item, checks by editors and producers to 
determine whether news stories are properly written and accurate, a 
headquarters-level review of all materials produced in MBN's overseas 
offices before broadcast,[Footnote 33] and postbroadcast discussions. 
In addition, we observed that MBN employs an experienced journalist 
from the Arab world to review all of Alhurra's and Alhurra-Iraq's 
weekday newscasts for technical and stylistic errors, a control not 
implemented by other U.S. broadcasting entities. The journalist watches 
the newscasts just before they are aired, then provides MBN management 
with an evaluation of the newscast's quality. MBN provided us with 
records of these evaluations, which assess whether the newscasts are 
presentable, balanced, and free of technical errors. In some cases, 
technical mistakes can be caught before the piece is aired. However, 
while MBN management follows up on critiques of individual journalists, 
they do not systematically review and assess the journalist's 
evaluations. 

MBN officials told us that the organization places a high value on 
journalistic controls, particularly due to the volatility of certain 
areas in the Middle East and the impact news reports can have. In 
addition, MBN's controls can serve as an assurance to its audiences and 
others that they are broadcasting quality programming. Since Radio Sawa 
and Alhurra were established, MBN has not had to retract a single story 
or apologize for any error, according to MBN officials. 

MBN Could More Fully Develop Some Quality Control Measures and Lacks 
Regular Editorial Training: 

There are several areas in which MBN could more fully develop some 
quality control measures for its programming. These areas include using 
listener and viewer feedback to improve program quality, making better 
use of weekly compilations, and ensuring its style guide is distributed 
to all staff. Unlike other U.S. international broadcasters, MBN 
typically only partially utilizes the following measures: 

* Although MBN collects feedback from its listeners and viewers, it 
does not rely extensively on this feedback as a program quality 
control. 

* Although MBN produces weekly compilations that are distributed to 
interested parties summarizing what was broadcast on Radio Sawa and 
Alhurra that week, it does not use these compilations to do any formal, 
long-term analysis of errors and programming. 

* MBN has developed a style guide to provide critical guidance on the 
use of sensitive terms and to help staff avoid grammatical mistakes. 
However, Radio Sawa staff did not receive the guide until early 2006, 
and MBN has not distributed it to all of its overseas offices, 
inhibiting its use by some staff. 

In addition, MBN has not provided regular training to its journalists 
and producers. MBN told us that it does not provide regular training to 
help its editorial staff maintain and increase their professional 
competence. MBN's level of training for its editorial staff is also not 
on a par with the training offered by other grantees. For example, RFE/ 
RL and RFA both have extensive training programs for their employees at 
their headquarters and in their bureaus in different countries, 
according to RFE/RL and RFA officials. While MBN provides some initial 
technical training to its journalists, an informal mentoring program, 
and sporadic training overseas, according to MBN officials, the 
organization does not have an ongoing training program to educate 
journalists throughout the organization about editorial and ethical 
issues they might encounter on the job. According to the MBN CFO, the 
organization does not have a well-established training program because 
the network is relatively new and lacks training resources. MBN 
officials have also noted that their most important controls are the 
editors themselves, and that MBN tries to hire experienced staff. 
However, regardless of experience, staff can and do make mistakes, and 
MBN's lack of regular training increases the risk that correspondents 
and editors will make mistakes. 

MBN Lacks Comprehensive, Regular Program Reviews to Determine 
Compliance with Mission and Journalistic Standards: 

The BBG's main mechanism to determine whether its broadcasting services 
comply with its mission and journalistic standards is a regular program 
review, which is designed to improve programming and ensure quality 
control. However, only one review has been conducted for Radio Sawa, 
and none have been conducted for Alhurra. The Radio Sawa review was 
more limited in scope than program reviews conducted by other entities. 
In addition, Radio Sawa's program quality score is inconsistent with 
other BBG entities, and without a program review to develop a program 
quality score for Alhurra, the BBG will not be able to measure the 
contribution of these efforts to the goals of the organization, or be 
able to ensure that the quality of Alhurra's broadcasts conforms to 
applicable standards. Finally, many MBN staff were not extensively 
involved in the Radio Sawa review. 

Annual Program Reviews Have Not Been Conducted for MBN's Services: 

According to written guidance from the International Broadcasting 
Bureau, which coordinates and supports all VOA program reviews, program 
review is an annual process by which an institution judges itself and 
solicits the judgment of others to make improvements and fulfill its 
mission with regard to U.S. national interests. The process enables the 
broadcasting entity to better connect its mission to the market where 
it is broadcasting and assess whether its editorial procedures are 
functioning effectively, while also allowing the BBG to fulfill its 
requirements from OMB that it conduct regular program evaluations to 
capture a program's impact over time. In addition, the Senate Committee 
on Foreign Relations Report on the Fiscal Year 2003 Foreign Relations 
Authorization Act called for significant resources to be dedicated to 
postbroadcast analysis of Radio Sawa programming to ensure that 
broadcasts are consistent with U.S. interests and values and with the 
standards in the U.S. International Broadcasting Act. Program review 
typically includes a study of the target area's media environment, 
analyses by internal and external reviewers, background quantitative 
research, reports on marketing and transmission, and target area 
profiles compiled by the broadcaster itself. Reviewers rate programs 
based on criteria[Footnote 34] for content--such as accuracy, 
timeliness, objectivity, relevance, and quality of analysis and 
interviews--and for presentation, such as pace and liveliness, 
presentation style, sound quality, and host interaction. These inputs 
are then discussed at a meeting that includes the program review 
coordinators and the management and staff of the entity being reviewed. 
Following the main program review meeting, key participants develop an 
action plan, and 3 months later the group meets again to determine to 
what extent the action plan has been carried out. 

Although the BBG calls for program reviews to be conducted annually, 
MBN has not complied with this guidance. Neither the BBG nor MBN has a 
regular mechanism in place to systematically review MBN programs. In 
December 2004, the BBG convened a program review meeting for Radio 
Sawa, which had begun operating in March 2002. However, there has been 
no Radio Sawa review since then, while Alhurra, which has been 
operating for more than 2 years, has had no program review at all. The 
BBG is planning to initiate a program review of Alhurra by the end of 
this year, but has not set a firm date. 

Radio Sawa's Program Review Was Limited by Its Emphasis on Audience 
Input: 

BBG officials have stated that program quality should be sampled and 
assessed by both internal and external evaluators.[Footnote 35] This is 
designed to produce a balanced and robust review.[Footnote 36] In 2000, 
in response to our report that found a lack of consistency in how 
program quality scores were developed, the BBG stated that it intended 
to harmonize and standardize program reviews across broadcasting 
entities.[Footnote 37] BBG guidance now calls for all U.S. 
international broadcasting entities to be evaluated using the same 
standards, definitions, and scoring methods. 

However, when conducting the Radio Sawa program review, the BBG relied 
only on audience monitoring panels to assess program quality and did 
not utilize internal analysts or external control listeners, as is 
common practice among other U.S. international broadcasters. Using only 
audience monitoring panels gives the audience more weight in the review 
results and, in turn, more potential to influence the strategic 
direction of the organization. In addition, since the audience tends to 
be unfamiliar with a broadcasting service's journalistic standards and 
editorial procedures, having input only from monitoring panels makes it 
more difficult for a service to provide reasonable assurance that its 
editorial procedures are working and that it is broadcasting quality 
programming. Also, external control listeners are specifically tasked 
with examining the programming in light of the service's mission, 
something audience monitoring panels are not asked to do. As a result, 
Radio Sawa's program review strongly emphasized the audience's 
perspective and therefore provided a less thorough evaluation of Radio 
Sawa's mission and standards. (For more information on Radio Sawa's 
program review, see appendix III.) 

Radio Sawa's Program Quality Score Is Inconsistent with Other BBG 
Entities: 

Furthermore, MBN's program score for Radio Sawa is not comparable with 
other BBG broadcaster scores for program quality--the percentage of a 
station's language services judged on both content and presentation 
criteria as being of good-or-better quality. The BBG says that to 
measure Radio Sawa's program quality, it has developed standardized 
criteria applicable to different media and methods of delivery, while 
minimizing subjective judgments on content and presentation. Although 
the criteria used to measure Radio Sawa's program quality are similar 
to those used by other broadcasting entities, the BBG did not use as 
many inputs when calculating the program quality scores for Radio Sawa, 
leading to a less robust result (see table 2). 

Table 2: Inputs to Program Quality Scores for BBG Broadcasting 
Entities: 

Broadcaster: MBN (Radio Sawa); 
Program review input: Monitoring panels; 
Percentage of total program quality score: 100. 

Broadcaster: RFE/RL; 
Program review input: Internal analysis; 
Percentage of total program quality score: 45. 

Broadcaster: RFE/RL; 
Program review input: Monitoring panels; 
Percentage of total program quality score: 35. 

Broadcaster: RFE/RL; 
Program review input: External experts; 
Percentage of total program quality score: 20. 

Broadcaster: RFA; 
Program review input: Internal analysis; 
Percentage of total program quality score: 45. 

Broadcaster: RFA; 
Program review input: Monitoring panels; 
Percentage of total program quality score: 35. 

Broadcaster: RFA; 
Program review input: External experts; 
Percentage of total program quality score: 20. 

Broadcaster: VOA; 
Program review input: Internal analysis; 
Percentage of total program quality score: 45. 

Broadcaster: VOA; 
Program review input: Monitoring panels; 
Percentage of total program quality score: 35. 

Broadcaster: VOA; 
Program review input: External experts; 
Percentage of total program quality score: 20. 

Sources: BBG, International Broadcasting Bureau, RFE/RL, RFA. 

[End of table] 

Without a consistent process for broadcaster program reviews, the BBG 
is limited in its ability to assess and compare broadcaster 
performance. In addition, without a program review to develop a program 
quality score for Alhurra, the BBG will not be able to measure the 
contribution of these efforts to the goals of the organization, or be 
able to ensure that the quality of Alhurra's broadcasts conforms to 
applicable standards. Currently, the BBG is considering how it will 
conduct the future program reviews for Radio Sawa, but officials could 
not yet provide specifics about the approach BBG will use. 

MBN Staff Had Limited Awareness of the Radio Sawa Program Review and 
Follow-up: 

MBN staff in the United States and overseas do not appear to have much 
knowledge of the Radio Sawa review or its follow-up. Program review can 
be a learning experience for staff, who are usually encouraged to sit 
in on their language service's program review meeting, and the process 
can also bring people together who normally do not interact, which can 
help generate ideas and improvements. However, according to MBN 
management, most Radio Sawa staff did not attend the Radio Sawa program 
review. In addition, there was also not much awareness about the 
program review in MBN's overseas offices. 

While Radio Sawa staff lacked awareness about their review, some staff 
who had worked at VOA in the past spoke positively of the program 
review process in general. Other broadcasters routinely involve staff 
in their program reviews. For example, RFA requires all staff from the 
service that is being reviewed to be present at the program review 
meeting. According to an RFA official, program review is one of the few 
times that all of the key players for a service are in one place, and 
it therefore presents a good opportunity for communication. In 
addition, a BBC official noted that program review can be a good way to 
expose staff to an organization's values. Increasing staff involvement 
in program review could therefore increase chances for communication 
throughout all levels of the organization, as well as provide a forum 
for discussing potential programming improvements. 

BBG Has Developed Several Performance Indicators, Although Weaknesses 
in Audience Surveys Call into Question Whether MBN Audience Size and 
Credibility Targets Have Been Met: 

The BBG has established several standard performance indicators and 
targets for MBN programs, including measures of audience size and 
credibility; however, it has not implemented some performance 
indicators fully, including a program quality measure. Additionally, we 
were unable to determine the accuracy of MBN's audience size and 
program credibility estimates due to weaknesses in MBN's methodology 
and documentation. Therefore, it is not clear whether the Radio Sawa 
and Alhurra performance targets have been met. 

MBN Performance Indicators and Targets Have Been Established, but Some 
Are Not Fully Developed: 

The Government Performance and Results Act of 1993 (GPRA) requires that 
all government agencies establish performance indicators, or measures, 
that provide a meaningful reading of how well the organization is 
progressing towards its goals. The BBG has developed a standard set of 
performance indicators for its broadcasting entities, which it says are 
a best effort to measure its level of effectiveness now and where its 
performance is targeted to be in the future. The BBG established common 
indicators for its entities to allow it to better assess overall 
progress for the organization. Although the BBG has made progress in 
establishing these standard performance indicators, as well as targets 
for MBN's programs, it has not formally established or implemented all 
of them for Radio Sawa and Alhurra.[Footnote 38] 

According to BBG, the three most important standard indicators for its 
entities, referred to as primary performance indicators, are (1) 
audience size, or the overall weekly audience of a station; (2) 
credibility, which represents the percentage of viewers in a target 
area that consider the station's news programs somewhat or very 
reliable; and (3) program quality, or the percentage of a station's 
programs judged on standard criteria as being of good-or-better 
quality. These three indicators are tied to the current BBG mission and 
strategic plan for U.S. international broadcasting.[Footnote 39] In 
addition to its standard primary performance indicators, the BBG has a 
number of "secondary" measures that provide management with additional 
information for gauging cost-effectiveness, marketing and promotion 
activities, and transmission efforts, including the number of 
transmitters and affiliates, cost per listener, signal strength (radio 
only), and awareness. 

The BBG has established and implemented two primary indicators-audience 
size and credibility--for both Radio Sawa and Alhurra.[Footnote 40] The 
BBG has also established performance targets for Radio Sawa and Alhurra 
for these two indicators. For example, MBN's reported fiscal year 2005 
audience size performance target for Radio Sawa was 18 million 
listeners and for Alhurra was 12.8 million viewers.[Footnote 41] 
However, the BBG has not yet established Alhurra's program quality 
indicator, or consistently implemented this measure for Radio Sawa. 
Without a measure of program quality for Alhurra and Radio Sawa, the 
BBG will not be able to consistently assess MBN's performance against 
that of other grantees, or fully assess MBN's contribution to the 
overall goals of the BBG organization. The BBG has established all of 
its standard secondary performance indicators for MBN's services except 
for audience awareness.[Footnote 42] See table 3 for a list of the 
performance indicators implemented for Alhurra and Radio Sawa. 

Table 3: BBG's Standard Performance Indicators and MBN's Status: 

Indicator type: Primary; indicators; 
BBG performance indicator name: Audience size (also called overall 
weekly audience); 
Definition: Total number of all adults listening or viewing a service's 
programming at least once a week in target areas; 
Source of data used to calculate measure: This figure is calculated by 
the office of research and is an estimate derived from audience surveys 
and estimates of national population; 
Current Alhurra performance indicator: Yes; 
Current Sawa performance indicator: Yes. 

Indicator type: Primary; indicators; 
BBG performance indicator name: Credibility; 
Definition: Percentage of total estimated weekly viewers/listeners in 
target areas that determined the reliability of news and information of 
the station to be somewhat or very reliable; 
Source of data used to calculate measure: This figure is calculated by 
the office of research and is an estimate derived from the results of 
audience surveys and estimates of overall weekly audience; 
Current Alhurra performance indicator: Yes; 
Current Sawa performance indicator: Yes. 

Indicator type: Primary; indicators; 
BBG performance indicator name: Program quality; 
Definition: Percentage of a station's language services whose 
programming is assessed as being of good-or-better quality; 
Source of data used to calculate measure: This figure is calculated by 
the office of strategic planning and performance management. For Radio 
Sawa, this figure is determined by scoring from a monitoring panel of 
regular listeners using predefined criteria for content and 
presentation; 
Current Alhurra performance indicator: No; 
Current Sawa performance indicator: Yes. 

Indicator type: Secondary indicators; 
BBG performance indicator name: Awareness; 
Definition: Percentage of adults in the target area that have prompted 
and unprompted awareness of the station; 
Source of data used to calculate measure: This figure is calculated by 
the office of research and is an estimate derived from the results of 
audience surveys and estimates of overall weekly audience; 
Current Alhurra performance indicator: No; 
Current Sawa performance indicator: No. 

Indicator type: Secondary indicators; 
BBG performance indicator name: Affiliations and transmitters; 
Definition: Total number of affiliates that regularly rebroadcast 
identified programs on competitive media and number of owned and 
operated local transmitters; 
Source of data used to calculate measure: This figure is calculated by 
the office of marketing and program placement using affiliates data and 
the office of engineering using transmitter data; 
Current Alhurra performance indicator: Yes; 
Current Sawa performance indicator: Yes. 

Indicator type: Secondary indicators; 
BBG performance indicator name: Signal strength; 
Definition: Quality of radio signal monitored and rated on a scale of 
nil to excellent; 
Source of data used to calculate measure: This figure is calculated by 
the office of engineering using data sampled periodically from overseas 
monitoring stations; 
Current Alhurra performance indicator: N/A; 
Current Sawa performance indicator: Yes. 

Indicator type: Secondary indicators; 
BBG performance indicator name: Cost per listener; 
Definition: Budget divided by weekly audience; 
Source of data used to calculate measure: This figure is calculated by 
BBG's office of strategic planning and performance management using 
budget data and audience size figures; 
Current Alhurra performance indicator: Yes; 
Current Sawa performance indicator: Yes. 

Source: GAO and BBG. 

[End of table] 

BBG Should Increase Steps to Ensure Reliability and Transparency of Its 
Performance Estimates: 

We and others have noted that agencies' performance indicators and data 
should provide a reliable means to assess progress. However, we were 
unable to determine the accuracy of MBN's reported audience size and 
program credibility estimates due to weaknesses in MBN's methodology 
and documentation. Therefore, it is not clear whether the Radio Sawa 
and Alhurra performance targets have been met. While BBG has taken 
several important steps to ensure the validity and reliability of its 
performance measurement approach, it has primarily used a methodology 
that cannot be reliably projected to the broader population. Although 
it is difficult to conduct probability sampling in many locations in 
the Middle East, the BBG has not taken steps to explain and increase 
the reliability of MBN's performance information, such as by 
maintaining more detailed documentation to support its estimates, 
reporting significant data limitations, limiting the scope of its 
projections to areas actually covered by its surveys, and developing 
BBG policies and procedures for verifying performance data. 

Measurement and Research Methods Should Allow for Confidence in 
Reliability of Performance Information: 

We have previously reported that performance indicators should provide 
a reliable way to assess progress.[Footnote 43] In particular, agencies 
should implement quality control procedures to mitigate errors that can 
occur at various points in the collection, maintenance, processing, and 
reporting of performance data and can impact its reliability. In 
addition, agencies should select sampling methods that ensure 
representative samples, where possible. For example, probability 
surveys are designed to ensure each person in the population has a 
measurable chance of being selected for the survey, enabling the 
results to be reliably projected to the larger population with known 
levels of precision. Additionally, agency performance reporting should 
provide sources, disclose limitations, and discuss the implications of 
them. Explaining the limitations of performance information--as well as 
actions taken to compensate for low-quality data-can provide context 
for understanding and assessing agencies' performance and the costs and 
challenges that agencies face in gathering, processing, and analyzing 
needed data.[Footnote 44] 

The U.S. International Broadcasting Act of 1994 requires the BBG's use 
of audience research to guide its decisions about its language 
services. For the past 5 years, the BBG has contracted with Intermedia 
to serve as the primary research contractor for the BBG and its 
broadcasting entities, including MBN. Intermedia works with 
subcontractors, partners, or both to gather audience research survey 
data from citizens in various locations overseas. This survey 
information is used by the BBG to develop estimates for MBN and other 
entities' audience size and credibility performance indictors. For 
example, for fiscal year 2005, the BBG estimated that Radio Sawa and 
Alhurra had each achieved an audience size of about 21.6 million 
people. Table 4 shows the breakdown of individual survey data reported 
by the BBG. 

Table 4: BBG's Reported Audience Size and Credibility Estimates for 
Radio Sawa and Alhurra, Fiscal Year 2005: 

Country surveyed: Algeria; 
Year surveyed: 2004; 
Sawa estimated audience size: N/A; 
Alhurra estimated audience size: 2,466,888; 
Sawa credibility rate: N/A; 
Alhurra credibility rate: N/A. 

Country surveyed: Bahrain; 
Year surveyed: 2005; 
Sawa estimated audience size: 127,008; 
Alhurra estimated audience size: 132,814; 
Sawa credibility rate: 78; 
Alhurra credibility rate: 86. 

Country surveyed: Djibouti; 
Year surveyed: 2003; 
Sawa estimated audience size: 151,620; 
Alhurra estimated audience size: N/A; 
Sawa credibility rate: N/A; 
Alhurra credibility rate: N/A. 

Country surveyed: Egypt; 
Year surveyed: 2005; 
Sawa estimated audience size: 4,791,552; 
Alhurra estimated audience size: 2,019,968; 
Sawa credibility rate: 82; 
Alhurra credibility rate: 92. 

Country surveyed: Iraq; 
Year surveyed: 2005; 
Sawa estimated audience size: 8,336,320; 
Alhurra estimated audience size: 6,860,435; 
Sawa credibility rate: 63; 
Alhurra credibility rate: 73. 

Country surveyed: Jordan; 
Year surveyed: 2005; 
Sawa estimated audience size: 965,216; 
Alhurra estimated audience size: 729,120; 
Sawa credibility rate: 75; 
Alhurra credibility rate: 68. 

Country surveyed: Kuwait; 
Year surveyed: 2005; 
Sawa estimated audience size: 299,739; 
Alhurra estimated audience size: 249,612; 
Sawa credibility rate: 82; 
Alhurra credibility rate: 72. 

Country surveyed: Lebanon; 
Year surveyed: 2005; 
Sawa estimated audience size: 68,400; 
Alhurra estimated audience size: 738,720; 
Sawa credibility rate: 68; 
Alhurra credibility rate: 79. 

Country surveyed: Morocco; 
Year surveyed: 2005; 
Sawa estimated audience size: 4,072,455; 
Alhurra estimated audience size: 2,679,367; 
Sawa credibility rate: 64; 
Alhurra credibility rate: 70. 

Country surveyed: Qatar; 
Year surveyed: 2004; 
Sawa estimated audience size: 324,786; 
Alhurra estimated audience size: N/A; 
Sawa credibility rate: 79; 
Alhurra credibility rate: N/A. 

Country surveyed: Saudi Arabia; 
Year surveyed: 2005; 
Sawa estimated audience size: 734,928; 
Alhurra estimated audience size: 2,204,784; 
Sawa credibility rate: 72; 
Alhurra credibility rate: 73. 

Country surveyed: Syria; 
Year surveyed: 2004; 
Sawa estimated audience size: 1,468,800; 
Alhurra estimated audience size: 2,661,120; 
Sawa credibility rate: 76; 
Alhurra credibility rate: 60. 

Country surveyed: Tunisia; 
Year surveyed: 2005; 
Sawa estimated audience size: N/A; 
Alhurra estimated audience size: 676,800; 
Sawa credibility rate: N/A; 
Alhurra credibility rate: 75. 

Country surveyed: UAE; 
Year surveyed: 2005; 
Sawa estimated audience size: 265,842; 
Alhurra estimated audience size: 138,510; 
Sawa credibility rate: 79[A]; 
Alhurra credibility rate: 82. 

Country surveyed: Regionwide fiscal year 2005 estimate[B,C]; 
Year surveyed: [Empty]; 
Sawa estimated audience size: 21.6 million[D]; 
Alhurra estimated audience size: 21.6 million; 
Sawa credibility rate: 69; 
Alhurra credibility rate: 73. 

Source: BBG. 

[A] The BBG originally reported this figure as 55 percent, but later 
determined that the correct figure was 79 percent. 

[B] To develop the regional estimate for audience size, 
listening/viewing rates are gathered from the national surveys and are, 
in most cases, multiplied by each country's respective national 
population estimate to obtain a total audience size estimate for each 
country. For some countries--such as Bahrain, Kuwait, Qatar, UAE and 
Saudi Arabia--the results are projected to nationals and Arab 
expatriate populations only. These estimates are developed for all 
countries surveyed and then added to develop a regional audience 
figure, which is the service's overall weekly audience estimate. 

[C] To develop the regional estimate for credibility, percentage 
credibility scores are gathered from national surveys and multiplied by 
that country's audience size estimate to obtain a total number of 
listeners or viewers for a given country that view the station's news 
and information as very or somewhat reliable. These estimates are 
developed for all countries surveyed and then used to create a weighted 
average of credibility. 

[D] The BBG reported in its fiscal year 2007 budget request that its 
audience size for Alhurra was 21.5 million. 

[End of table] 

BBG Has Not Taken Sufficient Steps to Ensure Reliability and 
Transparency of Its Performance Estimates for MBN: 

Although the BBG has taken several important steps to enhance the 
validity and reliability of its audience survey designs, several 
factors call into question the accuracy of the data used for BBG's 
audience size and credibility performance indicators for Radio Sawa and 
Alhurra. First, we observed that, in 12 out of 14 cases, the BBG used 
nonprobability surveys, which cannot be reliably projected to the 
broader population, to develop its regional estimates for audience size 
and credibility. While it is difficult to conduct probability surveys 
in hostile environments, such as those in Iraq and Saudi Arabia, the 
BBG did not take certain steps that could have increased the accuracy 
of its estimates and explained their limitations, thereby increasing 
confidence in the data. Therefore, we are unable to determine whether 
MBN actually met its performance targets for fiscal year 2005. 

BBG has taken several important steps to enhance the validity and 
reliability of its audience survey designs. For example, the BBG's 
questionnaires are reviewed by multiple parties, and its contractors 
extensively pretest the questionnaires in the field, conduct pilot 
studies, and use throwaway surveys[Footnote 45] for training new 
subcontractors. In addition, the BBG's contractors exercise quality 
control when collecting data and receiving it from the field, including 
by conducting preliminary electronic testing of the data, among other 
things. 

However, in 12 of 14 cases, we found the country-level estimates used 
for generating Radio Sawa and Alhurra's fiscal year 2005 performance 
indicators of audience size and credibility were not based on 
probability survey[Footnote 46] results. Many of the surveys conducted 
used judgment sampling, a form of nonprobability sampling, 
instead.[Footnote 47] BBG officials told us that its use of 
nonprobability surveys for certain countries is due to either the 
cultural, political, or security situation in those countries, which 
limits the selection of individuals or the geographical areas that can 
be surveyed.[Footnote 48] In addition, the International Broadcasting 
Bureau director of research stated that in many developing countries, 
existing map and population data is not adequate to support pure 
probability-based sampling. The Conference of International 
Broadcasters' Audience Research Services, called CIBAR, whose standards 
are specified as a source of guidance for BBG research contractors, 
requires that audience measurement use samples based on the principles 
of random probability and that other sampling methods should only be 
used in cases where, for reasons of practicality or cost, proper random 
samples cannot be used. (See appendix IV for more on the CIBAR 
standards.) 

We recognize that many agencies face challenges in collecting credible 
performance data and that, due to security risks and political 
considerations in many Middle Eastern countries, it is not always 
possible to expect BBG to use random samples. However, the BBG did not 
take certain steps that could have explained and increased the 
reliability of its estimates, such as fully documenting its research 
methods, measuring the level of uncertainty surrounding its estimates, 
disclosing significant limitations, limiting the scope of its 
projections to areas actually covered by its surveys, and developing 
and implementing procedures for verifying data. 

First, the BBG and its research contractors were unable to provide us 
with certain documentation commonly required by international 
broadcasting research standards. CIBAR requires that, in all 
measurement research, the sampling methods used and other technical 
aspects of the survey be both fully and accurately described in the 
project documentation and open to independent scrutiny. We asked the 
BBG and its contractors to provide us with detailed documentation- 
including clear information on sampling plans and related assumptions, 
response rates, and adjustments applied to the data to reflect the 
effects of the survey design-for all 14 of its Middle Eastern country 
surveys used to develop its fiscal year 2005 performance indicator 
estimates. For two cases, the BBG was unable to provide us with any 
survey documentation,[Footnote 49] and for all but one case, the BBG 
and its contractors were unable to provide us all the detailed 
information we requested.[Footnote 50] BBG officials acknowledged the 
lack of complete documentation in contractor technical reports, and 
said it was due in part to their failure to follow up with the 
contractors to obtain the details, as well as to the contractors' 
general practice of not generating such detailed documentation. 

Second, the BBG has not sufficiently measured the level of uncertainty 
surrounding MBN's performance estimates. CIBAR requires that in all 
measurement research, technical aspects of the survey, including 
margins of error and confidence levels where appropriate, be both fully 
and accurately described in the project documentation. The BBG has not 
been able to measure sampling errors[Footnote 51] for its surveys, in 
part because it has not required contractors to document the 
information that is needed to calculate the sampling errors accurately. 
Moreover, officials said that it is not customary for their contractors 
to maintain this information. Therefore, the BBG only has a rough idea 
of what the margins of error[Footnote 52] might be for its surveys, 
further limiting confidence in the reliability of its current 
performance information. One research official told us that he believed 
that the overall margins of error for Radio Sawa and Alhurra's audience 
size and credibility estimates are large, but said that currently the 
agency cannot accurately calculate them.[Footnote 53] 

Third, the BBG lacks transparency in reporting data sources and 
significant limitations affecting MBN's audience size and credibility 
performance information. In reporting performance data, agencies should 
provide data sources, disclose limitations, and discuss the 
implications of them. CIBAR standards also recommend that proper care 
be exercised when reporting estimates, to ensure that the type of 
audience covered by the estimate is clearly stated and that, at all 
stages in the calculation and extrapolation process, sources, 
assumptions, and methods be fully documented and available for 
independent scrutiny. However, the BBG has not sufficiently explained 
the specific methods used for generating estimates for its performance 
indicators, such as the number and names of the countries surveyed, 
methods of sampling used, sources of the population data, and basic 
procedures used to create the estimates. Moreover, the BBG has reported 
only two limitations for MBN performance indicators to date: (1) that 
credibility ratings are highly dependent on volatile political factors; 
and (2) that, depending on political, social, and media conditions, 
measurement of audience size may either be easily attained or 
impossible. However, the BBG has not explained many significant 
limitations, and their resulting implications, on the data reliability 
of the performance information for MBN. The largest identifiable 
limitations not reported are that many of the BBG surveys are not based 
on probability sampling, cover only part of the country, have very low 
response rates, or have high substitution rates; therefore, those 
results cannot be reliably projected to be representative of the larger 
population.[Footnote 54] In the case of the survey in Morocco, we 
calculated that the survey only covered 35 percent of the general 
population and had a 48 percent substitution rate, but the results were 
projected to represent a broader population. In addition, we calculated 
low response rates for a number of MBN's surveys; in the case of the 
survey of Egypt, the response rate was about 19 percent. BBG research 
staff explained that their stakeholders to date, including the BBG, 
have not required such a level of detail in reporting, and specifically 
have not required margins of error. However, a discussion on data 
limitations in performance reporting can help decision makers determine 
their level of confidence in the agency's ability to report on its 
performance goals and indicators and identify actions needed to improve 
its ability to measure performance. 

Fourth, we found the BBG has not taken sufficient steps to avoid 
projections to areas outside the population surveyed. For example, the 
elimination of projections to the rural population of Morocco, when 
only urban areas were surveyed, or to those governorates of Jordan not 
contained in the survey, would have resulted in estimates of audience 
size and credibility with fewer limitations for those countries. In 
addition, BBG project documentation does not clearly describe what 
steps have been taken to restrict the scope of MBN's surveys to 
provinces or areas that have sufficient map or population data. 
Probability sampling projected to those areas only would result in more 
reliable estimates of audience size and credibility. 

Fifth, we found the lack of verification procedures inhibits 
assessments of BBG's data quality. Data reliability is increased by the 
use of verification and validation procedures, such as checking 
performance data for significant errors by formal evaluation or audit. 
However, BBG has not fully implemented such procedures or formally 
documented policies and procedures governing its research. BBG 
officials said they have implemented some forms of verification, such 
as a research director review, at various stages of performance data 
analysis. However, it is not clear that the BBG always thoroughly 
verifies performance indicator estimates and calculations used for 
reporting. For example, we identified some errors in internal 
calculation spreadsheets for performance estimates, in official 
external reporting, and in an informal presentation on MBN performance. 
BBG officials have acknowledged that they need to develop policies and 
procedures and implement more rigorous verification of performance 
data. They said that, as a result of our review, they have taken some 
steps to expand their verification procedures and will document those 
and other research procedures in the BBG's research contract and manual 
of operations. 

Conclusions: 

MBN was established to broadcast accurate and relevant news and 
information to the Middle East in order to advance freedom, democracy, 
and long-term U.S. interests in the region. MBN's programs are 
challenged by existing television competitors, such as Al Jazeera and 
Al Arabiya, and planned initiatives such as the BBC's planned Arabic- 
language television network; by numerous local radio stations across 
the region; and by limited opportunities for increasing coverage of 
radio transmission. It has attempted to address these challenges 
through some planning efforts, such as by developing proposals to 
increase its news time for Alhurra and expand its number of news 
bureaus in the Middle East. However, MBN has not developed a strategic 
plan or taken a detailed strategic approach to addressing certain 
issues, including identifying opportunities for additional gains that 
could be made from further integrating Radio Sawa and Alhurra or fully 
utilizing MBN's overseas offices. Developing a strategic plan that 
establishes specific objectives, provides an overall shared vision or 
framework for decision-making, and comprehensively addresses 
competitive challenges could enable MBN to identify efficiencies or 
opportunities to address its competitors more effectively and increase 
congressional confidence in its operations. 

Although MBN has developed a number of financial and administrative 
controls, it could take additional steps to ensure its system of 
internal control is fully implemented. MBN's planned internal control 
board needs to establish protocols to oversee and monitor its internal 
control structure and to ensure timely completion of MBN's financial 
audits. Additionally, MBN should develop an internal control plan, 
conduct a comprehensive risk assessment, and develop an organized 
training program for its staff. 

Further, MBN has developed journalistic standards for its broadcast 
operations and has put in place a number of editorial procedures. 
However, the network has not conducted annual program reviews called 
for in BBG guidance. These reviews are a key mechanism for improving 
programming and ensuring quality control. 

Finally, the BBG has established performance indicators and targets for 
MBN services related to measures of audience size and program 
credibility. For example, in fiscal year 2005, the BBG reported that 
Radio Sawa and Alhurra had each achieved radio and television audiences 
of about 21.6 million. However, limitations in the survey methods and 
documentation used for developing these estimates raise questions about 
the accuracy of MBN's performance estimates. In addition, the BBG has 
not put in place policies and procedures for verifying the accuracy of 
its performance information. These weaknesses in methodology and 
documentation inhibit an accurate assessment of whether the Radio Sawa 
and Alhurra performance targets have been met. The accuracy of the MBN 
estimates could be improved by more rigorous reliance on probability 
sampling, wherever possible, and avoidance of projections to areas not 
contained within the survey. At the same time, insistence on proper 
fieldwork documentation from contractors, full disclosure of survey 
methods and limitations, and greater transparency in the development of 
aggregate audience estimates would further enhance confidence in MBN 
performance estimates. 

Recommendations for Executive Action: 

To improve efforts to monitor performance and the efficiency and 
effectiveness of the broadcasting activities of the Middle East 
Broadcasting Networks, Inc., we recommend that the chairman of the BBG 
do the following: 

* Require that MBN develop a long-term strategic plan, which 
incorporates a shared vision for Alhurra and Radio Sawa operations and 
details specific measurable objectives and implementation strategies 
for the successful implementation of the goals in the plan. 

* Require that MBN implement the remaining recommendations from the 
Grant Thornton LLP report and require that its internal control board 
meet on a regular basis to coordinate MBN's single audits and oversee 
MBN's ongoing efforts to use sound internal control procedures. 

* Develop a process for analyzing risk as part of strategic planning 
that identifies approaches to mitigate the potential obstacles to 
efficiently and effectively achieving MBN's operational objectives. 

* Require MBN to develop a comprehensive training program covering both 
internal controls and editorial procedures to meet the continuing needs 
of all employees. 

* Initiate a schedule of annual program reviews for Radio Sawa and 
Alhurra to regularly ensure that the quality of Alhurra's broadcasts 
conforms to applicable standards. 

* Implement program quality performance indicators for MBN's broadcast 
services, consistent with other BBG entities, to assess and compare 
their performance and measure the contribution of these efforts to the 
goals of the overall organization. 

* Require research contractors to improve the methods used in audience 
research to allow for probability sampling and document the sample 
selection so that survey sampling errors can be calculated, where 
possible. 

* Identify and report significant methodological limitations and the 
implications of them for performance indicators, including, where 
applicable, sampling errors, margins of error, or confidence intervals. 

* Develop, document, and report policies and procedures for 
verification and analysis of performance indicator estimates. 

Agency Comments and Our Evaluation: 

The Broadcasting Board of Governors (BBG) provided written comments on 
a draft of this report. The BBG's comments, along with our response to 
specific points, are reprinted in appendix V. The BBG also provided 
technical comments, which we incorporated where appropriate. 

In general, the BBG concurred with all of our recommendations and said 
it looked forward to implementing them. The BBG said that MBN has made 
significant progress in the 2 years of its operations in establishing a 
sound journalistic organization with financial and administrative 
controls. However, the BBG raised a number of concerns about the 
report's criticisms of the audience research conducted by the BBG and 
its contractors. Specifically, the BBG said that we did not fully 
understand the difficulty in surveying audiences in Middle Eastern 
countries and that the research practices used by the BBG and its 
contractors follow industry standards for commercial and media 
research. 

Our report examines the reliability of the BBG's fiscal year 2005 
performance information in order to determine whether or not MBN's 
performance targets have been met. We acknowledge, in our report, that 
BBG has taken positive steps to enhance the validity and reliability of 
its audience survey designs. We also acknowledge that there are 
challenges to conducting audience research in the Middle East, and that 
there are tradeoffs between cost and data reliability when conducting 
research. In conducting our data reliability assessment, as referenced 
in the draft, we largely used the international audience research 
guidelines published by the Conference of International Broadcasters' 
Audience Research Services (CIBAR), which are specified as a source of 
guidance for BBG research contractors. For example, CIBAR standards 
recommend that the proper care be exercised when reporting estimates to 
ensure that the type of audience covered by the estimate is clearly 
stated, and that, at all stages in the calculation and extrapolation 
process, sources, assumptions, and methods are fully documented and 
available for independent scrutiny. We were unable to determine the 
accuracy of MBN's fiscal year 2005 audience size and program 
credibility estimates due to weaknesses in MBN's methodology and 
documentation. As noted in the report, in several instances the BBG and 
its contractors departed from CIBAR research standards. In particular, 
the BBG did not take certain steps that could have explained and 
increased the reliability of its estimates, such as by fully 
documenting its research methods, measuring the level of uncertainty 
surrounding its estimates, disclosing significant limitations, limiting 
the scope of its projections to areas actually covered by its surveys, 
and developing and consistently implementing policies and procedures 
for verifying data. For these reasons, it is not clear whether the 
Radio Sawa and Alhurra performance targets have been met. 

As agreed with your office, unless you publicly announce the contents 
of this report earlier, we plan no further distribution of it until 30 
days from the date of this letter. We will then send copies to others 
who are interested and make copies available to others who request 
them. 

If you or your staff have any questions about this report, please 
contact me on (202) 512-4128. Contact points for our Offices of 
Congressional Relations and Public Affairs may be found on the last 
page of this report. Key contributors are listed in appendix VI. 

Sincerely yours, 

Signed by: 

Jess T. Ford: 
Director, International Affairs and Trade: 

[End of section] 

Appendix I: Scope and Methodology: 

To accomplish our objectives, we reviewed documentation and spoke with 
officials from the Department of State; the Broadcasting Board of 
Governors (BBG) and its grantees; including the Middle East 
Broadcasting Networks; Inc. (MBN); Radio Free Europe/Radio Liberty 
(RFE/RL); Radio Free Asia (RFA); the Voice of America (VOA); and the 
International Broadcasting Bureau. We also spoke with several outside 
experts, including representatives from National Public Radio, the 
InterMedia research organization, Zogby International, and foreign 
international broadcasters such as the British Broadcasting Corporation 
(BBC) and Deutsche Welle. 

In addition to our audit work in the Washington, D.C., area, we visited 
MBN's offices in Amman, Jordan, and Dubai, United Arab Emirates. We 
held telephone interviews with MBN's current affairs contractor in 
Beirut, Lebanon, and sent questions to the MBN office in Baghdad, Iraq. 

Because our work was focused on reviewing the BBG's MBN initiatives-- 
Alhurra and Radio Sawa-we did not include RFE/RL Arabic-language media 
activities in Iraq in the scope of our work. 

Review of MBN Internal Controls: 

To address our objective of assessing MBN's internal controls, we used 
criteria contained in MBN's grant agreement, OMB Circulars No. A-110 
and A-133, GAO published standards on internal controls, and a report 
on MBN's internal controls completed by Grant Thornton LLP accountants 
and management consultants. We also met with other broadcasters, 
including RFE/RL, RFA, and National Public Radio, to obtain an 
understanding of their internal controls and to make comparisons with 
MBN. We held discussions with MBN officials and reviewed documentation 
provided by them to determine whether the organization was complying 
with appropriate criteria. We spoke with PriceWaterhouseCoopers 
officials responsible for the 2003, 2004, and 2005 single audits to 
obtain their perspectives on the status of problems cited in the Grant 
Thornton LLP report and to learn about any new issues arising from 
their audit work. We did not test MBN's controls, since doing so would 
have been beyond the scope of our review. 

Review of MBN's Editorial Procedures: 

To address our objective of assessing the procedures MBN has developed 
to ensure it complies with its journalistic standards, we spoke with 
other broadcasters, including RFE/RL, RFA, VOA, and National Public 
Radio, to compare their editorial standards and procedures with those 
of MBN. We also met with the BBG general counsel, as well as MBN 
producers, editors, and journalists. To better understand the program 
review process, we attended several VOA program reviews. We also held 
detailed discussions and obtained documentation on program review from 
the BBG officials who conducted Radio Sawa's review, the International 
Broadcasting Bureau office of performance review, and the research 
directors of RFA and RFE/RL. We did not test MBN's editorial 
procedures. 

Review of MBN Audience Survey Methods and Performance Indicator 
Estimates: 

To address our objective of determining the extent to which the BBG met 
or exceeded its fiscal year 2005 performance MBN targets, we reviewed 
the reliability of BBG audience research that was used to generate the 
estimates of credibility and audience size contained in the BBG's 
Fiscal Year 2007 Budget Request, which was the BBG's most up-to-date 
source of officially-reported GPRA performance information for MBN for 
fiscal year 2005. We did not review the inputs used to generate 
performance information for fiscal years other than 2005, nor did we 
review the reliability of the data used to develop other MBN 
performance indictors reported in the 2007 Budget Request. 

To conduct our assessment of the reliability of BBG estimates of 
credibility and audience size for fiscal year 2005, we reviewed 
available documentation provided as of May 2005 against applicable 
standards and common professional research practices, including 
Conference of International Broadcasters' Audience Research Services 
(CIBAR) guidelines and the professional standards of the American 
Association for Public Opinion Research. We reviewed relevant surveys, 
including technical reports and printouts of the survey results, as 
well as spreadsheets used to calculate the performance indicator 
estimates. We also conducted a series of interviews and corresponded 
extensively with the BBG's coordinator for performance planning and 
research, the head of the International Broadcasting Bureau office of 
research, and officials from Intermedia to discuss the survey methods 
and how performance indicator estimates were generated. In addition, we 
requested information from the BBG's research subcontractors, including 
AC Nielsen and D3 Systems, to obtain details on the survey sampling 
plans and related assumptions, response rates, and adjustments applied 
to the data to reflect the effects of the survey design. As a result of 
our review, we determined that MBN's reported audience size and 
credibility estimates are not statistically reliable and are rather 
rough estimates of performance. 

[End of section] 

Appendix II: Radio Sawa and Alhurra Programming: 

Radio Sawa currently has seven programming streams,[Footnote 55] with 
the Iraq programming stream airing more news and features than the 
other streams. We reviewed one week of programming and found that the 
Iraq stream aired about 50 hours of news and features a week, while 
five other streams aired about 40 hours a week (see table 5). 

Table 5: Radio Sawa's Weekly News and Features Programming: 

Programs: Total news programming time; 
Jordan: 31 hrs 2 min; 
Iraq: 34 hrs 48 min; 
Persian Gulf: 31 hrs 2 min; 
Morocco: 31 hrs 2 min; 
Egypt: 31 hrs 2 min; 
Sudan: 31 hrs 2 min. 

Programs: Total features programming[A] time; 
Jordan: 8 hrs 22.5 min; 
Iraq: 14 hrs 47 min; 
Persian Gulf: 8 hrs 22.5 min; 
Morocco: 7 hrs 57.5 min; 
Egypt: 8 hrs 29.5 min; 
Sudan: 7 hrs 57.5 min. 

Programs: Total time for news and features; 
Jordan: 39 hrs 25 min; 
Iraq: 49 hrs 36 min; 
Persian Gulf: 39 hrs 25 min; 
Morocco: 39 hrs; 
Egypt: 39 hrs 32 min; 
Sudan: 39 hrs. 

Source: GAO analysis of sample MBN programming schedule. 

Note: The sample programming schedule was provided for the week of 
October 19, 2005. The exact breakdown of programming may differ from 
week to week. 

[A] Features programming includes the following Radio Sawa shows: Ask 
the World Now, Sawa Chat, Call Us Answers, Call Us Questions, Health 
News, What's New, Free Zone, Challenges, To the Point, A Poet's Story, 
Window on the Iraqi Constitution, Law and the Citizen, and Heart of the 
Matter. 

[End of table] 

Alhurra's programming focuses on news and information, including hourly 
news updates, daily hour-long newscasts, and current affairs talk shows 
(see table 6). In addition, Alhurra broadcasts current affairs shows on 
subjects including health and fitness, entertainment, sports, and 
science and technology. Alhurra also airs documentaries on a diverse 
range of topics designed to appeal to a broad audience. In a sample 
week, Alhurra's acquired programming, which mainly includes 
documentaries, accounted for the largest number of hours aired, about 
35 percent of the total weekly programming hours--or approximately 60 
hours in a week. News and news updates amounted to 22 percent of weekly 
programming hours. 

Table 6: Alhurra's Weekly Programming Schedule: 

Programming time: Per week; 
News: 26 hours; 
Talk shows: 29 hrs, 40 min; 
Current affairs: 32 hrs, 50 min; 
News updates: 10 hrs, 21 min; 
Economic news updates: 36 min; 
Acquired programming[A]: 58 hrs, 20 min; 
Change in programming[B]: 9 hours. 

Programming time: 
Total: 166 hours and 47 minutes. 

Source: GAO analysis of sample MBN programming schedule for the week of 
February 19-25, 2006. 

Note: The breakdown of shows may differ from week to week. 

[A] Acquired programming includes documentaries, historical programs, 
profiles, and travel and adventure programs. 

[B] Change in programming is a deviation from the normal schedule, 
including broadcasts of National Basketball Association games. 

[End of table] 

In contrast, Alhurra-Iraq's talk shows accounted for the largest number 
of hours aired---about 46 hours, or 27 percent of total programming 
time (see table 7). 

Table 7: Alhurra-Iraq's Weekly Programming Schedule: 

Programming time: Per week; 
News: 34 hours; 
Talk shows: 45 hrs, 40 min; 
Current affairs: 31 hrs, 20 min; 
News updates: 8 hrs, 24 min; 
Economic news updates: 36 min; 
Acquired programming: 42 hrs; 
Change in programming: 5 hours. 

Programming time: 
Total: 167 hours.  

Source: GAO analysis of sample MBN programming schedule for the week of 
February 19-25, 2006. 

Note: The breakdown of shows may differ from week to week. 

[End of table]

[End of section] 

Appendix III: Radio Sawa Program Review: 

The BBG initiated the Radio Sawa review because MBN did not have the in-
house capacity to do so, according to a BBG official. Moreover, both 
the BBG and MBN wanted to ensure that the review was conducted by an 
entity other than MBN to make it more independent. 

To plan and conduct this review, BBG officials reported that they held 
preliminary discussions with the International Broadcasting Bureau's 
office of performance review, hired an outside expert to serve as a 
consultant to the project, and contracted with the InterMedia research 
organization to convene panels of regular listeners in Iraq, Morocco, 
Egypt, and Jordan. According to representatives of InterMedia, they 
recruited a large number of panelists because the BBG wanted more input 
than usual. Panelists listened to and evaluated 6 days worth of 
programming, responded to a questionnaire that evaluated all of Radio 
Sawa's broadcast criteria, answered open-ended questions, provided 
detailed feedback, and made recommendations about the programming. 
Overall, the results of Radio Sawa's monitoring panels were generally 
positive, with an overall program quality score of 2.9 out of 4, which 
is considered "good" by the BBG.[Footnote 56] To follow up on the 
program review, MBN is attempting to implement action points drafted by 
the BBG after the review, such as reducing abrupt transitions between 
music and news and localizing Radio Sawa's streams. 

The BBG told us it placed great emphasis on the audience's response, 
noting that if there had been any problems, they would have heard about 
them from the listeners. The BBG's research contractor concurred that 
it is important to obtain the audience's perspective, since the 
audience can judge programming in the context of the alternatives 
available to them in the local media market. According to a BBG 
official, it was decided that information obtained from the audience 
monitoring panel would capture the issues that would normally be 
covered by the internal analysts in the content and production 
analyses. 

However, even a detailed, well-thought-out questionnaire does not 
replace the different perspectives internal and external control 
reviewers bring to the review, according to a grantee official. 
According to an International Broadcasting Bureau official, even though 
the three components of a program review-audience panels and internal 
and external reviewers-use nearly the same criteria, they provide 
different insights into the program's quality, direction, and context. 
According to several media experts,[Footnote 57] while audience 
panelists can evaluate news and information from their own perspective, 
they are less qualified to evaluate a service's mission or judge the 
extent to which a show complies with journalistic standards. In 
particular, the absence of an internal review meant that Radio Sawa's 
programming was not evaluated by people familiar with BBG and MBN 
standards and controls. Moreover, the external control listener's 
evaluation is intended to give the service a sense of what the panel's 
responses would be like if the environment were freer and more open, 
information that a monitoring panel cannot provide. 

The BBG told us that it did not believe an internal evaluation was 
necessary because the English translations of each of the panel reports 
enabled them to examine and assess MBN's programming. The BBG also told 
us that it did not use external control listeners in the review for 
several reasons: (1) Radio Sawa's focus on youth and popular music 
would make it difficult for a typical older control listener to 
evaluate Sawa's programs; (2) the BBG lacked resources to use control 
listeners for the multiple countries in Radio Sawa's review; (3) it 
would have been hard to find impartial listeners; and (4) the four in- 
country panels ensured enough diversity of opinion so that there was 
less need for a control reviewer. However, according to an 
International Broadcasting Bureau official, the VOA is able to 
successfully evaluate youth shows using its regular internal reviewers. 
Moreover, it may be possible to use younger control listeners, or to 
use the older ones selectively. For example, RFE/RL uses a control 
listener to evaluate the news of Radio Farda. In addition, the 
challenge of finding impartial control listeners is not unique to the 
Middle East, but is potentially present to some degree for every 
language service. 

[End of section] 

Appendix IV: CIBAR Standards: 

The Conference of International Broadcasters' Audience Research 
Services (CIBAR) guidelines were developed by an international group of 
broadcasters - including the BBG's International Broadcasting Bureau -
-to encourage, among other things, the appropriate use of audience and 
market data for decision-making within international broadcasting 
organizations, the establishment and maintenance of standards in 
international audience research, and the efficient and responsible use 
and application within member organizations of all forms of audience 
feedback. By providing a context for international audience research 
and a clear statement of the minimum standards required, the guidelines 
attempt to address the challenges faced by international broadcasters, 
including the tension between the needs of proper audience measurement 
and survey practice and the conditions and expertise in many of the 
countries where the research is carried out. 

The third edition of the guidelines, published in 2001, covers the 
nature of international audience research measures; survey design, 
sampling, and fieldwork; and data management and reporting.[Footnote 
58] Specific guidelines relevant to this report include the following: 

4.1: Samples and method: 

* The basis for audience measurement should be samples based on the 
principles of random probability. 

* The principles of random selection should be applied at all stages of 
the sampling process, from initial sampling point to selection of 
individuals. 

* Quota samples should be used only in cases where, for reasons of 
practicality, cost, or both, proper random samples cannot be used. When 
quota samples are used, this should be clearly labeled in the reporting 
and documentation of the data. 

* It should be a requirement of all measurement research that the 
sampling methods used and other technical aspects of the survey, 
including margins of error and confidence levels where appropriate, be 
both fully and accurately described in the project documentation, and 
open to independent scrutiny. 

4.2: Survey coverage: 

* Where certain groups are disproportionately sampled and weighting 
techniques are applied at the analysis stage to correct for this, 
project documentation should give full details of the weights applied. 

5.2: Data reporting: 

* The research agency should provide the following information to a 
contracting broadcaster: 

* Background information - client for whom the study was conducted; 
purpose of the study; names of subcontractors and consultants 
performing any substantial part of the work. 

* Intended and achieved sample --universe covered; statistics used 
(e.g. census data; size, nature and geographical distribution of the 
sample; sampling method and weighting methods used; response rates and 
possible bias due to non-response). 

* Data collection - method of collection; field staff; briefing and 
field quality control; method of recruiting respondents; and fieldwork 
dates. 

* Presentation of results - relevant factual findings obtained; bases 
of percentages (weighted and unweighted); margins of error; and 
questionnaire and other documents used. 

* Proper care be exercised when reporting such estimates, to ensure 
that the type of audience covered by the estimate is clearly stated. 

* At all stages in the calculation and extrapolation process, it is 
vital that sources, assumptions, and methods be fully documented and 
available for independent scrutiny. 

* All reporting of worldwide and regional audiences should be 
accompanied by a technical appendix giving details of the sources, 
assumptions, and measures used. 

[End of section] 

Appendix V: Comments from the Broadcasting Board of Governors: 

Broadcasting Board Of Governors United States Of America: 

July 20, 2006: 

Mr. Jess T. Ford: 
Director: 
International Affairs: 
Government Accountability Office: 

Thank you for the opportunity to comment on the report of the 
Government Accountability Office, "U.S. International Broadcasting: 
Management of Middle East Broadcasting Services Could Be Improved." We 
appreciate the GAO's recommendations and we will move to implement 
them, wherever practicable. 

While we are in general agreement with the GAO's recommendations, we 
are concerned that, in some cases, the text of the report overlooks 
Middle East Broadcasting Networks' (MBN) tremendous accomplishments 
over its short history, as well as the soundness of its operations. In 
any evaluation of MBN, it is important to remember that this is a new 
organization incorporated in April 2003 to respond quickly to the 
threat of international terrorism. This urgency was reflected in 
President Bush's January 21, 2004 State of the Union address when he 
announced that ".soon, a new television service will begin providing 
reliable news and information across the region." On February 14, 2004, 
the BBG responded with Alhurra Television available via satellite 
throughout the Arab world. In both the private and public sectors, 
launching a 24/7 TV news and information network generally requires 18 
to 24 months from initial funding to going on the air. The BBC recently 
announced that its new Arabic TV channel, which will broadcast 12 hours 
a day, will take two years to get on the air. 

In just over two years of operations, we believe we have both a sound 
journalistic organization, and one that is firmly on its feet with 
respect to financial and administrative controls. As always, these 
systems can be improved, and the GAO recommendations can assist this 
process. In the paragraphs below, we provide our comments on specific 
GAO recommendations. 

GAO Recommendation: Require that MBN develop a long-term strategic 
plan, which incorporates a shared vision for Alhurra and Radio Sawa 
operations and details specific measurable objectives and 
implementation strategies for the successful implementation of the 
goals in the plan. 

From start-up through its first years of operations, Alhurra planning 
has been encompassed within the BBG's broader strategic planning and 
has been a key focal point of the agency's overall strategic planning 
exercises. We agree it is time for MBN to transition to a fully 
developed, stand-alone, long-term strategic plan. 

The GAO report leaves the impression, however, that the BBG and MBN did 
not engage in extensive long-range planning, before and after the 
launch of Alhurra TV. In fact, considerable long-term planning has 
taken place during MBN's short life, both under the umbrella of the 
BBG's larger strategic plan, and the strategic plan established by 
Radio Sawa in FY 2002, and in accordance with the specific strategic 
direction laid out by the Administration and the Congress. MBN is a key 
element of the BBG annual performance report included in the BBG 
Performance and Accountability Report (PAR), and has participated in 
the development of BBG's new five-year strategic plan. 

The Office of Management and Budget in its annual Program Assessment 
Rating Tool (PART) has evaluated MBN's annual performance plan since 
PART's inception in 2002. The MBN annual performance plan sets long- 
term targets and contains all the elements typically found in a stand- 
alone strategic plan. For example, the MBN plan contains a program 
description; goals and objectives; key factors affecting goals and 
objectives, evaluating progress and linking performance measures to the 
agency's goals; and performance baselines and targets. 

Based on the BBG plans for the Alhurra initiative, the Bush 
Administration approved the plan and included it in the proposed FY 04 
budget. After the Alhurra TV plans were discussed with the Congress, 
both through the agency's budget request and briefings on goals and 
objectives, BBG's strategic planning for the regional Alhurra TV 
network was validated when the Congress initiated an additional 
appropriation for $40 million to expand Alhurra's plan to include a 
television network devoted to Iraq. 

As noted above, MBN is currently participating in the development of 
BBG's new five-year strategic plan (FY 2008 - 2012) that will inform 
the BBG's FY 2008 budget request. As part of this exercise, MBN has 
conducted extensive long-range planning exercises that address the 
elements identified by GAO as key components of a strategic plan. MBN, 
under the leadership of its new president, has made establishing a 
stand-alone strategic plan a top priority. 

GAO Recommendation: Require that MBN implement the remaining 
recommendations from the Grant Thornton LLP report and require that its 
internal control board meet on a regular basis to coordinate MBN's 
single audits and oversee MBN's ongoing efforts to use sound internal 
control procedures. 

We agree with GAO's recommendation. However, we believe many of GAO's 
broader comments on internal controls are misleading, leaving the 
impression that there are a number of federal requirements that MBN is 
required to implement, but has not yet done so. We would like to 
emphasize that MBN is in full compliance with all appropriate 
requirements for recipients of federal funds. 

GAO's findings acknowledge that MBN's 2005 Single Audit by independent 
auditor Price Waterhouse Coopers, evaluating the effectiveness of MBN's 
internal control program, resulted in an unqualified opinion without 
material weaknesses in internal controls. This audit was completed, 
provided an unqualified opinion on MBN's financial statements, and 
demonstrated improvement over previous years. 

GAO's broader discussion of internal controls sometimes blurs the 
distinction between those controls that are actually mandated by OMB 
requirements, and those that are voluntary or recommended options used 
by MBN management to comply with the requirements. For example, the 
report correctly cites OMB Circulars A-110, A-122, and A-133 as a basis 
for evaluating MBN financial management operations and internal 
controls, but rather than evaluating MBN achievement based on these 
requirements, the GAO appears to use the recommendations of a BBG- 
commissioned management review by Grant Thornton as a standard for 
compliance. 

The BBG initiated the Grant Thornton review in 2003 to assist MBN in 
its efforts to develop a fully mature internal financial management 
operation. While the MBN has used the Grant Thornton recommendations as 
a guide to improve MBN's internal controls, and plans to implement the 
remaining recommendations, it is important to note that these 
recommendations are not obligatory (i.e. not required by law or 
regulation). 

Developing an effective internal control program for MBN required 
prioritizing the Grant Thornton recommendations, implementing the 
fundamental and required components first, and building an effective 
internal control program from there. When reviewed chronologically, the 
results of the Grant Thornton review, the 2004 Single Audit, and the 
2005 Single Audit indicate that MBN has continually improved and 
strengthened its internal control program and at no time did its 
internal control program result in misstatements of its financial 
reports or financial status. Grant Thornton's recommendations were made 
less than one year after MBN had been established. As MBN has matured, 
the organization's environment and needs have changed. Initially, MBN 
procured and implemented an appropriate financial management system, 
hired skilled, knowledgeable staff, and established and documented 
policies and procedures. These were critical steps that required 
immediate attention and could materially affect MBN's ability to comply 
with relevant regulations and laws. 

GAO Recommendation: Develop a process for analyzing risk as part of 
strategic planning that identifies approaches to mitigate the potential 
obstacles to efficiently and effectively achieving MBN's operational 
objectives. 

MBN has been very aware since the start-up of Radio Sawa of risk 
factors that could adversely affect fulfillment of the Sawa/Alhurra 
mission. For example, Sawa's program of broadcasting redundancy - in 
which local FM transmission is backed up by cross-border, high-powered 
medium-wave service and further supported with digital audio via 
satellite - was a direct response to the threat of governments' 
silencing Sawa by interrupting local FM broadcasts. There are many 
similar examples. The lack of a written MBN strategic plan 
incorporating risk assessment is not to suggest, therefore, that such 
assessment did not take place. This said, MBN is committed to 
undertaking this analysis more formally. Indeed, it has already begun 
to do so in conjunction with the BBG's development of the Agency's 2008-
2013 strategic plan. MBN has now prepared a comprehensive 
organizational SWOT analysis (strengths, weaknesses, opportunities, and 
threats). It will now use this work to help drive implementation 
strategies that respond to the specific challenges Sawa and Alhurra 
face across the Middle East. 

GAO Recommendation: Require MBN to develop a comprehensive training 
program covering both internal controls and editorial procedures to 
meet the continuing needs of all employees. 

We appreciate the GAO's discussion of MBN's emphasis on strong 
journalistic standards and editorial procedures, as well as its 
recommendation that further training is needed at MBN. It is our goal 
to expand training options in a number of areas, and continued training 
on editorial procedures and controls is a requirement for any 
journalistic organization. 

One of the greatest challenges in setting up Alhurra Television was to 
recruit strong journalists with essential language skills and knowledge 
of the region under tight deadlines. We believe we have been successful 
in attracting substantial talent to our networks. This does not mean, 
however, that ongoing training and development is not necessary, and we 
are looking at ways to strengthen our competitive base by attracting 
and developing knowledgeable, professional employees. 

This commitment applies to training on internal controls as well. 
During the week of July 11, 2006 MBN completed a training program (as 
suggested by Grant Thornton) on internal controls for all managers 
responsible for making resource decisions. 

GAO Recommendation: Initiate a schedule of annual program reviews for 
Radio Sawa and Alhurra to regularly ensure that the quality of 
Alhurra's broadcasts conforms to applicable standards. 

The BBG agrees that annual program reviews should be regular and 
comprehensive. While we agree broadly with GAO's conclusions on this 
issue, we also feel that the overall discussion of the program review 
process in the report would benefit from clarifications that Alhurra is 
only two years old, and that Radio Sawa's first program review was 
designed in a manner to assist Sawa's specific operational needs. In 
some ways, Sawa's first program review was more exacting than the 
process followed by other BBG broadcasters. As the GAO report is 
currently drafted, the reader will have to comb through the GAO's 
appendix to learn that the BBG retained an outside expert to serve as 
consultant to Radio Sawa's program review in order to provide 
objectivity, and convened an unusually large number of listener panels 
in Iraq, Morocco, Egypt and Jordan, in order to gain more intensive 
feedback. A comprehensive program review of Radio Sawa and Alhurra TV 
is scheduled for later this year. 

GAO Recommendation: Implement program quality performance indicators 
for MBN's broadcast services consistent with other BBG entities to 
assess and compare their performance and measure the contribution of 
these efforts to the goals of the overall organization. 

Building on its first successful experience with monitoring panels in 
FY 2005, MBN in the future will conduct regular monitoring panels on 
Sawa and Alhurra which will permit computation of the program quality 
performance indicators consistent with those for other BBG entities. 
These may include regular listeners/viewers and external control 
listeners/viewers, where feasible, on program samples using content and 
presentation criteria. Professional, objective, in-house review of the 
material may also be used to provide additional information and 
perspective. Because MBN broadcasts to a very large number of Middle 
East countries and regions, the exact geographic and demographic make-
up of these panels will necessarily depend on available funding. 

GAO Recommendations: Require research contractors to improve the 
methods used in audience research to allow for probability sampling and 
document the sample selection so that survey sampling errors can be 
calculated, where possible; Identify and report significant 
methodological limitations and the implications of them on performance 
indicators, including where applicable, sampling errors, margins of 
error, or confidence intervals; Develop, document, and report policies 
and procedures for verification and analysis of performance indicator 
estimates. 

We concur with GAO that BBG research can be improved with respect to 
MBN and all of the BBG broadcast entities. However, we are concerned 
that the report ignores the many positive aspects of the BBG research 
program and displays a lack of understanding of field conditions and 
practical considerations that often require departures from "textbook" 
survey methods. 

We believe that our standards yield data of sufficient reliability to 
allow us to estimate our audiences and our credibility. Our contracts 
with ACNielsen and other well-recognized research firms who do business 
in the Middle East, follow widely-accepted industry standards for 
commercial and media research. These are the standards relied upon by 
government and business in the region and around the world. 

All surveys involve trade-offs between cost, time and field conditions 
on. the one hand, and reliability and precision on the other. Every 
organization conducting survey work has to consider what level of 
reliability and precision is required to meet its objectives, while 
taking into account available resources and difficult survey 
conditions. While recognizing that greater degrees of reliability are 
possible and desirable and that technical improvements may be made to 
the methodologies used thus far, it remains clear that MBN is reaching 
a very large audience. 

Since 2002, the BBG has commissioned 35 surveys conducted by respected 
audience research firms on behalf of MBN, involving more than 55,000 
respondents across 13 countries. While we agree with GAO's overall 
recommendations regarding the future of BBG research, we believe that 
the report misrepresents the integrity and professionalism of these 
surveys. In fact, we would argue that, in such key areas as interviewer 
training, field observation, and questionnaire design, the MBN surveys 
have been superior to the vast majority of comparable projects carried 
out in the region. In addition, the report downplays the very real 
difficulties of implementing statistically pure survey methods in 
countries such as Iraq, among others. 

Much of GAO's criticism is directed at the use of non-probability 
sampling. In the text of the draft report, GAO staff has defined 
probability samples as those in which "every person in the population 
has a measurable chance or probability of being selected." This is 
indeed the textbook definition. Application of "pure" probability 
sampling, however, requires perfect information regarding the 
population of sampled units at each stage of selection - conditions 
that rarely exist in practice, and certainly not in the countries 
targeted by MBN. In our experience, most samples advertised as having 
been drawn with "probability" designs in fact involve elements of both 
probability and non-probability methods. At the lower stages of 
selection, in particular, judgment or similar methods are often 
required in the face of non-existent population and geographic data. 

Security considerations can also play a role in the ability of 
contractors to implement perfect probability samples. D3 Systems, which 
does work in Iraq for the BBG and other USG agencies, notes that 
following GAO's recommendation to count all households at the block 
level so as to satisfy formal probability sampling methods would have 
jeopardized the lives of their interviewers. Interviewers working for 
other contractors in Iraq have been killed at checkpoints simply for 
doing their jobs. 

The contractor for most of the MBN studies - ACNielsen - specifically 
designed the samples to be as representative as possible of the survey 
universe in each country; typically by stratifying each country into 
socio-economic regions and then purposively selecting representative 
elements within each. As a senior ACNielsen executive confirmed in a 
letter to us and shared with GAO, they use this method for all of their 
clients in the region, and while survey methodologists can argue about 
the merits of such an approach, it does represent a good faith effort 
to obtain maximally representative samples at reasonable cost. 
Ultimately, the aim of these studies is not to produce academic work 
but to provide decision-makers with data they can use. Other 
international broadcasters - notably BBC and Deutsche Welle - appear to 
agree about the utility of the MBN surveys, as they have purchased the 
studies to provide guidance for their own broadcasting efforts in the 
region. 

The GAO also fails to note the significant differences between the 
various BBG surveys. First, in five of the countries studied - Saudi 
Arabia, Kuwait, Qatar, Bahrain, and UAE - probability sampling is not 
possible and is not, to our knowledge, practiced by any market research 
firm. To fault this research for use of non-probability sampling is to 
demand the impossible. In Jordan, following purposive selection of five 
governorates representing 76% of the country's population, 
straightforward probability techniques were used at the lower levels of 
selection. In Tunisia, I I out of the 24 governorates were purposively 
selected after stratification of the country to provide adequate 
representation of each region. In Lebanon, first stage selection 
covered all six of the country's six provinces, with probability 
techniques used for selection at subsequent stages. In Iraq, 
probability techniques were used throughout, though as noted earlier, 
lack of population data makes it impossible to calculate probability of 
selection at the block level. 

The standards GAO is using to evaluate BBG research are not fully 
transparent in the report. On the one hand, the report appears to 
recognize that deviations from textbook methods are necessary in some 
circumstances; on the other hand, its criticisms are so sweeping that 
it is impossible to know what trade-offs the GAO would consider 
"acceptable." The absence of clarification could easily lead the reader 
of the report to conclude that BBG research lacks significant 
professional standards. We believe that the methodologies used for the 
MBN studies yield data of sufficient reliability to allow us to 
estimate our audiences and our credibility. As noted earlier, ACNielsen 
and other well-recognized research firms that have conducted these 
studies for us followed the same widely accepted industry standards for 
us as they do for other commercial and media clients. 

Aside from the issue of non-probability sampling, the report criticizes 
BBG for lack of methodological detail in its reporting. In general, we 
believe the level of methodological detail we have provided in public 
documents is comparable to that typically used by other large research 
organizations for studies in the Islamic world. A comparison of 
disclosures from recent research reports indicates that the BBG is 
consistent with industry practice. As noted earlier, samples advertised 
as probability designs often include non-probability elements at 
various stages, though it is impossible to tell without examination of 
the detailed sampling plans. That said, we will ensure that 
descriptions of sample designs are included in future public documents 
reporting on MBN survey results. 

The GAO report maintains that the BBG has not attempted to measure 
sampling error for its surveys, and that BBG has only a rough idea of 
what the margins of error might be. This is untrue. BBG has indeed 
calculated sampling errors for the MBN surveys. GAO concerns have 
centered on the fact that we have used assumptions of simple random 
sampling (SRS) when making such calculations, rather than incorporating 
design effects, which typically have the effect of increasing standard 
errors. While this is a valid concern, we note that the use of design 
effects when calculating sampling error has never been practiced in the 
international broadcasting community. Moreover, such revised 
calculations of sampling error would have no effect on the point 
estimates from which audience size in each country is determined and 
hence would not affect the total estimate of the MBN audience. 

Finally, we would note, in attempting to minimize sampling error, a key 
factor is sample size, and in that respect, BBG's surveys have been 
superior to those conducted by other leading research organizations in 
the region. The following table shows the sample sizes for the recent 
polls undertaken in the Islamic world by Pew, Zogby, and Gallup, next 
to those commissioned by BBG. 

Table: 

Egypt; 
Pew, 2006[A]: 1000; 
Pew, 2005[B]: [Empty]; 
Zogby/Telhami, 2005[C]: 800; 
Gallup, 2002[D]: [Empty]; 
BBG 2005: 2101. 

Jordan; 
Pew, 2006[A]: 1000; 
Pew, 2005[B]: 1000; 
Zogby/Telhami, 2005[C]: 500; 
Gallup, 2002[D]: 797; 
BBG 2005: 1500. 

Lebanon; 
Pew, 2006[A]: [Empty]; 
Pew, 2005[B]: 1000; 
Zogby/Telhami, 2005[C]: 500; 
Gallup, 2002[D]: 1050; 
BBG 2005: 1507. 

Morocco; 
Pew, 2006[A]: [Empty]; 
Pew, 2005[B]: 1000; 
Zogby/Telhami, 2005[C]: 800; 
Gallup, 2002[D]: 1000; 
BBG 2005: 2000. 

Saudi Arabia; 
Pew, 2006[A]: [Empty]; 
Pew, 2005[B]: [Empty]; 
Zogby/Telhami, 2005[C]: 800; 
Gallup, 2002[D]: 754; 
BBG 2005: 2001. 

UAE; 
Pew, 2006[A]: [Empty]; 
Pew, 2005[B]: [Empty]; 
Zogby/Telhami, 2005[C]: 217; 
Gallup, 2002[D]: [Empty]; 
BBG 2005: 1202. 

Kuwait; 
Pew, 2006[A]: [Empty]; 
Pew, 2005[B]: [Empty]; 
Zogby/Telhami, 2005[C]: [Empty]; 
Gallup, 2002[D]: 790; 
BBG 2005: 1506. 

Iraq; 
Pew, 2006[A]: [Empty]; 
Pew, 2005[B]: [Empty]; 
Zogby/Telhami, 2005[C]: [Empty]; 
Gallup, 2002[D]: [Empty]; 
BBG 2005: 2500. 

Syria; 
Pew, 2006[A]: [Empty]; 
Pew, 2005[B]: [Empty]; 
Zogby/Telhami, 2005[C]: [Empty]; 
Gallup, 2002[D]: [Empty]; 
BBG 2005: 1516. 

Tunisia; 
Pew, 2006[A]: [Empty]; 
Pew, 2005[B]: [Empty]; 
Zogby/Telhami, 2005[C]: [Empty]; 
Gallup, 2002[D]: [Empty]; 
BBG 2005: 1501. 

Bahrain; 
Pew, 2006[A]: [Empty]; 
Pew, 2005[B]: [Empty]; 
Zogby/Telhami, 2005[C]: [Empty]; 
Gallup, 2002[D]: [Empty]; 
BBG 2005: 1200. 

[A] The Great Divide: How Westerners and Muslim View Each Other" (Pew 
Global Attitudes Project, June 2006) 

[B] "Islamic Extremism: Common Concern for Muslim and Western Publics" 
(Pew Global Attitudes Project, April=May, 2005): 

[C] "Arab Attitudes Towards Political and Social Issues" (Professor 
Shibley Telhami and Zogby Int'l, October, 2005) 

[D] "Gallup Poll of the Islamic World", (Gallup International, 
December, 2002): 

[End of table] 

As shown, sample sizes in every BBG survey exceeded those of other 
leading survey organizations. All other things being equal, the BBG 
surveys would therefore yields margins of error 20 - 30% lower than 
those for surveys by the other organizations. 

We acknowledge and regret that the contractors for these projects were 
not able to provide all requested documentation within the time frame 
requested by GAO. Some of this delay was due to the fact that the 
ACNielsen project manager was hospitalized at the time of, and for 
several weeks after, the original request, in addition to which the 
required information had to be gathered from each of the ACNielsen 
field offices in the countries where the survey work was conducted. 
Subsequent to GAO's deadline, we did receive additional documentation 
regarding sampling methods, which was duly passed on to GAO. While not 
as complete as we would have liked, we believe that in most instances 
the totality of documentation supplied thus far provides a reasonable 
basis on which to assess the survey results. 

In sum, we fully accept GAO's recommendation to require research 
contractors to improve the methods used in audience research to allow 
for probability sampling and document the sample selection so that 
survey sampling errors can be calculated, where possible. This 
acknowledges that full probability sampling is not possible everywhere. 
It also allows the BBG/MBN research program in the Middle East to work 
and deliver the best results possible given the circumstances. 
Insisting on full probability sampling, with no exceptions, would 
nullify the program. 

Again, the BBG appreciates the opportunity to provide detailed comments 
on the GAO's recommendations. Although we may disagree regarding some 
of the GAO's analysis and conclusions, we are in general agreement with 
GAO's recommendations, and look forward to implementing them. 

Sincerely, 

Signed by: 

Kenneth Y. Tomlinson: 
Chairman: 

The following are GAO's comments on the Broadcasting Board of 
Governor's (BBG) letter dated July 20, 2006. 

GAO Comments: 

1. GAO acknowledges in the report that MBN has participated in long- 
range planning in coordination with the BBG's update of its 2008-2012 
strategic plan, and that MBN has developed a 2006 annual performance 
plan as part of the OMB Program Assessment Rating Tool process. 
However, contrary to the BBG's comments, the fiscal year 2006 
performance plan document for MBN that we obtained did not contain all 
of the elements of a stand-alone strategic plan. In addition, we were 
told by the former executive director of the BBG, who is currently the 
president of MBN, that the Radio Sawa strategic plan drafted in 2002 by 
the BBG-which the BBG stated in its comments contributed to its 
considerable long-term planning for MBN--was never adopted by the BBG. 
We were also told by MBN's former president that he had never received 
a copy of the draft Radio Sawa strategic plan. 

2. The BBG stated that MBN's Single Audit evaluates the effectiveness 
of its internal control program. However, while MBN's Single Audits 
address the fair presentation of the organization's financial 
statements, its compliance with certain laws and regulations, and the 
organization's internal control as it relates to financial reporting, 
MBN's 2005 Single Audit as prepared by PriceWaterhouseCoopers does not- 
contrary to what the BBG stated in its comments--offer an opinion on 
the overall effectiveness of MBN's internal control. Thus, the 
unqualified opinion that MBN received on its fiscal year 2005 Single 
Audit does not provide a broad assessment or opinion of MBN's internal 
control system. For the purposes of our report, it was therefore 
necessary to consider other assessments of MBN's financial and 
administrative controls, including the Grant Thornton report. 

3. MBN's grant agreement states that MBN should make every reasonable 
effort to achieve the purpose of the grant in accordance with OMB 
Circulars A-110, A-122, and A-133. Our report did not attempt to make 
an assessment of MBN's compliance with its grant agreement, but rather 
focused on the progress MBN has made in developing its system of 
internal control and the ways in which MBN can continue to strengthen 
this system. To that end, we drew upon the work of Grant Thornton LLP, 
which assessed the status of MBN's controls against relevant OMB 
Circulars and GAO guidance on internal control. 

4. While the Grant Thornton recommendations are not obligatory, they 
are based on best management practices from OMB and GAO. According to 
officials, both the BBG and MBN accepted the results of the report, 
which provides detailed insight into MBN's financial operations, when 
it was completed in May 2004. Adopting the report's remaining 
recommendations would help MBN in its efforts to build a fully mature 
internal financial management operation. 

5. The BBG stated that it believes its standards yield data of 
sufficient reliability to allow it to estimate MBN's audiences and 
credibility. However, the BBG offers no evidence for this. BBG's 
estimates for MBN are based on judgment and not statistics, and its 
current methods do not and cannot estimate the error in its estimates. 
As noted in the report, in several instances the BBG and its 
contractors departed from CIBAR research standards. For example, we 
analyzed 12 of the 14 country surveys conducted by BBG contractors for 
2005. In the BBG's fiscal year 2007 budget request, the BBG reported 
the results of each country survey and an overall estimate for audience 
size of 21.5 for Alhurra and 21.6 million for Radio Sawa. However, our 
analysis of the 12 surveys identified a number of methodological 
weaknesses, including BBG's failure to fully document research methods, 
measure the level of uncertainty surrounding its estimates, disclose 
significant limitations, limit the scope of its projections to areas 
actually covered by its surveys, and develop and consistently implement 
policies and procedures for verifying data. These limitations were not 
reported in BBG's audience survey estimates and prevent us from 
concluding that the estimates are accurate and reliable. 

6. We acknowledge in the report that there are challenges to conducting 
audience research in the Middle East, and that there are tradeoffs 
between cost and data reliability when conducting research. 

7. As we noted in the report, it is difficult to conduct probability 
sampling in many locations in the Middle East. But, it is not 
impossible. Organizations, including the United States Census Bureau, 
have collaborated on probability surveys in the Middle East, including 
in Saudi Arabia and Jordan. The major problem of the BBG estimates is 
their lack of transparency, and the lack of an explanation for the 
methodology behind BBG's estimates. 

8. In conducting our data reliability assessment, as referenced in the 
report, we largely used the international audience research guidelines 
published by CIBAR. The BBG participated in drafting these standards, 
which are specified as a source of guidance for BBG research 
contractors. CIBAR standards recommend that proper care be exercised 
when reporting estimates to ensure that the type of audience covered by 
the estimate is clearly stated, and that, at all stages in the 
calculation and extrapolation process, sources, assumptions, and 
methods are fully documented and available for independent scrutiny. As 
noted in the report, in several instances the BBG and its contractors 
departed from CIBAR research standards, leading to weaknesses in MBN's 
methodology and documentation. As a result, we were unable to determine 
the accuracy of MBN's fiscal year 2005 audience size and program 
credibility estimates. 

9. The BBG stated that it believes the level of methodological detail 
it has provided in public documents is comparable with that typically 
used by other research organizations for studies in the Islamic world; 
therefore, it believes it is consistent with industry practice. 
However, the Reports Consolidation Act of 2000[Footnote 59] requires 
federal agencies, such as the BBG, to assess the completeness and 
reliability of the performance data in their performance reports and to 
discuss any material inadequacies in the completeness and reliability 
of their performance data, as well as actions to address the 
inadequacies. In performance reporting such as the BBG fiscal year 2007 
budget request, the BBG reported the results of its research for MBN as 
21.5 million viewers for Alhurra and 21.6 million for Radio Sawa. 
However, the BBG did not sufficiently explain the specific methods used 
for generating estimates for its performance indicators, such as the 
number and names of the countries surveyed (including the sizes of its 
samples), methods of sampling used, sources of the population data, and 
basic procedures used to create the estimates. Moreover, the BBG 
reported only two limitations for MBN performance indicators to date: 
(1) that credibility ratings are highly dependent on volatile political 
factors; and (2) that, depending on political, social, and media 
conditions, measurement of audience size may either be easily attained 
or impossible. Significant limitations not reported are that many of 
the BBG surveys are not based on probability sampling, cover only part 
of the country, have very low response rates, or have high substitution 
rates. Therefore, the BBG does not provide a full level of confidence 
in the credibility of its performance data for MBN. In contrast, the 
Department of Transportation, has a separate compendiums available 
online that provides source and accuracy statements, which provide 
detail on the methods used to collect performance data, sources of 
variation and bias in the data, and methods used to verify and validate 
the data. 

10. The BBG cannot calculate the sampling error for the BBG estimates 
of audience size and credibility because the probability of selection 
is not known. Although increased sample size will generally decrease 
sampling error, it is impossible to accurately estimate the sampling 
error of the BBG surveys because they are nonprobability surveys. As 
the BBG has stated, its use of a simple random sampling formula when 
calculating sampling errors for its surveys underestimates the sampling 
error. This formula is not appropriate for the sample designs used by 
MBN. 

11. Although our recommendation asks the BBG to require research 
contractors to improve the methods used in audience research to allow 
for probability sampling and document the sample selection so that 
survey sampling errors can be calculated, where possible, it is the 
responsibility of the BBG to explain and justify the need to conduct 
nonprobability samples. 

[End of section] 

Appendix VI: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Jess Ford, (202) 512-4268: 

Staff Acknowledgments: 

Diana Glod, Melissa Pickworth, Eve Weisberg, Dorian Herring, and Joe 
Carney made key contributions to this report. Chanetta Reed, Jay Smale, 
Karen O'Conor, and Jackie Nowicki provided technical assistance. 

(320376): 

FOOTNOTES 

[1] The Alhurra networks include Alhurra and Alhurra-Iraq. 

[2] GAO, Internal Control Management and Evaluation Tool, GAO-01-1008G 
(Washington, D.C.: August 2001). 

[3] Internal control is a major part of managing any organization and 
comprises the plans, methods, and procedures used to meet missions, 
goals, and objectives. Effective internal control helps in managing 
change to cope with shifting environments and evolving demands and 
priorities. 

[4] The BBG's reported fiscal year 2005 audience size performance 
target for Radio Sawa was 18 million listeners and for Alhurra 12.8 
million viewers. In addition, the BBG's reported program credibility 
target for Radio Sawa was 69 percent and for Alhurra 73 percent. 

[5] P.L. 108-11. 

[6] P.L.108-106. 

[7] $2.1 million of this amount was appropriated to support the start- 
up of Alhurra Europe. The administration proposed this effort, which 
would provide Arabic-speaking audiences in Europe with balanced and 
accurate news and information, as part of its budget request for the 
Fiscal Year 2005 Supplemental. MBN officials told us that they intend 
Alhurra Europe's 24-hour a day broadcast to include a 4-hour repeating 
cycle of Alhurra's and Alhurra-Iraq's original current affairs programs 
and other shows, with fresh news every hour. According to MBN 
officials, the channel will be broadcast on satellite and eventually on 
cable. 

[8] The 19 countries and territories are Algeria, Bahrain, Djibouti, 
Egypt, Iraq, Jordan, Kuwait, Lebanon, Libya, Morocco, Oman, Qatar, 
Saudi Arabia, Sudan, Syria, Tunisia, United Arab Emirates, Yemen, and 
Palestine. 

[9] MBN also has an overseas production facility that develops some 
Alhurra current affairs programs in Beirut, Lebanon. 

[10] MBN broadcasts additional newscasts and current affairs programs 
dealing with the challenges facing postwar Iraq on the Iraq stream. In 
addition, the Sawa Chat call-in show is currently localized for the 
Iraq, Gulf, Jordan, and Morocco streams. 

[11] This figure can vary weekly depending on events. 

[12] Alhurra transmitters are located in Baghdad, Basra, and Mosul. 

[13] In February 2004, Alhurra began broadcasting on the Arabsat and 
Nilesat satellites, which are also used by the major regional Arabic 
channels. Arabsat and Nilesat cover the entire Middle East. Their 
footprints extend from well into Northern Europe to as far south as 
Angola and Mozambique in Africa. From west to east, the footprints 
extend from the Atlantic Ocean to southern Pakistan. In the future, in 
order to serve the estimated 10 million Arabic speakers in Europe, the 
Alhurra channel will be distributed via the Hotbird satellite, which is 
popular in Europe. Its footprint spans across Europe and reaches well 
into Russia at its eastern edge. Hotbird satellite coverage also 
extends to the Middle East and North Africa. 

[14] Al Jazeera claims it reaches over 45 million viewers in the Middle 
East. However, based on lack of additional information about the 
measure and how the estimates for the figure were developed by Al 
Jazeera, we could not determine this figure's reliability. 

[15] Additional proposals MBN submitted internally to BBG included 
expanding purchases of acquired programming, as well as in-house 
production of town hall debates and Arabic-language documentaries to 
provide greater variety of current affairs content of regional interest 
to viewers. 

[16] MBN officials have noted that, according to survey research and 
evaluative interviews conducted in the Middle East, Alhurra's audience 
does not currently view it as a top source of news and information. MBN 
officials also noted that their assessments have found that Alhurra 
does not have the same level of credibility as Al Jazeera. 

[17] The streams identified for this initiative include Morocco, 
Jordan/West Bank/Syria, Egypt, Gulf/Saudi Arabia, and Sudan. 

[18] GAO-01-1008G. 

[19] With the exception of the Baghdad bureau, MBN has not developed an 
approach for maximizing the use of its existing overseas bureaus and 
offices for Alhurra's news operations. MBN's Dubai production center, 
for example, currently only produces a few of Alhurra's current affairs 
shows. 

[20] Some MBN officials have expressed concern that the BBC's favorable 
programming library and licensing arrangements could potentially 
interfere with Alhurra's ability to secure critical acquired 
programming content. A significant proportion (or 35 percent) of 
Alhurra's content is acquired programming. 

[21] MBN is required to undergo a Single Audit annually to comply with 
OMB Circular No. A-133, which is included as a requirement in MBN's 
grant agreement with the BBG. All nonfederal entities that expend 
$500,000 or more of federal awards in a year are required to obtain an 
annual audit in accordance with the Single Audit Act Amendments of 
1996. The Single Audit provides information on whether MBN's financial 
statements fairly present the organization's financial situation; 
whether MBN is maintaining internal control over its federal program; 
whether MBN is complying with relevant laws, regulations, and 
provisions of its grant agreement; whether there are any audit findings 
or questioned costs; and whether corrective action plans have been 
established. The A-133 audit has several components, including (1) an 
opinion on the organization's financial statements; (2) a report on 
internal control; (3) a report on compliance with laws and regulations, 
and the provisions of the grant agreement; and (4) a schedule of 
findings and questioned costs. It also requires a corrective action 
plan that describes how the auditee will address findings. 

[22] The control environment includes the tone that an organization's 
management and employees set toward internal control and conscientious 
management. 

[23] MBN's grant agreement requires compliance with the following OMB 
circulars: Circular No. A-110, Uniform Administrative Requirements for 
Grants and Agreements with Institutions of Higher Education, Hospitals 
and Other Non-Profit Organizations, which covers administrative 
requirements for grants to nonprofit organizations; OMB Circular No. A- 
122, Cost Principles for Non-Profit Organizations, which covers cost 
principles for nonprofit organizations; and OMB Circular No. A-133, 
Audits of States, Local Governments, and Non-Profit Organizations, 
which covers audits of states, local governments, and nonprofit 
organizations. 

[24] GAO, Standards for Internal Control in the Federal Government,GAO/ 
AIMD-00-21.3.1 (Washington, D.C.: November 1999). 

[25] Grant Thornton LLP, Status of Funds and Internal Control Report 
with Analysis and Recommendations for Middle East Television Network 
(MTN), May 28, 2004. 

[26] MBN's Fiscal Year 2003 and 2004 Single Audits were due June 2004 
and 2005, respectively. MBN officials stated that the auditors were not 
able to provide us with the finalized audits until May 2006 because 
they were not yet completed due to staffing and time constraints. OMB 
Circular No. A-133, Subpart C, .320, notes that the audit shall be 
completed and submitted within the earlier of 30 days after receipt of 
the auditor's report (s), or 9 months after the end of the audit 
period, unless a longer period is agreed to in advance by the federal 
agency that provided the funding. 

[27] In particular, (1) the auditors provided an opinion that MBN's 
fiscal year 2005 financial statements fairly presented MBN's financial 
position; (2) in its consideration of MBN's internal control over 
financial reporting, the auditor did not find any material weaknesses; 
(3) the auditor's tests did not reveal any instances of noncompliance 
with relevant laws, regulations, contracts, and grant agreements, nor 
did it find any material weaknesses in internal control over compliance 
with regard to the controls that they tested; and (4) the auditor found 
that all of the findings raised in the Fiscal Year 2004 Single Audit 
had been addressed. These findings included undocumented policies and 
procedures; inadequate training; improperly recorded, accounted, and 
executed transactions; a lack of assurance that assets were safeguarded 
against loss; a lack of periodic, formal account reconciliation and 
analysis; a failure to review entries in MBN's financial information 
system; lack of control over payroll; and lack of compliance with 
equipment and property management requirements. 

[28] GAO-01-1008G. 

[29] GAO-01-1008G. 

[30] The BBG Middle East Committee is authorized to approve all actions 
or expenditures requiring the approval of MBN's board of directors 
between board of directors' meetings. 

[31] GAO-01-1008G. 

[32] P.L. 103-236. 

[33] The exception is the Dubai office, which produces 8 hours of Radio 
Sawa programming during the daytime hours in the Middle East. 

[34] These program quality criteria are used in program reviews of VOA 
and RFE/RL broadcasters, while RFA applies broadly similar criteria. 
The Radio Sawa program review applied them as well, with the exception 
of "length of individual items," among others. The Radio Sawa audience 
panels also used music quality criteria, including selection of music, 
variety of music, uniqueness of music, balance in types of music, 
length of individual songs, transitions between songs, transitions 
between music and other segments, and overall appeal of music. 

[35] Internal reviewers, who measure a service's content and production 
standards against its journalistic standards and the program quality 
criteria, tend to be International Broadcasting Bureau program analysts 
or grantee staff. External reviewers include audience monitoring panels 
consisting of a number of regular listeners or viewers, as well as 
control listeners. Control listeners are language-qualified and are 
knowledgeable about target country politics and culture, but are 
usually located in the United States or other Western countries and are 
not affected by politics on the ground. 

[36] The internal analysts, audience monitoring panels, and external 
control listeners develop their own program quality scores, which feed 
into the BBG standard program quality score, a performance indicator. 

[37] In 2000, in response to our report that found a lack of 
consistency in how program quality scores were developed, the BBG 
stated that it intended to harmonize and standardize program reviews 
across broadcasting entities. See U.S. International Broadcasting: 
Strategic Planning and Performance Management System Could Be Improved, 
GAO/NSIAD-00-222 (Washington, D.C.: Sept. 27, 2000). 

[38] The BBG sets performance indicators and targets for Alhurra and 
Radio Sawa separately, rather than for the MBN organization as a whole. 

[39] The U.S. International Broadcasting Act of 1994 requires that the 
BBG ensure that the quality of the broadcasts conforms to applicable 
standards and reaches significant audiences. 

[40] The performance indicators for Alhurra include Alhurra-Iraq. 

[41] The BBG's reported program credibility target for fiscal year 2005 
for Radio Sawa was 69 percent and for Alhurra was 73 percent. 

[42] We have previously recommended that the BBG measure awareness and 
reported that the information necessary to generate the measure of 
audience awareness is already collected in national surveys. See GAO, 
U.S. International Broadcasting: New Strategic Approach Focuses on 
Reaching Large Audiences but Lacks Measurable Program Objectives, GAO- 
03-772 (Washington, D.C.: July 15, 2003). 

[43] GAO, The Results Act: An Evaluator's Guide to Assessing Agency 
Annual Performance Plans, GAO/GGD-10.1.20 (Washington, D.C.: April 
1998). 

[44] GAO, Agencies' Annual Performance Plans Under the Results Act: an 
Assessment Guide to Facilitate Congressional Decisionmaking, GAO/GGD/ 
AIMD-10.1.18 (Washington, D.C.: February 1998). 

[45] Throwaway surveys are uncounted surveys used in interviews with 
respondents as a training mechanism for the interviewer. The 
respondents are those individuals who were in the population that could 
have been selected, but were not included in the sample. 

[46] Probability surveys by definition mean that, due to the design of 
the survey, every person in the population has a measurable chance, or 
probability, of being selected and that, subsequently, the survey may 
be projected to the broader population with known levels of precision. 

[47] ACNielsen typically used sampling techniques where several stages 
of sampling are conducted. In the case of Egypt and Morocco, 
interviewers used their judgment for allocating starting points--the 
first stage of sampling selection. 

[48] BBG officials acknowledged that social and cultural factors in 
Saudi Arabia, Kuwait, Bahrain, UAE, and Qatar, in particular, place 
constraints on the ability of any research contractor to implement 
probability sampling. 

[49] Country surveys with no available documentation included Algeria 
and Djibouti, which used data purchased by the BBG from a French 
commercial source. 

[50] We obtained some technical documentation from Intermedia's 
subcontractor ACNielsen for the following 11 country surveys: Bahrain, 
Egypt, Lebanon, Qatar, Saudi Arabia, Tunisia, Morocco, Jordan, Syria, 
UAE, and Kuwait. D3 Systems provided us with the requested 
documentation for its research subcontract for Iraq. 

[51] Sampling errors do not reflect the total error of a survey 
statistic or estimate, which is a function of many different features 
of a given survey. 

[52] The margin of error is a measure, set at a specific confidence 
level, of the possible difference between the sample estimate and the 
actual population value. 

[53] Officials from the BBG's research contractor Intermedia said that 
they have used the approach of calculating a rough margin of error of 
+/-2.6 percent for all the Middle Eastern surveys. 

[54] CIBAR standards recommend that the proper care be exercised when 
reporting estimates, to ensure that the type of audience covered by the 
estimate is clearly stated and, at all stages in the calculation and 
extrapolation process, that sources assumptions and methods be fully 
documented and available for independent scrutiny. 

[55] In February 2006, Radio Sawa launched a seventh stream aimed at 
Lebanon, which was not included in the programming schedule that we 
received. 

[56] The panelists reported that they liked Radio Sawa's mixture of 
music and news, generally viewed the news as balanced and credible, and 
felt the station was culturally sensitive. In particular, panelists 
praised Sawa Chat for posing thought-provoking questions and providing 
interesting discussion. Panelists also found that some of the 
transitions between news and music were too abrupt; that there should 
be less repetition of songs; and that news should be more tailored to 
specific countries, while including more cultural and other "soft" 
news. 

[57] The experts included staff from InterMedia, RFE/RL, and RFA. 

[58] CIBAR, Harmonized Audience Research Measurement for International 
Broadcasting (3rd edition, 2001). 

[59] P.L. 106-531. 

GAO's Mission: 

The Government Accountability Office, the investigative arm of 
Congress, exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO's commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics. 

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as "Today's Reports," on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order 
GAO Products" heading. 

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office 

441 G Street NW, Room LM 

Washington, D.C. 20548: 

To order by Phone: 

Voice: (202) 512-6000: 

TDD: (202) 512-2537: 

Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm 

E-mail: fraudnet@gao.gov 

Automated answering system: (800) 424-5454 or (202) 512-7470: 

Public Affairs: 

Jeff Nelligan, managing director, 

NelliganJ@gao.gov 

(202) 512-4800 

U.S. Government Accountability Office, 

441 G Street NW, Room 7149 

Washington, D.C. 20548: