Analysis of Methodologies in Reports to the Secretaries of Defense and the Army
NSIAD-98-125: Published: Mar 16, 1998. Publicly Released: Mar 16, 1998.
GAO reviewed three studies on gender-related issues affecting initial entry training in the Department of Defense (DOD), focusing on: (1) how the groups conducted their work; (2) how well the work supported making conclusions and recommendations; (3) the availability of documentation supporting the report; and (4) the extent to which the final report described the study methodology and disclosed limitations.
GAO noted that: (1) the Army's Senior Review Panel on Sexual harassment used four methods to collect data: individual interviews, focus groups, surveys, and observations; (2) during its 8 months of work, the panel visited 59 installations worldwide, conducted interviews with 808 military and civilian Army personnel, ran focus groups with over 8,000 soldiers and civilians, and surveyed 22,952 individuals; (3) the use of multiple methods of data gathering, the rigor with which the various methods were conducted, and the publication of the data in the report provides ample support for making conclusions and recommendations; (4) the Federal Advisory Committee (FAC) on Gender-Integrated Training and Related Issues used focus groups as its primary method of data gathering; (5) although FAC conducted over 300 focus groups and individual interviews, their value for making conclusions and recommendations is limited because the Committee did not: (a) systematically collect the same information from all groups; (b) document the information generated in each of the interviews and focus groups; or (c) explain how what was heard in the interviews and focus groups led to their conclusions and recommendations; (6) in addition, the length of the focus group sessions, the number of participants, and the number of questions addressed may not have provided adequate time for full participation of the respondents on all issues; (7) given these limitations, the extent to which the Committee's work supports its conclusions and recommendations cannot be determined; (8) the Defense Advisory Committee on Women in the Services (DACOWITS) also used focus groups of trainees, trainers, and supervisors in the Army, Air Force, Navy, Marine Corps, and Coast Guard to identify what issues concerned women and men at training installations; (9) members of the DACOWITS held focus group discussions at 12 schools at 9 installations in the United States and prepared a summary report of the results at each installation; (10) the DACOWITS Chair used these to prepare a report to the Secretary of Defense that accurately reflected the opinions and perceptions cited in the individual installation reports; (11) the DACOWITS focus groups were: (a) larger than recommended in the literature; (b) were sometimes not long enough to allow meaningful participation; and (c) were not recorded or documented on a group-by-group basis; and (12) the DACOWITS report summarized the opinion and perception data obtained from focus groups; and (13) it made no conclusions or recommendations on military training based on that information.