This is the accessible text file for GAO report number GAO-13-306 entitled 'Consumer Product Safety Commission: Awareness, Use, and Usefulness of SaferProducts.gov' which was released on March 11, 2013. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States Government Accountability Office: GAO: Report to Congressional Committees: March 2013: Consumer Product Safety Commission: Awareness, Use, and Usefulness of SaferProducts.gov: GAO-13-306: GAO Highlights: Highlights of GAO-13-306, a report to congressional committees. Why GAO Did This Study: In the wake of increased product recalls in 2007-2008, Congress passed the Consumer Product Safety Improvement Act of 2008 (CPSIA). Among other things, CPSIA required CPSC to establish a database on the safety of consumer products that is publicly available, searchable, and accessible through the CPSC website. In response, CPSC launched SaferProducts.gov [hyperlink, http://www.saferproducts.gov] in March 2011, which has two main functions—to provide (1) a mechanism for online reporting of product safety issues and (2) the ability to search for these issues or others, such as recalls. CPSIA also required GAO to study the general utility of the website. This report examines (1) CPSC’s efforts to inform the public about SaferProducts.gov, (2) who is using the website and to what extent, and (3) the extent to which consumers have found the website to be useful. To do this, GAO analyzed agency documents and data from 2011 to 2012; interviewed CPSC officials, researchers, and consumer and industry groups; reviewed federal standards, guidance, and best practices for website usability; and conducted website usability tests with 37 consumers in three locations. What GAO Found: The Consumer Product Safety Commission (CPSC) has used various approaches to inform the public about SaferProducts.gov, including using social media, public service announcements, and printed materials, and promoting the site during speeches and events. CPSC’s efforts to inform the public about SaferProducts.gov have been part of a larger effort to raise awareness about the agency as a whole. While CPSC has employed many key practices for consumer education planning, it has not established metrics for measuring the success of its efforts. Without such metrics, the agency cannot determine which efforts have had the most impact on increasing awareness and use of the site. While CPSC collects some data on the category of persons, such as consumers or health care professionals, who submit reports (one of the main functions of the site), it does not collect data about who is using the site to search for information (the other main function). In addition, to minimize the reporting burden on users, CPSC has not asked for demographic data about the users (such as their age, gender, or income level). Therefore, it was difficult for GAO to assess, as mandated by Congress, whether a broad range of the public has used the site. Moreover, without such data, CPSC has been limited in its ability to target its marketing and outreach efforts to increase use of the site. Many consumers in GAO’s usability tests thought the site generally was easy to use and had helpful information, but identified areas for improvement. The consumers generally could perform basic searches and follow instructions to report an unsafe product, and although none were aware of the site before the tests, most said they would use the site again. However, some of the search functions posed challenges. In addition, some consumers expressed concern about registering with the site and said this might prevent them from completing a report. Other consumers were not clear about the site’s purpose, thinking it would focus on safe rather than unsafe products. By addressing the usability challenges GAO identified, CPSC could help users take full advantage of all the available features of SaferProducts.gov. Furthermore, cost- effective federal resources exist across the government to help agencies improve the usefulness of their sites. [Figure: Refer to PDF for image: web page illustration] Source: SaferProducts.gov. [End of figure] What GAO Recommends: CPSC should (1) establish and incorporate metrics to assess efforts to increase awareness and use of SaferProducts.gov, (2) look for cost- effective ways of gathering additional data about site use, and (3) implement cost-effective usability improvements to the site. CPSC supported these recommendations. View [hyperlink, http://www.gao.gov/products/GAO-13-306]. For more information, contact Alicia Puente Cackley at (202) 512-8678 or cackleya@gao.gov. [End of section] Contents: Letter: Background: CPSC Has Used Various Approaches to Inform the Public about SaferProducts.gov, but Has Not Established Metrics to Assess Its Efforts: CPSC Collects Limited Data about the Use of SaferProducts.gov: Consumer Testing Showed SaferProducts.gov to Be Generally Useful, but Identified Areas for Improvement: Conclusions: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendix I: Objectives, Scope, and Methodology: Appendix II: GAO Script, Including Questions and Tasks, for Usability Testing with Consumers: Appendix III: Comments from the U.S. Consumer Product Safety Commission: Appendix IV: GAO Contact and Staff Acknowledgments: Tables: Table 1: Types of Submitters of Incident Reports, for Reports Published from April 2011 through January 3, 2013: Table 2: Relationship between Self-Reported Consumer and Victim, for Incident Reports Published from April 2011 through January 3, 2013: Table 3: Relationship between Self-Reported Consumer and Victim for Those Specifying a Relationship, for Incident Reports Published from April 2011 through January 3, 2013: Table 4: Location of Reported Incident by Country, as of January 3, 2013: Figures: Figure 1: Consumer Product Safety Commission's (CPSC) Electronic Process for Publishing an Incident Report Submitted on SaferProducts.gov: Figure 2: Monthly Visits, from March 2011 through December 2012: Figure 3: Most Frequently Visited Pages Each Month, from March 2011 through December 2012: Figure 4: Number of Incident Reports Received, from March 2011 through December 2012: Figure 5: Total Number of Incident Reports Published by State of Incident, from April 2011 through January 3, 2013: Figure 6: Screenshot of Registration Page for Report Submission on SaferProducts.gov: Figure 7: Screenshot of SaferProducts.gov Home Page: Abbreviations: CPSRMS: Consumer Product Safety Risk Management System: CPSC: Consumer Product Safety Commission: CPSIA: Consumer Product Safety Improvement Act of 2008: GSA: General Services Administration: HHS: Department of Health and Human Services: PSA: public service announcement: [End of section] United States Government Accountability Office: Washington, DC 20548: March 11, 2013: The Honorable John D. Rockefeller IV: Chairman: The Honorable John Thune: Ranking Member: Committee on Commerce, Science, and Transportation: United States Senate: The Honorable Fred Upton: Chairman: The Honorable Henry A. Waxman: Ranking Member: Committee on Energy and Commerce: House of Representatives: A large number of children's product recalls in 2007 and 2008 led to heightened scrutiny of the Consumer Product Safety Commission's (CPSC) regulation of consumer products and public access to product safety information. In response to these concerns, Congress enacted the Consumer Product Safety Improvement Act (CPSIA) on August 14, 2008, to strengthen CPSC's authority to enforce safety standards and provide greater public access to product safety information.[Footnote 1] CPSIA required CPSC to establish a publicly available, searchable database of consumer and other products and substances the Commission regulates that are reported to be unsafe.[Footnote 2] On March 11, 2011, CPSC launched the mandated database at SaferProducts.gov [hyperlink, http://www.saferproducts.gov]. The website has two portals--one for consumers and one for businesses. For the consumer portal, website users can perform two main functions on the site. First, consumers and members of other statutorily defined groups can submit reports of harm or the risk of harm from products. [Footnote 3] Second, consumers and others can search for information on products reported to be unsafe that they own or may wish to purchase. SaferProducts.gov also shows manufacturers' comments alongside incident reports if the manufacturer requests their comments be published. The act also requires that GAO analyze the utility of the website, including an assessment of its use by consumers and efforts by CPSC to inform the public about the database.[Footnote 4] In this report, we examine (1) CPSC's efforts to inform the public about SaferProducts.gov, (2) who has used SaferProducts.gov and to what extent, and (3) the extent to which consumers have found SaferProducts.gov to be useful.[Footnote 5] To address the first objective, we reviewed CPSC marketing materials and budget, evaluation, and planning documents, and interviewed CPSC officials to determine the status of the agency's efforts for SaferProducts.gov. This included reviewing the targets of CPSC's outreach, how the agency evaluated the outcomes of these efforts, and any future plans. We compared CPSC's efforts with key practices for consumer education planning identified in a prior report and interviewed other entities (such as consumer groups) to determine what additional steps, if any, CPSC could take to better inform the public about SaferProducts.gov.[Footnote 6] For context, we reviewed documentation and interviewed officials about CPSC's strategies to increase overall public awareness of the agency. To address the second objective, we analyzed data for 2011 through 2012 on users who filed incident reports through various methods, including SaferProducts.gov, phone, e-mail, postal mail, and fax, and analyzed other data CPSC maintains to measure the amount of website use. To assess data reliability, we drew on our prior work assessing SaferProducts.gov and reviewed relevant documentation, performed manual tests to look for errors, and interviewed CPSC officials responsible for the data. We determined that, for the purposes of this report, the data were sufficiently reliable. We also interviewed CPSC officials about efforts to collect any demographic data--such as age, gender, and income--about users, and reviewed existing studies, including those by consumer groups and a law firm, related to use of SaferProducts.gov. To address the third objective, we conducted website usability tests with 37 consumers in three locations--Washington, D.C., Dallas, Texas, and San Francisco, California--to obtain their views on how easy it was to use SaferProducts.gov and how helpful the information on the website was. We chose these locations for geographic dispersion and ease of testing. We followed the protocols and used the Washington, D.C., facilities of the General Services Administration (GSA) for our testing conducted through the First Fridays Usability Testing Program. [Footnote 7] At this location, we conducted three one-on-one tests with consumers whom GSA recruited. We asked consumers to complete usability tasks we developed and discuss their general experiences using the site. In the other two locations, we worked with a contractor to recruit prospective website testers having a mix of characteristics in terms of age, gender, education, and ethnicity. We conducted two separate focus groups (of 8-9 consumers) in each of the other locations, asking them to complete specified usability tasks and discuss their experiences using the website. While the results of the tests are not generalizable to all U.S. consumers, they provided us with in-depth interactive feedback and detailed perspectives from a range of consumers about the usability challenges associated with SaferProducts.gov. We also analyzed other usability resources to determine key practices on how to make websites easy to use and helpful. Appendix I contains a more detailed description of our objectives, scope, and methodology. Appendix II contains the script used in our website usability tests. We conducted this performance audit from July 2012 to March 2013 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Background: CPSC was created in 1972 under the Consumer Product Safety Act to regulate certain consumer products and address those that pose an unreasonable risk of injury; assist consumers in evaluating the comparative safety of consumer products; and promote research and investigation into the causes and prevention of product-related deaths, injuries, and illnesses.[Footnote 8] CPSC's jurisdiction is broad, covering thousands of types of manufacturers and consumer products used in and around the home and in sports, recreation, and schools. CPSC does not have jurisdiction over some categories of products, including automobiles and other on-road vehicles, tires, boats, alcohol, tobacco, firearms, food, drugs, cosmetics, medical devices, and pesticides. Other federal agencies--including the National Highway Traffic Safety Administration; Coast Guard; Bureau of Alcohol, Firearms, Tobacco, and Explosives; Department of Agriculture; Food and Drug Administration; and Environmental Protection Agency-- have jurisdiction over these products. As noted above, in 2011 CPSC made the CPSIA-mandated database available through the SaferProducts.gov website. Users of SaferProducts.gov can perform two main functions on the website. They can (1) submit a report describing the harm or risk of harm related to the use of consumer products, and (2) search for these reports, as well as recall notices and other safety information on products. [Footnote 9] SaferProducts.gov allows the public to review incident reports that previously were available only to CPSC unless requested under the Freedom of Information Act. The public also may view manufacturers' comments on reports when manufacturers request that their comments be published. As required by statute, CPSC disclaims any responsibility to guarantee the accuracy of a report.[Footnote 10] The submitter of an incident report on SaferProducts.gov must fit into one of five categories: (1) consumers; (2) local, state, and federal government agencies; (3) health care professionals; (4) child service providers; and (5) public safety entities. CPSC regulations specify that "consumers" include, but not be limited to, users of consumer products, family members, relatives, parents, guardians, friends, attorneys, investigators, professional engineers, agents of a user of a consumer product, and observers of the consumer products being used. [Footnote 11] Once CPSC receives an incident report through SaferProducts.gov, it reviews each report to determine if the submitter included all the information required by CPSIA for publication in the public database. [Footnote 12] Reports that do not meet the minimum criteria for publication are reviewed by CPSC staff and saved for internal use, and CPSC is not required to contact the submitters for further information.[Footnote 13] CPSC then transmits a copy to manufacturers, importers, and private labelers identified in the reports, to provide them with the opportunity to comment. Qualifying reports and manufacturer comments submitted for publication are then available on SaferProducts.gov (see figure 1). Figure 1: Consumer Product Safety Commission's (CPSC) Electronic Process for Publishing an Incident Report Submitted on SaferProducts.gov: [Refer to PDF for image: illustration] Submitter: SaferProducts.gov public portal: Submitter enters information in all eight required data fields, and any additional (optional) information. Incident report: Sent to CPSC's central repository[A]. Eight required fields of information verified. Manufacturer: Manufacturer's response(s): * General comment; * Confidential Business Information claim; * Materially Inaccurate Information claim. Optional response using business portal: CPSC. CPSC makes a determination whether to publish the incident report, including the manufacturer's comment, on SaferProducts.gov. CPSC publishes the report on SaferProducts.gov. Consumer can also search the public database to look for prior safety issues for a specific consumer product. Source: GAO analysis of CPSC regulations and CPSIA, as amended; Art Explosion (images). [A] The repository also contains reports where required fields of information are missing (reports stored and not published). [End of figure] CPSC's public database, SaferProducts.gov, is a part of CPSC's larger Consumer Product Safety Risk Management System (CPSRMS), which was designed to replace CPSC's historically segmented data systems with a unified information technology system. The updated system is designed to allow CPSC to study data from multiple sources in a centralized location to identify emerging consumer product safety hazards.[Footnote 14] SaferProducts.gov is one of five official public websites CPSC maintains. For example, SaferProducts.gov is separate from CPSC's primary website, CPSC.gov [hyperlink, http://www.cpsc.gov], which provides information on the agency and consumer product recalls, and also includes a link to SaferProducts.gov.[Footnote 15] While SaferProducts.gov also contains product recall information in addition to incident reports by consumers, the recall notices originate from CPSC.gov. Therefore, website users who click on a link for a recall notice on SaferProducts.gov leave the website and are automatically directed to CPSC.gov. CPSC also has a Twitter feed, @OnSafety, which it uses to disseminate product safety information. CPSC Has Used Various Approaches to Inform the Public about SaferProducts.gov, but Has Not Established Metrics to Assess Its Efforts: CPSC's efforts to promote SaferProducts.gov formed part of a larger effort to increase the public's awareness of the agency. CPSC has taken a variety of approaches to inform the public about SaferProducts.gov, many of which are consistent with key practices for consumer education planning. However, CPSC has not established metrics for its efforts. As a result, the agency does not know which of its efforts have had the most impact on increasing awareness and use of SaferProducts.gov. Outreach about SaferProducts.gov Has Been Part of an Effort to Increase Awareness of CPSC: CPSC's efforts to inform the public about SaferProducts.gov have been part of a larger effort to increase the public's awareness of the agency. According to CPSC officials, certain segments of the public may not be aware of the agency or its mission in product safety, much less be aware of SaferProducts.gov. Likewise, roughly one-third of the 37 consumers who participated in our website usability tests were aware of CPSC or its mission. To promote awareness of CPSC, officials have conducted public information campaigns related to various product safety hazards such as fire hazards and those involving children's products, issued press releases about product recalls, and used social media.[Footnote 16] Officials said that media stories promoting the use of SaferProducts.gov have had the benefit of promoting CPSC as a resource not only for information about product recalls (for which the agency is most commonly known), but also as a place where consumers can raise concerns about the safety of consumer products. In addition to the outreach efforts noted above, CPSC has planned initiatives to assess the public's awareness of the agency as a whole. In fiscal year 2011, CPSC's Office of Communications received funds to award a contract to plan and conduct field surveys to assess consumer awareness of the agency. CPSC and a contractor are developing the survey tool. These surveys are to cover such areas as the public's knowledge and awareness of the safety issues for which CPSC is responsible, how the agency's work affects consumers, and how the public responds to product recalls and other safety hazards that CPSC communicates. CPSC officials told us that they plan to administer the survey in 2013, but have been awaiting approval of the survey from the Office of Management and Budget. CPSC also recently redesigned its main website, CPSC.gov, based on feedback from the public. According to CPSC officials, this redesign allowed the agency to provide a more visible link to SaferProducts.gov.[Footnote 17] CPSC Has Used a Variety of Approaches to Inform the Public about SaferProducts.gov: As it has for publicizing the agency, CPSC has used a variety of approaches to inform the public about SaferProducts.gov, including the use of social and other media. Before launching SaferProducts.gov, CPSC hosted a web conference on January 11, 2011, to inform interested stakeholders such as consumer groups and the public about the site's search function and the information required to submit an incident report.[Footnote 18] Around the time it launched SaferProducts.gov in March 2011, CPSC promoted the new website through print and other media. According to CPSC officials, the agency's promotional strategy emphasized both the public's ability to search SaferProducts.gov for reports and submit such reports. In addition, near the 1-year anniversary of SaferProducts.gov, CPSC launched three public service announcements (PSAs) about SaferProducts.gov, sending these PSAs to local and national media and making them available on online media channels, such as YouTube [hyperlink, http://www.youtube.com]. According to CPSC officials, the agency has a contract with a video production company to produce and distribute the videos. CPSC officials said that the PSAs have been among the 10 most-viewed videos on CPSC's YouTube channel. They added that it was difficult to attract extensive television coverage or the best airtime slots given CPSC's PSA budget of about $50,000 for fiscal year 2012. Further, the officials said that PSAs can cost from $700,000 to $1 million to produce, distribute, and air during prime viewing or listening times. The agency also has distributed informational materials to target audiences at conferences and community events; referenced the site in speeches and presentations by the Chairman, Commissioners, and staff; and held press interviews to promote the site, according to CPSC officials. For example, CPSC developed a series of brochures, including some tailored to specific professional sectors, such as health care, child care, public safety, and government. CPSC officials noted that they have mentioned the site at conferences, particularly those aimed at minority populations and professional groups. The agency also has made a data feed of the incident reports available to third-party software developers to create mobile applications and provided information for developers in a frequently asked questions page on SaferProducts.gov. In conducting its public information efforts, CPSC has employed a number of strategies consistent with key practices for consumer education planning that we identified in a prior report.[Footnote 19] For example, CPSC has worked with stakeholders such as consumer groups (a key practice) to promote SaferProducts.gov, and used a variety of media (another key practice) to promote the site. CPSC also has identified "messengers" such as consumer groups and state attorneys general to assist with publicity, and identified the resources needed for publicity (other key practices). Most of the consumer product safety experts we interviewed from nine groups representing consumers, researchers, and various industries stated that CPSC has been taking appropriate measures to promote the site. However, some also suggested that CPSC could conduct more targeted outreach to other professional groups, such as those in health care, and other populations, such as parents. CPSC Has Not Established Metrics to Evaluate Its Outreach on SaferProducts.gov: While CPSC has employed many of the key practices for consumer education planning as described previously, it has not employed one of the key practices that could further improve the efficacy of its outreach for SaferProducts.gov. Specifically, CPSC has not established metrics, such as process and outcome metrics, to measure the success of its outreach efforts.[Footnote 20] In its 2013 performance budget request, as part of an effort to increase awareness of the agency, CPSC has a goal for the number of visits to CPSC.gov. However, CPSC does not have a similar goal for the number of visits to SaferProducts.gov, although it collects such data (as discussed in the next section of this report). Similarly, CPSC has not determined whether its efforts to publicize SaferProducts.gov at conferences or through PSAs have led to increased use of SaferProducts.gov after the events. CPSC also has not incorporated tools or features on the site (such as a drop-down menu on the homepage that would ask users to select an option such as "conferences," "PSAs," "printed materials," or "media") to identify how the user learned about and arrived at the site. The information generated by such tools also may provide CPSC with ideas for additional metrics to measure awareness and use of the site. CPSC has not established metrics to evaluate its outreach efforts for SaferProducts.gov because the agency has been focused on increasing awareness of CPSC and improving the functionality of CPSC.gov. CPSC officials said that in comparison with SaferProducts.gov, CPSC.gov received almost 10 times as many visits each month. Officials have said they may focus on evaluating outreach efforts for SaferProducts.gov in the future. However, without current metrics to assess the efficacy of its outreach for SaferProducts.gov, CPSC will not know which of its efforts--for instance, promoting the site at conferences and using PSAs--have had the most impact on increasing awareness and use of SaferProducts.gov, or be able to best target its limited resources to increase use of the site. CPSC Collects Limited Data about the Use of SaferProducts.gov: CPSC collects limited data about the use of SaferProducts.gov. To track use, CPSC collects data on the number of visitors, most frequently visited pages, and number of reports received, among other metrics. CPSC also collects some data about the category of person who is submitting a report. However, CPSC does not collect any data about who is using the site to search for information. In particular, CPSC has not sought to collect demographic data, such as age, gender, or income. In mandating this report, Congress required us to assess whether a broad range of the public uses the site. However, CPSC's limited data collection related to use of the site made it difficult to conduct such an assessment. Website Use Remained Relatively Constant, with Searches and Submission of Reports Constituting Frequent Uses: According to CPSC officials, the agency's primary measure of the extent of use of SaferProducts.gov is the number of visitors each month. CPSC collects these data through web analytics software. According to CPSC's data, visits to SaferProducts.gov exceeded 100,000 each month since June 2011, a few months after the launch of the site (see figure 2), peaking at about 238,000 in November 2012.[Footnote 21] CPSC officials have not been able to identify the reasons for the increase in visits. Figure 2: Monthly Visits, from March 2011 through December 2012: [Refer to PDF for image: vertical bar graph] 2011: Date: March; Visits: 58,582. Date: April; Visits: 37,291. Date: May; Visits: 41,150. Date: June; Visits: 177,708. Date: July; Visits: 165,364. Date: August; Visits: 119,376. Date: September; Visits: 130,998. Date: October; Visits: 135,886. Date: November; Visits: 139,061. Date: December; Visits: 146,794. 2012: Date: January; Visits: 151,942. Date: February; Visits: 166,093. Date: March; Visits: 167,117. Date: April; Visits: 141,506. Date: May; Visits: 157,065. Date: June; Visits: 126,821. Date: July; Visits: 135,528. Date: August; Visits: 188,382. Date: September; Visits: 184,146. Date: October; Visits: 216,430. Date: November; Visits: 237,523. Date: December; Visits: 201,542. Source: GAO analysis of CPSC web analytics data. [End of figure] CPSC also collects data on the most frequently visited pages each month (see figure 3). These data show that users frequently used the site to search for information--for example, to search for recalled products or incident reports submitted by other users of the site. Figure 3: Most Frequently Visited Pages Each Month, from March 2011 through December 2012: [Refer to PDF for image: horizontal bar graph] Page: Home; Number of months page was among the top 5 pages visited per month: 22. Page: Search; Number of months page was among the top 5 pages visited per month: 22. Page: FAQ; Number of months page was among the top 5 pages visited per month: 19. Page: Display documents[A]; Number of months page was among the top 5 pages visited per month: 18. Page: Others; Number of months page was among the top 5 pages visited per month: 12. Page: Advanced search; Number of months page was among the top 5 pages visited per month: 11. Page: About; Number of months page was among the top 5 pages visited per month: 6. Source: GAO analysis of CPSC web analytics data. [A] According to CPSC, this page is for streaming file content, such as documents, from its server to its browser. [End of figure] CPSC also collects data on the number of reports received each month through SaferProducts.gov, as well as by phone, e-mail, postal mail, and fax. These data show that users submitted more than 1,000 reports from all sources each month from March 2011 through December 2012 (see figure 4). Figure 4: Number of Incident Reports Received, from March 2011 through December 2012: [Refer to PDF for image: 2 stacked vertical bar graphs, combined] 2011: Date: March; Reports through the SaferProducts.gov site: 669; 54%; Reports by phone, mail, email, fax: 561; 46%. Date: April; Reports through the SaferProducts.gov site: 703; 44%; Reports by phone, mail, email, fax: 900; 56%. Date: May; Reports through the SaferProducts.gov site: 596; 45%; Reports by phone, mail, email, fax: 719; 55%. Date: June; Reports through the SaferProducts.gov site: 653; 42%; Reports by phone, mail, email, fax: 892; 58%. Date: July; Reports through the SaferProducts.gov site: 751; 59%; Reports by phone, mail, email, fax: 518; 41%. Date: August; Reports through the SaferProducts.gov site: 668; 41%; Reports by phone, mail, email, fax: 972; 59%. Date: September; Reports through the SaferProducts.gov site: 719; 43%; Reports by phone, mail, email, fax: 953; 57%. Date: October; Reports through the SaferProducts.gov site: 750; 51%; Reports by phone, mail, email, fax: 732; 49%. Date: November; Reports through the SaferProducts.gov site: 700; 45%; Reports by phone, mail, email, fax: 854; 55%. Date: December; Reports through the SaferProducts.gov site: 726; 48%; Reports by phone, mail, email, fax: 776; 52%. 2012: Date: January; Reports through the SaferProducts.gov site: 783; 48%; Reports by phone, mail, email, fax: 851; 52%. Date: February; Reports through the SaferProducts.gov site: 744; 42%; Reports by phone, mail, email, fax: 1,023. 58%. Date: March; Reports through the SaferProducts.gov site: 662; 44%; Reports by phone, mail, email, fax: 857; 56%. Date: April; Reports through the SaferProducts.gov site: 587; 39%; Reports by phone, mail, email, fax: 920; 61%. Date: May; Reports through the SaferProducts.gov site: 652; 48%; Reports by phone, mail, email, fax: 698; 52%. Date: June; Reports through the SaferProducts.gov site: 633; 52%; Reports by phone, mail, email, fax: 589; 48%. Date: July; Reports through the SaferProducts.gov site: 656; 45%; Reports by phone, mail, email, fax: 809; 55%. Date: August; Reports through the SaferProducts.gov site: 731; 41%; Reports by phone, mail, email, fax: 1,046; 59%. Date: September; Reports through the SaferProducts.gov site: 651; 49%; Reports by phone, mail, email, fax: 686; 51%. Date: October; Reports through the SaferProducts.gov site: 645; 59%; Reports by phone, mail, email, fax: 456; 41%. Date: November; Reports through the SaferProducts.gov site: 643; 51%; Reports by phone, mail, email, fax: 625; 49%. Date: December; Reports through the SaferProducts.gov site: 746; 62%; Reports by phone, mail, email, fax: 456; 38%. Source: GAO analysis of CPSC's CPSRMS data. [End of figure] CPSC Collects Some Information on Users Who Submit Reports, but Has Collected Little Information on Other Users: CPSC collects some data about the categories of persons using SaferProducts.gov to submit incident reports but does not collect additional data such as age, gender, or income level of the submitters or others who use the site to search for information. When completing a report, CPSC requires submitters to state whether they are consumers, represent a government agency, or are health care or other professionals, among other categories of user. As shown in table 1, our analysis of more than 12,000 reports posted on SaferProducts.gov as of January 2013 found that most report submitters--about 97 percent--identified themselves as consumers, results consistent with our prior reporting.[Footnote 22] As stated previously, "consumers" include, but are not limited to, users of consumer products, family members, relatives, friends, attorneys, investigators, and others. Representatives of government agencies and public safety entities, as well as health care professionals, child service providers, and medical examiners and coroners also submitted reports. Table 1: Types of Submitters of Incident Reports, for Reports Published from April 2011 through January 3, 2013: Submitter: Consumer; Number[A]: 11,696; Percent of total[B]: 97.22%. Submitter: Federal government agency; Number[A]: 11; Percent of total[B]: 0.09%. Submitter: State government agency; Number[A]: 34; Percent of total[B]: 0.28%. Submitter: Local government agency; Number[A]: 71; Percent of total[B]: 0.59%. Submitter: Child service provider; Number[A]: 29; Percent of total[B]: 0.24%. Submitter: Health care professional; Number[A]: 63; Percent of total[B]: 0.52%. Submitter: Medical examiner and coroner; Number[A]: 19; Percent of total[B]: 0.16%. Submitter: Public safety entity; Number[A]: 107; Percent of total[B]: 0.89%. Submitter: Total; Number[A]: 12,030; Percent of total[B]: 100.0%. Source: GAO analysis of CPSC data on SaferProducts.gov. Note: CPSC provides notice on SaferProducts.gov, as required by statute, that it does not guarantee the accuracy, completeness, or adequacy of the contents on SaferProducts.gov, including incident reports. [A] These include reports submitted through SaferProducts.gov as well as by phone, postal mail, e-mail, and fax. [B] Percentages may not equal 100 because of rounding. [End of table] CPSC also asks report submitters to state their relationship to the victim of the incident (such as self, parent, or spouse). As shown in table 2, of those who identified themselves as consumers, most identified themselves as the victims of an incident. However, many submitters did not specify a relationship. Table 2: Relationship between Self-Reported Consumer and Victim, for Incident Reports Published from April 2011 through January 3, 2013: Victim: Self; Number: 4,463. Victim: Spouse; Number: 603. Victim: Child; Number: 1,867. Victim: Parent; Number: 108. Victim: Other relative; Number: 196. Victim: Client, patient, student, etc.[A]; Number: 58. Victim: Friend, neighbor, co-worker; Number: 78. Victim: No relationship; Number: 33. Victim: Unspecified; Number: 4,290. Victim: Total; Number: 11,696. Source: GAO analysis of CPSC data on SaferProducts.gov. Note: CPSC provides notice on SaferProducts.gov, as required by statute, that it does not guarantee the accuracy, completeness, or adequacy of the contents on SaferProducts.gov, including incident reports. [A] According to CPSC, "etc." refers to professional relationships similar to a client, patient, or student. [End of table] Of those who did specify a relationship to the victim, 4,463, or 60 percent, reported that they were the victims, and 1,867, or 25 percent, reported that their child was the victim (see table 3). Table 3: Relationship between Self-Reported Consumer and Victim for Those Specifying a Relationship, for Incident Reports Published from April 2011 through January 3, 2013: Victim[A]: Self; Number: 4,463; Percent: 60.26%. Victim[A]: Spouse; Number: 603; Percent: 8.14%. Victim[A]: Child; Number: 1,867; Percent: 25.21%. Victim[A]: Parent; Number: 108; Percent: 1.46%. Victim[A]: Other relative; Number: 196; Percent: 2.65%. Victim[A]: Client, patient, student, etc.[B]; Number: 58; Percent: 0.78%. Victim[A]: Friend, neighbor, co-worker; Number: 78; Percent: 1.05%. Victim[A]: No relationship; Number: 33; Percent: 0.45%. Victim[A]: Total; Number: 7,406; Percent: 100.0%. Source: GAO analysis of CPSC data on SaferProducts.gov. Note: CPSC provides notice on SaferProducts.gov, as required by statute, that it does not guarantee the accuracy, completeness, or adequacy of the contents on SaferProducts.gov, including incident reports. [A] 4,290 consumers did not specify their relationship to the victim. [B] According to CPSC, "etc." refers to professional relationships similar to a client, patient, or student. [End of table] CPSC asks that submitters specify the location of the reported incident, including the country and state. Most submitters providing this information--about 90 percent--reported that the incident took place in the United States (see table 4). Submitters also reported that incidents took place in other countries or did not specify where the incident took place.[Footnote 23] Table 4: Location of Reported Incident by Country, as of January 3, 2013: Location: United States of America[A]; Number: 10,863; Percent: 90.30%. Location: Canada[B]; Number: 84; Percent: 0.70%. Location: Other countries[C]; Number: 36; Percent: 0.30%. Location: Unspecified; Number: 1,047; Percent: 8.70%. Location: Total; Number: 12,030; Percent: 100.0%. Source: GAO analysis of CPSC data on SaferProducts.gov. Note: CPSC provides notice on SaferProducts.gov, as required by statute, that it does not guarantee the accuracy, completeness, or adequacy of the contents on SaferProducts.gov, including incident reports. [A] The United States of America category consists of reports from the 50 states (10,822), District of Columbia (33), Guam (1), Puerto Rico (6), and Virgin Islands (1). [B] Locations cited were Canada (75), British Columbia (1), Ontario (5), Quebec (1), and Saskatchewan (2). [C] The reports were from Australia (2), Austria (1), Belgium (1), Bermuda (1), Chile (1), China (1), Columbia (2), France (1), Germany (6), Italy (3), Jamaica (1), Japan (1), Malaysia (1), Mexico (2), New Zealand (2), Papua New Guinea (1), Philippines (1), Singapore (1), Sweden (1), Thailand (1), United Kingdom (4), and Uruguay (1). [End of table] In addition, states with the highest population--such as California, Texas, New York, Florida, and Illinois--had the most reported incidents (see figure 5). Figure 5: Total Number of Incident Reports Published by State of Incident, from April 2011 through January 3, 2013: [Refer to PDF for image: illustrated U.S. map] Less than 100: Alaska; Arkansas; Delaware; District of Columbia; Hawaii; Idaho; Iowa; Kansas; Louisiana; Maine; Mississippi; Montana; Nebraska; Nevada; New Mexico; North Dakota; Oklahoma; Rhode Island; South Dakota; Utah; Vermont; West Virginia; Wyoming. 100 to less than 200: Alabama; Connecticut; Indiana; Kentucky; New Hampshire; Oregon; South Carolina; Tennessee. 200 to less than 300: Arizona; Colorado; Georgia; Minnesota; Missouri; Wisconsin. 300 or more: California; Florida; Illinois; Maryland; Massachusetts; Michigan; New Jersey; New York; North Carolina; Ohio; Pennsylvania; Texas; Virginia; Washington. Source: GAO analysis of CPSC data on SaferProducts.gov. Note: CPSC provides notice on SaferProducts.gov, as required by statute, that it does not guarantee the accuracy, completeness, or adequacy of the contents on SaferProducts.gov, including incident reports. As of January 2013, there were more than 12,000 incident reports published on SaferProducts.gov. Of these, 1,216 were for incidents in which the state of the incident was not specified or specified as "United States." Forty-one incidents were reported to have occurred in the District of Columbia or a U.S. territory, and 120 incidents were reported to have occurred outside the United States. [End of figure] Beyond these data, CPSC does not request or obtain additional details about the users of SaferProducts.gov. According to CPSC officials, the agency also cannot distinguish new users from returning users because CPSC's web analytics software has not been configured with "cookies" to capture these data.[Footnote 24] Officials have cited resource and privacy concerns as reasons for not collecting these data, although they said they have been considering using cookies in the future. In addition, CPSC does not collect more specific demographic information such as age, gender, or income level from the submitters of reports or other site users, citing an interest in minimizing the reporting burden on users. As an example, CPSC has not requested that site users voluntarily provide this information during the report submission process or after submitting a report. Congress required us to assess whether a broad range of the public uses SaferProducts.gov, but CPSC's limited data collection made it difficult to conduct such an assessment. In addition, standards for internal control in the federal government state that agencies should have timely, relevant information for management decision-making purposes.[Footnote 25] As a result of its limited data collection about users of the site, CPSC has been limited in its ability to target its marketing and outreach efforts on specific groups, populations, or areas to achieve the goal of increasing use of the site. Consumer Testing Showed SaferProducts.gov to Be Generally Useful, but Identified Areas for Improvement: As discussed earlier in this report, our website usability tests focused on asking consumers in our testing sessions to judge if SaferProducts.gov was easy to use. We had the consumers perform various tasks (such as searching for recalled products and submitting mock incident reports) and asked for opinions about the site's usefulness. A moderator facilitated the sessions and we elicited feedback from participants. In addition, a GSA official with expertise in website usability assessed SaferProducts.gov, and another GSA official reviewed the site for website accessibility.[Footnote 26] Consumers Said the Site Was Easy to Use, but Noted a Few Search Functions and a Registration Page as Problematic: Many consumers in our testing sessions generally found SaferProducts.gov easy to use, but they encountered difficulties with certain aspects of the two main functions: searching for information and submitting incident reports.[Footnote 27] Of the 37 consumers who participated in our testing sessions, 20 found SaferProducts.gov easy to use as indicated by their responses to a questionnaire we administered following each test session. For example, almost all the consumers were easily able to determine what initial steps to take to search for or report a product that may be unsafe. In addition, the expert evaluator reviewing the site at our request described the site as clean and easy to navigate. In conducting the search tasks, consumers generally were able to find recalled products using basic key word searches. But some search functions, including those that required more complicated searches such as use of an advanced search function to narrow results, posed challenges. For example, in one testing session, no consumers were able to complete a task that required them to narrow their search by injury, time period, and location. In another session, the calendar function, which filters the results by time period, posed particular challenges. Five of the eight consumers in that session experienced difficulties in having to enter and, when seeking to make one change, re-enter all the dates to focus their search on products recalled within a particular time period. The expert evaluator from GSA experienced similar challenges in using the calendar function. In addition, when asked to search for and compare safety information for two products--one for which there were search results and one for which there were none--almost all the testers had difficulty interpreting the lack of search results for the latter product. For example, while some testers assumed that a search for a product that produced no results indicated that the product was safe, others did not make this presumption. In our testing sessions, most consumers were not sure which product to purchase based on their searches and roughly a quarter indicated that they would leave SaferProducts.gov to search other websites if they found no results on SaferProducts.gov. In contrast to SaferProducts.gov, other websites inform users of a possibly incorrect search term, such as a typographical error, which helps users interpret the results of their searches and identify potential errors. During the usability tests, consumers experienced fewer challenges using the reporting function than the search function. To submit an incident report, consumers must enter information on a series of pages that include a combination of required and optional fields. During our testing, consumers found the instructions for submitting a report to be generally clear. For example, almost all the testers thought the instructions for submitting information about the incident, product, and victims were clear. However, 15 of the 37 consumers in our test sessions expressed concern about apparently needing to register before submitting a report and generally did not notice that they could continue without registering (see figure 6). By registering on SaferProducts.gov, site users can save their reports to complete at a later time and receive updates on the status of reports. When reaching the registration page, over a third of the consumers in our focus group sessions said that they would not be inclined to register. In one session, seven of nine consumers said they would not be inclined to register and thought that having to register was a deterrent to completing a report. In another session, none of the participants noticed the option to skip registration. Some of those who noticed that they could skip registration emphasized that the option should be more prominent--for example, placed alongside the registration box rather than below it where it might not be immediately visible. Likewise, as an issue of website usability, the expert evaluator from GSA reviewing the site on our behalf noted that the "continue without registering" option was not prominent enough and stated that registration may deter users from continuing the report submission process. Figure 6: Screenshot of Registration Page for Report Submission on SaferProducts.gov: [Refer to PDF for image: screenshot] By not scrolling to the bottom of the page, consumers did not always notice "continue without registering" and suggested that it be placed alongside the registration boxes. Source: GAO and SaferProducts.gov. Note: CPSC, "SaferProducts.gov" (Bethesda, Md.), accessed February 26, 2013, [hyperlink, https://www.saferproducts.gov/CPSRMSPublic/Incidents/IncidentReportingRe gistration.aspx?Affiliation=2&ProductCategory=3&PCode=&UserProduct=]. [End of figure] In addition, some consumers in our testing sessions said that the reporting pages contained too many questions and described the submission process as cumbersome, particularly for busy individuals such as parents.[Footnote 28] To address this, one consumer suggested grouping all of the required fields on one page. The expert evaluator from GSA also suggested that all the questions in the reporting process should be reviewed to determine if each was necessary. When CPSC first developed SaferProducts.gov, the agency conducted three focus groups--one with consumers and one with professionals--to test the site and assess users' experience with it.[Footnote 29] CPSC's focus group testing only addressed the incident reporting function, not the search function, and focused on (1) awareness of how and where to submit a safety complaint and (2) general reactions to the site. CPSC has not conducted additional usability testing since launching SaferProducts.gov in March 2011. As mentioned previously, CPSC officials have said that issues such as assessing the level of awareness of CPSC and redesigning CPSC.gov were higher priorities than assessing and improving SaferProducts.gov. A number of resources across the federal government are available to help agencies in making their websites more usable. For example, as cited previously, GSA's First Fridays Usability Testing Program is designed to teach agency officials how to find and fix usability problems at no cost to the agency. The program's services are (1) formal tests, (2) quick tests, (3) mobile tests, (4) observation, and (5) expert evaluation.[Footnote 30] GSA also offers DigitalGov University, which includes courses in web design and usability best practices. In addition, the Department of Health and Human Services (HHS) operates two usability labs, both of which are free of charge to other federal agencies, to evaluate websites to ensure that they are easy-to-use and useful. Furthermore, GSA and HHS maintain HowTo.gov and Usability.gov, respectively, to provide guidance and resources to help agencies create websites that are usable, useful, and accessible. Because of the usability issues in the areas we identified, consumers may not take advantage of all the features of SaferProducts.gov, and consumers may be dissuaded from completing and submitting incident reports. As a result, CPSC may not be obtaining all possible information from consumers that can help inform its safety assessments and other regulatory efforts. Consumers Initially Were Not Aware of SaferProducts.gov and Thought the Site's Purpose Was Unclear, but Found the Site Generally Useful after Using It: None of the consumers in our test sessions previously had heard of SaferProducts.gov, although a few were familiar with CPSC as an agency involved in recalling certain products. In addition, 5 of the 37 consumers who participated in our tests said that the purpose of SaferProducts.gov was not clear based on its name and the initial information on the home page. In our testing sessions, roughly a third of the consumers commented that the name of the website-- SaferProducts.gov--and the home page did not accurately convey what consumers could and could not do on the site. For example, when asked about their impressions of SaferProducts.gov, over a quarter of the testers thought that they would find information about safe products, such as a list of products that meet certain standards or a rating of products. These consumers did not appear to notice information on the home page indicating that they would only find information on unsafe products (see figure 7). In our testing sessions, several consumers commented that the website would be more aptly named UnsafeProducts.gov. In addition, during our one-on-one testing sessions in Washington, D.C., two of the testers had difficulty distinguishing between recall notices and incident reports, which serve different purposes.[Footnote 31] Likewise, although the expert evaluator from GSA was able to obtain a general sense of the purpose of the site, he noted that a tagline (brief text that gives users an immediate idea of what the site does) would help reinforce the site's purpose. Figure 7: Screenshot of SaferProducts.gov Home Page: [Refer to PDF for image: screenshot] Tabs: Reporting unsafe products; Searching recalls and reports. Many consumers thought SaferProducts.gov would list safe products, rather than those that were unsafe. Source: GAO and SaferProducts.gov. Note: CPSC, "SaferProducts.gov" (Bethesda, Md.), accessed February 26, 2013, [hyperlink, http://www.saferproducts.gov]. [End of figure] In one testing session, a few consumers also said that it was not apparent from looking at the home page that CPSC did not regulate certain categories of products, such as automobiles and medications, although more than half of the consumers in our testing sessions said at the outset that they routinely searched online for safety information on particular products. Similarly, none of the consumers in our testing sessions noticed that they could be directed to the agency's main site, CPSC.gov, by clicking on certain links in SaferProducts.gov. Only the expert evaluator noticed that by opening a recall notice, the website user would leave SaferProducts.gov and go to CPSC.gov. However, as consumers completed the various tasks in the testing sessions, they better understood the website's features and functions. In responding to our closing questions about their overall experiences in using the site, most said that they would use SaferProducts.gov again now that they were aware of it. For example, some consumers found information about product recalls the most useful component of the site and said they would give more weight to this information. In our testing sessions, about one-quarter of the consumers also found value in the incident reports, noting that they helped website users understand whether products that had not yet been recalled had safety issues. Two of the consumers commented that they found the content of the reports to be more credible than other websites that provide a forum for consumer complaints. In addition, a few consumers pointed to the amount of detail in the reports, such as the incident description, location, and date as particularly helpful. Nevertheless, because of the usability issues in the areas we identified (for example, not having a clear and "up-front" statement of what the site contains and how it can be used), consumers may not use all of the site's available features and be dissuaded from completing and submitting reports. CPSC officials also acknowledged that awareness of the agency could be heightened if consumers were informed about CPSC while using or searching SaferProducts.gov. Conclusions: CPSC has used many approaches to inform the public about SaferProducts.gov, employing many key practices of consumer education planning in the process. Incorporating the promotion of SaferProducts.gov into its broader effort to increase awareness of the agency has represented a logical approach that has prevented duplication. However, our work confirms CPSC's perception that public awareness of SaferProducts.gov is likely low. For example, none of the participants in our usability tests had heard of SaferProducts.gov prior to the testing. Although CPSC has employed many of the key practices for consumer education planning, it has not established metrics to measure the success of its efforts. By establishing such metrics, the agency would be better able to determine which of its outreach efforts had the most impact on increasing awareness and use of the site and thus could more effectively target its limited resources to increase use of the site. In addition to establishing and using metrics, more data about the use of SaferProducts.gov could help CPSC target its marketing and outreach. Currently, CPSC collects limited data about the use of SaferProducts.gov. For example, it collects data on the number of visitors, but not whether they are using the site to search for information--one of the main functions of the site. It also does not collect demographic data about the users' age, gender, or income. These types of data could help CPSC identify groups, populations, or areas on which to focus to further increase use of the site. Our usability testing with consumers identified other ways in which CPSC may increase the use of SaferProducts.gov. Although our testing revealed that many consumers found the site generally easy to use, it also revealed that certain search functions, site registration, and lack of a clear statement of purpose posed challenges for some users. By improving the site in these areas, CPSC could help ensure that consumers take advantage of all the features of the site and are able to search for and report information in an easy and convenient manner. Making these improvements also may provide CPSC with additional reports from consumers to inform its safety assessments and other regulatory efforts. Recommendations for Executive Action: To improve the awareness, use, and usefulness of SaferProducts.gov, CPSC should take the following three actions: * establish and incorporate metrics to assess efforts to increase awareness and use of SaferProducts.gov, * look for cost-effective ways of gathering additional data about the users and their use of SaferProducts.gov, and: * implement cost-effective usability improvements to SaferProducts.gov, taking into account the results of any existing usability testing or any new testing CPSC may choose to conduct. Agency Comments and Our Evaluation: We provided a draft of this report to CPSC for review and comment. In commenting on the draft report, the Chairman and Commissioners stated that they support the report's recommendations. Specifically, they stated that CPSC staff will look for cost-effective ways to improve awareness of SaferProducts.gov, improve the usability of the site based on research on best practices in web design, and gather additional metrics about users. The Chairman and Commissioners' comments are reprinted in appendix III. CPSC also provided technical comments that we incorporated in the report as appropriate. We are sending copies of this report to interested congressional committees and to the Chairman and Commissioners of CPSC. In addition, the report will be available at no charge on GAO's website at [hyperlink, http://www.gao.gov]. If you or your staff members have any questions about this report, please contact me at (202) 512-8678 or cackleya@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV. Signed by: Alicia Puente Cackley, Director: Financial Markets and Community Investment: [End of section] Appendix I: Objectives, Scope, and Methodology: The objectives of our report were to examine (1) the Consumer Product Safety Commission's (CPSC) efforts to inform the public about SaferProducts.gov, (2) who has been using the website and to what extent, and (3) the extent to which consumers have found the website to be useful. For the first objective, we reviewed CPSC marketing, budget, evaluation, and planning documents to determine the status of the agency's public information efforts related to SaferProducts.gov. We sought to determine whom CPSC had been targeting; how, if at all, the agency was evaluating the outcomes of its efforts; and any future plans to promote awareness and use of the site. We compared CPSC's efforts with criteria on key practices for consumer education planning.[Footnote 32] We interviewed consumer product safety experts from nine groups representing consumers, researchers, and various industries to determine what additional steps, if any, CPSC could take to better inform the public about SaferProducts.gov. We identified these experts through our prior work or based on recommendations from those we interviewed. For context, we interviewed an official in CPSC's Office of Communications and reviewed CPSC's strategies to increase awareness of the agency as a whole. For the second objective, we analyzed data, published on SaferProducts.gov as of January 3, 2013, about users who filed incident reports on the site. We also analyzed other data CPSC maintains to measure the extent of website use. The data on the extent of website use covered the period from March 2011 through December 2012. We interviewed CPSC officials about efforts to collect any demographic data--such as age, gender, or income level--about site users. Furthermore, we reviewed existing studies, such as those by consumer groups and a law firm, related to the use of SaferProducts.gov. To assess data reliability, we drew on our prior work assessing SaferProducts.gov and reviewed relevant documentation, performed manual tests to look for errors, and interviewed CPSC officials responsible for the data.[Footnote 33] As noted in the body of the report, CPSC has a statement on SaferProducts.gov, to provide notice as required by statute, that it does not guarantee the accuracy of incident reports submitted by users of the site. We considered this statement as part of our overall assessment of the data's reliability and determined that, for the purposes of this report, the data were sufficiently reliable. We did not review specific incident descriptions that individuals filed and do not attest to the reliability of that information. For the third objective, we conducted website usability tests with 37 consumers--who represented a mix of demographic characteristics in terms of age, gender, and educational level--to obtain their views on how easy it was to use SaferProducts.gov and how useful they found the website. We conducted the tests in Washington, D.C., Dallas, Texas, and San Francisco, California. We chose these locations for geographic dispersion and ease of testing. We followed the protocols and used the Washington, D.C. facilities of the General Services Administration (GSA) for our testing conducted through the First Fridays Usability Testing Program. At this location, GSA recruited three volunteer testers on our behalf. Consistent with the GSA program protocols, a moderator facilitated the testers' execution of various website tasks, such as searching for recalled products and submitting mock incident reports. We followed similar protocols in San Francisco and Dallas. To identify the participants in San Francisco and Dallas, we worked with a contractor to recruit prospective testers who had a mix of demographic characteristics. We held two focus groups in each location, for a total of four groups. Two groups had eight participants per group and the other two groups had nine participants per group. In all four groups, a moderator facilitated the testers' execution of various tasks, as was done in Washington, D.C. Although the results of our usability tests are not generalizable to all U.S. consumers, they provided us with in-depth, interactive feedback and detailed perspectives from a range of website users about the usability challenges associated with SaferProducts.gov. To supplement our approach, we requested and reviewed an expert evaluation conducted by the First Fridays program manager. The GSA official evaluated SaferProducts.gov based on the following criteria: (1) accessibility--the ability of people with physical or mental disabilities to use the site; (2) identity and purpose--whether the site clearly presents its purpose, including what the site offers and what a user can do on it; (3) clarity--the ability to read and digest content; (4) navigation--how easily users can find information; and (5) design and content--focusing on the layout, headers, and design. Another GSA official provided a more in-depth accessibility review of SaferProducts.gov to identify issues that users with disabilities might encounter when navigating the site. According to GSA, although an expert evaluation can be a useful starting point for determining a website's usability strengths and weaknesses, the expert evaluation emphasizes the importance of the user experience. In addition, we reviewed various other website usability resources and criteria, including Usability.gov, to understand the key practices for making websites easy to use and helpful.[Footnote 34] We conducted this performance audit from July 2012 to March 2013 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. [End of section] Appendix II: GAO Script, Including Questions and Tasks, for Usability Testing with Consumers: Questions about CPSC/SaferProducts.gov: 1. How many of you have ever gone online to search for information about the safety of a particular product? a. Of those who have where do you go? 2 If you had a safety concern or issue with a consumer product, where would you typically go first to report these? a. Why? 3. Has anyone ever heard of the Consumer Product Safety Commission, or CPSC? a. If Yes: i. What would you say it does? b. If No: i. What would you say it does? ii. Skip to #5. 4. Have you used the CPSC website before? a. If Yes: i. Can you talk about why and how you used it? ii. How did you find out about it? 5. Have you ever heard of SaferProducts.gov? a. If Yes: i. Where did you hear about it? ii. Have you used the site? 1. If yes, can you talk about why and how you used it? b. If No: i. What would you say it does? First Impressions: First, I'm going to ask you to look at this page and tell me what you make of it. Feel free to scroll around, but don't click anything yet. 1. What strikes you first about the site? 2. What does the purpose of the site seem to be? a. What would you expect to find here? b. What types of information or reports? 3. What do you think you can do here? 4. Who do you think the target audience is? 5. What would you be inclined to click on first? a. Why? First Clicks: Now I'm going to ask you some questions and I'm hoping you can tell me where you would click first to try and find an answer. You don't have to actually click, but just tell me where you would probably begin looking. Where would you click to...(please write it down on your paper then we'll discuss): * Search for a safety report submitted by a consumer or recall notice about a consumer product? - Conduct an advanced search? * Find out more about what SaferProducts.gov is? * Submit a report about a safety issue with a consumer product? * Find help using the site? * Contact the agency? Thanks. Now I am going to ask you to try doing some specific tasks. I'll lay out a scenario for you, and then you can move through the site to try and accomplish the task. Feel free to make notes as you run into any difficulties or problems. We will discuss each task as a group. Tasks: Browse and Search: OK. Here is the first task: 1. You are in the market for a dishwasher and are trying to decide between two possible products. One is the GE Adora dishwasher and the other is the Amana Tall Tub dishwasher. You want to determine which one is safer. Try to find an answer to your question. a. How would you describe the process of tying to answer your question? b. How useful was the information that was returned? 2. You recently heard on the radio that consumers have been having problems with Mr. Coffee coffeemakers, but you didn't hear all the details. You have a Mr. Coffee single cup brewer, and you come to SaferProducts.gov to see if the government has issued any information about it. a. Try to find answer to your question. * What was the process you used to find your answer? * Were you able to find an answer to your question? b. Many/all of you found a recall notice for the Mr. Coffee coffeemaker. Take a look at the Recall page. What strikes you about this page? c. Where do you think this information came from? d. Would this information prompt you to take any action? How many say yes (show of hands): How many say no (show of hands): Those who say yes — why yes? Those who say no — why no? e. You're also interested in what other consumers are saying about this product. Can you find anything that would help you learn more about this? * Take a look at a Report page. What strikes you about this page? * Where do you think this information came from? * Would this information prompt you to take any action? * Did you notice the manufacturer comments? Do they have any impact on the value of this information to you? * How would you describe the process of trying to answer your question? * Did you notice a distinction between actions by CPSC and reports from consumers? * Which did you find more useful? * Do you think you would value one more than the other? * How useful was the information that was returned? 3. You are the new parent of a young child, and you are looking for information about products that could pose a choking hazard to children ages one to five. How would you go about trying to find this information? If you haven't done so already, please enter "choking hazard" In the search field. What if you wanted to narrow this search to find reports from consumers that involved choking hazards and: - occurred in the past 2 years; - in California. - resulted in an injury that required a trip to the Emergency Room? Take a few minutes to find the information and then we will discuss. * How easy was it to find this information? * How useful was the information that was returned? Reporting Functions: Now I want move from tasks that relate to finding information to tasks that involve reporting information on the site. The reporting task is as follows: 4. You recently had a very jarring experience related to a product you had purchased. You want to make sure others are aware of the incident. After doing an Internet search, you have arrived at SaferProducts.gov. Please read the full scenario below: Four months ago, you purchased an RCA-DVD Player on sale at Target in Washington, DC. The product has worked as intended, but shortly after you bought it you noticed that the plastic lid to the battery compartment on the remote control popped open once when you picked it up, and the battery fell out. Last week, your 11-month old son was playing In your living room near the DVD player. You left the room to answer the phone. Suddenly, you heard a sound and rushed back to find him choking. A battery was missing from the DVD player remote control, so you called 911 and an ambulance arrived and took him to the hospital. Your son survived, but suffered burns in his esophagus from the battery. He was treated and released from the hospital after several days. He is restricted to a liquid diet, but otherwise has no long- term effects. * How would you submit information about this incident? * What information do you think you'll need in order to submit a report? (Is it clear?) * You want to submit a report but you don't have all day. Is there anything here that may help you decide whether or not to start the process? * You're concerned about the confidentiality of some of the information you submit. Is there anything here that might address your concerns? Please submit a report to CPSC about the incident. Additional Facts: * Incident Date: October 2, 2012 at around 7:00 p.m. * Victim Information: Boy; 11-months old; Asian; * Product Information: RCA DVD Player and Remote, Model #DRC277; * Purchase Information: Purchased on May 26, 2012 at Target in Washington, D.C. for $45; * Actions taken by Submitter Following Incident: So far you have not reported the incident to Target where you purchased the DVD player or to the manufacturer of the unit, RCA, although you intend to do both. The DVD player and remote is still in your possession; * Submitter Contact Information: Ron Ito, itor@gao.gov; write "Test" in all other required fields (In addition to entering the additional facts above, note that the required fields include: incident description, incident date, product category, product description, name, and address). Questions as users test the reporting process: * Registration Page: Would you be inclined to register? Why or Why not? * Tell Us What Happened Page: Enter "Test" in incident description; fill out date and location; skip all other fields. * People Involved and Their Injuries Page: On the injury page, is It clear which option to select from the drop down menu? Skip victim name and contact information fields. * Tell Us About the Product Page: On the product information page, fill out whatever information you can: If it appears that users are unsure, pause to ask if they have questions about the product category selections] - Is it clear what the difference is between brand, manufacturer, and private labeler? - If you answered yes to important questions about the product, is it clear what information you have to provide in the field provided? 5. For research purposes, you would like to find a full list of all the incident reports submitted on SaferProducts,.gov to date. Try to find this information. How easy was it to find this information? * What did you do to find the information? * How useful was the information that was returned? Final impressions: 1. Having performed these tasks, what are your overall impressions of how easy it is to use SaferProducts.gov (e.g., navigate the site, clarity of instructions, etc.)? 2. What are your overall impressions about the reports and notices that you found and read about during your searches? 3. What did you find most useful about SaferProducts.gov and why? Anything else? 4. What did you find least useful about SaferProducts.gov and why? Anything else? 5. Do you think you would use this site? IF so, for what purposes? If not, why not? [End of section] Appendix III: Comments from the U.S. Consumer Product Safety Commission: U.S. Consumer Product Safety Commission: 4330 East West Highway: Bethesda, MD 20814: CPSC Hotline: 1-800-638-CP5C (2772): CPSC's Web Site: [hyperlink, http://www.cpsc.gov] February 25, 2013: Ms. Alicia Puente Cackley: Director, Financial Markets and Community Investment: U.S. Government Accountability Office: 411 G Street, NW: Washington, DC 20548: Dear Ms. Cackley: The U.S. Consumer Product Safety Commission (CPSC) appreciates the opportunity to review and provide comments on the draft U.S. Government Accountability Office report "Awareness, Use, and Usefulness of SaferProducts.gov. The draft report reviews the CPSC's efforts to inform the public about SaferProducts.gov, our publicly available consumer product safety database. The report addresses who is using the database and the extent to which consumers have found the website to be useful. We support the recommendations to: (1) establish and incorporate metrics to assess efforts to increase awareness and use of SaferProducts.gov; (2) look for cost-effective ways of gathering additional data about the users and their use of SaferProducts.gov; and (3) implement cost-effective usability improvements to SaferProducts.gov, taking into account the results of any existing usability testing or any new testing CPSC may choose to conduct. Staff will look for cost-effective ways to improve awareness of SaferProducts.gov, improve the usability of the she based on research on best practices in web design, and gather additional metrics about users. Thank you again for providing us with the opportunity to comment on the draft report. Sincerely, Signed by: Inez M. Tenenbaum: Chairman: Signed by: Robert S. Adler: Commissioner: Signed by: Nancy A. Nord: Commissioner: [End of section] Appendix IV: GAO Contact and Staff Acknowledgments: GAO Contact: Alicia Puente Cackley, (202) 512-8678 or cackleya@gao.gov: Staff Acknowledgments: In addition to the individual named above, Debra Johnson, Assistant Director; Meghana Acharya; Mark Bird; William Carrigg; Jeremy Cluchey; Meredith Graves; Ronald Ito; Sarah Kaczmarek; May Lee; Marc Molino; Patricia Moye; Barbara Roesmann; Andrew Stavisky; and Julie Trinder made key contributions to this report. [End of section] Footnotes: [1] Pub. L. No. 110-314, 122 Stat. 3016 (2008) (codified in scattered sections of Title 15 of the U.S. Code). [2] 15 U.S.C. § 2055a. [3] Although CPSIA uses the term "reports of harm," in this report we refer to "incident reports." [4] Pub. L. No. 110-314, § 212. [5] In the Department of Defense and Full Year Continuing Appropriations Act of 2011, Congress required us to report on data in CPSC's safety information database. We issued a report in October 2011 that examined (1) the information required for submitting a report of harm to SaferProducts.gov, (2) the extent to which the information required for submitting a report of harm is sufficient to identify the product and to allow CPSC to review a manufacturer's claim that a report of harm contains materially inaccurate information, and (3) the length of time CPSC takes to review and resolve manufacturers' claims of material inaccuracy in a report of harm. GAO, Consumer Product Safety Commission: Action Needed to Strengthen Identification of Potentially Unsafe Products, [hyperlink, http://www.gao.gov/products/GAO-12-30] (Washington, D.C.: Oct. 12, 2011). [6] GAO, Digital Television Transition: Increased Federal Planning and Risk Management Could Further Facilitate the DTV Transition, [hyperlink, http://www.gao.gov/products/GAO-08-43] (Washington, D.C.: Nov. 19, 2007), p. 26. [7] GSA's testing program is free and focused exclusively on federal websites. According to GSA, it has tested more than 37 websites, including those of the Internal Revenue Service, Bureau of the Census, and Department of Transportation. [8] Pub. L. No. 92-573, 86 Stat. 1207 (1972) (codified, as amended, at 15 U.S.C. §§ 2051-2089). [9] Consumers and others previously were able to report safety problems or concerns about consumer products through CPSC's toll-free hotline; the U.S. mail, or a form on CPSC's website submitted through e-mail and they can continue to use these methods in lieu of submitting reports through SaferProducts.gov. CPSIA and CPSC define "harm" as injury, illness, or death or risk of injury, illness or death. 15 U.S.C. § 2055a(g); 16 C.F.R. § 1102.6(b)(4). [10] Specifically, SaferProducts.gov includes the following disclaimer to provide notice to users as required by CPSIA: "CPSC does not guarantee the accuracy, completeness, or adequacy of the contents of the Publicly Available Consumer Product Safety Information Database on SaferProducts.gov, particularly with respect to information submitted by people outside of CPSC." See 15 U.S.C. § 2055a(b)(5). [11] 16 C.F.R. § 1102.10(a). Local, state, or federal agencies include, but are not limited to, local government agencies, school systems, social services, child protective services, state attorneys general, state agencies, and all executive and independent federal agencies as defined in Title 5 of the United States Code. Health care professionals include, but are not limited to, medical examiners, coroners, physicians, nurses, physicians' assistants, hospitals, chiropractors, and acupuncturists. Child service providers include, but are not limited to, child care centers, child care providers, and prekindergarten schools. Public safety entities include, but are not limited to, police, fire, ambulance, emergency medical services, federal, state, and local law enforcement entities, and other public safety officials and professionals, including consumer advocates or individuals who work for nongovernmental organizations, consumer advocacy organizations, and trade associations, so long as they have a public safety purpose. [12] CPSIA requires the following information when submitting a report of harm: (1) description of the consumer product sufficient to distinguish the product as a product or component part regulated by CPSC; (2) identity of the manufacturer or private labeler by name; (3) description of the harm related to use of the consumer product; (4) approximate or actual date of the incident; (5) category of submitter; (6) submitter's contact information; (7) submitter's verification that the information contained therein is true and accurate; and (8) consent to publication of the report of harm. 15 U.S.C. § 2055a(b)(2)(B); C.F.R. § 1102.10(d). Subject to §§ 1102.24 and 1102.26, CPSC will publish reports of harm containing all the required information. [13] According to CPSC officials, it will mail letters to submitters who did not include all of the information required under CPSIA if the submitters provided contact information and asked that the report be published on SaferProducts.gov. The letters describe the information missing from the report of harm, and officials explained that some submitters contact CPSC to add the missing information to the report. Although CPSC is not required to contact submitters for further information when submitters' reports do not meet the minimum criteria for publication, CPSC is required to contact submitters whose reports do not contain the product model or serial number, to seek such information or, in the alternative, a photograph of the product. However, publication of an incident report is not contingent on CPSC obtaining the model or serial number (or photograph). 15 U.S.C. § 2055a(c)(5). [14] We previously reported on how CPSC assesses consumer product risks. See GAO, Consumer Product Safety Commission: Agency Faces Challenges in Responding to New Product Risks, [hyperlink, http://www.gao.gov/products/GAO-13-150] (Washington, D.C.: Dec. 20, 2012). [15] CPSC also maintains PoolSafely.gov, Recalls.gov, and ATVsafety.gov, which are dedicated to pool safety, government recalls, and the safety of all-terrain vehicles, respectively. [16] CPSC officials conduct a variety of public information campaigns. They use different media and work with various partners such as schools and the Neighborhood Safety Network--comprising hospitals, day care centers, fire stations, and parent organizations among others--to inform the public about hazardous consumer products. [17] CPSC launched its redesigned website on January 28, 2013. [18] CPSC held a second webinar on January 20, 2011, which focused on business use of the site and included demonstrations of the form and process for filing an incident report, the search function of the site, and manufacturer registration and comment features. [19] [hyperlink, http://www.gao.gov/products/GAO-08-43]. The key practices are (1) define goals and objectives; (2) analyze the situation (such as market conditions); (3) identify stakeholders; (4) identify resources; (5) research target audiences; (6) develop consistent, clear messages; (7) identify credible messenger(s); design media mix; and (8) establish metrics to measure success. [20] Process metrics ensure the quality, quantity, and timeliness of outreach work. Outcome metrics evaluate how well the campaign influenced the attitudes and behaviors of the target audience(s). See [hyperlink, http://www.gao.gov/products/GAO-08-43]. [21] CPSC defines a visit as a series of actions, beginning when a visitor views the first page and ending when the visitor leaves the site or remains idle for more than 30 minutes. [22] [hyperlink, http://www.gao.gov/products/GAO-12-30]. [23] In addition, CPSC requires that submitters provide contact information, including name, address, city, and state so that the agency can follow up with the submitter if CPSC has questions about the report. [24] A cookie is a short string of text that is sent from a web server to a web browser when the browser accesses a web page. The use of cookies allows the server to recognize returning users, track online purchases, or maintain and serve customized web pages. [25] GAO, Standards for Internal Control in the Federal Government, [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1] (Washington, D.C.: November 1999). [26] According to GSA, "web accessibility" is ensuring that people with any disability type--including motor, auditory, cognitive, seizure/neurological, and visual impairments--are able to use web content, and ensuring that content is perceivable, operable, understandable, and robust. Section 508 of the Rehabilitation Act of 1973 requires that members of the public with disabilities have access to and use of information and data from federal agencies that is comparable to the access of those without disabilities. See 29 U.S.C. § 794d. In GSA's website accessibility review, the official identified some helpful features of the site and a few areas for improvement, including enhancing some aspects of the site's navigation, recoding the search and submission forms, and adding alternative text for screen readers and Braille display users. [27] We have used the term "usability" to encompass the concepts of ease of use and usefulness or utility. We reviewed various website usability resources and criteria to understand key practices for making websites easy to use and helpful. For instance, at usability.gov, the federal government has published criteria to assess a website's usability and issued guidance and best practices for designing federal websites. Usability criteria include ease of navigation, readability, and having a clear statement of purpose on the site's home page. [28] As stated in the background section of this report, submitters must include the following when submitting a report: (1) description of the product; (2) identity of the manufacturer or private labeler by name; (3) description of the harm related to use of the product; (4) approximate or actual date of the incident; (5) category of submitter (such as consumer, government agency official, or public official); (6) submitter's contact information; (7) submitter's verification that the information contained therein is true and accurate; and (8) consent to publication of the report of harm. 15 U.S.C. § 2055a(b)(2)(B); C.F.R. § 1102.10(d). Subject to §§ 1102.24 and 1102.26. CPSC will publish reports of harm containing all the required information. [29] CPSC conducted these focus groups in July and August 2010. The focus groups comprised of professionals included individuals who might use the site in their professional capacity and included a cross- section of health professional, child care providers, public safety officials, and government officials. [30] Formal tests are day-long tests of a live or staged website, which include recruiting three test participants and providing tasks for them to accomplish. Observers are located in a separate room where they monitor the tests and take notes. During quick tests, paper printouts of web pages are distributed to a large group of people for feedback, usually at random and in a public space. Mobile tests consist of tests of sites on smartphones and mobile applications. User behavior is recorded by a camera focusing on the mobile device and the hands of the user. Observation involves other staff watching GSA officials conduct usability tests of sites, either in person at GSA or remotely. Expert evaluation includes assessing a site for usability categories including navigation, identity, accessibility, content and design, and preparing an expert report. [31] Different parties also write each document, with CPSC writing recall notices and consumers and other users of SaferProducts.gov, such as health care professionals, writing incident reports. [32] For more information on key practices for consumer education planning, see [hyperlink, http://www.gao.gov/products/GAO-08-43. [33] See [hyperlink, http://www.gao.gov/products/GAO-12-30. [34] Usability.gov is a resource for federal web designers to make websites more usable, useful, and accessible. [End of section] GAO’s Mission: The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO’s website [hyperlink, http://www.gao.gov]. Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to [hyperlink, http://www.gao.gov] and select “E-mail Updates.” Order by Phone: The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO: Connect with GAO on facebook, flickr, twitter, and YouTube. Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts. Visit GAO on the web at [hyperlink, http://www.gao.gov]. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; E-mail: fraudnet@gao.gov; Automated answering system: (800) 424-5454 or (202) 512-7470. Congressional Relations: Katherine Siggerud, Managing Director, siggerudk@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, DC 20548. Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, DC 20548. [End of document]