Skip to Content


Generic Clearance for Proposed Information Collection; Comment Request; Generic Clearance for Internet Nonprobability Panel Pretesting and Qualitative Survey Methods Testing

Document Details

Information about this document as published in the Federal Register.

Published Document

This document has been published in the Federal Register. Use the PDF linked in the document sidebar for the official electronic format.

Start Preamble


U.S. Census Bureau, Commerce.




The Department of Commerce, as part of its continuing effort to reduce paperwork and respondent burden, invites the general public and other Federal agencies to take this opportunity to comment on proposed and/or continuing information collections, as required by the Paperwork Reduction Act of 1995.


To ensure consideration, written comments must be submitted on or before March 14, 2017.


Direct all written comments to Jennifer Jessup, Departmental Paperwork Clearance Officer, Department of Commerce, Room 6616, 14th and Constitution Avenue NW., Washington, DC 20230 (or via the Internet at

Start Further Info


Requests for additional information or copies of the information collection instrument(s) and instructions should be directed to Jennifer Hunter Childs, U.S. Census Bureau, 4600 Silver Hill Road, Center for Survey Measurement, Washington, DC 20233 or (202)603-4827.

End Further Info End Preamble Start Supplemental Information


I. Abstract

The Census Bureau is committed to conducting research in a cost efficient manner. Prior to this generic clearance, several stages of testing occurred in research projects at the Census Bureau. As a first stage of research, the Census Bureau pretests questions on surveys or censuses and evaluates the usability and ease of use of Web sites using a small number of subjects during focus groups, usability and cognitive testing. These projects are in-person and labor-intensive, but typically only target samples of 20 to 30 respondents. This small-scale work is done through another existing OMB generic clearance. Often the second stage is a larger-scale field test with a split-panel design of a survey or a release of a Census Bureau data dissemination product with a feedback mechanism. The field tests often involve a lot of preparatory work and often are limited in the number of panels tested due to the cost considerations. They are often targeted at very large sample sizes with over 10,000 respondents per panel. These are typically done using stand-alone OMB clearances.

Cost efficiencies can occur by testing some research questions in a medium-scale test, using a smaller number of participants than what we typically use in a field test, yet a larger and more diverse set of participants than who we recruit for cognitive and usability tests. Using Internet panel pretesting, we can answer some research questions more thoroughly than in the small-scale testing, but less expensively than in the large-scale field test. This clearance established a medium-scale (defined as having sample sizes from 100-2000 per study), cost-efficient method of testing Start Printed Page 4285questions and contact strategies over the Internet through different types of nonprobability samples.

This research program will be used by the Census Bureau and survey sponsors to test alternative contact methods, including emails and text messages (via an opt-in strategy), improve online questionnaires and procedures, reduce respondent burden, and ultimately increase the quality of data collected in the Census Bureau censuses and surveys. We will use the clearance to conduct pretesting of decennial and demographic census and survey questionnaires prior to fielding them as well as communications and/or marketing strategies and data dissemination tools for the Census Bureau. The primary method of identifying measurement problems with the questionnaire or survey procedure is split panel tests. This will encompass both methodological and subject matter research questions that can be tested on a medium-scale nonprobability panel.

This research program will also be used by the Census Bureau for remote usability testing of electronic interfaces and to perform other qualitative analyses such as respondent debriefings. An advantage of using remote, medium-scale testing is that participants can test products at their convenience using their own equipment, as opposed to using Census Bureau-supplied computers. A diverse participant pool (geographically, demographically, or economically) is another advantage. Remote usability testing would use click through rates and other paradata, accuracy and satisfaction scores, and written qualitative comments to determine optimal interface designs and to obtain feedback from respondents.

The public is currently offered an opportunity to participate in this research remotely, by signing up for an online research panel. If a person opts in, the Census Bureau will occasionally email (or text, if applicable) the person an invitation to complete a survey for one of our research projects. Invited respondents will be told the topic of the survey, and how long it will take to complete it. Under this clearance, we will also conduct similar-scale and similarly designed research using other email lists to validate preliminary findings and expand the research.

II. Method of Collection

Split sample experiments. This involves testing alternative versions of questionnaires, invitations to questionnaires (e.g., emails or text messages), or Web sites, at least some of which have been designed to address problems identified in draft versions or versions from previous waves. The use of multiple questionnaires, invitations, or Web sites, randomly assigned to permit statistical comparisons, is the critical component here; data collection will be via the Internet. Comparison of revised questionnaires (or invitations) against a control version, preferably, or against each other facilitates statistical evaluation of the performance of alternative versions of the questionnaire (or invitation or Web site).

The number of versions tested and the number of cases per version will depend on the objectives of the test. We cannot specify with certainty a minimum panel size, although we would expect that no questionnaire versions would be administered to less than fifty respondents.

Split sample tests that incorporate methodological questionnaire design experiments will have a larger maximum sample size (up to several hundred cases per panel) than other pretest methods. This will enable the detection of statistically significant differences, and facilitate methodological experiments that can extend questionnaire design knowledge more generally for use in a variety of Census Bureau data collection instruments.

Usability Interviews: This method involves getting respondent input to aid in the development of automated questionnaires and Web sites and associated materials. The objective is to identify problems that keep respondents from completing automated questionnaires accurately and efficiently with minimal burden, or that prevent respondents from successfully navigating Web sites and finding the information they seek. Remote usability testing may be conducted under this clearance, whereby a user would receive an invitation to use a Web site or survey, then answer targeted questions about that experience.

Qualitative Interviews: This method involves one-on-one (or sometimes group) interviews in which the respondent is typically asked questions about survey content areas, survey questions or the survey process. A number of different techniques may be involved, including cognitive interviews and focus groups. The objective is to identify problems of ambiguity or misunderstanding, or other difficulties respondents may have answering survey questions in order to improve the information ultimately collected in large scale surveys and censuses.

Data collection for this project is authorized under the authorizing legislation for the questionnaire being tested. This authorization may be Title 13, United States Code (U.S.C.), Sections 131, 141, 161, 181, 182, 193, and 301 for Census Bureau-sponsored surveys, and Title 13 and 15 for surveys sponsored by other Federal agencies. We do not now know what other titles will be referenced, since we do not know what survey questionnaires will be pretested during the course of the clearance.

Literature on and considerations about the use of nonprobability samples for this type of work have recently been thoroughly covered by a Task Force commissioned by the American Association for Public Opinion Research and are well documented there (Baker, et al., 2013).

The information collected in this program of developing and testing questionnaires will be used by staff from the Census Bureau and sponsoring agencies to evaluate and improve the quality of the data in the surveys and censuses that are ultimately conducted. Because the questionnaires being tested under this clearance are still in the process of development, the data that result from these collections are not considered official statistics of the Census Bureau or other Federal agencies. Data will be included in research reports prepared for sponsors inside and outside of the Census Bureau. The results may also be prepared for presentations related to survey methodology at professional meetings or publications in professional journals.

III. Data

OMB Control Number: 0607-0978.

Form Number(s): TBD.

Type of Review: Extension of a Currently Approved Collection.

Affected Public: Individuals and households.

Number of Respondents: 60,000.

Average Hours per Response: 0.167

Burden Hours: 10,000.

Estimated Total Annual Cost to Public: None.

Respondent's Obligation: Voluntary.

Legal Authority: Data collection for this project is authorized under the authorizing legislation for the questionnaire being tested. This may be Title 13, U.S.C., Sections 131, 141, 161, 181, 182, 193, and 301 for Census Bureau-sponsored surveys, and Title 13 and 15 for surveys sponsored by other Federal agencies. We do not now know what other titles will be referenced, since we do not know what survey questionnaires will be pretested during the course of the clearance.

IV. Request for Comments

Comments are invited on: (a) Whether the proposed collection of information Start Printed Page 4286is necessary for the proper performance of the functions of the agency, including whether the information shall have practical utility; (b) the accuracy of the agency's estimate of the burden (including hours and cost) of the proposed collection of information; (c) ways to enhance the quality, utility, and clarity of the information to be collected; and (d) ways to minimize the burden of the collection of information on respondents, including through the use of automated collection techniques or other forms of information technology.

Comments submitted in response to this notice will be summarized and/or included in the request for OMB approval of this information collection; they also will become a matter of public record.

Start Signature

Sheleen Dumas,

PRA Departmental Lead, Office of the Chief Information Officer.

End Signature End Supplemental Information

[FR Doc. 2017-00584 Filed 1-12-17; 8:45 am]