Skip to Content


Submission for OMB Review; Comment Request

Document Details

Information about this document as published in the Federal Register.

Document Statistics
Document page views are updated periodically throughout the day and are cumulative counts for this document including its time on Public Inspection. Counts are subject to sampling, reprocessing and revision (up or down) throughout the day.
Published Document

This document has been published in the Federal Register. Use the PDF linked in the document sidebar for the official electronic format.

Start Preamble

The Department of Commerce will submit to the Office of Management and Budget (OMB) for clearance the following proposal for collection of information under the provisions of the Paperwork Reduction Act (44 U.S.C. chapter 35).

Agency: U.S. Census Bureau.

Title: Generic Clearance for Questionnaire Pretesting Research.

OMB Control Number: 0607-0725.

Form Number(s): Various.

Type of Request: Extension of a currently approved collection.

Burden Hours: 16,500 over the next three years.

Number of Respondents: 5,500 annually.

Average Hours per Response: 1 hour.

Needs and Uses: In recent years, there has been an increased interest among federal agencies and others in the importance of testing questionnaires. This interest has been spurred by a recognition that the traditional methods of pretesting are weak tools for evaluating questionnaires and procedures. These methods consist of a small “hothouse” field test accompanied by interviewer debriefing, and the information collected through their use is quite limited in its ability to detect and diagnose problems with the instruments and the procedures being tested.

In response to this recognition, new methods have come into popular use, which are useful for identifying questionnaire and procedural problems, suggesting solutions, and measuring the relative effectiveness of alternative solutions. Through the use of these kinds of techniques, employed routinely in the testing phase of Census Bureau surveys, questionnaires can be simplified for respondents, respondent burden can be reduced, and the quality of the questionnaires used in continuing and one-time surveys can be improved. Thus an increase in the quality of the data collected through these surveys can be achieved as well.

In September 1991, the Census Bureau requested and received a generic clearance (Number 0607-0725) on an experimental basis, which relaxed some of the time constraints and enabled the Census Bureau to begin conducting extended cognitive and questionnaire design research as part of testing for its censuses and surveys. The clearance covered data collections in the demographic, economic, and decennial areas of the Bureau, and specifically applied to research that is focused on questionnaire design and procedures aimed at reducing measurement errors in surveys. Research on paying respondents was specifically excluded from the clearance. As part of the experimental clearance, the Census Bureau submitted to the Office of Management and Budget (OMB) a report that contained indicators of the work that was conducted under the clearance. At the end of the experimental period, the Census Bureau requested and received a three-year renewal of the clearance (through December 1995), covering the same kinds of research activities. As part of the clearance, the Census Bureau has submitted to OMB a report of pretesting activities at the end of each year of the clearance.

Subsequently, the Census Bureau has received six more three-year renewals of the generic clearance for pretesting (through August 2013). The current clearance contains approval for three additional types of activities: Research about incentives, expanded field tests conducted to include split sample questionnaire experiments in multiple panels, and usability testing of electronic instruments.

At this time, the Census Bureau is seeking another three-year renewal of the generic clearance for pretesting, with the same conditions as the previous clearance. This will enable the Census Bureau to continue providing support for pretesting activities, which is important given the length of time required to plan the activities.

The specific methods proposed for coverage by this clearance are described below. Also outlined are the procedures in place for keeping the Economics and Statistics Administration and OMB informed about the identity of the surveys and the nature of the research activities being conducted.

The methods proposed for use in questionnaire development are as follows:

Field test. For the purposes of this clearance, we are defining field tests as small data collection efforts of 500 cases or less, conducted among either purposive or statistically representative samples, for which evaluation of the questionnaire and/or procedures is the main objective and no plans to publish the data other than for purely methodological purposes are envisioned.

Field tests are an essential component of this clearance package because they serve as the vehicle for conducting standardized behavior coding of the interaction between the respondent and the interviewer. This methodology does not require any additional data collection above and beyond the field test—it involves applying a standardized coding scheme to the completion of a field interview, either by a coder using a tape-recording of the interview or by a “live” observer at the time of the interview. The coding scheme is designed to identify situations that occur during the interview that reflect problems with the questionnaire. For example, if respondents frequently interrupt the interviewer before the question is completed, the question may be too long. If respondents frequently give inadequate answers, this suggests there are some other problems with the question. Quantitative data derived from this type of standardized coding scheme can provide valuable information to identify problem areas in a questionnaire, and research (“New Techniques for Pretesting Survey Questions” by Cannell, Kalton, Oksenberg, Bischoping, and Fowler, 1989) has demonstrated that this is a more objective and reliable method of identifying problems than the traditional interviewer debriefing, which is typically the sole tool used to evaluate the results of a traditional field test.

Interviewer debriefing has advantages as well, since it utilizes the knowledge of the employees who have the closest contact with our respondents. In conjunction with other methods, we plan to use this method in our field tests to collect information about how interviewers react to the survey instruments.

Field tests conducted under this clearance will involve either purposive or statistically representative samples. Under this clearance a variety of surveys will be pretested, and the exact nature Start Printed Page 41030of the surveys and the samples is undetermined at present. However, due to the small nature of the tests, we expect that some will not involve representative samples. In these cases, samples will basically be convenience samples, which will be limited to specific geographic locations and may involve expired rotation groups of a current survey or census blocks that are known to have specific aggregate demographic characteristics. The needs of the particular sample will vary based on the content of the survey being tested, but the selection of sample cases will not be completely arbitrary in any instance.

Respondent debriefing questionnaire. In this method, standardized debriefing questionnaires are administered to respondents who have participated in a field test. The debriefing form is administered at the end of the questionnaire being tested, and contains questions that probe to determine how respondents interpret the questions and whether they have problems in completing the survey/questionnaire. This structured approach to debriefing enables quantitative analysis of data from a representative sample of respondents, to learn whether respondents can answer the questions, and whether they interpret them in the manner intended by the questionnaire designers.

Split sample experiments. This involves testing alternative versions of questionnaires, at least some of which have been designed to address problems identified in draft questionnaires or questionnaires from previous survey waves. The use of multiple questionnaires, randomly assigned to permit statistical comparisons, is the critical component here; data collection can include mail, telephone, or personal visit interviews or group sessions at which self-administered questionnaires are completed. Comparison of revised questionnaires against a control version, preferably, or against each other facilitates statistical evaluation of the performance of alternative versions of the questionnaire.

In any split sample experiments conducted under this clearance, alternative questionnaire versions will be tested. The number of versions tested and the number of cases per version will depend on the objectives of the test. We cannot specify with certainty a minimum panel size, although we would expect that no questionnaire versions would be administered to less than fifty persons in a split sample test.

Split sample tests that incorporate methodological questionnaire design experiments will have a larger maximum sample size (up to several hundred cases per panel) than field tests using other pretest methods. This will enable the detection of statistically significant differences, and facilitate methodological experiments that can extend questionnaire design knowledge more generally for use in a variety of Census Bureau data collection instruments. The Census Bureau will consult with OMB prior to submission regarding split sample tests with sample sizes over 1000.

Cognitive interviews. This method involves intensive, one-on-one interviews in which the respondent is typically asked to “think aloud” as he or she answers survey questions. A number of different techniques may be involved, including asking respondents to paraphrase questions, probing questions asked to determine how respondents came up with their answers, and so on. The objective is to identify problems of ambiguity or misunderstanding, or other difficulties respondents may have answering questions. This is frequently the first stage of revising a questionnaire.

Usability Interviews. This method involves getting respondent input to aid in the development of automated questionnaires and Web sites and associated materials. A number of different techniques may be involved, such as one-on-one usability interviews with think aloud, probing, and paraphrasing tasks, card-sorting techniques, and disability accommodation testing. The objective is to identify problems that keep respondents from completing automated questionnaires accurately and efficiently, with minimal burden or that prevent respondents from successfully navigating Web sites and finding the information they seek.

Focus groups. This method involves group sessions guided by a moderator, who follows a topical outline containing questions or topics focused on a particular issue, rather than adhering to a standardized questionnaire. Focus groups are useful for surfacing and exploring issues (e.g., confidentiality concerns) which people may feel some hesitation about discussing.

This clearance will only cover pretests that involve more extensive testing than the traditional field test with interviewer debriefing as the only evaluative component. Since the types of surveys included under the umbrella of the clearance are so varied, it is impossible to specify at this point what kinds of activities would be involved in any particular test. But at a minimum, one of the types of testing described above or some other form of cognitive pretesting would be incorporated into the testing program for each survey.

We will provide OMB with a copy of questionnaires, protocols, and debriefing materials in advance of any testing activity. Depending on the stage of questionnaire development, this may be the printed questionnaire from the last round of a survey or a revised draft based on analysis of other evaluation data. When the time schedule for a single survey permits multiple rounds of testing, the questionnaire(s) for each round will be provided separately. When split sample experiments are conducted, either in small group sessions or as part of a field test, all the questionnaires to be used will be provided. For a test of alternative procedures, the description and rationale for the procedures would be submitted. A brief description of the planned field activity will also be provided. OMB will endeavor to provide comments on substantive issues within 10 working days of receipt.

Any large field tests or dress rehearsals that follow from the initial questionnaire development activity included here are not covered by this generic clearance. Separate submissions for any such data collection efforts will be made.

The Census Bureau will consult with the Economics and Statistics Administration (ESA) and OMB prior to submission on the appropriateness of submissions under this clearance that may raise policy or substantive issues. With respect to ESA, this will include all research and testing related to the American Community Survey (ACS) and any testing of any activities directly related to the 2020 decennial. In addition, the Census Bureau will consult with ESA on any research and testing proposals that are presented to the Data Stewardship Executive Policy (DSEP) Committee. Consultation with ESA includes the Census Bureau providing copies of questionnaires, protocols, and debriefing materials in advance of any of the above-mentioned activities.

The Census Bureau will send ESA and OMB an annual report at the end of each year summarizing the number of hours used, as well as the nature and results of the activities completed under this clearance.

The information collected in this program of developing and testing questionnaires will be used by staff from the Census Bureau and sponsoring agencies to evaluate and improve the quality of the data in the surveys and censuses that are ultimately conducted. None of the data collected under this Start Printed Page 41031clearance will be published for its own sake.

Because the questionnaires being tested under this clearance are still in the process of development, the data that result from these collections are not considered official statistics of the Census Bureau or other Federal agencies. Data will be included in research reports prepared for sponsors inside and outside of the Census Bureau. The results may also be prepared for presentations related to survey methodology at professional meetings or publications in professional journals.

Affected Public: Individuals or households, business or other for-profit.

Frequency: On occasion.

Respondent's Obligation: Voluntary.

Legal Authority: Data collection for this project is authorized under the authorizing legislation for the questionnaire being tested. This may be Title 13, Sections 131, 141, 161, 181, 182, 193, and 301 for Census Bureau-sponsored surveys, and Title 13 and 15 for surveys sponsored by other Federal agencies. We do not now know what other titles will be referenced, since we do not know what survey questionnaires will be pretested during the course of the clearance.

OMB Desk Officer: Brian Harris-Kojetin, (202) 395-7314.

Copies of the above information collection proposal can be obtained by calling or writing Jennifer Jessup, Departmental Paperwork Clearance Officer, (202) 482-0336, Department of Commerce, Room 6616, 14th and Constitution Avenue NW., Washington, DC 20230 (or via the Internet at

Written comments and recommendations for the proposed information collection should be sent within 30 days of publication of this notice to Brian Harris-Kojetin, OMB Desk Officer either by fax (202-395-7245) or email (

Start Signature

Dated: July 3, 2013.

Glenna Mickelson,

Management Analyst, Office of the Chief Information Officer.

End Signature End Preamble

[FR Doc. 2013-16489 Filed 7-8-13; 8:45 am]