Volume 32 - Article 26 | Pages 797–828  

Do low survey response rates bias results? Evidence from Japan

By Ronald R. Rindfuss, Minja K. Choe, Noriko O. Tsuya, Larry L. Bumpass, Emi Tamaki

Abstract

Background: In developed countries, response rates have dropped to such low levels that many in the population field question whether the data can provide unbiased results.

Objective: The paper uses three Japanese surveys conducted in the 2000s to ask whether low survey response rates bias results. A secondary objective is to bring results reported in the survey response literature to the attention of the demographic research community.

Methods: Using a longitudinal survey as well as paradata from a cross-sectional survey, a variety of statistical techniques (chi square, analysis of variance (ANOVA), logistic regression, ordered probit or ordinary least squares regression (OLS), as appropriate) are used to examine response-rate bias.

Results: Evidence of response-rate bias is found for the univariate distributions of some demographic characteristics, behaviors, and attitudinal items. But when examining relationships between variables in a multivariate analysis, controlling for a variety of background variables, for most dependent variables we do not find evidence of bias from low response rates.

Conclusions: Our results are consistent with results reported in the econometric and survey research literatures. Low response rates need not necessarily lead to biased results. Bias is more likely to be present when examining a simple univariate distribution than when examining the relationship between variables in a multivariate model.

Comments: The results have two implications. First, demographers should not presume the presence or absence of low response-rate bias; rather they should test for it in the context of a specific substantive analysis. Second, demographers should lobby data gatherers to collect as much paradata as possible so that rigorous tests for low response-rate bias are possible.

Author’s Affiliation

• Ronald R. Rindfuss - University of North Carolina at Chapel Hill, United States of America EMAIL
• Minja K. Choe - East-West Center, United States of America EMAIL
• Noriko O. Tsuya - Keio University, Japan EMAIL
• Larry L. Bumpass - University of Wisconsin–Madison, United States of America EMAIL
• Emi Tamaki - Ritsumeikan University, Japan EMAIL

Other articles by the same author/authors in Demographic Research

Employment and household tasks of Japanese couples, 1994-2009
Volume 27 - Article 24

Cohabitation and children's living arrangements: New estimates from the United States
Volume 19 - Article 47

Marital Dissolution in Japan: Recent Trends and Patterns
Volume 11 - Article 14

The topography of the divorce plateau: Levels and trends in union stability in the United States after 1980
Volume 8 - Article 8

Most recent similar articles in Demographic Research

Fertility differences across immigrant generations in the United Kingdom
Volume 52 - Article 33    | Keywords: event history analysis, fertility, immigrant, second generation, United Kingdom

Amish fertility in the United States: Comparative evidence from the American Community Survey and Amish population registries
Volume 52 - Article 26    | Keywords: American Community Survey (ACS), Amish, fertility, natural fertility, total fertility rate (TFR)

Examining the relationships between education, coresidential unions, and the fertility gap by simulating the reproductive life courses of Dutch women
Volume 52 - Article 24    | Keywords: contraception, education, fertility, GGS, life course, LISS, microsimulation, Netherlands, physiology, unions

Decomposing delayed first marriage and birth across cohorts: The role of increased employment instability among men in Japan
Volume 52 - Article 21    | Keywords: cohort analysis, delayed effects, delayed marriage, delayed parenthood, Japan, precarious employment

Job creation, job destruction, and fertility in Germany
Volume 52 - Article 13    | Keywords: fertility, gender, Germany, job creation, job destruction, labor market, spatial modelling, unemployment