Volume 54 - Article 42 | Pages 1375–1412
Click, collect, compare: Evaluating a nonprobability web survey for family demography
By Leonie Diffené, Thomas Leopold, Zafer Buyukkececi, Marcel Raab
References
Adler, M.A. (2023). The complexities of family diversity in the contemporary US. In: Adler, M.A. and Lenz, K. (eds.). The changing faces of families. Routledge: 187–212.
Albertini, M. and Kohli, M. (2013). The generational contract in the family: An analysis of transfer regimes in Europe. European Sociological Review 29(4): 828–840.
Alburez-Gutierrez, D., Williams, I., and Caswell, H. (2023). Projections of human kinship for all countries. Proceedings of the National Academy of Sciences 120(52): e2315722120.
Ansolabehere, S. and Rivers, D. (2013). Cooperative survey research. Annual Review of Political Science 16(1): 307–329.
Ansolabehere, S. and Schaffner, B.F. (2014). Does survey mode still matter? Findings from a 2010 multi-mode comparison. Political Analysis 22(3): 285–303.
Arndt, A.D., Ford, J.B., Babin, B.J., and Luong, V. (2022). Collecting samples from online services: How to use screeners to improve data quality. International Journal of Research in Marketing 39(1): 117–133.
Baker, R., Brick, J.M., Bates, N.A., Battaglia, M., Couper, M.P., Dever, J.A., Gile, K.J., and Tourangeau, R. (2013). Summary report of the AAPOR task force on non-probability sampling. Journal of Survey Statistics and Methodology 1(2): 90–143.
Batrinca, B. and Treleaven, P.C. (2015). Social media analytics: A survey of techniques, tools and platforms. AI and Society 30(1): 89–116.
Bengtson, V.L. and Roberts, R.E.L. (1991). Intergenerational solidarity in aging families: An example of formal theory construction. Journal of Marriage and the Family 53(4): 856–870.
Benson, H. (2020). Divorce rates are falling across Europe and the UK is leading the way. Romford: Marriage Foundation.
Bergmann, M. (2011). IPFWEIGHT: Stata module to create adjustment weights for surveys. Boston College Department of Economics.
Bishop, C.D., Leite, W.L., and Snyder, P.A. (2018). Using propensity score weighting to reduce selection bias in large-scale data sets. Journal of Early Intervention 40(4): 347–362.
Bornstein, M.H., Jager, J., and Putnick, D.L. (2013). Sampling in developmental science: Situations, shortcomings, solutions, and standards. Developmental Review 33(4): 357–370.
Buhrmester, M., Kwang, T., and Gosling, S.D. (2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science 6(1): 3–5.
Callegaro, M., Baker, R., Bethlehem, J., Göritz, A.S., Krosnick, J.A., and Lavrakas, P.J. (2014). Online panel research: Data quality perspective. Wiley.
Campisi, N., Kulu, H., Mikolai, J., Klüsener, S., and Myrskylä, M. (2020). Spatial variation in fertility across Europe: Patterns and determinants. Population, Space and Place 26(4): e2308.
Castrén, A.M., Česnuitytė, V., Crespi, I., Gauthier, J.A., Gouveia, R., Martin, C., Moreno Mínguez, A., and Suwada, K. (2021). The Palgrave handbook of family sociology in Europe. Palgrave Macmillan.
Caswell, H. (2020). The formal demography of kinship II: Multistate models, parity, and sibship. Demographic Research 42(38): 1097–1146.
Caswell, H. (2022). The formal demography of kinship IV: Two-sex models and their approximations. Demographic Research 47(13): 359–396.
Caswell, H. (2019). The formal demography of kinship: A matrix formulation. Demographic Research 41(24): 679–712.
Caswell, H. and Song, X. (2021). The formal demography of kinship III: Kinship dynamics with time-varying demographic rates. Demographic Research 45(16): 517–546.
Chen, Y., Li, P., and Wu, C. (2020). Doubly robust inference with nonprobability survey samples. Journal of the American Statistical Association 115(532): 2011–2021.
Chmielewski, M. and Kucker, S.C. (2020). An MTurk crisis? Shifts in data quality and the impact on study results. Social Psychological and Personality Science 11(4): 464–473.
Chong, M., Alburez-Gutierrez, D., Alexander, M.J., and Zagheni, E. (2022). Correcting historical mortality rate bias in big crowd-sourced online genealogies. Max Planck Institute for Demographic Research Working Paper WP-2022-005, Max Planck Institute for Demographic Research Working Paper WP-2022-005.
Cochran, W.G. (1977). Sampling techniques. Wiley.
Colasurdo, A. and Omenti, R. (2024). Using online genealogical data for demographic research: An empirical examination of the FamiLinx database. Demographic Research 51(41): 1299–1350.
De Bel, V., Bokányi, E., Hank, K., and Leopold, T. (2024). A parallel kinship universe? Using Dutch kinship network data to replicate Kolk et al.’s (2023) demographic account of kinship networks in Sweden. OSF preprint.
Dudel, C. and Klüsener, S. (2021). Male–female fertility differentials across 17 high-income countries: Insights from a new data resource. European Journal of Population 37(2): 417–441.
Elliott, M.R. and Valliant, R. (2017). Inference for nonprobability samples. Statistical Science 32(2): 249–264.
Erens, B., Burkill, S., Couper, M.P., Conrad, F., Clifton, S., Tanton, C., Phelps, A., Datta, J., Mercer, C.H., Sonnenberg, P., Prah, P., Mitchell, K.R., Wellings, K., Johnson, A.M., and Copas, A.J. (2014). Nonprobability web surveys to measure sexual behaviors and attitudes in the general population: A comparison with a probability sample interview survey. Journal of Medical Internet Research 16(12): e276.
Feng, K., Song, X., and Caswell, H. (2025). The present and future dementia burden in China: Kinship-based projections and global comparison. Demography 62(6): 1917–1944.
Fingerman, K.L., Huo, M., and Birditt, K.S. (2020). Mothers, fathers, daughters, and sons: Gender differences in adults’ intergenerational ties. Journal of Family Issues 41(9): 1597–1625.
Freese, J. and Jin, O. (2025). Online nonprobability samples. Annual Review of Sociology 51: 109–128.
Gil-Clavel, S. and Mulder, C.H. (2024). Does Twitter data mirror the European North–South family ties divide? A comparative analysis of tweets about family. Population Research and Policy Review 43(4): 48.
Gittelman, S.H., Thomas, R.K., Lavrakas, P.J., and Lange, V. (2015). Quota controls in survey research: A test of accuracy and intersource reliability in online samples. Journal of Advertising Research 55(4): 368–379.
Hank, K. (2007). Proximity and contacts between older parents and their children: A European comparison. Journal of Marriage and Family 69(1): 157–173.
Hays, R.D., Liu, H., and Kapteyn, A. (2015). Use of internet panels to conduct surveys. Behavior Research Methods 47(3): 685–690.
Herbell, K. and Zauszniewski, J.A. (2018). Facebook or Twitter? Effective recruitment strategies for family caregivers. Applied Nursing Research 41: 1–4.
Hünteler, B., Hank, K., Alburez-Guiterrez, D., and Leopold, T. (2026). A typology of younger adults’ nuclear and extended family relations in the United States. Journal of Marriage and Family 88(1): 284–298.
Jerit, J. and Barabas, J. (2023). Are nonprobability surveys fit for purpose? Public Opinion Quarterly 87(3): 816–840.
Kaplanis, J., Gordon, A., Shor, T., Weissbrod, O., Geiger, D., Wahl, M., Gershovits, M., Markus, B., Sheikh, M., Gymrek, M., Bhatia, G., MacArthur, D.G., Price, A.L., and Erlich, Y. (2018). Quantitative analysis of population-scale family trees with millions of relatives. Science 360(6385): 171–175.
Kennedy, C., Mercer, A., Keeter, S., Hatley, N., McGeeney, K., and Gimenez, A. (2016). Evaluating online nonprobability surveys. Vendor choice matters; widespread errors found for estimates based on blacks and Hispanics. Pew Research Center (Numbers, Facts and Trends Shaping the World).
Kocar, S. and Baffour, B. (2023). Comparing and improving the accuracy of nonprobability samples: Profiling Australian surveys. Methods, Data, Analyses 17(2): 36.
Kohut, A., Keeter, S., Doherty, C., Dimock, M., and Christian, L. (2012). Assessing the representativeness of public opinion surveys. Pew Research Center.
Kolk, M., Andersson, L., Pettersson, E., and Drefahl, S. (2023). The Swedish kinship universe: A demographic account of the number of children, parents, siblings, grandchildren, grandparents, aunts/uncles, nieces/nephews, and cousins using national population registers. Demography 60(5): 1359–1385.
Kuhnt, A.K. and Steinbach, A. (2014). Diversität von Familie in Deutschland. In: Steinbach, A., Hennig, M., and Arránz Becker, O. (eds.). Familie im Fokus der Wissenschaft. Wiesbaden: Springer Fachmedien: 41–70.
Lavrakas, P.J., Pennay, D., Neiger, D., and Phillips, B. (2022). Comparing probability-based surveys and nonprobability online panel surveys in Australia: A total survey error perspective. Survey Research Methods 16(2): 241–266.
Lee, S. and Valliant, R. (2009). Estimation for volunteer panel web surveys using propensity score adjustment and calibration adjustment. Sociological Methods and Research 37(3): 319–343.
Lehdonvirta, V., Oksanen, A., Räsänen, P., and Blank, G. (2021). Social media, web, and panel surveys: Using non‐probability samples in social and policy research. Policy and Internet 13(1): 134–155.
Leopold, T., Becker, C.C., Buyukkececi, Z., Çineli, B., and Raab, M. (2025). KINMATRIX - Uncovering the kinship matrix (Version 1.0.0) [electronic resource]. GESIS.
Leopold, T., Becker, C.C., Buyukkececi, Z., Çineli, B., and Raab, M. (2024). KINMATRIX: A new data resource for studies of families and kinship. Demographic Research 51(25): 789–808.
Leopold, T. and Kalmijn, M. (2026). Kinship in Black and White families: The strength of horizontal ties. American Journal of Sociology (forthcoming).
Leopold, T., Raab, M., Becker, C.C., Buyukkececi, Z., and Çineli, B. (2025). Mapping modern kinship networks: First results from the KINMATRIX survey. Journal of Marriage and Family 87(2): 478–504.
Levy, P.S. and Lemeshow, S. (2008). Sampling of populations: Methods and applications. Wiley.
Lindner, J.R., Murphy, T.H., and Briers, G.E. (2001). Handling nonresponse in social research. Journal of Agricultural Education 42(4): 43–53.
MacInnis, B., Krosnick, J.A., Ho, A.S., and Cho, M.J. (2018). The accuracy of measurements with probability and nonprobability survey samples: Replication and extension. Public Opinion Quarterly 82(4): 707–744.
Mercer, A.W., Kreuter, F., Keeter, S., and Stuart, E.A. (2017). Theory and practice in nonprobability surveys. Public Opinion Quarterly 81(S1): 250–271.
Mocho, H., Martins, C., Dos Santos, R., and Nunes, C. (2024). Parental involvement and stress in children’s quality of life: A longitudinal study with Portuguese parents during the COVID-19 pandemic period. Children 11(4): 440.
Mondal, M., Punuru, A., Cheng, T.W.S., Vargas, K., Gundry, C., Driggs, N.S., Schill, N., Carlson, N., Bedwell, J., Lorenc, J.Q., Ghosh, I., Li, Y., Fulda, N., and Page, X. (2023). A tale of two cultures: Comparing interpersonal information disclosure norms on Twitter. Proceedings of the ACM on Human-Computer Interaction 7(CSCW2): 1–40.
Mönkediek, B. and Bras, H. (2014). Strong and weak family ties revisited: Reconsidering European family structures from a network perspective. The History of the Family 19(2): 235–259.
Nandram, B., Choi, J.W., and Liu, Y. (2021). Integration of nonprobability and probability samples via survey weights. International Journal of Statistics and Probability 10(6): 5.
Paolacci, G. and Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science 23(3): 184–188.
Pasek, J. (2016). When will nonprobability surveys mirror probability surveys? Considering types of inference and weighting strategies as criteria for correspondence. International Journal of Public Opinion Research 28(2): 269–291.
Peer, E., Rothschild, D., Gordon, A., Evernden, Z., and Damer, E. (2021). Data quality of platforms and panels for online behavioral research. Behavior Research Methods 54(4): 1643–1662.
Pesando, L.M., Dorélien, A., St-Denis, X., and Santos, A. (2023). Demography as a field: Where we came from and where we are headed. Canadian Studies in Population 50(4).
Pruchno, R.A., Brill, J.E., Shands, Y., Gordon, J.R., Genderson, M.W., Rose, M., and Cartwright, F. (2008). Convenience samples and caregiving research: How generalizable are the findings? The Gerontologist 48(6): 820–827.
Reher, D.S. (1998). Family ties in Western Europe: Persistent contrasts. Population and Development Review 24(2): 203–234.
Reher, D.S. (2004). The demographic transition revisited as a global process. Population, Space and Place 10(1): 19–41.
Rohr, B., Silber, H., and Felderer, B. (2025). Comparing the accuracy of univariate, bivariate, and multivariate estimates across probability and non-probability surveys with population benchmarks. Sociological Methodology 55(1): 121–154.
Rosenbaum, P.R. and Rubin, D.B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika 70(1): 41–55.
Salvatore, C. (2023). Inference with non-probability samples and survey data integration: A science mapping study. METRON 81(1): 83–107.
Sivak, E. and Smirnov, I. (2018). Gender bias in sharenting: Both men and women mention sons more often than daughters on social media.
Smith, M.B., Blakemore, J.K., Ho, J.R., and Grifo, J.A. (2021). Making it (net)work: A social network analysis of “fertility” in Twitter before and during the COVID-19 pandemic. F&S Reports 2(4): 472–478.
Smith, T.W. (1995). Trends in non-response rates. International Journal of Public Opinion Research 7(2): 157–171.
Stelter, R. and Alburez-Gutierrez, D. (2022). Representativeness is crucial for inferring demographic processes from online genealogies: Evidence from lifespan dynamics. Proceedings of the National Academy of Sciences 119(10): e2120455119.
Sytkina, K. and Leopold, T. (2026). Parental separation and the quality of ties to extended kin. (forthcoming).
Tofallis, C. (2015). A better measure of relative prediction accuracy for model selection and model estimation. Journal of the Operational Research Society 66(8): 1352–1362.
Tourangeau, R., Michael Brick, J., Lohr, S., and Li, J. (2017). Adaptive and responsive survey designs: A review and assessment. Journal of the Royal Statistical Society Series A: Statistics in Society 180(1): 203–223.
United Nations (2024). World population prospects 2024: Summary of results. United Nations, UN DESA/POP/2024/TR/NO. 9.
Vehovar, V., Toepoel, V., and Steinmetz, S. (2016). Non-probability sampling. In: Wolf, C., Joye, D., Smith, T., and Fu, Y. (eds.). The SAGE handbook of survey methodology. Newcastle: SAGE: 329–345.
Vicente, P. (2023). Sampling Twitter users for social science research: Evidence from a systematic review of the literature. Quality and Quantity 57(6): 5449–5489.
Williams, I. and Alburez-Gutierrez, D. (2025). DemoKin: Estimate population kin distribution. GitHub.
Williams, I., Alburez-Gutierrez, D., Song, X., and Caswell, H. (2021). DemoKin: An R package to implement demographic matrix kinship models. CRAN.
Wiśniowski, A., Sakshaug, J.W., Perez Ruiz, D.A., and Blom, A.G. (2020). Integrating probability and nonprobability samples for survey inference. Journal of Survey Statistics and Methodology 8(1): 120–147.
Yeager, D.S., Krosnick, J.A., Chang, L., Javitz, H.S., Levendusky, M.S., Simpser, A., and Wang, R. (2011). Comparing the accuracy of RDD telephone surveys and internet surveys conducted with probability and non-probability samples. Public Opinion Quarterly 75(4): 709–747.
Zack, E.S., Kennedy, J., and Long, J.S. (2019). Can nonprobability samples be used for social science research? A cautionary tale. Survey Research Methods 13(2): 215–227.