Theory and Practice in Nonprobability Surveys

Public Opinion Quarterly

Volumn 81, Issue , 2017, Pages 250-279

  Mercer, Andrew W ^a,b   Kreuter, Frauke ^b,c,d   Keeter, Scott ^a   Stuart, Elizabeth A ^e  

^a Pew Research Center   (United States)

^b UNIVERSITY OF MARYLAND   (United States)

^c UNIVERSITY OF MANNHEIM   (Germany)

^d INSTITUTE FOR EMPLOYMENT RESEARCH IAB   (Germany)

^e JOHNS HOPKINS BLOOMBERG SCHOOL OF PUBLIC HEALTH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 85019183550     PISSN: 0033362X     EISSN: 15375331     Source Type: Journal    
DOI: 10.1093/poq/nfw060     Document Type: Conference Paper

References (78)

1. Andridge, Rebecca R., Roderick J. A. Little. 2011. "Proxy Pattern-Mixture Analysis for Survey Nonresponse. " Journal of Official Statistics 27(2):153-80.
2. Angrist, Joshua D., Jörn-Steffen Pischke. 2009. Mostly Harmless Econometrics: An Empiricist's Companion. Princeton, NJ: Princeton University Press.
3. Ansolabehere, Stephen, Douglas Rivers. 2013. "Cooperative Survey Research. " Annual Review of Political Science 16(1):307-29.
4. Bremer, John. 2013. "Research Quality: The Interaction of Sampling and Weighting in Producing a Representative Sample Online: An Excerpt from the ARF's 'Foundations of Quality 2' initiative. " Journal of Advertising Research 53(4):363-71.
5. Brick, J. Michael. 2013. "Unit Nonresponse and Weighting Adjustments: A Critical Review. " Journal of Official Statistics 29(3):329-53.
6. -. 2015. "Compositional Model Inference. " In JSM Proceedings (Survey Research Methods Section), 299-307. Alexandria, VA: American Statistical Association.
7. Brigham, Nancy, Michael Fallig, Chuck Miller. 2014. "The Impact of Survey Routers on Sampling and Surveys: Unraveling the Mysteries of Survey-Router Design and Deployment. " Journal of Advertising Research 54(4):381-88.
8. Buskirk, Trent D., David J. Dutwin. 2015. "Selected or Self-Selected Part 2: Exploring Non-Probability and Probability Samples from Response Propensities to Participant Profiles to Outcome Distributions. " Paper presented at the Conference of the American Association for Public Opinion Research, Hollywood, FL, USA.
9. Buskirk, Trent D., Stanislav Kolenikov. 2015. "Finding Respondents in the Forest: A Comparison of Logistic Regression and Random Forest Models for Response Propensity Weighting and Stratification. " Survey Methods: Insights from the Field, Weighting: Practical Issues, "How To" Approach. Available at http://surveyinsights. org/p=5108.
10. Callegaro, Mario, Reg Baker, Jelke Bethlehem, Anja S. Goritz, Jon A. Krosnick, Paul J. Lavrakas. 2014. "Online Panel Research: History, Concepts, Applications and a Look at the Future. " In Online Panel Research: A Data Quality Perspective, edited by Mario Callegaro, Reg Baker, Jelke Bethlehem, Anja S. Goritz, Jon A. Krosnick, Paul J. Lavrakas, 1-22. New York: John Wiley and Sons.
11. Callegaro, Mario, Charles DiSogra. 2008. "Computing Response Metrics for Online Panels. " Public Opinion Quarterly 72(5):1008-32.
12. Callegaro, Mario, Katja Lozar Manfreda, Vasja Vehovar. 2015. Web Survey Methodology. Los Angeles: Sage.
13. Callegaro, Mario, Ana Villar, David Yeager, Jon A. Krosnick. 2014. "A Critical Review of Studies Investigating the Quality of Data Obtained with Online Panels Based on Probability and Nonprobability Samples. " In Online Panel Research: A Data Quality Perspective, edited by Mario Callegaro, Reg Baker, Jelke Bethlehem, Anja S. Goritz, Jon A. Krosnick, Paul J. L. avrakas, 23-53. West Sussex: Wiley and Sons.
14. Clark, Julia, Clifford Young, Robert Petrin. 2015. "Meta-Analysis of Online Panel and Non-Panel Sampling: Electoral and Non-Electoral Behavior Metrics. " Paper presented at the Conference of the American Association for Public Opinion Research, Hollywood, FL, USA.
15. Cole, Stephen R., Miguel A. Hernán, James M. Robins, Kathryn Anastos, Joan Chmiel, Roger Detels, Carolyn Ervin, et al. 2003. "Effect of Highly Active Antiretroviral Therapy on Time to Acquired Immunodeficiency Syndrome or Death Using Marginal Structural Models. " American Journal of Epidemiology 158(7):687-94.
16. Cole, Stephen R., Elizabeth A. Stuart. 2010. "Generalizing Evidence from Randomized Clinical Trials to Target Populations: The ACTG 320 Trial. " American Journal of Epidemiology 172(1):107-15.
17. Cook, Thomas D., William R. Shadish, Vivian C. Wong. 2008. "Three Conditions under Which Experiments and Observational Studies Produce Comparable Causal Estimates: New Findings from Within-Study Comparisons. " Journal of Policy Analysis and Management 27(4):724-50.
18. Couper, Mick P. 2000. "Web Surveys: A Review of Issues and Approaches. " Public Opinion Quarterly 64(4):464-94.
19. Craig, Benjamin M., Ron D. Hays, A. Simon Pickard, David Cella, Dennis A. Revicki, Bryce B. Reeve. 2013. "Comparison of US Panel Vendors for Online Surveys. " Journal of Medical Internet Research 15(11):e260.
20. Deville, Jean-Claude, Carl-Erik Sarndal. 1992. "Calibration Estimators in Survey Sampling. " Journal of the American Statistical Association 87(418):376-82.
21. DiSogra, Charles, Curtiss Cobb, Elisa Chan, J. Michael Dennis. 2011. "Calibrating Non-Probability Internet Samples with Probability Samples Using Early Adopter Characteristics. " In JSM Proceedings (Survey Methods Section), 4501-15. Alexandria, VA: American Statistical Association.
22. Eckman, Stephanie. 2016. "Does the Inclusion of Non-Internet Households in a Web Panel Reduce Coverage Bias" Social Science Computer Review 34(1):41-58.
23. Erens, Bob, Sarah Burkill, Mick P Couper, Frederick Conrad, Soazig Clifton, Clare Tanton, Andrew Phelps, et al. 2014. "Nonprobability Web Surveys to Measure Sexual Behaviors and Attitudes in the General Population: A Comparison with a Probability Sample Interview Survey. " Journal of Medical Internet Research 16(12):e276.
24. Fahimi, Mansour, Frances M. Barlas, Randall K Thomas, Nicole Buttermore. 2015. "Scientific Surveys Based on Incomplete Sampling Frames and High Rates of Nonresponse. " Survey Practice 8(6). Available at http://www.surveypractice. org/index. php/SurveyPractice/article/view/324.
25. Ghitza, Yair, Andrew Gelman. 2013. "Deep Interactions with MRP: Election Turnout and Voting Patterns among Small Electoral Subgroups. " American Journal of Political Science 57(3):762-76.
26. Gittelman, Steven H., Randall K. Thomas, Paul J. Lavrakas, Victor Lange. 2015. "Quota Controls in Survey Research: A Test of Accuracy and Intersource Reliability in Online Samples. " Journal of Advertising Research 55(4):368-79.
27. Greenland, Sander, James M. Robins. 1986. "Identifiability, Exchangeability, Epidemiological Confounding. " International Journal of Epidemiology 15(3):413-19.
28. -. 2009. "Identifiability, Exchangeability and Confounding Revisited. " Epidemiologic Perspectives & Innovations 6(4). Available at http://epi-perspectives. biomedcentral. com/articles/10. 1186/1742-5573-6-4.
29. Groves, Robert. 2006. "Nonresponse Rates and Nonresponse Bias in Household Surveys. " Public Opinion Quarterly 70(5):646-75.
30. Hernán, Miguel A., James M. Robins. 2006. "Estimating Causal Effects from Epidemiological Data. " Journal of Epidemiology and Community Health 60(7):578-86.
31. Ho, Daniel E., Kosuke Imai, Gary King, Elizabeth A. Stuart. 2011. "MatchIt: Nonparametric Preprocessing for Parametric Causal Inference. " Journal of Statistical Software 42(8):1-28.
32. Horvitz, Daniel G., Donovan J. Thompson. 1952. "A Generalization of Sampling Without Replacement from a Finite Universe. " Journal of the American Statistical Association 47(260):663-85.
33. Imbens, Guido W., Donald B. Rubin. 2015. Causal Inference for Statistics, Social, Biomedical Sciences. New York: Cambridge University Press.
34. Kalton, Graham, Ismael Flores-Cervantes. 2003. "Weighting Methods. " Journal of Official Statistics 19(2):81-97.
35. Keeter, Scott, Kyley McGeeney, Andrew Mercer, Nick Hatley, Eileen Patten, Andrew Perrin. 2015. "Coverage Error in Internet Surveys: Who Web-Only Surveys Miss and How That Affects Results. " Pew Research Center. Available at http://www.pewresearch. org/files/2015/09/2015-09-22-coverage-error-in-internet-surveys.pdf.
36. Keiding, Niels, Thomas A. Louis. 2016. "Perils and Potentials of Self-Selected Entry to Epidemiological Studies and Surveys. " Journal of the Royal Statistical Society, Series A: Statistics in Society 179(2):319-76.
37. Kennedy, Courtney, Andrew Mercer, Scott Keeter, Nick Hatley, Kyley Mcgeeney, Alejandra Gimenez. 2016. "Evaluating Online Nonprobability Surveys. " Pew Research Center. Available at http://www.pewresearch. org/files/2016/04/Nonprobability-report-May-2016-FINAL.pdf.
38. Kern, Holger L., Elizabeth A. Stuart, Jennifer Hill, Donald P. Green. 2016. "Assessing Methods for Generalizing Experimental Impact Estimates to Target Populations. " Journal of Research on Educational Effectiveness 9(1):103-27.
39. Lax, Jeffrey R., Justin H. Phillips. 2009. "Gay Rights in the States: Public Opinion and Policy Responsiveness. " American Political Science Review 103(3):367-86.
40. Lee, Brian K., Justin Lessler, Elizabeth A. Stuart. 2010. "Improving Propensity Score Weighting Using Machine Learning. " Statistics in Medicine 29(3):337-46.
41. Lee, Sunghee. 2006. "Propensity Score Adjustment as a Weighting Scheme for Volunteer Panel Web Surveys. " Journal of Official Statistics 22(2):329-49.
42. Lee, Sunghee, Richard Valliant. 2009. "Estimation for Volunteer Panel Web Surveys Using Propensity Score Adjustment and Calibration Adjustment. " Sociological Methods & Research 37(3):319-43.
43. Little, Roderick J. A., Donald B. Rubin. 2002. Statistical Analysis with Missing Data, 2nd ed. Hoboken, NJ: John Wiley and Sons.
44. Lorch, Jackie, Kristin Cavallaro, Robert van Ossenbruggen. 2010. "Sample Blending: 1 + 1 = 2. " Survey Sampling International. Available at https://www.surveysampling. com/site/assets/files/1584/sample-blending-1-1-2.pdf.
45. Manski, Charles F. 2007. Identification for Prediction and Decision. Cambridge, MA: Harvard University Press.
46. Myers, Jessica A., Jeremy A. Rassen, Joshua J. Gagne, Krista F. Huybrechts, Sebastian Schneeweiss, Kenneth J. Rothman, Marshall M. Joffe, Robert J. Glynn. 2011. "Effects of Adjusting for Instrumental Variables on Bias and Precision of Effect Estimates. " American Journal of Epidemiology 174(11):1213-22.
47. Park, David K., Andrew Gelman, Joseph Bafumi. 2004. "Bayesian Multilevel Estimation with Poststratification: State-Level Estimates from National Polls. " Political Analysis 12(4):375-85.
48. Pearl, Judea. 2009. Causality: Models, Reasoning and Inference, 2nd ed. Cambridge: Cambridge University Press.
49. Pearl, Judea, Elias Bareinboim. 2014. "External Validity: From Do-Calculus to Transportability across Populations. " Statistical Science 29(4):579-95.
50. Petersen, Maya L., Kristin E. Porter, Susan Gruber, Yue Wang, Mark J. van der Laan. 2012. "Diagnosing and Responding to Violations in the Positivity Assumption. " Statistical Methods in Medical Research 21(1):31-54.
51. Petrin, Robert A., Neale El-Dash. 2015. "Reaching Wider, Going Deeper: Incorporating Sample Source Variation and Other Considerations into MRP Adjustments of Polling Estimates from Blended River Samples. " Paper presented at the Conference of the American Association for Public Opinion Research, Hollywood, FL, USA.
52. Pettit, Annie. 2015. "Building a Quality Nonprobability Panel: Methods, Problems, Being Innovative. " Paper presented at the 2015 Conference of the American Association for Public Opinion Research, Hollywood, FL, USA.
53. Rivers, Douglas. 2007. "Sampling for Web Surveys. " Paper presented at the 2007 Joint Statistical Meetings, Salt Lake City, UT, USA.
54. Robins, James M. 1999a. "Association, Causation, Marginal Structural Models. " Synthese: An International Journal for Epistemology, Methodology, Philosophy of Science 121(1-2):151-79.
55. -. 1999b. "Marginal Structural Models versus Structural Nested Models as Tools for Causal Inference. " In Statistical Models in Epidemiology: The Environment and Clinical Trials, edited by M. Elizabeth Halloran and Donald Berry, 116:95-134. New York: Springer-Verlag.
56. Robins, James M., Andrea Rotnitzky, Daniel O. Scharfstein. 1999. "Sensitivity Analysis for Selection Bias and Unmeasured Confounding in Missing Data and Causal Inference Models. " In Statistical Models in Epidemiology: The Environment and Clinical Trials, edited by M. Elizabeth Halloran and Donald Berry, 1-92. New York: Springer-Verlag.
57. Rosenbaum, Paul R. 2002. Observational Studies. Springer Series in Statistics. New York: Springer.
58. -. 2005. "Sensitivity Analysis in Observational Studies. " In Encyclopedia of Statistics in Behavioral Science, edited by Brian S Everitt and David C Howell, 4:1809-14. Chichester: John Wiley & Sons.
59. Rosenbaum, Paul R., Donald B. Rubin. 1983. "The Central Role of the Propensity Score in Observational Studies for Causal Effects. " Biometrika 70(1):41-55.
60. Rubin, Donald B. 1974. "Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies. " Journal of Educational Psychology 66(5):688-701.
61. -. 1978. "Bayesian Inference for Causal Effects: The Role of Randomization. " Annals of Statistics 6(1):34-58.
62. Schnorf, Sebastian, Aaron Sedley, Martin Ortlieb, Allison Woodruff. 2014. "A Comparison of Six Sample Providers Regarding Online Privacy Benchmarks. " Symposium on Usable Privacy and Security Workshop on Privacy Personas and Segmentation. Available at http://static. googleusercontent. com/media/research. google. com/en//pubs/archive/42558. pdf.
63. Schonlau, Matthias, Arthur van Soest, Arie Kapteyn. 2007. "Are 'Webographic' or Attitudinal Questions Useful for Adjusting Estimates from Web Surveys Using Propensity Scoring" Survey Research Methods 1(3):155-63.
64. Schonlau, Matthias, Kinga Zapert, Lisa P. Simon, Katherine Haynes Sanstad, Sue M. Marcus, John Adams, Mark Spranca, Hongjun Kan, Rachel Turner, Sandra H. Berry. 2004. "A Comparison Between Responses from a Propensity-Weighted Web Survey and an Identical RDD Survey. " Social Science Computer Review 22(1):128-38.
65. Shadish, William R., Thomas D. Cook, Donald T. Campbell. 2002. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Belmont: Wadsworth Cengage Learning.
66. Steiner, Peter M., Thomas D. Cook, William R. Shadish, M. H. Clark. 2010. "The Importance of Covariate Selection in Controlling for Selection Bias in Observational Studies. " Psychological Methods 15(3):250-67.
67. Stuart, Elizabeth A. 2010. "The Use of Propensity Scores to Assess Generalizability. " Journal of the Royal Statistical Society, Series A: Statistics in Society 174(2):369-86.
68. Stuart, Elizabeth A., Catherine P. Bradshaw, Philip J. Leaf. 2015. "Assessing the Generalizability of Randomized Trial Results to Target Populations. " Prevention Science 16(3):475-85.
69. Taylor, Humphrey. 2000. "Does Internet Research Work" International Journal of Market Research 42(1):51-63.
70. Terhanian, George, John Bremer. 2000. "Confronting the Selection-Bias and Learning Effects Problems Associated with Internet Research. " Rochester, NY: Harris Interactive.
71. -. 2012. "A Smarter Way to Select Respondents for Surveys" International Journal of Market Research 54(6):751-80.
72. Valliant, Richard, Jill A. Dever. 2011. "Estimating Propensity Adjustments for Volunteer Web Surveys. " Sociological Methods & Research 40(1):105-37.
73. Valliant, Richard, Jill A. Dever, Frauke Kreuter. 2013. Practical Tools for Designing and Weighting Survey Samples. New York: Springer.
74. Valliant, Richard, Alan H. Dorfman, Richard M. Royall. 2000. Finite Population Sampling and Inference: A Prediction Approach. New York: John Wiley & Sons.
75. Wang, Wei, David Rothschild, Sharad Goel, Andrew Gelman. 2014. "Forecasting Elections with Non-Representative Polls. " International Journal of Forecasting 31(3):980-91.
76. West, Stephen G., Naihua Duan, Willo Pequegnat, Paul Gaist, Don C. Des Jarlais, David Holtgrave, José Szapocznik, et al. 2008. "Alternatives to the Randomized Controlled Trial. " American Journal of Public Health 98(8):1359-66.
77. Yeager, David S., Jon A. Krosnick, LinChiat Chang, Harold S. Javitz, Matthew S. Levendusky, Alberto Simpser, Rui Wang. 2011. "Comparing the Accuracy of RDD Telephone Surveys and Internet Surveys Conducted with Probability and Non-Probability Samples. " Public Opinion Quarterly 75(4):709-47.
78. Young, Clifford, John Vidmar, Julia Clark, Neale El-Dash. 2012. "Our Brave New World: Blended Online Samples and the Performance of Nonprobablity Approaches. " Ipsos Public Affairs. Available at http://www. ipsos-na. com/knowledge-ideas/public-affairs/points-of-view/q=brave-new-world.