Comparing the accuracy of RDD telephone surveys and internet surveys conducted with probability and non-probability samples

Public Opinion Quarterly

Volumn 75, Issue 4, 2011, Pages 709-747

  Yeager, David S ^a   Krosnick, Jon A ^a   Chang, Linchiat ^b   Javitz, Harold S ^c   Levendusky, Matthew S ^d   Simpser, Alberto ^e   Wang, Rui ^a  

^a STANFORD UNIVERSITY   (United States)

^b LinChiat Chang Consulting LLC   (United States)

^c SRI INTERNATIONAL   (United States)

^d UNIVERSITY OF PENNSYLVANIA   (United States)

^e UNIVERSITY OF CHICAGO   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 80755155713     PISSN: 0033362X     EISSN: 15375331     Source Type: Journal    
DOI: 10.1093/poq/nfr020     Document Type: Article

References (93)

1. Adams, Swann Arp, Charles E. Matthews, Cara B. Ebbeling, Charity G. Moore, Joan E. Cunningham, Jeanette Fulton, and James R. Hebert. 2005. "The Effect of Social Desirability and Social Approval on Self-Reports of Physical Activity." American Journal of Epidemiology 161:389-98. (Pubitemid 40256210)
2. Aguinis, Herman, Charles A. Pierce, and Brian M. Quigley. 1993. "Conditions Under Which a Bogus Pipeline Procedure Enhances the Validity of Self-Reported Cigarette Smoking: A Meta-Analytic Review." Journal of Applied Social Psychology 23:352-73.
3. -. 1995. "Enhancing the Validity of Self-Reported Alcohol and Marijuana Consumption Using a Bogus Pipeline Procedure: A Meta-Analytic Review." Basic and Applied Social Psychology 16:515-27.
4. Baim, Julian, Michal Galin, Martin R. Frankel, Risa Becker, and Joe Agresti. 2009. "Sample Surveys Based on Internet Panels: 8 Years of Learning." Paper presented at the Worldwide Readership Symposium, Valencia, Spain.
5. Baker, Reg, Dan Zahs, and George Popa. 2004. "Health Surveys in the 21st Century: Telephone vs. Web." Paper presented at the Eighth Conference on Health Survey Research Methods, Peachtree City, GA.
6. Battaglia, Michael P., David Izrael, David C. Hoaglin, and Martin R. Frankel. 2009. "Practical Considerations in Raking Survey Data." Survey Practice, (June). http://surveypractice.org/2009/06/29/raking-survey- data/.
7. Bender, Bruce G., Susan J. Bartlett, Cynthia S. Rand, Charles Turner, Frederick S. Wamboldt, and Lening Zhang. 2007. "Impact of Reporting Mode on Accuracy of Child and Parent Report of Adherence with Asthma Controller Medication." Pediatrics 120:e471-77. (Pubitemid 47358258)
8. Berrens, Robert P., Alok K. Bohara, Hank Jenkins-Smith, Carol Silva, and David L. Weimer. 2003. "The Advent of Internet Surveys for Political Research: A Comparison of Telephone and Internet Samples." Political Analysis 11:1-22.
9. Bethell, Christina, John Fiorillo, David Lansky, Michael Hendryx, and James Knickman. 2004. "Online Consumer Surveys as a Methodology for Assessing the Quality of the United States Health Care System." Journal of Medical Internet Research 6(1):e2.
10. Braunsberger, Karin, Hans Wybenga, and Roger Gates. 2007. "A Comparison of Reliability Between Telephone and Web-Based Surveys." Journal of Business Research 60:758-64.
11. Carbone, Enrica. 2005. "Demographics and Behavior." Experimental Economics 8:217-32.
12. Chang, LinChiat, and Jon A. Krosnick. 2002. "RDD Telephone vs. Internet Survey Methodology for Studying American Presidential Elections: Comparing Sample Representativeness and Response Quality." Paper presented at the American Political Science Association Annual Meeting, Boston.
13. -. 2009. "National Surveys via RDD Telephone Interviewing versus the Internet: Comparing Sample Representativeness and Response Quality." Public Opinion Quarterly 73:641-78.
14. -. 2010. "Comparing Oral Interviewing with Self-Administered Computerized Question naires: An Experiment." Public Opinion Quarterly 74:154-67.
15. Chatt, Cindy, Mike Dennis, Rick Li, and Paul Pulliam. 2003. "Data Collection Mode Effects Controlling for Sample Origins in a Panel Survey: Telephone versus Internet." Paper presented at the Annual Meeting of the Midwest Chapter of the American Association for Public Opinion Research, Chicago.
16. Cooley, Philip C, Heather G Miller, James N. Gribble, and Charles F. Turner. 2000. "Automating Telephone Surveys: Using T-ACASI to Obtain Data on Sensitive Topics." Computers in Human Behavior 16:1-11. (Pubitemid 30562174)
17. Couper, Mick P., Arie Kapteyn, Matthias Schonlau, and Joachim Winter. 2007. "Noncoverage and Nonresponse in an Internet Survey." Social Science Research 36:131-48. (Pubitemid 44734600)
18. Crampton, Thomas. 2007. "About Online Surveys, Traditional Pollsters Are: (C) Somewhat Disappointed." New York Times, May 31. http://www.nytimes.com/2007/05/31/business/media/31adco.html?pagewanted=print.
19. Crete, Jean, and Laura Stephenson. 2008. "Internet and Telephone Survey Methodology: An Evaluation of Mode Effects." Paper presented at the Annual Meeting of the Midwest Political Science Association, Chicago.
20. Curtin, Richard, Stanley Presser, and Eleanor Singer. 2000. "The Effects of Response Rate Changes on the Index of Consumer Sentiment." Public Opinion Quarterly 64:413-28. (Pubitemid 33241089)
21. DeBell, Matthew, and Jon A. Krosnick. 2009. "Weighting Plan for the American National Election Studies." American National Election Studies Technical Report, Ann Arbor, MI.
22. Dever, Jill A., Ann Rafferty, and Richard Valliant. 2008. "Internet Surveys: Can Statistical Adjustments Eliminate Coverage Bias?" Survey Research Methods 2:47-62.
23. Diamond, Alexis, and Jasjeet Sekhon. 2005. "Genetic Matching for Estimating Causal Effects: A General Multivariate Matching Method for Achieving Balance in Observational Studies." Working Paper, University of California at Berkeley. http://sekhon.berkeley.edu/papers/GenMatch.pdf.
24. Duffy, Bobby, Kate Smith, George Terhanian, and John Bremer. 2005. "Comparing Data from Online and Face-to-Face Surveys." International Journal of Market Research 47:615-39.
25. Elmore-Yalch, Rebecca, Jeffrey Busby, and Cynthia Britton. 2008. "Know Thy Customer? Know Thy Research! Comparison of Web-Based and Telephone Responses to a Public Service Customer Satisfaction Survey." Paper presented at the Transportation Research Board 2008 Annual Meeting, Washington, D.C.
26. Faas, Thorsten, and Harald Schoen. 2006. "Putting a Questionnaire on the Web Is Not Enough: A Comparison of Online and Offline Surveys Conducted in the Context of the German Federal Election 2002." Journal of Official Statistics 22:177-90.
27. Ferrell, Dan, and James C. Peterson. 2010. "The Growth of Internet Research Methods and the Reluctant Sociologist." Sociological Inquiry 80:114-25.
28. Gelman, Andrew. 2007. "Struggles with Survey Weighting and Regression Modeling." Statistical Science 22:153-64.
29. Ghanem, Khalil, Heidi E. Hutton, Jonathan M. Zenilman, Rebecca Zimba, and Emily J. Erbelding. 2005. "Audio Computer Assisted Self-Interview and Face-to-Face Interview Modes in Assessing Response Bias Among STD Clinic Patients." Sexually Transmitted Infections 81:421-25. (Pubitemid 41494359)
30. Graham, Patricia. 2005. "The Decision-Maker's Guide to Online Research." http://www.knowledge networks.com/dmg/dmg-09010.html.
31. -. 2007. "Using Known Benchmarks to Inform the Accuracy of Online Research." Presen tation at the Advertising Research Foundation Online Research Quality Meeting, New York. http://s3.amazonaws.com/thearf-org-aux- assets/downloads/cnc/orqc/09-10-07-ORQC-Graham. pdf.
32. Harmon, Thomas, Charles F. Turner, Susan M. Rogers, Elizabeth Eggleston, Anthony M. Roman, Maria A. Villarroel, James R. Chromy, Laxminarayana Ganapathi, and Sheping Li. 2009. "Impact of T-ACASI on Survey Measurements of Subjective Phenomena." Public Opinion Quarterly 73:255-80.
33. Harris Interactive. 2004. "Final Pre-Election Harris Polls: Still Too Close to Call But Kerry Makes Modest Gains." http://www.prnewswire.com/ news-releases/final-pre-election-harris-polls-still-too-close-to-call-but- internet-poll-results-suggest-a-kerry-victory-75137942.html.
34. -. 2008. "Election Results Further Validate Efficacy of Harris Interactive's Online Methodology." Business Wire, November 6. http://ir.harrisinteractive.com/releasedetail.cfm?ReleaseID=396524.
35. Henry, Peter J. 2008. "College Sophomores in the Laboratory Redux: Influences of a Narrow Data Base on Social Psychology's View of the Nature of Prejudice." Psychological Inquiry 19:49-71.
36. Hochstim, Joseph R. 1967. "A Critical Comparison of Three Strategies of Collecting Data from Households." Journal of the American Statistical Association 62:976-89.
37. Holbrook, Allyson, Melanie Green, and Jon A. Krosnick. 2003. "Telephone versus Face-to-Face Interviewing of National Probability Samples with Long Questionnaires: Comparisons of Respondent Satisficing and Social Desirability Response Bias." Public Opinion Quarterly 67:79-125.
38. Holbrook, Allyson, and Jon A. Krosnick. 2010. "Social Desirability Bias in Voter Turnout Reports: Tests Using the Item Count Technique." Public Opinion Quarterly 74:37-67.
39. Holbrook, Allyson, Jon A. Krosnick, and Alison Pfent. 2007. "The Causes and Consequences of Response Rates in Surveys by the News Media and Government Contractor Survey Research Firms." In Advances in Telephone Survey Methodology, eds. James M. Lepkowski, Clyde Tucker, J. Michael Brick, Edith de Leeuw, Lilli Japec, Paul J. Lavrakas Michael W. Link, and Roberta L. Sangster. New York: Wiley-Interscience.
40. Kapteyn, Arie, James P. Smith, and Arthur van Soest. 2007. "Vignettes and Self-Reports of Work Disability in the United States and the Netherlands." American Economic Review 97:461-73.
41. Keeter, Scott, Courtney Kennedy, Michael Dimock, Jonathan Best, and Peyton Craighill. 2006. "Gauging the Impact of Growing Nonresponse on Estimates from a National RDD Telephone Survey." Public Opinion Quarterly 70:759-79.
42. Keeter, Scott, Carolyn Miller, Andrew Kohut, Robert M. Groves, and Stanley Presser. 2000. "Consequences of Reducing Nonresponse in a National Telephone Survey." Public Opinion Quarterly 64:125-48.
43. Kellner, Peter. 2004. "Can Online Polls Produce Accurate Findings?" International Journal of Market Research 46:3-23.
44. Kish, L. 1992. "Weighting for Unequal P." Journal of Official Statistics 8:183-200.
45. Klein, Jonathan D., Randall K. Thomas, and Erika J. Sutter. 2007. "Self-Reported Smoking in Online Surveys: Prevalence Estimate Validity and Item Format Effects." Medical Care 45: 691-95.
46. Krosnick, Jon A., and Douglas Rivers. 2005a. "Web Survey Methodologies: A Comparison of Survey Accuracy." Paper presented at the Annual Meeting of the American Association for Public Opinion Research, Miami Beach, FL.
47. -. 2005b. "Comparing Major Survey Firms in Terms of Survey Satisficing: Telephone and Internet Data Collection." Paper presented at the Annual Meeting of the American Association for Public Opinion Research, Miami Beach, FL.
48. Krosnick, Jon A., Douglas Rivers, Alberto Simpser, Matthew Levendusky, and LinChiat Chang. 2005. "Web Survey Methodologies: A Comparison and Evaluation." Unpublished manuscript draft, Stanford University, Stanford, CA.
49. Lee, Sunghee. 2006. "Propensity Score Adjustment as a Weighting Scheme for Volunteer Panel Web Surveys." Journal of Official Statistics 22:329-49.
50. Lee, Sunghee, and Valliant Richard. 2009. "Estimation for Volunteer Panel Web Surveys Using Propensity Score Adjustment and Calibration Adjustment." Sociological Methods Research 37:319-43.
51. Lensvelt-Mulders, Getty, Peter Lugtig, and Marianne Hubregtse. 2009. "Separating Selection Bias and Non-Coverage in Internet Panels Using Propensity Matching." Survey Practice. (August). http://surveypractice.org/ .
52. Lerner, Jennifer S., Roxana M. Gonzalez, Deborah A. Small, and Baruch Fischhoff. 2003. "Effects of Fear and Anger on Perceived Risks of Terrorism: A National Field Experiment." Psychological Science 14:144-50. (Pubitemid 37283099)
53. Link, Michael W., and Ali H. Mokdad. 2004. "Are Web and Mail Feasible Options for the Behavioral Risk Factor Surveillance System?" Paper presented at the Eighth Conference on Health Survey Research Methods, Peachtree City, GA.
54. -. 2005. "Alternative Modes for Health Surveillance Surveys: An Experiment with Web, Mail, and Telephone." Epidemiology 16:701-4. (Pubitemid 41233628)
55. Loosveldt, Geert, and Nathalie Sonck. 2008. "An Evaluation of the Weighting Procedures for an Online Access Panel Survey." Survey Research Methods 2:93-105.
56. Malhotra, Neil, and Jon A. Krosnick. 2007. "The Effect of Survey Mode and Sampling on Inferences About Political Attitudes and Behavior: Comparing the 2000 and 2004 ANES to Internet Surveys with Nonprobability Samples." Political Analysis 15:286-324.
57. Malhotra, Neil, and Alexander G Kuo. 2008. "Attributing Blame: The Public's Response to Hurricane Katrina." Journal of Politics 70:120-35.
58. McHorney, Colleen A., Mark Kosinski, and John Ware, Jr. 1994. "Comparisons of the Costs and Quality of Norms for the SF-36 Health Survey Collected by Mail versus Telephone Interview: Results from a National Survey." Medical Care 32:551-67.
59. Merkle, Daniel, and Murray Edelman. 2002. "Nonresponse in Exit Polls: A Comprehensive Analysis." In Survey Nonresponse, eds. Robert Groves, Don Dillman, John Eltinge, and Roderick Little, pp. 243-58. New York: Wiley.
60. Metzger, David S., Beryl Koblin, Charles Turner, Helen Navaline, Francesca Valenti, Sarah Holte, Michael Gross, Amy Sheon, Heather Miller, and Philip Cooley. HIVNET Vaccine Preparedness Study Protocol Team. 2000. "Randomized Controlled Trial of Audio Computer-Assisted Self-Interviewing: Utility and Acceptability in Longitudinal Studies." American Journal of Epidemiology 152:99-106. (Pubitemid 30463656)
61. Moskalenko, Sophia, and Clark McCauley. 2009. "Measuring Political Mobilization: The Distinction Between Activism and Radicalism." Terrorism and Political Violence 21:239-60.
62. Newman, Jessica Clark Don C. Des Jarlais, Charles F. Turner, Jay Gribble, Phillip Cooley, and Denise Paone. 2002. "The Differential Effects of Face-to-Face and Computer Interview Modes." American Journal of Public Health 92:294-97. (Pubitemid 34101445)
63. Niemi, Richard, Kent Portney, and David King. 2008. "Sampling Young Adults: The Effects of Survey Mode and Sampling Method on Inferences About the Political Behavior of College Students." Paper presented at the Annual Meeting of the American Political Science Association, Boston.
64. Petty, Richard E., and John T. Cacioppo. 1996. "Addressing Disturbing and Disturbed Consumer Behavior: Is It Necessary to Change the Way We Conduct Behavioral Science?" Journal of Marketing Research 33:1-8. (Pubitemid 126109035)
65. Rogers, Susan M., Gordon Willis, Alia Al-Tayyib, Maria A. Villarroel, Charles F. Turner, Lazminarayana Ganapathi, Jonathan Zenilman, and Rosemary Jadack. 2005. "Audio Computer-Assisted Interviewing to Measure HIV Risk Behaviors in a Clinic Population." Sexually Transmitted Infections 81:501-7. (Pubitemid 41811334)
66. Roster, Catherine A., Robert D. Rogers, Gerald Albaum, and Darin Klein. 2004. "A Comparison of Response Characteristics from Web and Telephone Surveys." International Journal of Market Research 46:359-73.
67. Sanders, David, Harold D. Clarke, Marianne C. Stewart, and Paul Whiteley. 2007. "Does Mode Matter for Modeling Political Choice? Evidence from the 2005 British Election Study." Political Analysis 15:257-85.
68. Schillewaert, Niels, and Pascale Meulemeester. 2005. "Comparing Response Distributions of Offline and Online Data Collection Methods." International Journal of Market Research 47:163-78. (Pubitemid 40366694)
69. Schonlau, Matthias, Beth J. Asch, and Can Du. 2003. "Web Surveys as Part of a Mixed Mode Strategy for Populations That Cannot Be Contacted by E-Mail." Social Science Computer Review 21:218-22.
70. Schonlau, Matthias, Arthur van Soest, Arie Kapteyn, and Mick Couper. 2009. "Selection Bias in Web Surveys and the Use of Propensity Scores." Sociological Methods and Research 37: 291-318.
71. Schonlau, Matthias, Kinga Zapert, Lisa P. Simon, Katherine H. Sanstad, Sue M. Marcus, John Adams, Mark Spranca, Hongjun Kan, Rachel Turner, and Sandra H. Berry. 2004. "A Comparison Between Responses from a Propensity-Weighted Web Survey and an Identical RDD Survey." Social Science Computer Review 22:128-38. (Pubitemid 38191979)
72. Sears, David O. 1986. "College Sophomores in the Laboratory: Influences of a Narrow Data Base on Social Psychology's View of Human Nature." Journal of Personality and Social Psychology 51:515-30.
73. Siemiatycki, Jack. 1979. "A Comparison of Mail, Telephone, and Home Interview Strategies for Household Health Surveys." American Journal of Public Health 69:238-45.
74. Skitka, Linda J., and Christopher W. Bauman. 2008. "Moral Conviction and Political Engagement." Political Psychology 29:29-54.
75. Smith, Tom W. 2003. "An Experimental Comparison of Knowledge Networks and the GSS." International Journal of Public Opinion Research 15:167-79.
76. Smith, Tom W., and Michael J. Dennis. 2005. "Online vs. In-Person: Experiments with Mode, Format, and Question Wordings. " Public Opinion Pros (December). http://www.publicopinionpros. norc.org/from-field/2005/dec/smith. asp.
77. Sparrow, Nick. 2006. "Developing Reliable Online Polls." International Journal of Market Research 48:659-80.
78. Spijkerman, Renske, Ronald Knibbe, Kim Knoops, Dike van de Mheen, and Regina van den Eijnden. 2009. "The Utility of Online Panel Surveys versus Computer-Assisted Interviews in Obtaining Substance-Use Prevalence Estimates in the Netherlands." Addiction 104:1641-45.
79. Stirton, John, and Euan Robertson. 2005. "Assessing the Viability of Online Opinion Polling During the 2004 Federal Election." Australian Market and Social Research Society.
80. Taylor, Humphrey, John Bremer, Cary Overmeyer, Jonathan W. Siegel, and George Terhanian. 2001. "The Record of Internet-Based Opinion Polls in Predicting the Results of 72 Races in the November 2000 U.S. Elections." International Journal of Market Research 43:127-36. (Pubitemid 33346783)
81. Taylor, Humphrey, David Krane, and Randall K. Thomas. 2005. "Best Foot Forward: Social Desirability in Telephone vs. Online Surveys." Public Opinion Pros (February). http://www. publicopinionpros.norc.org/from-field/2005/ feb/taylor-2.asp.
82. Terhanian, George, Renee Smith, John Bremer, and Randall K. Thomas. 2001. "Exploiting Analytical Advances: Minimizing the Biases Associated with Internet-Based Surveys of Non-Random Samples." ARF/ESOMAR: Worldwide Online Measurement 248:247-72.
83. Turner, Charles F., Alia Al-Tayyib, Susan M. Rogers, Elizabeth Eggleston, Maria A. Villarroel, Anthony M. Roman, James R. Chromy, and Phillip C. Cooley. 2009. "Improving Epidemiological Surveys of Sexual Behavior Conducted by Telephone." International Journal of Epidemiology 38:1118-27.
84. Turner, Charles F., Maria A. Villarroel, Susan M. Rogers, Elizabeth Eggleston, Laxminarayana Ganapathi, Anthony M. Roman, and Alia Al-Tayyib. 2005. "Reducing Bias in Telephone Survey Estimates of the Prevalence of Drug Use: A Randomized Trial of Telephone Audio-CASI." Addiction 100:1432-44. (Pubitemid 41442313)
85. Twyman, Joe. 2008. "Getting It Right: YouGov and Online Survey Research in Britain." Journal of Elections, Public Opinion, and Parties 18:343-54.
86. van Ryzin, Gregg G. 2008. "Validity of an Online Panel Approach to Citizen Surveys." Public Performance and Management Review 32:236-62.
87. Vavreck, Lynn, and Douglas Rivers. 2008. "The 2006 Cooperative Congressional Election Study." Journal of Elections, Public Opinion, and Parties 18:355-66.
88. Villarroel, Maria A., Charles F. Turner, Elizabeth Eggleston, Alia Al-Tayyib, Susan M. Rogers, Anthony M. Roman, Philip C. Cooley, and Harper Gordek. 2006. "Same-Gender Sex in the USA: Impact of T-ACASI on Prevalence Estimates." Public Opinion Quarterly 70:166-96. (Pubitemid 43992276)
89. Vonk, Ted, Robert Ossenbruggen, and Pieter Willems. 2006. "The Effects of Panel Recruitment and Management on Research Results." In Panel Research 2006. Amsterdam: ESOMAR.
90. Yeager, David S., and Jon A. Krosnick. 2010. "The Validity of Self-Reported Nicotine Product Use in the 2001-2008 National Health and Nutrition Examination Survey." Medical Care 48:1128-32.
91. Zogby, Jonathan. 2007. "The New Polling Revolution: Opinion Researchers Overcome Their Hangups with Online Polling." Campaigns and Elections (May):16-19.
92. -. 2009. "For Interactive Polling, the Future Is Now." Campaigns and Elections: Politics. (June). http://politicsmagazine.com/magazine- issues/june-2009/for-interactive-polling-the-future-is-now/.
93. -. 2007. "Down with RandomSamples." http://my.yougov.com/ commentaries/peter-kellner/down-with-random-samples.aspx.