A critical review of studies investigating the quality of data obtained with online panels based on probability and nonprobability samples1

Online Panel Research: A Data Quality Perspective

Volumn 9781119941774, Issue , 2014, Pages 23-53

  Callegaro, Mario ^a   Villar, Ana ^b   Yeager, David ^c   Krosnick, Jon A ^d  

^a GOOGLE   (United Kingdom)

^b CITY UNIVERSITY   (United Kingdom)

^c UNIVERSITY OF TEXAS AT AUSTIN   (United States)

^d STANFORD UNIVERSITY   (United States)

Author keywords

Life of an online panel member; Multiple panel membership; Nonprobability online panel; Point estimates; Pre election polls; Probability online panel; Relationship among variables; Reproducibility of study results; Weighting

Indexed keywords

EID: 84927800015     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1002/9781118763520.ch2     Document Type: Chapter

References (64)

1. Anich, B. (2002). Trends in marketing research and their impact on survey research sampling. Imprints, May.
2. Baim, J., Galin, M., Frankel, M. R., Becker, R., & Agresti, J. (2009). Sample surveys based on Internet panels: 8 years of learning. Worldwide Readership Symposium. Valencia, Spain. Retrieved January 1, 2013, from: http://www.gfkmri.com/PDF/WWRS-MRI_SampleSurveysBasedOnInternetPanels.pdf.
3. Baker, R., Blumberg, S. J., Brick, J. M., Couper, M. P., Courtright, M., Dennis, J. M., Dillman, D. A., et al. (2010). Research synthesis. AAPOR report on online panels. Public Opinion Quarterly, 74, 711-781.
4. Baker, R., Brick, M. J., Bates, N., Battaglia, M. P., Couper, M. P., Dever, J. A., Gile, K. J., et al. (2013, May). Report of the AAPOR task-force on non-probability sampling.
5. Berrens, R. P., Bohara, A. K., Jenkins-Smith, H., Silva, C., & Weimer, D. L. (2003). The advent of internet surveys for political research: A comparison of telephone and internet samples. Political Analysis, 11, 1-22.
6. Bourque, C., & Lafrance, S. (2007). Web survey and representativeness: Close to three in ten Canadians do not have access to the Internet. Should we care? Canadian Journal of Marketing Research, 24, 16-21.
7. Bradley, N. (1999). Sampling for Internet surveys. An examination of respondent selection for Internet research. Journal of the Market Research Society, 41, 387-395.
8. Bryan, C. J., Walton, G. M., Rogers, T., & Dweck, C. S. (2011). Motivating voter turnout by invoking the self. Proceedings of the National Academy of Sciences, 108, 12653-12656.
9. Callegaro, M., & DiSogra, C. (2008). Computing response metrics for online panels. Public Opinion Quarterly, 72, 1008-1032.
10. Casdas, D., Fine, B., & Menictas, C. (2006). Attitudinal differences. Comparing people who belong to multiple versus single panels. Panel research 2006. Amsterdam: ESOMAR. Retrieved from:internal-pdf://Casdas-Fine-Menictas_2006_ESOMAR-0928660738/Casdas-Fine-Menictas_2006_ESOMAR.pdf.
11. Cavallaro, K. (2013). Theory of adaptation or survival of the fittest? Quirk'sMarketing Research Review, 27, 24-27.
12. Chan, P., & Ambrose, D. (2011). Canadian online panels: Similar or different? Vue, (January/February), 16-20. Retrieved from: http://www.mktginc.com/pdf/VUE%20JanFeb%202011021.pdf.
13. Coen, T., Lorch, J., & Piekarski, L. (2005). The effects of survey frequency on panelist responses. ESOMAR World Research Conference 2005. Budapest: ESOMAR.
14. Comley, P. (2007). Onlinemarket research. In M. van Hamersveld,&C. de Bont (Eds.), Market research handbook (5th ed., pp. 401-419). Chichester: John Wiley & Sons, Ltd.
15. Crassweller, A., Rogers, J., Graves, F., Gauthier, E., & Charlebois, O. (2011). In search of a new approach to measure newspaper audiences in Canada: The journey continues. Paper presented at the Print and Digital Research Forum, San Francisco, CA. Retrieved from: http://nadbank .com/en/system/files/PDRFNADbankEKOS.pdf.
16. Crassweller, A., Rogers, J., & Williams, D. (2008). Between random samples and online panels:Where is the next.lily pad? Paper presented at the ESOMAR panel research 2008. Retrieved from:http://www.nadbank.com/en/system/files/Final%20paper.pdf.
17. Crassweller, A., Williams, D., & Thompson, I. (2006). Online data collection: Solution or band-aid? Paper presented at theWorldwide Readership Research Symposia,Vienna. Retrieved March 15, 2012, from: http://www.nadbank.com/en/system/files/Presentation%20WRRS%20Final.pdf.
18. Das, M. (2012). Innovation in online data collection for scientific research: The Dutch MESS project. Methodological Innovations Online, 7, 7-24.
19. Dillman, D. A., & Messer, B. L. (2010). Mixed-mode surveys. In P. V. Marsden, & J. D. Wright (Eds.), Handbook of survey research (2nd ed., pp. 551-574). Howard House: Emerald Group.
20. Duffy, B., & Smith, K. (2005). Comparing data from online and face-to-face surveys. International Journal of Market Research, 47, 615-639.
21. EKOS. (2010, January). Ekos' observation of MRIA study: Canadian online panels: similar or different? Retrieved August 28, 2011, from: http://www.ekos.com/admin/articles/MRIA-Comparison -Panel-Study-2010-01-27.pdf.
22. Farrell, D., & Petersen, J. C. (2010). The growth of Internet research methods and the reluctant sociologist. Sociological Inquiry, 80, 114-125.
23. Felix, J., & Sikkel, D. (1991). Attrition bias in telepanel research. Kwantitatiewe Methoden, 61.
24. Fulgoni, G. (2005). The professional respondent problem in online panel surveys today. Paper presented at the Market Research Association Annual Conference, Chicago, IL. Retrieved from: http://www.docstoc.com/docs/30817010/Partnership-Opportunity-Use-of-comScores-Survey-Panels.
25. Gailey, R., Teal, D., & Haechrel, E. (2008). Sample factors that influence data quality. Paper presented at theAdvertising Research FoundationOnline Research Quality Council conference (ORQC). Retrieved January 1, 2013, from: http://s3.amazonaws.com/thearf-org-aux-assets/downloads/cnc /orqc/2008-09-16_ARF_ORQC_WaMu.pdf.
26. Gittelman, S., & Trimarchi, E. (2010). Online research... and all that jazz!: The practical adaption of old tunes to make new music. ESOMAR Online Research 2010. Amsterdam: ESOMAR.
27. Grey Matter Research. (2009). Dirty little secrets of online panels: And how the one you select can make or break your study. Retrieved from: http://greymatterresearch.com/index_files/Online_Panels.htm.
28. Grey Matter Research. (2012). More dirty little secrets of online panel research. Retrieved from: http://www.greymatterresearch.com/index_files/Online_Panels_2012.htm.
29. Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33(2-3), 61-83.
30. Henry, P. J. (2008). College sophomores in the laboratory redux: Influences of a narrow data base on social psychology's view of the nature of prejudice. Psychological Inquiry, 19, 49-71.
31. Jones, E. E. (1986). Interpreting interpersonal behavior: The effects of expectancies. Science, 234(4772), 41-46.
32. Malhotra, N., & Krosnick, J. A. (2007). The effect of survey mode and sampling on inferences about political attitudes and behavior: Comparing the 2000 and 2004 ANES to internet surveys with nonprobability samples. Political Analysis, 15, 286-323.
33. Markus, H. R., & Kitayama, S. (1991). Culture and the self: Implications for cognition, emotion, and motivation. Psychological Review, 98, 224-253.
34. Martinsson, J., Dahlberg, S., & Lundmark, O. S. (2013). Is accuracy only for probability samples? Comparing probability and non-probability samples in a country with almost full internet coverage. Paper presented at the 68th Annual Conference of the American Association for Public Opinion Research, Boston, MA. Retrieved from: http://www.lore.gu.se/digitalAssets/1455/1455221_martinsson--dahlberg-and-lundmark--2013--aapor-is-accuracy-only-for-probability-samples.pdf.
35. Miller, J. (2007). Burke panel quality R&D summary. Retrieved from: http://s3.amazonaws.com /thearf-org-aux-assets/downloads/cnc/orqc/09-10-07_ORQC_Miller.pdf.
36. Miller, J. (2008). Burke panel quality R&D.
37. Morgan, S. L., & Winship, C. (2007). Counterfactuals and causal inference: Methods and principles for social research (1st ed.). Cambridge: Cambridge University Press.
38. Pasek, J., & Krosnick, J. A. (2010, December 28). Measuring intent to participate and participation in the 2010 census and their correlates and trends: Comparisons of RDD telephone and non-probability sample internet survey data. U.S. Census Bureau. Retrieved from: http://www.census.gov/srd/papers/pdf/ssm2010-15.pdf.
39. Peterson, R. A. (2001). On the use of college students in social science research: Insights from a second-order meta-analysis. Journal of Consumer Research, 28, 450-461.
40. Petty, R. E., & Cacioppo, J. T. (1996). Addressing disturbing and disturbed consumer behavior: Is it necessary to change the way we conduct behavioral science? Journal of Marketing Research, 33, 1-8.
41. Postoaca, A. (2006). The anonymous elect. Market research through online access panels. Berlin:Springer.
42. Sanders, D., Clarke, H. D., Stewart, M. C., & Whiteley, P. (2007). Does mode matter for modeling political choice? Evidence from the 2005 British Election Study. Political Analysis, 15, 257-285.
43. Scherpenzeel, A., & Bethlehem, J. (2011). How representative are online-panels? Problems of coverage and selection and possible solutions. In M. Das, P. Ester, & L. Kaczmirek (Eds.), Social and behavioral research and the internet: Advances in applied methods and research strategies (pp. 105-132). New York: Routledge.
44. Sears, D. O. (1986). College sophomores in the laboratory: Influences of a narrow data base on social psychology's view of human nature. Journal of Personality and Social Psychology, 51, 515-530.
45. Sikkel, D., & Hoogendoorn, A. (2008). Panel surveys. In E. De Leeuw, J. Hox, & D. A. Dillman (Eds.), International handbook of survey methodology (pp. 479-499). New York: Lawrence Erlbaum Associates.
46. Silver, N. (2012, November 10). Which polls fared best (and worst) in the 2012 Presidential Race. Retrieved from: http://fivethirtyeight.blogs.nytimes.com/2012/11/10/which-polls-fared-best-and -worst-in-the-2012-presidential-race/.
47. Smith, T. W. (1995). Little things matter: A sampler of how differences in questionnaire format can affect survey responses. In American Statistical Association (Ed.), Proceedings of the Joint Statistical Meeting, Survey Research Methods Section (pp. 1046-1051). Washington, DC: AMSTAT.
48. Smyth, J. D., & Pearson, J. E. (2011). Internet survey methods: A review of strengths, weaknesses, and innovations. In M. Das, P. Ester, & L. Kaczmirek (Eds.), Social and behavioral research and the internet. Advances in applied methods and research strategies (pp. 11-44). New York: Taylor and Francis.
49. Sudman, S. & Wansink, B. (2002). Consumer panels. Chicago: American Marketing Association.
50. Taylor, H. (2007, January 15). The case for publishing (some) online polls. Retrieved January 1, 2013, from: http://www.pollingreport.com/ht_online.htm.
51. Taylor,H., Bremer, J., Overmeyer, C., Siegel, J.W.,&Terhanian, G. (2001). The record of internet-based opinion polls in predicting the results of 72 races in the November 2000 U.S. Elections. International Journal of Market Research, 43, 127-136.
52. Terhanian, G. (2013). Comment to the summary report of the AAPOR task force on non-probability sampling. Journal of Survey Statistics and Methodology, 1, 124-129.
53. Tourangeau, R., Conrad, F. C., & Couper, M. P. (2013). The science of web surveys. Oxford: Oxford University Press.
54. Van Ossenbruggen, R., Vonk, T., & Willems, P. (2006). Results of NOPVO. Paper presented at the Online panels, close up, Utrecht, Netherlands. Retrieved from: http://www.nopvo.nl/page0 /files/Results_NOPVO_English.pdf.
55. Vonk, T., van Ossenbruggen, R.,&Willems, P. (2006). The effects of panel recruitment and management on research results: A study across 19 online panels. Paper presented at the Panel research 2006, ESOMAR, Barcelona.
56. Walker, R., Pettit, R.,&Rubinson, J. (2009). Aspecial report from the Advertising Research Foundation:The foundations of quality initiative:Afive-part immersion into the quality of online research. Journal of Advertising Research, 49, 464-485.
57. Wells, W. D. (1993). Discovery-oriented consumer research. Journal of Consumer Research, 19, 489-504.
58. West, R. F., Toplak,M. E.,&Stanovich, K. E. (2008). Heuristics and biases as measures of critical thinking:Associations with cognitive ability and thinking dispositions. Journal of Educational Psychology, 100, 930-941.
59. Williams, J. (2012). Survey methods in an age of austerity. Driving value in survey design. International Journal of Market Research, 54, 35-47.
60. Yeager, D. S., & Krosnick, J. A. (2011). Does mentioning "some people" and "other people" in a survey question increase the accuracy of adolescents' self-reports? Developmental Psychology, 47, 1674-1679.
61. Yeager, D. S., & Krosnick, J. A. (2012). Does mentioning "Some People" and "Other People" in an opinion question improve measurement quality? Public Opinion Quarterly, 76, 131-141.
62. Yeager, D. S., Krosnick, J. A., Chang, L., Javitz, A. S., Levendusky, M. S., Simpser, A., & Wang, R. (2011). Comparing the accuracy of RDD telephone surveys and internet surveys conducted with probability and non-probability samples. Public Opinion Quarterly, 75, 709-747.
63. Yeager, D. S., Larson, S. B., Krosnick, J. A., & Thompson, T. (2011). Measuring Americans' issue priorities: A new version of the most important problem question reveals more concern about global warming and the environment. Public Opinion Quarterly, 75, 125-138.
64. YouGov. (2011). YouGov's record. Public polling results compared to other pollsters and actual outcomes. Retrieved January 1, 2013, from: http://cdn.yougov.com/today_uk_import/yg-archives -pol-trackers-record2011.pdf.