A multi-group analysis of online survey respondent data quality: Comparing a regular USA consumer panel to MTurk samples

Journal of Business Research

Volumn 69, Issue 8, 2016, Pages 3139-3148

  Smith, Scott M ^a   Roster, Catherine A ^b   Golden, Linda L ^c   Albaum, Gerald S ^b  

^a BRIGHAM YOUNG UNIVERSITY   (United States)

^b UNIVERSITY OF NEW MEXICO   (United States)

^c UNIVERSITY OF TEXAS AT AUSTIN   (United States)

Author keywords

Consumer panels; Data quality; Internet surveys; Mechanical Turk; Respondents; Survey research

Indexed keywords

EID: 84955572928     PISSN: 01482963     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jbusres.2015.12.002     Document Type: Article

References (33)

1. Albaum G., Evangelista F., Medina N. Role of response behavior theory in survey research: A cross-national study. Journal of Business Research 1998, 42(2):115-125.
2. Baker R., Blumberg S.J., Brick M.J., Couper M.P., Courtright M., Dennis J.M., et al. Research Synthesis: AAPOR report on online panels. AAPOR executive council by a task force. AAPOR Standards Committee. Public Opinion Quarterly 2010, 74(4):711-781.
3. Braunsberger K., Wybenga H., Gates R. A comparison of reliability between telephone and web-based surveys. Journal of Business Research 2007, 60:758-764.
4. Brick M.J. The future of survey sampling. Public Opinion Quarterly 2011, 75(5):872-888.
5. Brüggen E., de Ruyter K., Wetzels M. What motivates respondents to participate in online panels?. Paper presented at the World Marketing Congress 2005, July, Academy of Marketing Science, Muenster, Germany.
6. Brüggen E., Wetzels M., de Ruyter K., Schillewaert N. Individual differences in motivation to participate in online panels: The effect on response rate and response quality perceptions. International Journal of Market Research 2011, 53(3):369-398.
7. Buhrmester M., Kwang T., Gosling S. Amazon's Mechanical Turk: A new source of inexpensive, yet high-quality, data?. Perspectives on Psychological Science 2011, 10.1177/1745691610393980.
8. Online panel research: A data quality perspective 2014, John Wiley & Sons, Inc., Chichester, West Sussex. 1st ed. M. Callegaro, R. Baker, J. Bethlehem, A.S. Göritz, J.A. Krosnick, P.J. Lavrakas (Eds.).
9. Comley P. Understanding the online panelist. Worldwide panel research: Developments and progress 2005, 409-424. ESOMAR, Amsterdam.
10. Couper M.P. Web surveys: A review of issues and approaches. Public Opinion Quarterly 2000, 64(4):464-494.
11. Downs J.S., Holbrook M.B., Sheng S., Cranor L.F. Are your participants gaming the system? Screening Mechanical Turk workers. CHI 2010: 1001 Users 2010, 2399-2402.
12. Golden L.L., Brockett P.L. Trials, tribulations and trust: Addressing issues in Internet surveys. Presented at the Academy of Marketing Science World Congress 2009, Oslo, Norway, July.
13. Golden L.L., Smith S. Data quality evidence for Internet survey use in intellectual property law. Presented at the Academy of Marketing Science 2010 Annual Conference 2010, May, Portland, Oregon.
14. Golden L.L., Larson J., Smith S. Data quality and sample integrity in Internet research. Presented at The 15th Biennial World Marketing Conference 2011, July, Academy of Marketing Science, Reims, France.
15. Gosling S.D., Vazire S., Srivastava S., John O.P. Should we trust web-based studies? A comparative analysis of six preconceptions about Internet questionnaires. American Psychologist 2004, 5993-104.
16. Greszki R., Meyer M., Schoen H. The impact of speeding on data quality innonprobability and freshly recruited probability-based online panels. Online panel research: A data quality perspective 2014, 238-262. John Wiley & Sons, Inc., Chichester, West Sussex. 1st. ed. Callegaro (Ed.).
17. Heerwegh D., Loosveldt G. Face-to-face versus web surveys in a high-Internet coverage population. Public Opinion Quarterly 2008, 72(5):836-846.
18. Hillygus D.S., Jackson N., Young M. Professional respondents innonprobability online panels. Online panel research: A data quality perspective 2014, 219-237. John Wiley & Sons, Inc., Chichester, West Sussex. 1st ed. Callegaro (Ed.).
19. Howe J. Crowdsourcing: A definition. Crowdsourcing: Tracking the rise of the amateur 2006, 2. Weblog.
20. Ipeirotis P. Turker demographics vs. Internet demographics Accessed February 25, 2014. http://www.behind-the-enemy-lines.com/2009/03/turker-demographics-vs-internet.html.
21. Karminska O., McCutcheon A.L., Billet J. Satisficing among reluctant respondents in a cross-national context. Public Opinion Quarterly 2010, 74:956-984.
22. Kazai G., Kamps J., Milic-Frayling N. The face of quality in crowdsourcing relevance labels: Demographics, personality and labeling accuracy. CIKM 2012: Proceedings of the 21st ACM International Conference on Information and Knowledge Management 2012, 2583-2586.
23. Kittur A., Chi E.H., Suh B. Crowdsourcing user studies with Mechanical Turk. Proc. CHI 2008 2008, 453-456. ACM Press.
24. Lee H., Lindquist J.D., Acito F. Managers' evaluation of research design and its impact on the use of research: An experimental approach. Journal of Business Research 1997, 39:231-240.
25. Menictas C., Wang P., Fine B. Assessing flat-lining response style bias in online research. Australasian Journal of Market & Social Research 2011, 19(2):34-44.
26. Morea S. Market research in the US. IBIS World Industry Report 54191 2014, April.
27. Murray G.R., Rugeley C.R., Mitchell D., Mondak J.J. Convenient yet not a convenience sample: Jury pools as experimental subject pools. Social Science Research 2013, 42:246-253.
28. Paolacci G., Chandler J., Ipeirotis P.G. Running experiments on Amazon Mechanical Turk, judgment and decision making 2010, 5(5):411-419.
29. Peterson R.A., Merunka D.R. Convenience samples of college students and research reproducibility. Journal of Business Research 2014, 67:1035-1041.
30. Poynter R. The handbook of online and social media research 2010, John Wiley & Sons, Inc., Chichester, West Sussex. 1st ed.
31. Poynter R., Compley P. Beyond online panels. Proceedings of ESOMAR Technovate Conference 2003, ESOMAR, Amsterdam.
32. de Rada V.D., Dominquez-Alvarez J.A. Response quality of self-administered questionnaires: A comparison between paper and web questionnaires. Social Science Computer Review 2013, 1-14. 10.1177/0894439313508516.
33. Ross J., Irani L., Silberman M.S., Zaldivar A., Tomlinson B. Who are the crowedworkers? Shifting demographics in Mechanical Turk. CHI 2010: Human factors in computing systems 2010, 2863-2872.