The pitfall of experimenting on the Web: How unattended selective attrition leads to surprising (yet false) research conclusions

Journal of Personality and Social Psychology

Volumn 111, Issue 4, 2016, Pages 493-504

  Zhou, Haotian ^a   Fishbach, Ayelet ^a  

^a UNIVERSITY OF CHICAGO   (United States)

Author keywords

Dropout rate; Mechanical Turk; Selective attrition; Web experiment

Indexed keywords

EXPERIMENTAL MODEL; EXPERIMENTAL STUDY; HUMAN; PSYCHOLOGY; PSYCINFO; WEIGHT REDUCTION; ADULT; BEHAVIORAL RESEARCH; INTERNET; METHODOLOGY; PATIENT SELECTION; SOCIAL PSYCHOLOGY; STANDARDS;

ADULT; BEHAVIORAL RESEARCH; HUMANS; INTERNET; PATIENT SELECTION; PSYCHOLOGY, SOCIAL; RESEARCH DESIGN;

EID: 84973864110     PISSN: 00223514     EISSN: None     Source Type: Journal    
DOI: 10.1037/pspa0000056     Document Type: Article

References (20)

1. Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon.com's Mechanical Turk. Political Analysis, 20, 351-368. http://dx.doi.org/10.1093/pan/ mpr057
2. Birnbaum, M. H., & Mellers, B. A. (1989). Mediated models for the analysis of confounded variables and self-selected samples. Journal of Educational Statistics, 14, 146-158. http://dx.doi.org/10.3102/ 10769986014002146
3. Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon's Mechanical Turk a new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 3-5. http://dx.doi.org/10.1177/ 1745691610393980
4. Fernbach, P. M., Rogers, T., Fox, C. R., & Sloman, S. A. (2013). Political extremism is supported by an illusion of understanding. Psychological Science, 24, 939-946. http://dx.doi.org/10.1177/0956797612464058
5. Goodman, J. K., Cryder, C. E., & Cheema, A. (2013). Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making, 26, 213-224. http://dx.doi.org/ 10.1002/bdm.1753
6. Göritz, A. S. (2014). Determinants of the starting rate and the completion rate in online panel studies. In R. Callegaro, R. Baker, J. Bethlehem, A. S. Göritz, J. A. Krosnick, & P. J. Lavrakas (Eds.), Online panel research (pp. 154-170). Retrieved from http://onlinelibrary.wiley.com/ doi/10.1002/9781118763520.ch7/summary
7. Hauser, D. J., & Schwarz, N. (2016). Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behavior Research Methods, 48, 400-407.
8. Henderson, M. D. (2013). When seeing the forest reduces the need for trees: The role of construal level in attraction to choice. Journal of Experimental Social Psychology, 49, 676-683. http://dx.doi.org/10.1016/j.jesp.2013.03.001
9. Horton, J. J., Rand, D. G., & Zeckhauser, R. J. (2011). The online laboratory: Conducting experiments in a real labor market. Experimental Economics, 14, 399-425. http://dx.doi.org/10.1007/s10683-011-9273-9
10. Kraut, R., Olson, J., Banaji, M., Bruckman, A., Cohen, J., & Couper, M. (2004). Psychological research online: Report of Board of Scientific Affairs' Advisory Group on the Conduct of Research on the Internet. American Psychologist, 59, 105-117. http://dx.doi.org/10.1037/0003-066X.59.2.105
11. May, F., & Monga, A. (2014). When Time Has a Will of Its Own, the Powerless Don't Have the Will to Wait: Anthropomorphism of Time Can Decrease Patience. Journal of Consumer Research, 40, 924-942. http://dx.doi.org/10.1086/673384
12. Musch, J., & Reips, U.-D. (2000). A brief history of Web experimenting. In Psychological experiments on the Internet (pp. 61-87). San Diego, CA: Academic Press. http://dx.doi.org/10.1016/B978-012099980-4/ 50004-6
13. Paolacci, G., & Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science, 23, 184-188. http://dx.doi.org/10.1177/0963721414531598
14. Reips, U.-D. (2000). The Web experiment method: Advantages, disadvantages, and solutions. Psychological Experiments on the Internet, 89-117.
15. Ross, J., Irani, L., Silberman, M., Zaldivar, A., & Tomlinson, B. (2010). Who are the crowdworkers?: Shifting demographics in Mechanical Turk. In CHI'10 extended abstracts on Human factors in computing systems (pp. 2863-2872). Retrieved from http://dl.acm.org/citation.cfm?id=1753873
16. Schwarz, N., Bless, H., Strack, F., Klumpp, G., Rittenauer-Schatka, H., & Simons, A. (1991). Ease of retrieval as information: Another look at the availability heuristic. Journal of Personality and Social Psychology, 61, 195-202. http://dx.doi.org/10.1037/0022-3514.61.2.195
17. Wisman, A., Heflick, N., & Goldenberg, J. L. (2015). The great escape: The role of self-esteem and self-related cognition in terror management. Journal of Experimental Social Psychology, 60, 121-132. http://dx.doi.org/10.1016/j.jesp.2015.05.006
18. Wolpert, R. M., & Gimpel, J. G. (1998). Self-interest, symbolic politics, and public attitudes toward gun control. Political Behavior, 20, 241-262. http://dx.doi.org/10.1023/A:1024814624070
19. Woltin, K.-A., & Yzerbyt, V. (2015). Regulatory focus in predictions about others. Personality and Social Psychology Bulletin, 41, 379-392. http:// dx.doi.org/10.1177/0146167214566188
20. Yam, K. C., Chen, X.-P., & Reynolds, S. J. (2014). Ego depletion and its paradoxical effects on ethical decision making. Organizational Behavior and Human Decision Processes, 124, 204-214. http://dx.doi.org/10.1016/j.obhdp.2014.03.008