Conducting online behavioral research using crowdsourcing services in Japan

Frontiers in Psychology

Volumn 8, Issue MAR, 2017, Pages

  Majima, Yoshimasa ^a   Nishiyama, Kaoru ^a   Nishihara, Aki ^a   Hata, Ryosuke ^a  

^a HOKUSEI GAKUEN UNIVERSITY   (Japan)

Author keywords

Instructional manipulation check; Non MTurk crowdsourcing; Online study; Personality; Reasoning

Indexed keywords

EID: 85016279918     PISSN: None     EISSN: 16641078     Source Type: Journal    
DOI: 10.3389/fpsyg.2017.00378     Document Type: Article

References (39)

1. Behrend, T. S., Sharek, D. J., Meade, A. W., and Wiebe, E. N. (2011). The viability of crowdsourcing for survey research. Behav. Res. Methods 43, 800-813. doi: 10.3758/s13428-011-0081-0
2. Buhrmester, M., Kwang, T., and Gosling, S. D. (2011). Amazon's mechanical turk: a new source of inexpensive, yet high-quality, data? Perspect. Psychol. Sci. 6, 3-5. doi: 10.1177/1745691610393980
3. Chandler, J., Mueller, P., and Paolacci, G. (2014). Nonnaïveté among Amazon Mechanical Turk workers: consequences and solutions for behavioral researchers. Behav. Res. Methods 46, 112-130. doi: 10.3758/s13428-013-0365-7
4. Chandler, J., Paolacci, G., Peer, E., Mueller, P., and Ratliff, K. A. (2015). Using nonnaive participants can reduce effect sizes. Psychol. Sci. 26, 1131-1139. doi: 10.1177/0956797615585115
5. Crump, M. J., McDonnell, J. V., and Gureckis, T. M. (2013). Evaluating amazon's mechanical turk as a tool for experimental behavioral research. PLoS ONE 8:e57410. doi: 10.1371/journal.pone.0057410
6. eMarketer (2014). Worldwide Smartphone Usage to Grow 25% in 2014. Available online at: http://www.emarketer.com/Article/Worldwide-Smartphone-Usage-Grow-25-2014/1010920 (Accessed).
7. Estelles-Arolas, E., and Gonzalez-Ladron-De-Guevara, F. (2012). Towards an integrated crowdsourcing definition. J. Inf. Sci. 38, 189-200. doi: 10.1177/0165551512437638
8. Evans, J. S., Barston, J. L., and Pollard, P. (1983). On the conflict between logic and belief in syllogistic reasoning. Mem. Cognit. 11, 295-306. doi: 10.3758/BF03196976
9. Frederick, S. (2005). Cognitive reflection and decision making. J. Econ. Perspect. 19, 25-42. doi: 10.1257/089533005775196732
10. Gleibs, I. H. (2016). Are all "research fields" equal? Rethinking practice for the use of data from crowdsourcing market places. Behav. Res. Methods. doi: 10.3758/s13428-016-0789-y. [Epub ahead of print].
11. Goodman, J. K., Cryder, C. E., and Cheema, A. (2013). Data collection in a flat world: the strengths and weaknesses of mechanical turk samples. J. Behav. Decis. Making 26, 213-224. doi: 10.1002/bdm.1753
12. Gosling, S. D., Rentfrow, P. J., and Swann, W. B. Jr. (2003). A very brief measure of the big-five personality domains. J. Res. Pers. 37, 504-528. doi: 10.1016/S0092-6566(03)00046-1
13. Hauser, D. J., and Schwarz, N. (2015). It's a trap! Instructional manipulation checks prompt systematic thinking on "Tricky" tasks. SAGE Open 5:2158244015584617. doi: 10.1177/2158244015584617
14. Hauser, D. J., and Schwarz, N. (2016). Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behav. Res. Methods 48, 400-407. doi: 10.3758/s13428-015-0578-z
15. Kawamoto, T., Oshio, A., Abe, S., Tsubota, Y., Hirashima, T., Ito, H., et al. (2015). Big Five personality tokusei no nenreisa to seisa: daikibo oudan chousa ni yoru kentou [Age and gender differences of Big Five personality traits in a cross-sectional Japanese sample]. Jpn. J. Develop. Psychol. 26, 107-122.
16. Kling, K. C., Hyde, J. S., Showers, C. J., and Buswell, B. N. (1999). Gender differences in self-esteem: a meta-analysis. Psychol. Bull. 125, 470-500. doi: 10.1037/0033-2909.125.4.470
17. Lutz, J. (2016). The validity of crowdsourcing data in studying anger and aggressive behavior. Soc. Psychol. 47, 38-51. doi: 10.1027/1864-9335/a000256
18. Majima, Y. (2015). Belief in pseudoscience, cognitive style and science literacy. Appl. Cognit. Psychol. 29, 552-559. doi: 10.1002/acp.3136
19. Markovits, H., and Nantel, G. (1989). The belief-bias effect in the production and evaluation of logical conclusions. Mem. Cognit. 17, 11-17. doi: 10.3758/BF03199552
20. Mason, W., and Suri, S. (2012). Conducting behavioral research on Amazon's mechanical Turk. Behav. Res. Methods 44, 1-23. doi: 10.3758/s13428-011-0124-6
21. McCrae, R. R., Costa, P. T. Jr., Ostendorf, F., Angleitner, A., Hrebícková, M., Avia, M. D., et al. (2000). Nature over nurture: temperament, personality, and life span development. J. Pers. Soc. Psychol. 78, 173-186. doi: 10.1037/0022-3514.78.1.173
22. Okada, R., Oshio, A., Mogaki, M., Wakita, T., and Namikawa, T. (2015). Nihon-jin ni okeru jison kanjou no seisa ni kansuru meta bunseki [A meta-analysis of gender differences in self-esteem in Japanese]. Jpn. J. Pers. 24, 49-60. doi: 10.2132/personality.24.49
23. Oppenheimer, D. M., Meyvis, T., and Davidenko, N. (2009). Instructional manipulation checks: detecting satisficing to increase statistical power. J. Exp. Soc. Psychol. 45, 867-872. doi: 10.1016/j.jesp.2009.03.009
24. Oshio, A., Abe, S., and Cutrone, P. (2012). Nihongoban ten item personality inventory (TIPI-J) sakusei no kokoromi [Development, reliability, and validity of the Japanese version of ten item personality inventory (TIPI-J)]. Jpn. J. Pers. 21, 40-52. doi: 10.2132/personality.21.40
25. Paolacci, G., and Chandler, J. (2014). Inside the turk: understanding mechanical turk as a participant pool. Curr. Dir. Psychol. Sci. 23, 184-188. doi: 10.1177/0963721414531598
26. Paolacci, G., Chandler, J., and Ipeirotis, P. (2010). Running experiments on Amazon mechanical turk. Judge. Decis. Making 5, 411-419. Retrieved from: http://journal.sjdm.org/10/10630a/jdm10630a.pdf
27. Peer, E., Brandimarte, L., Samat, S., and Acquisti, A. (2017). Beyond the Turk: alternative platforms for crowdsourcing behavioral research. J. Exp. Soc. Psychol. 70, 153-163. doi: 10.1016/j.jesp.2017.01.006
28. Richins, M. L. (2004). The material values scale: measurement properties and development of a short form. J. Consum. Res. 31, 209-219. doi: 10.1086/383436
29. Robins, R. W., Trzesniewski, K. H., Tracy, J. L., Gosling, S. D., and Potter, J. (2002). Global self-esteem across the life span. Psychol. Aging 17, 423-434. doi: 10.1037/0882-7974.17.3.423
30. Rosenberg, M. (1965). Society and the Adolescent Self-Image. Princeton, NJ: Princeton University Press.
31. Srivastava, S., John, O. P., Gosling, S. D., and Potter, J. (2003). Development of personality in early and middle adulthood: set like plaster or persistent change? J. Pers. Soc. Psychol. 84, 1041-1053. doi: 10.1037/0022-3514.84.5.1041
32. Stewart, N., Ungemach, C., Harris, A. J. L., Bartels, D. M., Newell, B. R., Paolacci, G., et al. (2015). The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers. Judge. Decis. Making 10, 479-491. Retrieved from: http://journal.sjdm.org/14/14725/jdm14725.pdf
33. Toplak, M. E., West, R. F., and Stanovich, K. E. (2011). The cognitive reflection test as a predictor of performance on heuristics-and-biases tasks. Mem. Cogn. 39, 1275-1289. doi: 10.3758/s13421-011-0104-1
34. Tversky, A., and Kahneman, D. (1974). Judgment under uncertainty: heuristics and biases. Science 185, 1124-1131. doi: 10.1126/science.185.4157.1124
35. Tversky, A., and Kahneman, D. (1981). The framing of decisions and the psychology of choice. Science 211, 453. doi: 10.1126/science.7455683
36. Tversky, A., and Kahneman, D. (1983). Extensional versus intuitive reasoning: the conjunction fallacy in probability judgment. Psychol. Rev. 90, 293-315. doi: 10.1037/0033-295X.90.4.293
37. Vandewalle, D. (1997). Development and validation of a work domain goal orientation instrument. Educ. Psychol. Meas. 57, 995-1015. doi: 10.1177/0013164497057006009
38. Yamamoto, M., Matsui, Y., and Yamanari, Y. (1982). Ninchi sareta jiko no shosokumen no kouzou [The structure of perceived aspects of self]. Jpn. J. Educ. Psychol. 30, 64-68. doi: 10.5926/jjep1953.30.1_64
39. Zhou, H., and Fishbach, A. (2016). The pitfall of experimenting on the web: How unattended selective attrition leads to surprising (yet false) research conclusions. J. Pers. Soc. Psychol. 111, 493-504. doi: 10.1037/pspa0000056