The validity of crowdsourcing data in studying anger and aggressive behavior: A comparison of online and laboratory data

Social Psychology

Volumn 47, Issue 1, 2016, Pages 38-51

  Lutz, Johannes ^a  

^a UNIVERSITY OF POTSDAM   (Germany)

Author keywords

Aggression; Anger; Crowdsourcing; Frustration; Mechanical turk; Online research

Indexed keywords

EID: 84961139984     PISSN: 18649335     EISSN: 21512590     Source Type: Journal    
DOI: 10.1027/1864-9335/a000256     Document Type: Article

References (56)

1. Andover, M. S. (2014). Non-suicidal self-injury disorder in a community sample of adults. Psychiatry Research, 219, 305-310. doi:10.1016/j.psychres.2014.06.001
2. Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. A., Fiedler, K., ⋯ Wicherts, J. M. (2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27, 108-119. doi:10.1002/per.1919
3. Baumeister, R. F., Bratslavsky, E., Muraven, M., & Tice, D. M. (1998). Ego depletion: Is the active self a limited resource? Journal of Personality and Social Psychology, 74, 1252-1265. doi:10.1037/0022-3514.74.5.1252
4. Berkowitz, L. (1989). Frustration-aggression hypothesis: Examination and reformulation. Psychological Bulletin, 106, 59-73. doi:10.1037/0033-2909.106.1.59
5. Bizer, G. Y., Hart, J., & Jekogian, A. M. (2012). Belief in a just world and social dominance orientation: Evidence for a mediational pathway predicting negative attitudes and discrimination against individuals with mental illness. Personality and Individual Differences, 52, 428-432. doi:10.1016/j.paid.2011.11.002
6. Boynton, M. H., & Richman, L. S. (2014). An online daily diary study of alcohol use using Amazon's Mechanical Turk. Drug and Alcohol Review, 33, 456-461. doi:10.1111/dar.12163
7. Bradley, M. M., & Lang, P. J. (1994). Measuring emotion: The Self-Assessment Manikin and the semantic differential. Journal of Behavior Therapy and Experimental Psychiatry, 25, 49-59. doi:10.1016/0005-7916(94)90063-9
8. Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon's Mechanical Turk: A new source of inexpensive, yet highquality, data? Perspectives on Psychological Science, 6, 3-5. doi:10.1177/1745691610393980
9. Burns, D. J., Hart, J., Kramer, M. E., & Burns, A. D. (2014). Dying to remember, remembering to survive: Mortality salience and survival processing. Memory, 22, 36-50. doi:10.1080/09658211.2013.788660
10. Buss, A. H. (1963). Physical aggression in relation to different frustrations. Journal of Abnormal and Social Psychology, 67, 1-7. doi:10.1037/h0040505
11. Campbell, T. H., & Kay, A. C. (2014). Solution aversion: On the relation between ideology and motivated disbelief. Journal of Personality and Social Psychology, 107, 809-824. doi:10.1037/a0037963
12. Casler, K., Bickel, L., & Hackett, E. (2013). Separate but equal? A comparison of participants and data gathered via Amazon's MTurk, social media, and face-to-face behavioral testing. Computers in Human Behavior, 29, 2156-2160. doi:10.1016/j.chb.2013.05.009
13. Chandler, J., Mueller, P., & Paolacci, G. (2014). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, 46, 112-130. doi:10.3758/s13428-013-0365-7
14. Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015). Using nonnaive participants can reduce effect sizes. Psychological Science, 26, 1131-1139. doi:10.1177/095679761558511
15. Chopik, W. J., & Edelstein, R. S. (2014). Death of a salesman: Webpage-based manipulations of mortality salience. Computers in Human Behavior, 31, 94-99. doi:10.1016/j.chb.2013.10.022
16. Chow, J. T., & Lau, S. (2015). Nature gives us strength: Exposure to nature counteracts ego-depletion. The Journal of Social Psychology, 155, 70-85. doi:10.1080/00224545.2014.972310
17. Costa, S., Ntoumanis, N., & Bartholomew, K. J. (2015). Predicting the brighter and darker sides of interpersonal relationships: Does psychological need thwarting matter? Motivation and Emotion, 39, 11-24. doi:10.1007/s11031-014-9427-0
18. Davis, J. R., & Reyna, C. (2015). Seeing red: How perceptions of social status and worth influence hostile attributions and endorsement of aggression. The British Journal of Social Psychology. Advance online publication. doi:10.1111/bjso.12109
19. DeJoy, D. M. (1985). Information input rate, control over task pacing, and performance during and after noise exposure. Journal of General Psychology, 112, 229-242. doi:10.1080/00221309.1985.9711008
20. DeWall, C. N., & Anderson, C. A. (2011). The general aggression model. In P. R. Shaver & M. Mikulincer (Eds.), Human aggression and violence: Causes, manifestations, and consequences (pp. 15-33). Washington, DC: American Psychological Association. doi:10.1037/12346-001
21. Dynamo. (2014). Guidelines for academic requesters (Version 1.1). Retrieved from http://guidelines.wearedynamo.org/
22. ∗Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41, 1149-1160.
23. Geen, R. G. (1968). Effects of frustration, attack, and prior training in aggressiveness upon aggressive behavior. Journal of Personality and Social Psychology, 9, 316-321. doi:10.1037/h0026054
24. Gentile, D. A., and Anderson., C. A., Yukawa, S., Ihori, N., Saleem, M., and Ming., L. K., ⋯ Sakamoto, A. (2009). The effects of prosocial video games on prosocial behaviors: International evidence from correlational, longitudinal, and experimental studies. Personality and Social Psychology Bulletin, 35, 752-763. doi:10.1177/0146167209333045
25. Giancola, P. R., & Zeichner, A. (1995). Construct validity of a competitive reaction-time aggression paradigm. Aggressive Behavior, 21, 199-204. doi:10.1002/1098-2337(1995)21:3<199::AID-AB2480210303>3.0.CO;2-Q
26. Goldin, G., & Darlow, A. (2013). TurkGate (Version 0.4.0) [Software]. Available from http://gideongoldin.github.com/TurkGate/
27. Goodman, J. K., and Cryder., C. E., & Cheema, A. (2013). Data collection in a flat world: The strengths and weaknesses of mechanical Turk samples. Journal of Behavioral Decision Making, 26, 213-224. doi:10.1002/bdm.1753
28. Harwell, M. R., and Rubinstein., E. N., Hayes, W. S., & Olds, C. C. (1992). Summarizing Monte Carlo results in methodological research: The one- and two-factor fixed effects ANOVA cases. Journal of Educational Statistics, 17, 315-339. doi:10.2307/1165127
29. Hauser, D. J., & Schwarz, N. (2015). Attentive Turkers: MTurk participants perform better on online attention checks than subject pool participants. Behavior Research Methods. Advance online publication. doi:10.3758/s13428-015-0578-z
30. Hirth, M., Hoßfeld, T., & Tran-Gia, P. (2011). Anatomy of a crowdsourcing platform - Using the example of Microworkers.com. Paper presented at the Workshop on Future Internet and Next Generation Networks (FINGNet), Seoul, Korea. Retrieved from http://www3.informatik.uni-wuerzburg.de/staff/hossfeld/Transparentia/pubs/conf-410.pdf
31. Horton, J. J., and Rand., D. G., & Zeckhauser, R. J. (2011). The online laboratory: Conducting experiments in a real labor market. Experimental Economics, 14, 399-425. doi:10.1007/s10683-011-9273-9
32. Jonason, P. K., Wee, S., Li, N. P., & Jackson, C. (2014). Occupational niches and the dark triad traits. Personality and Individual Differences, 69, 119-123. doi:10.1016/j.paid.2014.05.024
33. Krahé, B., & Bieneck, S. (2012). The effect of music-induced mood on aggressive affect, cognition, and behavior. Journal of Applied Social Psychology, 42, 271-290. doi:10.1111/j.1559-1816.2011.00887.x
34. Krahé, B., & Fenske, I. (2002). Predicting aggressive driving behavior: The role of macho personality, age, and power of car. Aggressive Behavior, 28, 21-29. doi:10.1002/ab.90003
35. Lickel, B., Kushlev, K., Savalei, V., Matta, S., & Schmader, T. (2014). Shame and the motivation to change the self. Emotion, 14, 1049-1061. doi:10.1037/a0038235
36. Lieberman, J. D., Solomon, S., Greenberg, J., & McGregor, H. A. (1999). A hot new way to measure aggression: Hot sauce allocation. Aggressive Behavior, 25, 331-348. doi:10.1002/(SICI)1098-2337(1999)25:5<331::AID-AB2>3.0.CO;2-1
37. Martin, M. (1990). On the induction of mood. Clinical Psychology Review, 10, 669-697. doi:10.1016/0272-7358(90)90075-L
38. Mason, W., & Suri, S. (2012). Conducting behavioral research on Amazon's mechanical Turk. Behavior Research Methods, 44, 1-23. doi:10.3758/s13428-011-0124-6
39. Paolacci, G., & Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science, 23, 184-188. doi:10.1177/0963721414531598
40. Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411-419.
41. Peer, E., Vosgerau, J., & Acquisti, A. (2014). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 46, 1023-1031. doi:10.3758/s13428-013
42. Phillips, L. H., and Henry., J. D., Hosie, J. A., & Milne, A. B. (2006). Age, anger regulation and well-being. Aging & Mental Health, 10, 250-256. doi:10.1080/13607860500310385
43. Reips, U. (2002). Standards for Internet-based experimenting. Experimental Psychology, 49, 243-256. doi:10.1026/1618-3169.49.4.243
44. Rick, S. I., Pereira, B., & Burson, K. A. (2014). The benefits of retail therapy: Making purchase decisions reduces residual sadness. Journal of Consumer Psychology, 24, 373-380. doi:10.1016/j.jcps.2013.12.004
45. Rogstadius, J., Kostakos, V., Kittur, A., Smus, B., Laredo, J., & Vukovic, M. (2011). An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets. Paper presented at the fifth International AAAI Conference on Weblogs and Social Media), Barcelona, Spain. Retrieved from http://www.aaai.org/ocs/index.php/ICWSM/IC-WSM11/paper/view/2778/3295
46. Routledge, C., Ostafin, B., Juhl, J., Sedikides, C., Cathey, C., & Liao, J. (2010). Adjusting to death: The effects of mortality salience and self-esteem on psychological well-being, growth motivation, and maladaptive behavior. Journal of Personality and Social Psychology, 99, 897-916. doi:10.1037/a0021431
47. Saleem, M., and Anderson., C. A., & Barlett, C. P. (2015). Assessing helping and hurting behaviors through the Tangram help/hurt task. Personality and Social Psychology Bulletin, 41, 1345-1362. doi:10.1177/0146167215594348
48. Saleem, M., and Anderson., C. A., & Gentile, D. A. (2012). Effects of prosocial, neutral, and violent video games on children's helpful and hurtful behaviors. Aggressive Behavior, 38, 281-287. doi:10.1002/ab.21428
49. Schmider, E., Ziegler, M., Danay, E., Beyer, L., & Bühner, M. (2010). Is it really robust? Reinvestigating the robustness of ANOVA against violations of the normal distribution assumption. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 6, 147-151. doi:10.1027/1614-2241/a000016
50. Severance, L., Bui-Wrzosinska, L., Gelfand, M. J., Lyons, S., Nowak, A., Borkowski, W., ⋯ Yamaguchi, S. (2013). The psychological structure of aggression across cultures. Journal of Organizational Behavior, 34, 835-865. doi:10.1002/job.1873
51. Shapiro, D. N., Chandler, J., & Mueller, P. A. (2013). Using Mechanical Turk to study clinical populations. Clinical Psychological Science, 1, 213-220. doi:10.1177/2167702612469015
52. Simcox, T., & Fiez, J. A. (2014). Collecting response times using Amazon Mechanical Turk and Adobe Flash. Behavior Research Methods, 46, 95-111. doi:10.3758/s13428-013-0345-y
53. Spielberger, C. D., and Sydeman., S. J., Owen, A. E., & Marsh, B. J. (1999). Measuring anxiety and anger with the State-Trait Anxiety Inventory (STAI) and the State-Trait Anger Expression Inventory (STAXI). In M. E. Maruish (Ed.), The use of psychological testing for treatment planning and outcomes (2nd ed., pp. 993-1021). Mahwah, NJ: Erlbaum.
54. Summerville, A., & Chartier, C. R. (2013). Pseudo-dyadic "interaction" on Amazon's Mechanical Turk. Behavior Research Methods, 45, 116-124. doi:10.3758/s13428-012-0250-9
55. Williams, K. D., & Nida, S. A. (2011). Ostracism: Consequences and coping. Current Directions in Psychological Science, 20, 71-75. doi:10.1177/0963721411402480
56. Wolf, W., Levordashka, A., Ruff, J. R., Kraaijeveld, S., and Lueckmann., J. M., & Williams, K. D. (2015). Ostracism online: A social media ostracism paradigm. Behavior Research Methods, 47, 361-373. doi:10.3758/s13428-014-0475-x