Using mechanical turk to study clinical populations

Clinical Psychological Science

Volumn 1, Issue 2, 2013, Pages 213-220

  Shapiro, Danielle N ^a   Chandler, Jesse ^b   Mueller, Pam A ^b  

^a UNIVERSITY OF MICHIGAN   (United States)

^b PRINCETON UNIVERSITY   (United States)

Author keywords

Clinical methods; Crowdsourcing; Internet research; Mechanical Turk

Indexed keywords

EID: 84885135505     PISSN: 21677026     EISSN: 21677034     Source Type: Journal    
DOI: 10.1177/2167702612469015     Document Type: Article

References (58)

1. Aharoni, E., Sinnott-Armstrong, W., & Kiehl, K. A. (2012). Can psychopathic offenders discern moral wrongs? A new look at the moral/conventional distinction. Journal of Abnormal Psychology, 121, 484-497. doi: 10. 1037/a0024796.
2. Altman, E. G., Hedeker, D., Peterson, J. L., & Davis, J. M. (1997). The Altman Self-Rating Mania Scale. Biological Psychiatry, 42, 948-955. doi: 10. 1016/S0006-3223(96)00548-3.
3. Andersson, G., Carlbring, P., Holmström, A., Sparthan, E., Furmark, T., Nilsson-Ihrfelt, E.,. Ekselius, L. (2006). Internet-based self-help with therapist feedback and in vivo group exposure for social phobia: A randomized controlled trial. Journal of Consulting and Clinical Psychology, 74, 677-686. doi: 10. 1037/0022-006X. 74. 4. 677.
4. Arbisi, P. A., & Ben-Porath, Y. S. (1995). An MMPI-2 infrequent response scale for use with psychopathological populations: The Infrequency-Psychopathology Scale, F(p). Psychological Assessment, 7, 424-431. doi: 10. 1037/1040-3590. 7. 4. 424.
5. Bargh, J. A., & McKenna, K. Y. A. (2004). The Internet and social life. Annual Review of Psychology, 55, 573-590. doi: 10. 1146/annurev. psych. 55. 090902. 141922.
6. Beck, A. T., Epstein, N., Brown, G., & Steer, R. A. (1988). An inventory for measuring clinical anxiety: Psychometric properties. Journal of Consulting and Clinical Psychology, 56, 893-897. doi: 10. 1037/0022-006X. 56. 6. 893.
7. Beck, A. T., Ward, C. H., Mendelson, M., Mock, J., & Erbaugh, J. (1961). An inventory for measuring depression. Archives of General Psychiatry, 4, 561-571. doi: 10. 1001/archpsyc. 1961. 01710120031004.
8. Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon. com's Mechanical Turk. Political Analysis, 20, 351-368. doi: 10. 1093/pan/mpr057.
9. Bernat, J. A., Ronfeldt, H. M., Calhoun, K. S., & Arias, I. (1998). Prevalence of traumatic events and peritraumatic predictors of posttraumatic stress symptoms in a nonclinical sample of college students. Journal of Traumatic Stress, 11, 645-664. doi: 10. 1023/A: 1024485130934.
10. Binik, Y. M., Mah, K., & Kiesler, S. (1999). Ethical issues in conducting sex research on the Internet. Journal of Sex Research, 36, 82-90. doi: 10. 1080/00224499909551971.
11. Blake, D. D., Weathers, F. W., Nagy, L. M., Kaloupek, D. G., Gusman, F. D., Charney, D. S., & Keane, T. M. (1995). The development of a Clinician-Administered PTSD Scale. Journal of Traumatic Stress, 8, 75-90. doi: 10. 1007/BF02105408.
12. Brown, R. L., & Rounds, L. A. (1995). Conjoint screening questionnaires for alcohol and other drug abuse: Criterion validity in a primary care practice. Wisconsin Medical Journal, 94, 135-140.
13. Buhrmester, M., Kwang, T., & Gosling, S. (2011). Amazon's Mechanical Turk: A new source of inexpensive, yet highquality, data? Perspectives on Psychological Science, 6, 3-5. doi: 10. 1177/1745691610393980.
14. Butcher, J. N., Dahlstrom, W. G., Graham, J. R., Tellegen, A., & Kaemmer, B. (1989). MMPI-2: Manual for administration and scoring. Minneapolis, MN: University of Minnesota Press.
15. Byrnes, J. P., Miller, D. C., & Schafer, W. D. (1999). Gender differences in risk taking: A meta-analysis. Psychological Bulletin, 125, 367-383. doi: 10. 1037/0033-2909. 125. 3. 367.
16. Carlbring, P., Gunnarsdóttir, M., Hedensjö, L., Andersson, G., Ekselius, L., & Furmark, T. (2007). Treatment of social phobia: Randomised trial of Internet-delivered cognitive-behavioural therapy with telephone support. British Journal of Psychiatry, 190, 123-128. doi: 10. 1192/bjp. bp. 105. 020107.
17. Chandler, J., Mueller, P., & Paolacci, G. (2013). Methodological concerns and advanced uses of Amazon Mechanical Turk in psychological research. Manuscript submitted for publication.
18. Corrigan, P. (2004). How stigma interferes with mental health care. American Psychologist, 59, 614-625. doi: 10. 1037/0003-066X. 59. 7. 614.
19. Couper, M. (2008). Designing effective Web surveys. New York, NY: Cambridge University Press.
20. Diener, E., Emmons, R. A., Larsen, R. J., & Griffin, S. (1985). The Satisfaction With Life Scale. Journal of Personality Assessment, 49, 71-75. doi: 10. 1207/s15327752jpa4901_13.
21. Dooley, D., Catalano, R., & Wilson, G. (1994). Depression and unemployment: Panel findings from the Epidemiologic Catchment Area Study. American Journal of Community Psychology, 22, 745-765. doi: 10. 1007/BF02521557.
22. Downs, J. S., Holbrook, M., & Peel, E. (2012, October). Screening participants on Mechanical Turk: Techniques and justifications. Paper presented at the annual conference of the Association for Consumer Research, Vancouver, BC, Canada.
23. Estellés-Arolas, E., & Ladrün-de-Guevara, F. (2012). Towards an integrated crowdsourcing definition. Journal of Information Science, 38, 189-200.
24. Gates, G. (2011). How many people are lesbian, gay, bisexual, and transgender? Retrieved from http://williamsinstitute. law. ucla. edu/wp-content/uploads/Gates-How-Many-People-LGBTApr-2011. pdf.
25. Horton, J. J., & Chilton, L. B. (2010). The labor economics of paid crowdsourcing. In Proceedings from EC '10: The 11th ACM Conference on Electronic Commerce (pp. 209-218). New York, NY: ACM. doi: 10. 1145/1807342. 1807376.
26. Joinson, A. N. (2001). Self-disclosure in computer mediated communication: The role of self-awareness and visual anonymity. European Journal of Social Psychology, 31, 177-192. doi: 10. 1002/ejsp. 36.
27. Jones, D. N., & Paulhus, D. L. (2011). The role of impulsivity in the dark triad of personality. Personality and Individual Differences, 51, 679-682. doi: 10. 1016/j. paid. 2011. 04. 011.
28. Kessler, R. C., Berglund, P. A., Demler, O., Jin, R., & Walters, E. E. (2005). Lifetime prevalence and age-of-onset distributions of DSM-IV disorders in the National Comorbidity Survey Replication (NCS-R). Archives of General Psychiatry, 62, 593-602. doi: 10. 1001/archpsyc. 62. 6. 593.
29. Kessler, R. C., Chiu, W. T., Demler, O., & Walters, E. E. (2005). Prevalence, severity, and comorbidity of twelve-month DSM-IV disorders in the National Comorbidity Survey Replication (NCSR). Archives of General Psychiatry, 62, 617-627. doi: 10. 1001/archpsyc. 62. 6. 617.
30. Kittur, A., Chi, E. H., & Suh, B. (2008). Crowdsourcing user studies with Mechanical Turk. In Proceedings of the ACM Conference on Human Factors in Computing Systems (pp. 453-456). New York, NY: ACM.
31. Lambert, M. V., Senior, C., Phillips, M. L., & David, A. S. (2000). Depersonalization in cyberspace. Journal of Nervous and Mental Disease, 188, 764-771. doi: 10. 1097/00005053-200011000-00007.
32. Lex, B. W. (1991). Some gender differences in alcohol and polysubstance users. Health Psychology, 10, 121-132. doi: 10. 1037/0278-6133. 10. 2. 121.
33. Liebowitz, M. R. (1987). Social phobia. Modern Problems of Pharmacopsychiatry, 22, 141-173.
34. Mason, W., & Suri, S. (2012). Conducting behavioral research on Amazon's Mechanical Turk. Behavior Research Methods, 44, 1-23. doi: 10. 3758/s13428-011-0124-6.
35. Morris, R., & Picard, R. (2012). Crowdsourcing collective emotional intelligence. In T. W. Malone & L. von Ahn (Eds.), Proceedings of Collective Intelligence 2012. arXiv: 1204. 2991.
36. Mueller, P., & Chandler, J. (2012). Emailing workers using Python. Available at SSRN: http://ssrn. com/abstract=2100601.
37. Nunnally, J. C. (1978). Psychometric theory. New York, NY: McGraw-Hill.
38. O'Neil, K. M. & Penrods, D. (2001). Methodological variables in web-based research that may affect results: Sample type, monetary incentives, and personal information. Behavioral Research Methods, Instruments, & Computers, 33, 226-233. doi: 10. 3758/BF03195369.
39. Paolacci, G., Chandler, J., & Ipeirotis, P. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411-419.
40. Patel, M. X., Doku, V., & Tennakoon, L. (2003). Challenges in recruitment of research participants. Advances in Psychiatric Treatment, 9, 229-238. doi: 10. 1192/apt. 9. 3. 229.
41. Rand, D. G. (2012). The promise of Mechanical Turk: How online labor markets can help theorists run behavioral experiments. Journal of Theoretical Biology, 299, 172-179. doi: 10. 1016/j. jtbi. 2011. 03. 004.
42. Rinck, M., & Becker, E. S. (2005). A comparison of attentional biases and memory biases in women with social phobia and major depression. Journal of Abnormal Psychology, 114, 62-74. doi: 10. 1037/0021-843X. 114. 1. 62.
43. Rodriguez, E., Allen, J. A., Frongillo, E. A., & Chandra, P. (1999). Unemployment, depression, and health: A look at the African-American community. Journal of Epidemiology and Community Health, 53, 335-342. doi: 10. 1136/jech. 53. 6. 335.
44. Rosenthal, R., & Rosnow, R. L. (2009). Artifacts in behavioral research. New York, NY: Oxford University Press.
45. Sartorius, N. (1998). Stigma: What can psychiatrists do about it? Lancet, 352, 1058-1059. doi: 10. 1016/S0140-6736(98)08008-8.
46. Shapiro, D. N., & Stewart, A. J. (2011). Parenting stress, perceived child regard, and depression in stepmothers and biological mothers. Family Relations, 60, 533-544. doi: 10. 1111/j. 1741-3729. 2011. 00665. x.
47. Shepherd, R. M., & Edelmann, R. J. (2005). Reasons for Internet use and social anxiety. Personality and Individual Differences, 39, 949-958. doi: 10. 1016/j. paid. 2005. 04. 001.
48. Simmons, J., Nelson, L., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-1366. doi: 10. 1177/0956797611417632.
49. Snowden, L. R., & Cheung, F. K. (1990). Use of inpatient mental health services by members of ethnic minority groups. American Psychologist, 45, 347-355. doi: 10. 1037/0003-066X. 45. 3. 347.
50. Stevens, S. B., & Morris, T. L. (2007). College dating and social anxiety: Using the Internet as a means of connecting to others. CyberPsychology & Behavior, 10, 680-688. doi: 10. 1089/cpb. 2007. 9970.
51. Suri, S., Goldstein, D. G., & Mason, W. A. (2011, August). Honesty in an online labor market. In L. von Ahn, & P. G. Ipeirotis (Eds.), Papers from the 2011 AAAI Workshop. Menlo Park, CA: AAAI Press.
52. Tamir, D. (2011). 50, 000 worldwide mechanical Turk workers. Techlist. Retrieved from http://techlist. com/mturk/global-mturkworker-map. php.
53. Tsai, J., Ford, E. S., Li, C., Pearson, W. S., & Zhao, G. (2010). Binge drinking and suboptimal self-rated health among adult drinkers. Alcoholism: Clinical and Experimental Research, 34, 1465-1471. doi: 10. 1111/j. 1530-0277. 2010. 01231. x.
54. U. S. Bureau of Labor Statistics (2012). Labor force statistics from the Current Population Survey. Retrieved from http://data. bls. gov/timeseries/LNS14000000.
55. Vrana, S., & Lauterbach, D. (1994). Prevalence of traumatic events and post-traumatic psychological symptoms in a nonclinical sample of college students. Journal of Traumatic Stress, 7, 289-302. doi: 10. 1007/BF02102949.
56. Weidman, A. C., Fernandez, K. C., Levinson, C. A., Augustine, A. A., Larsen, R. J., & Rodebaugh, T. L. (2012). Compensatory Internet use among individuals higher in social anxiety and its implications for well-being. Personality and Individual Differences, 53, 191-195. doi: 10. 1016/j. paid. 2012. 03. 003.
57. Weinstock, L. S. (1999). Gender differences in the presentation and management of social anxiety disorder. Journal of Clinical Psychiatry, 60, 9-13. doi: 10. 1007/BF02102949.
58. Yancey, A. K., Ortega, A. N., & Kumanyika, S. K. (2006). Effective recruitment and retention of minority research participants. Annual Review of Public Health, 27, 1-28. doi: 10. 1146/annurev. publhealth. 27. 021405. 102113.