Using online, crowdsourcing platforms for data collection in personality disorder research: The example of Amazon's mechanical turk

Personality Disorders: Theory, Research, and Treatment

Volumn 8, Issue 1, 2017, Pages 26-34

  Miller, Joshua D ^a   Crowe, Michael ^a   Weiss, Brandon ^a   Maples Keller, Jessica L ^a   Lynam, Donald R ^b  

^a University of Georgia   (United States)

^b PURDUE UNIVERSITY   (United States)

Author keywords

Crowdsourcing; Personality disorders; Reliability; Validity

Indexed keywords

CROWDSOURCING; HUMAN; INFORMATION PROCESSING; INTERNET; MEDICAL RESEARCH; PERSONALITY DISORDER; PROCEDURES; STANDARDS; STATISTICS AND NUMERICAL DATA;

BIOMEDICAL RESEARCH; CROWDSOURCING; DATA COLLECTION; HUMANS; INTERNET; PERSONALITY DISORDERS;

EID: 85008406160     PISSN: 19492715     EISSN: 19492723     Source Type: Journal    
DOI: 10.1037/per0000191     Document Type: Article

References (54)

1. Arditte, K. A., Çek, D., Shaw, A. M., & Timpano, K. R. (2015). The importance of assessing clinical phenomena in Mechanical Turk research. Psychological Assessment. Advance online publication. http: //dx. doi. org/10. 1037/pas0000217
2. Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. N. (2011). The viability of crowdsourcing for survey research. Behavior Research Methods, 43, 800-813. http: //dx. doi. org/10. 3758/s13428-011-0081-0
3. Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon. com's Mechanical Turk. Political Analysis, 20, 351-368. http: //dx. doi. org/10. 1093/pan/ mpr057
4. Boynton, M. H., & Richman, L. S. (2014). An online daily diary study of alcohol use using Amazon's Mechanical Turk. Drug and Alcohol Review, 33, 456-461. http: //dx. doi. org/10. 1111/dar. 12163
5. Buhrmester, M., Kwang, T., &Gosling, S. D. (2011). Amazon's Mechanical Turk a new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 3-5. http: //dx. doi. org/10. 1177/1745691610393980
6. Carter, N. T., Guan, L., Maples, J. L., Williamson, R. L., & Miller, J. D. (2015). The downsides of extreme conscientiousness for psychological wellbeing: The role of obsessive compulsive tendencies. Journal of Personality. Advance online publication http: //dx. doi. org/10. 1111/jopy. 12177
7. Chandler, J., Mueller, P., & Paolacci, G. (2014). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, 46, 112-130. http: //dx. doi. org/10. 3758/s13428-013-0365-7
8. Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015). Using nonnaive participants can reduce effect sizes. Psychological Science, 26, 1131-1139. http: //dx. doi. org/10. 1177/0956797615585115
9. Chandler, J., & Shapiro, D. (2016). Conducting clinical research using crowdsourced convenience samples. Annual Review of Clinical Psychology, 12, 53-81. http: //dx. doi. org/10. 1146/annurev-clinpsy-021815-093623
10. Clark, L. A., & Watson, D. (1995). Constructing validity: Basic issues in objective scale development. Psychological Assessment, 7, 309-319. http: //dx. doi. org/10. 1037/1040-3590. 7. 3. 309
11. Corker, K. S., Donnellan, M. B., Kim, S. Y., Schwartz, S. J., & Zamboanga, B. L. (2015). College student samples are not always equivalent: The magnitude of personality differences across colleges and universities. Journal of Personality. Advance online publication. http: //dx. doi. org/10. 1111/jopy. 12224
12. Crowe, M. L., LoPilato, A. C., Campbell, W. K., & Miller, J. D. (2015). Identifying two groups of entitled individuals: Cluster analysis reveals emotional stability and self-esteem distinction. Journal of Personality Disorders. Advance online publication. http: //dx. doi. org/10. 1521/ pedi-2015-29-229
13. Crump, M. J., McDonnell, J. V., & Gureckis, T. M. (2013). Evaluating Amazon's Mechanical Turk as a tool for experimental behavioral research. PLoS ONE, 8, e57410. http: //dx. doi. org/10. 1371/journal. pone. 0057410
14. DeYoung, C. G., Carey, B. E., Krueger, R. F., & Ross, S. R. (2016). Ten aspects of the Big Five in the Personality Inventory for DSM-5. Personality Disorders: Theory, Research, and Treatment, 7, 113-123. http: //dx. doi. org/10. 1037/per0000170
15. Eisenbarth, H., Lilienfeld, S. O., & Yarkoni, T. (2015). Using a genetic algorithm to abbreviate the Psychopathic Personality Inventory-Revised (PPI-R). Psychological Assessment, 27, 194-202. http: //dx. doi. org/10. 1037/pas0000032
16. Feitosa, J., Joseph, D. L., & Newman, D. A. (2015). Crowdsourcing and personality measurement equivalence: A warning about countries whose primary language is not English. Personality and Individual Differences, 75, 47-52. http: //dx. doi. org/10. 1016/j. paid. 2014. 11. 017
17. Goodman, J. K., Cryder, C. E., & Cheema, A. (2013). Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making, 26, 213-224. http: //dx. doi. org/ 10. 1002/bdm. 1753
18. Gore, W. L., & Widiger, T. A. (2015). Assessment of dependency by the FFDI: Comparisons to the PID-5 and maladaptive agreeableness. Personality and Mental Health, 9, 258-276. http: //dx. doi. org/10. 1002/pmh. 1308
19. Gosling, S. D., & Mason, W. (2015). Internet research in psychology. Annual Review of Psychology, 66, 877-902. http: //dx. doi. org/10. 1146/ annurev-psych-010814-015321
20. Harms, P. D., & DeSimone, J. A. (2015). Caution! MTurk workers ahead-Fines doubled. Industrial and Organizational Psychology: Perspectives on Science and Practice, 8, 183-190. http: //dx. doi. org/10. 1017/iop. 2015. 23
21. Hauser, D. J., & Schwarz, N. (2015). It's a trap! Instructional manipulation checks prompt systematic thinking on "tricky" tasks. SAGE Open. http: //dx. doi. org/10. 1177/2158244015584617
22. Hauser, D. J., & Schwarz, N. (2015). Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behavior Research Methods, 48, 400-407.
23. Hilbig, B. E. (2015). Reaction time effects in lab-versus web-based research: Experimental evidence. Behavior Research Methods. Advance online publication. http: //dx. doi. org/10. 3758/s13428-015-0678-9
24. Holden, C. J., Dennie, T., & Hicks, A. D. (2013). Assessing the reliability of the M5-120 on Amazon's mechanical Turk. Computers in Human Behavior, 29, 1749-1754. http: //dx. doi. org/10. 1016/j. chb. 2013. 02. 020
25. Horton, J. J., & Chilton, L. B. (2010, June). The labor economics of paid crowdsourcing. In Proceedings of the 11th ACM conference on Electronic commerce (pp. 209-218). New York, NY: ACM.
26. Johnson, D. R., & Borden, L. A. (2012). Participants at your fingertips using Amazon's Mechanical Turk to increase student-faculty collaborative research. Teaching of Psychology, 39, 245-251. http: //dx. doi. org/ 10. 1177/0098628312456615
27. Jones, D. N. (2013). What's mine is mine and what's yours is mine: The Dark Triad and gambling with your neighbor's money. Journal of Research in Personality, 47, 563-571. http: //dx. doi. org/10. 1016/j. jrp. 2013. 04. 005
28. Jones, D. N., & Paulhus, D. L. (2011). The role of impulsivity in the Dark Triad of personality. Personality and Individual Differences, 51, 679-682. http: //dx. doi. org/10. 1016/j. paid. 2011. 04. 011
29. Jones, D. N., & Paulhus, D. L. (2014). Introducing the short Dark Triad (SD3): A brief measure of dark personality traits. Assessment, 21, 28-41. http: //dx. doi. org/10. 1177/1073191113514105
30. Kosara, R., & Ziemkiewicz, C. (2010, April). Do Mechanical Turks dream of square pie charts? In Proceedings of the 3rd BELIV'10 Workshop: Beyond time and errors: Novel evaluation methods for information visualization (pp. 63-70). New York, NY: ACM.
31. Kotov, R., Gamez, W., Schmidt, F., & Watson, D. (2010). Linking "big" personality traits to anxiety, depressive, and substance use disorders: A meta-analysis. Psychological Bulletin, 136, 768-821. http: //dx. doi. org/ 10. 1037/a0020327
32. Krueger, R. F., Derringer, J., Markon, K. E., Watson, D., & Skodol, A. E. (2012). Initial construction of a maladaptive personality trait model and inventory for DSM-5. Psychological Medicine, 42, 1879-1890. http: // dx. doi. org/10. 1017/S0033291711002674
33. Landers, R. N., & Behrend, T. S. (2015). An inconvenient truth: Arbitrary distinctions between organizational, Mechanical Turk, and other convenience samples. Industrial and Organizational Psychology: Perspectives on Science and Practice, 8, 142-164. http: //dx. doi. org/10. 1017/iop. 2015. 13
34. Law, M. K., Furr, R. M., Arnold, E. M., Mneimne, M., Jaquett, C., & Fleeson, W. (2015). Does assessing suicidality frequently and repeatedly cause harm? A randomized control study. Psychological Assessment, 27, 1171-1181. http: //dx. doi. org/10. 1037/pas0000118
35. Litman, L., Robinson, J., & Rosenzweig, C. (2015). The relationship between motivation, monetary compensation, and data quality among U. S.-and India-based workers on Mechanical Turk. Behavior Research Methods, 47, 519-528. http: //dx. doi. org/10. 3758/s13428-014-0483-x
36. Maples, J. L., Carter, N. T., Few, L. R., Crego, C., Gore, W. L., Samuel, D. B., Miller, J. D. (2015). Testing whether the DSM-5 personality disorder trait model can be measured with a reduced set of items: An item response theory investigation of the Personality Inventory for DSM-5. Psychological Assessment, 27, 1195-1210. http: //dx. doi. org/10. 1037/pas0000120
37. Maples, J. L., Lamkin, J., & Miller, J. D. (2014). A test of two brief measures of the dark triad: The dirty dozen and short dark triad. Psychological Assessment, 26, 326-331. http: //dx. doi. org/10. 1037/ a0035084
38. Marcus, D. K., Zeigler-Hill, V., Mercer, S. H., & Norris, A. L. (2014). The psychology of spite and the measurement of spitefulness. Psychological Assessment, 26, 563-574. http: //dx. doi. org/10. 1037/a0036039
39. Mason, W., & Suri, S. (2012). Conducting behavioral research on Amazon's Mechanical Turk. Behavior Research Methods, 44, 1-23. http: // dx. doi. org/10. 3758/s13428-011-0124-6
40. Miller, J. D., Few, L. R., Wilson, L., Gentile, B., Widiger, T. A., Mackillop, J., & Keith Campbell, W. (2013). The Five-Factor Narcissism Inventory (FFNI): A test of the convergent, discriminant, and incremental validity of FFNI scores in clinical and community samples. Psychological Assessment, 25, 748-758. http: //dx. doi. org/10. 1037/a0032536
41. Miller, J. D., Price, J., Gentile, B., Lynam, D. R., & Campbell, W. K. (2012). Grandiose and vulnerable narcissism from the perspective of the interpersonal circumplex. Personality and Individual Differences, 53, 507-512. http: //dx. doi. org/10. 1016/j. paid. 2012. 04. 026
42. Paolacci, G., & Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science, 23, 184-188. http: //dx. doi. org/10. 1177/0963721414531598
43. Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411-419.
44. Peer, E., Samat, S., Brandimarte, L., & Acquisti, A. (2015). Beyond the Turk: An empirical comparison of alternative platforms for crowdsourcing online behavioral research. Retrieved from http: //ssrn. com/ abstract-2594183
45. Peer, E., Vosgerau, J., & Acquisti, A. (2014). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 46, 1023-1031. http: //dx. doi. org/10. 3758/s13428-013-0434-y
46. Rojas, S. L., & Widiger, T. A. (2014). Convergent and discriminant validity of the five factor form. Assessment, 21, 143-157. http: //dx. doi. org/10. 1177/1073191113517260
47. Samuel, D. B., & Widiger, T. A. (2008). A meta-analytic review of the relationships between the five-factor model and DSM-IV-TR personality disorders: A facet level analysis. Clinical Psychology Review, 28, 1326-1342. http: //dx. doi. org/10. 1016/j. cpr. 2008. 07. 002
48. Schönbrodt, F. D., & Perugini, M. (2013). At what sample size do correlations stabilize? Journal of Research in Personality, 47, 609-612. http: //dx. doi. org/10. 1016/j. jrp. 2013. 05. 009
49. Sears, D. O. (1986). College sophomores in the laboratory: Influences of a narrow data base on social psychology's view of human nature. Journal of Personality and Social Psychology, 51, 515-530. http: //dx. doi. org/10. 1037/0022-3514. 51. 3. 515
50. Shapiro, D. N., Chandler, J., & Mueller, P. A. (2013). Using Mechanical Turk to study clinical populations. Clinical Psychological Science, 1, 213-220. http: //dx. doi. org/10. 1177/2167702612469015
51. Simcox, T., & Fiez, J. A. (2014). Collecting response times using Amazon Mechanical Turk and Adobe Flash. Behavior Research Methods, 46, 95-111. http: //dx. doi. org/10. 3758/s13428-013-0345-y
52. Stewart, N., Ungemach, C., Harris, A. J., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015). The average laboratory samples a population of 7, 300 Amazon Mechanical Turk workers. Judgment and Decision Making, 10, 479-491.
53. Vakharia, D., & Lease, M. (2015). Beyond Mechanical Turk: An analysis of paid crowd work platforms. Retrieved from https: //www. ischool. utexas. edu/~ml/papers/donna-iconf15. pdf
54. Veilleux, J. C., Skinner, K. D., Reese, E. D., & Shaver, J. A. (2014). Negative affect intensity influences drinking to cope through facets of emotion dysregulation. Personality and Individual Differences, 59, 96-101. http: //dx. doi. org/10. 1016/j. paid. 2013. 11. 012