The importance of assessing clinical phenomena in Mechanical Turk research

Psychological Assessment

Volumn 28, Issue 6, 2016, Pages 684-691

  Arditte, Kimberly A ^a   Çek, Demet ^a   Shaw, Ashley M ^a   Timpano, Kiara R ^a  

^a Department of Biology   (United States)

Author keywords

Mechanical Turk; MTurk; Prevalence; Psychopathology; Symptoms

Indexed keywords

ADULT; ANXIETY; ANXIETY DISORDER; ANXIETY SENSITIVITY INDEX 3; ARTICLE; DEPRESSION; DEPRESSION ANXIETY STRESS SCALE; DIMENSIONAL OBSESSIVE COMPULSIVE SCALE; DISTRESS TOLERANCE SCALE; EATING DISORDER; EATING DISORDER INVENTORY; FEMALE; HOARDING DISORDER; HOARDING RATING SCALE; HUMAN; INTERNET; INTOLERANCE OF UNCERTAINTY SCALE; MALE; MENTAL DISEASE; MENTAL DISEASE ASSESSMENT; MOOD DISORDER; OBSESSIVE COMPULSIVE DISORDER; PREVALENCE; RUMINATIVE RESPONSES SCALE; SOCIAL INTERACTION ANXIETY SCALE; SOCIAL PHOBIA; BEHAVIORAL RESEARCH; INFORMATION PROCESSING; MENTAL DISORDERS; STANDARDS; STATISTICS AND NUMERICAL DATA;

ADULT; BEHAVIORAL RESEARCH; DATA COLLECTION; FEMALE; HUMANS; INTERNET; MALE; MENTAL DISORDERS;

EID: 84939863402     PISSN: 10403590     EISSN: 1939134X     Source Type: Journal    
DOI: 10.1037/pas0000217     Document Type: Article

References (42)

1. Abramowitz, J. S., Deacon, B. J., Olatunji, B. O., Wheaton, M. G., Berman, N. C., Losardo, D.,. Hale, L. R. (2010). Assessment of obsessivecompulsive symptom dimensions: Development and evaluation of the Dimensional Obsessive-Compulsive Scale. Psychological Assessment, 22, 180-198. http://dx.doi.org/10.1037/a0018260.
2. Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. N. (2011). The viability of crowdsourcing for survey research. Behavior Research Methods, 43, 800-813. http://dx.doi.org/10.3758/s13428-011-0081-0.
3. Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon.com's Mechanical Turk. Political Analysis, 20, 351-368. http://dx.doi.org/10.1093/pan/mpr057.
4. Breslau, J., Kendler, K. S., Su, M., Gaxiola-Aguilar, S., & Kessler, R. C. (2005). Lifetime risk and persistence of psychiatric disorders across ethnic groups in the United States. Psychological Medicine, 35, 317-327. http://dx.doi.org/10.1017/S0033291704003514.
5. Brown, T. A., Chorpita, B. F., Korotitsch, W., & Barlow, D. H. (1997). Psychometric properties of the Depression Anxiety Stress Scales (DASS) in clinical samples. Behaviour Research and Therapy, 35, 79-89. http://dx.doi.org/10.1016/S0005-7967(96)00068-X.
6. Buhr, K., & Dugas, M. J. (2006). Investigating the construct validity of intolerance of uncertainty and its unique relationship with worry. Journal of Anxiety Disorders, 20, 222-236. http://dx.doi.org/10.1016/j.janxdis.2004.12.004.
7. Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon's Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 3-5. http://dx.doi.org/10.1177/1745691610393980.
8. Carleton, N. R., Mulvogue, M. K., Thibodeau, M. A., McCabe, R. E., Antony, M. M., & Asmundson, G. J. G. (2012). Increasingly certain about uncertainty: Intolerance of uncertainty across anxiety and depression. Journal of Anxiety Disorders, 26, 468-479. http://dx.doi.org/10.1016/j.janxdis.2012.01.011.
9. Carleton, R. N., Sharpe, D., & Asmundson, G. J. G. (2007). Anxiety sensitivity and intolerance of uncertainty: Requisites of the fundamental fears? Behaviour Research and Therapy, 45, 2307-2316. http://dx.doi.org/10.1016/j.brat.2007.04.006.
10. Casler, K., Bickel, L., & Hackett, E. (2013). Separate but equal? A comparison of participants and data gathered via Amazon's MTurk, social media, and face-to-face behavioral testing. Computers in Human Behavior, 29, 2156-2160. http://dx.doi.org/10.1016/j.chb.2013.05.009.
11. Garner, D. M., Olmstead, M. P., & Polivy, J. (1983). Development and validation of a multidimensional eating disorder inventory for anorexianervosa and bulimia. International Journal of Eating Disorders, 2, 15-34. http://dx.doi.org/10.1002/1098-108X(198321)2:2<15::AIDEAT2260020203>3.0.CO;2-6.
12. Goodman, J. K., Cryder, C. E., & Cheema, A. (2013). Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making, 26, 213-224. http://dx.doi.org/10.1002/bdm.1753.
13. Gosling, S. D., Vazire, S., Srivastava, S., & John, O. P. (2004). Should we trust web-based studies? A comparative analysis of six preconceptions about internet questionnaires. American Psychologist, 59, 93-104.
14. Henry, J. D., & Crawford, J. R. (2005). The short-form version of the Depression Anxiety Stress Scales (DASS-21): Construct validity and normative data in a large non-clinical sample. British Journal of Clinical Psychology, 44, 227-239. http://dx.doi.org/10.1348/014466505X29657.
15. Horton, J. J., & Chilton, L. B. (2010). The labor economics of paid crowdsourcing. Paper presented at the proceedings of the 11th ACM conference on electronic commerce, Cambridge, MA.
16. Ipeirotis, P. G. (2010). Analyzing the Amazon Mechanical Turk marketplace. XRDS: Crossroads. The ACM Magazine for Students, 17, 16-21. http://dx.doi.org/10.1145/1869086.1869094.
17. Joormann, J., Levens, S. M., & Gotlib, I. H. (2011). Sticky thoughts: Depression and rumination are associated with difficulties manipulating emotional material in working memory. Psychological Science, 22, 979-983. http://dx.doi.org/10.1177/0956797611415539.
18. Jorm, A. F. (2000). Does old age reduce the risk of anxiety and depression? A review of epidemiological studies across the adult life span. Psychological Medicine, 30, 11-22. http://dx.doi.org/10.1017/S0033291799001452.
19. Kessler, R. C., Chiu, W. T., Demler, O., Merikangas, K. R., & Walters, E. E. (2005). Prevalence, severity, and comorbidity of 12-month DSM-IV disorders in the National Comorbidity Survey Replication. Archives of General Psychiatry, 62, 617-627. http://dx.doi.org/10.1001/archpsyc.62.6.617.
20. Leyro, T. M., Zvolensky, M. J., & Bernstein, A. (2010). Distress tolerance and psychopathological symptoms and disorders: A review of the empirical literature among adults. Psychological Bulletin, 136, 576-600. http://dx.doi.org/10.1037/a0019712.
21. Lovibond, P. F., & Lovibond, S. H. (1995). The structure of negative emotional states: Comparison of the Depression Anxiety Stress Scales (DASS) with the Beck Depression and Anxiety Inventories. Behavior Research Methods, 33, 335-343. http://dx.doi.org/10.1016/0005-7967(94)00075-U.
22. Mason, W., & Suri, S. (2012). Conducting behavioral research on Amazon's Mechanical Turk. Behavioral Research, 44, 1-23. http://dx.doi.org/10.3758/s13428-011-0124-6.
23. Mattick, R. P., & Clarke, J. C. (1998). Development and validation of measures of social phobia scrutiny fear and social interaction anxiety. Behaviour Research and Therapy, 36, 455-470. http://dx.doi.org/10.1016/S0005-7967(97)10031-6.
24. Mitchell, M. A., Riccardi, C. J., Keough, M. E., Timpano, K. R., & Schmidt, N. B. (2013). Understanding the associations among anxiety sensitivity, distress tolerance, and discomfort intolerance: A comparison of three models. Journal of Anxiety Disorders, 27, 147-154. http://dx.doi.org/10.1016/j.janxdis.2012.12.003.
25. Nolen-Hoeksema, S., Wisco, B. E., & Lyubomirsky, S. (2008). Rethinking rumination. Perspectives on Psychological Science, 3, 400-424. http://dx.doi.org/10.1111/j.1745-6924.2008.00088.x.
26. Osman, A., Wong, J. L., Bagge, C. L., Freedenthal, S., Gutierrez, P. M., & Lozano, G. (2012). The Depression Anxiety Stress Scales-21 (DASS-21): Further examination of dimensions, scale reliability, and correlates. Journal of Clinical Psychology, 68, 1322-1338. http://dx.doi.org/10.1002/jclp.21908.
27. Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411-419.
28. Peters, L. (2000). Discriminant validity of the Social Phobia and Anxiety Inventory (SPAI), the Social Phobia Scale (SPS) and the Social Interaction Anxiety Scale (SIAS). Behaviour Research and Therapy, 38, 943-950. http://dx.doi.org/10.1016/S0005-7967(99)00131-X.
29. Porter, R. J., Gallagher, P., Thompson, J. M., & Young, A. H. (2003). Neurocognitive impairment in drug-free patients with major depressive disorder. The British Journal of Psychiatry, 182, 214-220. http://dx.doi.org/10.1192/bjp.182.3.214.
30. Reis, H. T., & Gosling, S. D. (2010). Social psychological methods outside the laboratory. In S. T. Fiske, D. T. Gilbert, & G. Lindzey (Eds.), Handbook of social psychology (5th ed., Vol. 1, pp. 82-114). New York, NY: Wiley. http://dx.doi.org/10.1002/9780470561119.socpsy001003.
31. Schmidt, N. B., Zvolensky, M. J., & Maner, J. K. (2006). Anxiety sensitivity: Prospective prediction of panic attacks and Axis I pathology. Journal of Psychiatric Research, 40, 691-699. http://dx.doi.org/10.1016/j.jpsychires.2006.07.009.
32. Shapiro, D. N., Chandler, J., & Mueller, P. A. (2013). Using Mechanical Turk to study clinical populations. Clinical Psychological Science, 1, 213-220. http://dx.doi.org/10.1177/2167702612469015.
33. Simons, J. S., & Gaher, R. M. (2005). The distress tolerance scale: Development and validation of a self-report measure. Motivation and Emotion, 29, 83-102. http://dx.doi.org/10.1007/s11031-005-7955-3.
34. Steketee, G., & Frost, R. (2003). Compulsive hoarding: Current status of the research. Clinical Psychology Review, 23, 905-927. http://dx.doi.org/10.1016/j.cpr.2003.08.002.
35. Taylor, S., Zvolensky, M. J., Cox, B. J., Deacon, B., Heimberg, R. G., Ledley, D. R.,. Cardenas, S. J. (2007). Robust dimensions of anxiety sensitivity: Development and initial validation of the Anxiety Sensitivity Index-3. Psychological Assessment, 19, 176-188. http://dx.doi.org/10.1037/1040-3590.19.2.176.
36. Timpano, K. R., Exner, C., Glaesmer, H., Rief, W., Keshaviah, A., Brähler, E., & Wilhelm, S. (2011). The epidemiology of the proposed DSM-5 hoarding disorder: Exploration of the acquisition specifier, associated features, and distress. Journal of Clinical Psychiatry, 72, 780-786. http://dx.doi.org/10.4088/JCP.10m06380.
37. Tolin, D. F., Frost, R. O., & Steketee, G. (2010). A brief interview for assessing compulsive hoarding: The Hoarding Rating Scale-Interview. Psychiatry Research, 178, 147-152. http://dx.doi.org/10.1016/j.psychres.2009.05.001.
38. Treynor, W., Gonzalez, R., & Nolen-Hoeksema, S. (2003). Rumination reconsidered: A psychometric analysis. Cognitive Therapy and Research, 27, 247-259. http://dx.doi.org/10.1023/A:1023910315561.
39. Wheaton, M. G., Deacon, B. J., McGrath, P. B., Berman, N. C., & Abramowitz, J. S. (2012). Dimensions of anxiety sensitivity in the anxiety disorders: Evaluation of the ASI-3. Journal of Anxiety Disorders, 26, 401-408. http://dx.doi.org/10.1016/j.janxdis.2012.01.002.
40. Widiger, T. A., & Samuel, D. B. (2005). Diagnostic categories or dimensions? A question for the Diagnostic and Statistical Manual of Mental Disorders-Fifth ed. Journal of Abnormal Psychology, 114, 494-504. http://dx.doi.org/10.1037/0021-843X.114.4.494.
41. Yip, J. A., & Côté, S. (2013). The emotionally intelligent decision maker: Emotion-understanding ability reduces the effect of incidental anxiety on risk taking. Psychological Science, 24, 48-55. http://dx.doi.org/10.1177/0956797612450031.
42. Zetsche, U., D'Avanzato, C., & Joormann, J. (2012). Depression and rumination: Relation to components of inhibition. Cognition and Emotion, 26, 758-767. http://dx.doi.org/10.1080/02699931.2011.613919.