Conducting Clinical Research Using Crowdsourced Convenience Samples

Annual Review of Clinical Psychology

Volumn 12, Issue , 2016, Pages 53-81

  Chandler, Jesse ^a   Shapiro, Danielle ^b  

^a UNIVERSITY OF MICHIGAN   (United States)

^b UNIVERSITY OF MICHIGAN MEDICAL SCHOOL   (United States)

Author keywords

Amazon Mechanical Turk; Internet research methods

Indexed keywords

ARTICLE; ATTENTION; CLINICAL RESEARCH; CONVENIENCE SAMPLE; CROWDSOURCING; DEMOGRAPHY; HUMAN; INCOME; MALINGERING; MOTIVATION; NONPROBABILITY SAMPLE; PROBABILITY SAMPLE; RESEARCH ETHICS; SELF REPORT; TURK (PEOPLE); WORKER; MEDICAL RESEARCH; STANDARDS; STATISTICS AND NUMERICAL DATA;

BIOMEDICAL RESEARCH; CROWDSOURCING; HUMANS;

EID: 84962150075     PISSN: 15485943     EISSN: 15485951     Source Type: Book Series    
DOI: 10.1146/annurev-clinpsy-021815-093623     Document Type: Article

References (176)

1. Allan NP,Norr AM, Macatee RJ, Gajewska A, Schmidt NB. 2015. Interactive effects of anxiety sensitivity and emotion regulation on anxiety symptoms. J. Psychopathol. Behav. 37:663-72
2. Am. Psychol. Assoc. 2010. 2010 Amendments to the 2002 "Ethical principles of psychologists and code of conduct." Am. Psychol. 65:493
3. Andersson G. 2016. Internet-delivered psychological treatments. Annu. Rev. Clin. Psychol. 12:157-79
4. Andover MS. 2014. Non-suicidal self-injury disorder in a community sample of adults. Psychiatry Res. 219:305-10
5. Arbisi PA, Ben-Porath YS. 1995. An MMPI-2 infrequent response scale for use with psychopathological populations: The Infrequency-Psychopathology Scale, F(p). Psychol. Assess. 7:424-31
6. Arch JJ. 2014. Cognitive behavioral therapy and pharmacotherapy for anxiety: treatment preferences and credibility among pregnant and non-pregnant women. Behav. Res. Ther. 52:53-60
7. Arch JJ, Twohig MP, Deacon BJ, Landy LN, Bluett EJ. 2015. The credibility of exposure therapy: Does the theoretical rationale matter? Behav. Res. Ther. 72:81-92
8. Arditte KA, Çek D, Shaw AM, Timpano KR. 2015. The importance of assessing clinical phenomena in Mechanical Turk research. Psychol. Assess. doi:10.137/pas0000217
9. Austin EJ. 2005. Personality correlates of the broader autism phenotype as assessed by the Autism Spectrum Quotient (AQ). Personal. Individ. Differ. 38:451-60
10. Azzam T, Jacobson MR. 2013. Finding a comparison group: Is online crowdsourcing a viable option? Am. J. Eval. 34:372-84
11. Barnhoorn JS, Haasnoot E, Bocanegra BR, van Steenbergen H. 2014. QRTEngine: an easy solution for running online reaction time experiments using Qualtrics. Behav. Res. Methods 530:1-12
12. Behrend TS, Sharek DJ, Meade AW, Wiebe EN. 2011. The viability of crowdsourcing for survey research. Behav. Res. Methods 43:800-13
13. Benoit K, Conway D, Lauderdale BE, Laver M, Mikhaylov S. 2015. Crowd-sourced text analysis: reproducible and agile reproduction of political data. Am. Polit. Sci. Rev. In press
14. Beramendi P, Duch RM, Matsuo A. 2014. When lab subjects meet real people: comparing different modes of experiments. Presented at Asian Polit. Methodol. Conf., Taipai
15. Berinsky AJ, Huber GA, Lenz GS. 2012. Evaluating online labor markets for experimental research: Amazon.com's Mechanical Turk. Polit. Anal. 20:351-68
16. Berinsky AJ, Margolis MF, Sances MW. 2014. Separating the shirkers from the workers? Making sure respondents pay attention on self administered surveys. Am. J. Polit. Sci. 58:739-53
17. Bernal DR. 2014. Acculturation, acculturative stress, social status, and well-being among English language proficient immigrants. PhD Thesis, Univ. Miami, Coral Gables
18. Black D, Gates G, Sanders S, Taylor L. 2000. Demographics of the gay and lesbian population in the United States: evidence from available systematic data sources. Demography 37:139-54
19. Blumberg SJ, Luke JV. 2007. Coverage bias in traditional telephone surveys of low-income and young adults. Public Opin. Q. 71:734-49
20. BoyntonMH, Richman LS. 2014. An online daily diary study of alcohol use using Amazon's Mechanical Turk. Drug Alcohol. Rev. 33:456-61
21. BrawleyAM, Pury CL. 2016. Work experiences onMTurk: job satisfaction, turnover, and information sharing. Comput. Hum. Behav. 54:531-46
22. Buchanan T, Williams JE. 2010. Ethical issues in psychological research on the Internet. In AdvancedMethods for Conducting Behavioral Research, ed. SD Gosling, JA Johnson, pp. 255-71. Washington, DC: Am. Psychol. Assoc.
23. Buhrmester M, Kwang T, Gosling SD. 2011. Amazon's Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspect. Psychol. Sci. 6:3-5
24. BuiDC, Myerson J,Hale S. 2015. Age-related slowing in online samples. Psychol. Record. doi: 10.1007/s40732-015-0135-2
25. Burke SE, Wang K, Dovidio JF. 2014. Witnessing disclosure of depression:Gender and attachment avoidance moderate interpersonal evaluations. J. Soc. Clin. Psychol. 33:536-59
26. Cabrera LY, Fitz NS, Reiner PB. 2014. Empirical support for the moral salience of the therapy-enhancement distinction in the debate over cognitive, affective and social enhancement. Neuroethics 8:243-56
27. Carr A. 2014. An exploration of Mechanical Turk as a feasible recruitment platform for cancer survivors. Undergrad. honors thesis: Univ. Colo., Boulder
28. Casey L, Chandler J, Levine AS, Proctor A, StrolovitchD. 2015. Demographic characteristics of a large sample of US workers. Mathematica Policy Res. Rep. In press
29. Casler K, Bickel L, Hackett E. 2013. Separate but equal? A comparison of participants and data gathered via Amazon's MTurk, social media, and face-to-face behavioral testing. Comput. Hum. Behav. 29:2156-60
30. Cavanagh TM. 2014. Cheating on online assessment tests: prevalence and impact on validity. PhD Thesis, Colo. State Univ., Ft. Collins
31. Chandler J, Mueller P, Paolacci G. 2014. Nonnäiveté among AmazonMechanical Turk workers: consequences and solutions for behavioral researchers. Behav. Res. Methods 46:112-30
32. Chandler J, Paolacci G, Mueller P. 2013. Risks and rewards of crowdsourcing marketplaces. In Handbook of Human Computation, ed. P Michelucci, pp. 377-92. New York: Springer
33. Chandler J, Paolacci G, Peer E, Mueller P, Ratliff KA. 2015. Using nonnaive participants can reduce effect sizes. Psychol. Sci. 26:1131-39
34. Clifford S, Jerit J. 2014. Is there a cost to convenience? An experimental comparison of data quality in laboratory and online studies. J. Exp. Polit. Sci. 1:120-31
35. Cooper EA, Farid H. 2014. Does the Sun revolve around the Earth? A comparison between the general public and online survey respondents in basic scientific knowledge. Public Underst. Sci. doi: 10.1177/0963662514554354
36. Corrigan PW, Bink AB, Fokuo JK, Schmidt A. 2015. The public stigma of mental illness means a difference between you and me. Psychiatry Res. 226:186-91
37. Cougle JR, Hawkins KA, Macatee RJ, Zvolensky MJ, Sarawgi S. 2014. Multiple facets of problematic anger in regular smokers: exploring associations with smoking motives and cessation difficulties. Nicotine Tob. Res. 16:881-85
38. Crump MJ, McDonnell JV, Gureckis TM. 2013. Evaluating Amazon's Mechanical Turk as a tool for experimental behavioral research. PLOS ONE 8:e57410
39. Daly TM, Nataraajan R. 2015. Swapping bricks for clicks: crowdsourcing longitudinal data on Amazon Turk. J. Bus. Res. 68:2603-9
40. Davenport SW, Bergman SM, Bergman JZ, Fearrington ME. 2014. Twitter versus Facebook: exploring the role of narcissism in the motives and usage of different social media platforms. Comput. Hum. Behav. 32:212-20
41. de Leeuw JR. 2014. jsPsych: A JavaScript library for creating behavioral experiments in aWeb browser. Behav. Res. Methods 47:1-12
42. Devlin HC, Johnson SL, Gruber J. 2015. Feeling good and taking a chance? Associations of hypomania risk with cognitive and behavioral risk taking. Cogn. Ther. Res. 39:473-79
43. DeWall CN, Pond RS Jr, Carter EC, McCulloughME, LambertNM, et al. 2014. Explaining the relationship between religiousness and substance use: Self-control matters. J. Personal. Soc. Psychol. 107:339-51
44. Downes-Le Guin T, Mechling J, Baker R. 2006. Great results from ambiguous sources: cleaning Internet panel data. ESOMAR World Res. Conf. Panel Res. Amsterdam: ESOMAR
45. Eriksson K. 2013. Autism-spectrum traits predict humor styles in the general population. Humor 26:461-75
46. Farrell AM, Grenier JH, Leiby J. 2014. Scoundrels or stars? Theory and evidence on the quality of workers in online labor markets. SSRN. doi: 10.2139/ssrn.2424718
47. Feitosa J, Joseph DL,NewmanDA. 2015. Crowdsourcing and personalitymeasurement equivalence: awarning about countries whose primary language is not English. Personal. Individ. Differ. 75:47-52
48. Fergus TA. 2014a. The Cyberchondria Severity Scale (CSS): an examination of structure and relations with health anxiety in a community sample. J. Anxiety Disord. 28:504-10
49. Fergus TA. 2014b. Are "not just right experiences" (NJREs) specific to obsessive-compulsive symptoms? Evidence that NJREs span across symptoms of emotional disorders. J. Clin. Psychol. 70:353-63
50. Fergus TA, Bardeen JR. 2014. Emotion regulation and obsessive-compulsive symptoms: A further examination of associations. J. Obsessive-Compuls. Relat. Disord. 3:243-48
51. Fergus TA, Rowatt WC. 2015. Uncertainty, God, and scrupulosity: Uncertainty salience and priming God concepts interact to cause greater fears of sin. J. Behav. Ther. Exp. Psychiatry 46:93-98
52. Fergus TA, Valentiner DP, McGrath PB, Gier-Lonsway SL, Kim HS. 2012. Short forms of the Social Interaction Anxiety Scale and the Social Phobia Scale. J. Personal. Assess. 94:310-20
53. Fort K, Adda G, Cohen KB. 2011. Amazon Mechanical Turk: gold mine or coal mine? Comput. Linguist 37:413-20
54. Frederick S. 2005. Cognitive reflection and decision making. J. Econ. Perspect. 19:25-42
55. Gates GJ. 2014. LGBT demographics: comparisons among population-based surveys. Williams Inst., UCLA Sch. Law
56. Gerber AS, Green DP. 2012. Field Experiments: Design, Analysis, and Interpretation. New York: Norton
57. Goldin G, Darlow A. 2013. TurkGate, Version 0.4.0. Software. http://www.gideongoldin.github.com/TurkGate/
58. Goodman JK, Cryder CE, Cheema A. 2013. Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples. J. Behav. Decis. Mak. 26:213-24
59. Gootzeit JH. 2014. ACT process measures: specificity and incremental value. PhD Thesis, Univ. Iowa, Iowa City
60. Gosling SD, MasonW. 2015. Internet research in psychology. Annu. Rev. Psychol. 66:877-902
61. Green AE, Kenworthy L,MosnerMG, GallagherNM, Fearon EW, et al. 2014. Abstract analogical reasoning in high-functioning children with autism spectrum disorders. Autism Res. 7:677-86
62. Greenblatt JB. 2013. US residential consumer product information: validation of methods for post-stratification weighting of Amazon Mechanical Turk surveys. Lawrence Berkeley Natl. Lab., LBNL Pap. LBNL-6163E, Berkeley, CA
63. Greenblatt JB. 2014. US residential miscellaneous refrigeration products: results from Amazon Mechanical Turk surveys. Lawrence Berkeley Natl. Lab., LBNL Pap. LBNL-6537E, Berkeley, CA
64. Groves RM. 2004. Survey Errors and Survey Costs, Vol. 536. New York: Wiley
65. GuptaN,MartinD,Hanrahan BV, O'Neill J. 2014. Turk-life in India. In Proc. Int. Conf. Support. GroupWork, 18th, Sanibel Island, pp. 1-11. New York: ACM
66. Hauser DJ, Schwarz N. 2015a. Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behav. Res. Methods. In press
67. Hauser DJ, Schwarz N. 2015b. It's a trap! Instructional manipulation checks prompt systematic thinking on "tricky" tasks. SAGE Open 5:1-5
68. Hawkins RXD. 2014. Conducting real-time multiplayer experiments on the web. Behav. Res. Methods. doi: 10.3758/s13428-014-0515-6
69. Heer J, BostockM. 2010. Crowdsourcing graphical perception: using Mechanical Turk to assess visualization design. In Proc. SIGCHI Conf. Hum. Factors Comput. Syst., 30th, Atlanta, pp. 203-12. New York: ACM
70. Higgins C, McGrath E, Moretto L. 2010. MTurk crowdsourcing: A viable method for rapid discovery of Arabic nicknames. In Proc. NAACL HLT 2010 Workshop on Creat. Speech and Language Data with Amazon's Mechanical Turk, Los Angeles, pp. 89-92. Stroudburg, PA: Assoc. Comput. Linguist.
71. HillygusDS, JacksonN, YoungM. 2014. Professional respondents in non-probability online panels. In Online Panel Research: A Data Quality Perspective, ed.MCallegaro, R Baker, J Bethlehem, ASGöritz, JA Krosnick, PJ Lavrakas, pp. 219-37. Chichester, UK:Wiley
72. Horton JJ, Chilton LB. 2010. The labor economics of paid crowdsourcing. In Proc. ACM Conf. Electron. Commer., 11th, Cambridge, MA, pp. 209-18. New York: ACM
73. Horton JJ, Rand DG, Zeckhauser RJ. 2011. The online laboratory: conducting experiments in a real labor market. Exp. Econ. 14:399-425
74. Howard A, MacCalla J. 2014. Pilot study to evaluate the effectiveness of a mobile-based therapy and educational app for children. In Proc. Workshop on Mob. Med. Appl., 1st, Memphis, pp. 12-15. New York: ACM
75. Huang JL, Curran PG, Keeney J, Poposki EM, DeShon RP. 2012. Detecting and deterring insufficient effort responding to surveys. J. Bus. Psychol. 27:99-114
76. Huff C, TingleyD. 2014. "Who are these people?": evaluating the demographic characteristics and political preferences of MTurk survey respondents.Work. Pap., Dep. Gov., Harvard Univ.
77. Ipeirotis P. 2010. Demographics of Mechanical Turk. Work. Pap., Stern Sch. Bus., New YorkUniv.
78. Irani LC, Silberman M. 2013, April. Turkopticon: interrupting worker invisibility in Amazon Mechanical Turk. In Proc. SIGCHI Conf. Hum. Factors Comput. Syst., 31st, Paris, pp. 611-20. New York: ACM
79. Jahnke S, Imhoff R, Hoyer J. 2015. Stigmatization of people with pedophilia: two comparative surveys. Arch. Sex. Behav. 44:21-34
80. Jiang L, Wagner C. 2014. Participation in micro-task crowdsourcing markets as work and leisure: The impact of motivation and micro-time structuring. Presented at Collect. Intell. Conf., June 10-12, MIT, Cambridge, M.A.
81. Johnson DR, Borden LA. 2012. Participants at your fingertips using Amazon's Mechanical Turk to increase student-faculty collaborative research. Teach. Psychol. 39:245-51
82. Johnson PS, Herrmann ES, Johnson MW. 2015. Opportunity costs of reward delays and the discounting of hypothetical money and cigarettes. J. Exp. Anal. Behav. 103:87-107
83. Jones DN, Olderbak SG. 2014. The associations among dark personalities and sexual tactics across different scenarios. J. Interpers. Violence 29:1050-107
84. Jones DN, Paulhus DL. 2011. The role of impulsivity in the dark triad of personality. Personal. Individ. Differ. 51:679-82
85. Jurs SG, Glass GV. 1971. The effect of experimental mortality on the internal and external validity of the randomized comparative experiment. J. Exp. Educ. 40:62-66
86. Kang R, Brown S, Dabbish L, Kiesler S. 2014. Privacy attitudes of Mechanical Turk workers and the US public. Presented at SOUPS Workshop on Privacy Personas and Segmentation, 10th, Menlo Park, CA, July 9-11, pp. 37-47. Berkeley, CA: USENIX Assoc.
87. Kazai G, Kamps J, Milic-FraylingN. 2012. The face of quality in crowdsourcing relevance labels: demographics, personality and labeling accuracy. In Proc. Int. Conf. Inf. Knowl., 21st, Maui, pp. 2583-86. New York: ACM
88. Khanna S, Ratan A, Davis J, ThiesW. 2010. Evaluating and improving the usability of Mechanical Turk for low-income workers in India. In Proc. Symp. Comput. Dev., 1st, London, pp. 453-56. New York: ACM
89. Kim J. 2014. Does stigma against smokers really motivate cessation? A moderated mediation model on the effect of anti-smoking campaigns promoting smoker-related stigma on cessation intentions. Master's thesis, Univ. Wis., Milwaukee
90. Kittur A, Chi EH, Suh B. 2008. Crowdsourcing user studies with Mechanical Turk. In Proc. SIGCHI Conf. Hum. Factors Comput. Syst., 26th, Florence, pp. 453-56. New York: ACM
91. Klein RA, Ratliff KA, Vianello M, Adams RB Jr, Bahník S, et al. 2014. Investigating variation in replicability: A "many labs" replication project. Soc. Psychol. 45:142-52
92. Komarov S, Reinecke K, Gajos KZ. 2013. Crowdsourcing performance evaluations of user interfaces. In Proc. SIGCHI Conf. Hum. Factors Comput. Syst., 31st, Paris, pp. 207-16. New York: ACM
93. Konstam V, Tomek S, Celen-Demirtas S, Sweeney K. 2015. Volunteering and reemployment status in unemployed emerging adults a time-worthy investment? J. Career Assess. 23:152-65
94. Kosara R, Ziemkiewicz C. 2010.DoMechanical Turks dream of square pie charts? In Proc. BELIV'10Workshop. Beyond Time and Errors: Novel Eval. Methods Inf. Vis., 3rd, Atlanta, pp. 63-70. New York: ACM
95. Krische SD. 2014. Who is the average individual investor? Numerical skills and implications for accounting research. SSRN. http://www.dx.doi.org/10.2139/ssrn.2426570
96. Lanaj K, Johnson RE, Barnes CM. 2014. Beginning the workday yet already depleted? Consequences of late-night smartphone use and sleep. Organ. Behav. Hum. Dec. 124:11-23
97. Lang ASID, Rio-Ross J. 2011. Using Amazon Mechanical Turk to transcribe historical handwritten documents. Code4Lib J. 15:1
98. LeaseM, Hullman J, Bigham JP, Bernstein M, Kim J, et al. 2013. Mechanical Turk is not anonymous. SSRN. doi: 10.2139/ssrn.2228728
99. Lebowitz MS, AhnWK, Oltman K. 2015. Sometimes more competent, but always less warm: perceptions of biologically oriented mental-health clinicians. Int. J. Soc. Psychiatry 61:668-76
100. Lebowitz MS, Pyun JJ, Ahn WK. 2014. Biological explanations of generalized anxiety disorder: effects on beliefs about prognosis and responsibility. Psychiatr. Serv. 65:498-503
101. Lebreton P, Mäki T, Skodras E, Hupont I, Hirth M. 2015. Bridging the gap between eye tracking and crowdsourcing. In Proc. SPIE Conf. Hum. Vis. Electron. Imaging, 20th, San Francisco. Bellingham,WA: Int. Soc. Opt. Photonics. doi: 10.1117/12.2076745
102. Lintott C, Reed J. 2013. Human computation in citizen science. In Handbook of Human Computation, ed. P Michelucci, pp. 153-62. New York: Springer
103. Litman L, Robinson J, Rosenzweig C. 2015. The relationship between motivation, monetary compensation, and data quality among US-and India-based workers on Mechanical Turk. Behav. Res. Methods 47:519-28
104. Lynn BMD. 2014. Shared sense of purpose and well-being among veterans and non-veterans. PhD Thesis, North Carolina State Univ., Raleigh
105. Marge M, Banerjee S, Rudnicky A. 2010. Using the Amazon Mechanical Turk for transcription of spoken language. In Int. Conf. Acoust. Speech Signal Process. (ICASSP), 2010 IEEE, Dallas, pp. 5270-73. New York: IEEE
106. Marietta G, Viola J, Ibekwe N, Claremon J, Gehlbach H. 2014. Improving relationships through virtual environments: how seeing the world through victims' eyes may prevent bullying. Work. Pap., Grad. Sch. Educ., Harvard Univ.
107. Martin D, Hanrahan BV, O'Neill J, Gupta N. 2014. Being a Turker. In Proc. ACM Conf. Comput. Support. Coop. Work Social Comput., 17th, San Francisco, pp. 224-35. New York: ACM
108. MasonW, Suri S. 2012. Conducting behavioral research on Amazon's Mechanical Turk. Behav. Res. Methods 44:1-23
109. Mason W, Suri S, Watts DJ. 2014. Long-run learning in games of cooperation. In Proc. ACM Conf. Econ. Comput., 15th, Palo Alto, pp. 821-38. New York: ACM
110. MasonW,Watts DJ. 2010. Financial incentives and the performance of crowds. ACM SigKDD Explor. Newsl. 11:100-8
111. MasonW,Watts DJ. 2012. Collaborative learning in networks. PNAS 109:764-69
112. McAllister Byun TM, Halpin PF, Szeredi D. 2014. Online crowdsourcing for efficient rating of speech: A validation study. J. Commun. Disord. 53:70-83
113. Miller J. 2006. Online marketing research. In The Handbook of Marketing Research, ed. R Grover, M Vriens, pp. 110-32. Thousand Oaks, CA: SAGE
114. Miller JD, Few LR, Wilson L, Gentile B, Widiger TA, et al. 2013. The Five-Factor Narcissism Inventory (FFNI): A test of the convergent, discriminant, and incremental validity of FFNI scores in clinical and community samples. Psychol. Assess. 25:748-58
115. Mitchell GE, Locke KD. 2015. Lay beliefs about autism spectrum disorder among the general public and childcare providers. Autism 19:553-61
116. Morris RR, Picard R. 2014. Crowd-powered positive psychological interventions. J. Posit. Psychol. 9:509-16
117. Mullinix K, Drukman J, Freese J. 2014. The generalizability of survey experiments. Inst. Policy Res.,Work. Pap Ser., WP-14-19 Northwest. Univ., Evanston, IL
118. Nichols AL, Webster GD. 2015. Designing a brief measure of social anxiety: psychometric support for a three-item version of the Interaction Anxiousness Scale (IAS-3). Personal. Individ. Differ. 79:110-15
119. Norr AM, Allan NP, Boffa JW, Raines AM, Schmidt NB. 2015. Validation of the Cyberchondria Severity Scale (CSS): replication and extension with bifactor modeling. J. Anxiety Disord. 31:58-64
120. Nosek BA, Alter G, Banks GC, Borsboom D, Bowman SD, et al. 2015. Promoting an open research culture. Science 348(6242):1422-25
121. Nouri E, Traaum D. 2014. Cultural differences in playing repeated ultimatum game online with virtual humans. In Hawaii Int. Conf. System Sci. (HICSS), 47th, Waikoloa, Hawaii, pp. 1213-20. New York: IEEE
122. O'Connell BH, O'Shea D, Gallagher S. 2016. Enhancing social relationships through positive psychology activities: A randomised controlled trial. J. Posit. Psychol. 11:149-62
123. Oppenheimer DM, Meyvis T, Davidenko N. 2009. Instructional manipulation checks: detecting satisficing to increase statistical power. J. Exp. Soc. Psychol. 45:867-72
124. Palmer CJ, Paton B, Enticott PG, Hohwy J. 2015. "Subtypes" in the presentation of autistic traits in the general adult population. J. Autism Dev. Disord. 45:1291-301
125. Paolacci G, Chandler J. 2014. Inside the Turk: understanding Mechanical Turk as a participant pool. Curr. Dir. Psychol. Sci. 23:184-88
126. Paolacci G, Chandler J, Ipeirotis PG. 2010. Running experiments on AmazonMechanical Turk. Judgm. Decis. Mak. 5:411-19
127. Papa A, Lancaster NG, Kahler J. 2014. Commonalities in grief responding across bereavement and nonbereavement losses. J. Affect. Disord. 161:136-43
128. Parent J, McKee LG, Rough JN, Forehand R. 2015. The association of parentmindfulness with parenting and youth psychopathology across three developmental stages. J. Abnorm. Child Psychol. doi: 10.1007/s10802-015-9978-x
129. Pearl RL, Puhl RM, Dovidio JF. 2014. Differential effects of weight bias experiences and internalization on exercise among women with overweight and obesity. J. Health Psychol. doi: 10.1177/1359105313520338
130. Peer E, Paolacci G, Chandler J, Mueller P. 2012. Selectively recruiting participants from AmazonMechanical Turk using Qualtrics. SSRN. http://www.ssrn.com/abstract=092100631
131. Peer E, Vosgerau J, Acquisti A. 2014. Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behav. Res. Methods 46:1023-31
132. Pew Res. Cent. 2010. Blacks upbeat about black progress, prospects: A year after Obama's election. http://www. pewsocialtrends.org/2010/01/12/blacks-upbeat-about-black-progress-prospects/
133. Raihani NJ, Bshary R. 2012. A positive effect of flowers rather than eye images in a large-scale, cross-cultural dictator game. Proc. R. Soc. B 279:3556-64
134. Raines AM, Boffa JW, Allan NP, Short NA, Schmidt NB. 2015. Hoarding and eating pathology: The mediating role of emotion regulation. Compr. Psychiatry 57:29-35
135. Ranard BL, Ha YP, Meisel ZF, Asch DA, Hill SS, et al. 2014. Crowdsourcing-harnessing the masses to advance health and medicine, a systematic review. J. Gen. Intern. Med. 29:187-203
136. Rand DG. 2012. The promise of Mechanical Turk: how online labor markets can help theorists run behavioral experiments. J. Theor. Biol. 299:172-79
137. Rand DG, Peysakhovich A, Kraft-Todd GT,Newman GE, Wurzbacher O, et al. 2014. Social heuristics shape intuitive cooperation. Nat. Commun. 5:3677
138. Reese ED, Veilleux JC. 2016. Relationships between craving beliefs and abstinence self-efficacy are mediated by smoking motives and moderated by nicotine dependence. Nicotine Tob. Res. 18:48-55
139. Reidy DE, Berke DS, Gentile B, Zeichner A. 2014. Man enough? Masculine discrepancy stress and intimate partner violence. Personal. Individ. Differ. 68:160-64
140. Reips UD. 2002. Standards for Internet-based experimenting. Exp. Psychol. 49:243-56
141. Rojas SL, Widiger TA. 2014. Convergent and discriminant validity of the Five Factor Form. Assessment 21:143-57
142. Rose P, Segrist DJ. 2012. Difficulty identifying feelings, distress tolerance and compulsive buying: analyzing the associations to inform therapeutic strategies. J. Ment. Health Addict. 10:927-35
143. Rose P, Segrist DJ. 2014. Negative and positive urgency may both be risk factors for compulsive buying. J. Behav. Addict. 3:128-32
144. Ruzich E, Allison C, Smith P, Watson P, Auyeung B, et al. 2015. Measuring autistic traits in the general population: A systematic review of the Autism-Spectrum Quotient (AQ) in a nonclinical population sample of 6,900 typical adult males and females. Mol. Autism 6:2
145. Schleider JL, Weisz JR. 2015. Using Mechanical Turk to study family processes and youth mental health: A test of feasibility. J. Child Fam. Stud. 24:3235-46
146. Schnorf S, Sedley A, Ortlieb M,Woodruff A. 2014. A comparison of six sample providers regarding online privacy benchmarks. Presented at SOUPSWorkshop on Privacy Personas and Segmentation, 10th, Menlo Park, CA, July 9-11. Berkeley, CA: USENIX Assoc.
147. ShaoW, GuanW, Liu T, Clark MA, Merchant RC, et al. 2015. Variations in recruitment yield, costs, speed and participant diversity across Internet platforms in a global study examining the efficacy of an HIV/AIDs and HIV testing animated and live-action video among English-or Spanish-speaking Internet or social media users. Digit. Cult. Educ. 7(2)
148. Shapiro DN,Chandler J,Mueller PA. 2013. Using Mechanical Turk to study clinical populations. Clin. Psychol. Sci. 1:213-20
149. Shaw AD, Horton JJ, Chen DL. 2011. Designing incentives for inexpert human raters. In Proc. ACM 2011 Conf., Comput. Support. Coop. Work, Hangzhou, pp. 275-84. New York: ACM
150. SilbermanM, Irani L, Ross J. 2010. Ethics and tactics of professional crowdwork. XRDS Crossroads ACM Mag. Stud. 17:39-43
151. Simcox T, Fiez JA. 2014. Collecting response times using AmazonMechanical Turk and Adobe Flash. Behav. Res. Methods 46:95-111
152. Simons DJ, ChabrisCF. 2012.Common(mis) beliefs about memory: A replication and comparison of telephone and Mechanical Turk survey methods. PLOS ONE 7:e51876
153. Sina S, Kraus S, Rosenfeld A. 2014. Using the crowd to generate content for scenario-based serious-games. arXiv:1402.5034 [cs.AI]
154. Standish AJ, Benfield JA, Bernstein MJ, Tragesser S. 2014. Characteristics of borderline personality disorder and disgust sensitivity. Psychol. Rec. 64:869-77
155. Stewart N, Ungemach C, Harris AJL, Bartels DM, Newell BR, et al. 2015. The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers. Judgm. Decis. Mak. 10:479-91
156. Sturgis P, Allum N, Brunton-Smith I. 2009. Attitudes over time: The psychology of panel conditioning. In Methodology of Longitudinal Surveys, ed. P Lynn, pp. 113-26. Hoboken, NJ: Wiley
157. Suri S, Goldstein DG, Mason WA. 2011. Honesty in an online labor market. Presented at AAAI Workshops, 25th, San Francisco, pp. 61-66. Menlo Park, CA: AAAI Press
158. Suri S, Watts DJ. 2011. Cooperation and contagion in web-based, networked public goods experiments. PLOS ONE 6:e16836
159. ten Brinke L, Black PJ, Porter S, Carney DR. 2015. Psychopathic personality traits predict competitive wins and cooperative losses in negotiation. Personal. Individ. Differ. 79:116-22
160. Tobin SJ, Loxton NJ, Neighbors C. 2014. Coping with causal uncertainty through alcohol use. Addict. Behav. 39:580-85
161. Tylka TL, Wood-Barcalow NL. 2015. The Body Appreciation Scale-2: item refinement and psychometric evaluation. Body Image 12:53-67
162. Usinger T. 2014. Effect of internet administered mindfulness training on anxiety and sleep quality. Undergraduate thesis, Univ. Colo., Boulder
163. Veilleux JC, Salomaa AC, Shaver JA, Zielinski MJ, Pollert GA. 2015. Multidimensional assessment of beliefs about emotion: development and validation of the Emotion and Regulation Beliefs Scale. Assessment 22:86-100
164. Veilleux JC, Skinner KD, Reese ED, Shaver JA. 2014. Negative affect intensity influences drinking to cope through facets of emotion dysregulation. Personal. Individ. Differ. 59:96-101
165. Victor SE, Klonsky ED. 2014. Daily emotion in non-suicidal self-injury. J. Clin. Psychol. 70:364-75
166. Vlahovic TA, Wang YC, Kraut RE, Levine JM. 2014. Support matching and satisfaction in an online breast cancer support community. In Proc. ACM Conf. Hum. Factors Comput. Syst., 32nd, Toronto, pp. 1625-34. New York: ACM
167. VoracekM, Stieger S, Gindl A. 2001. Online replication of evolutionary psychology evidence: sex differences in sexual jealousy in imagined scenarios of mate's sexual versus emotional infidelity. In Dimensions of Internet Science, ed. U Reips, M Bosnjak, pp. 91-112. Lengerich: Pabst Sci.
168. Weatherly JN, Cookman ML. 2014. Investigating several factors potentially related to endorsing gambling as an escape. Curr. Psychol. 33:422-33
169. Weinberg JD, Freese J,McElhattan D. 2014. Comparing data characteristics and results of an online factorial survey between a population-based and a crowdsource-recruited sample. Sociol. Sci. 1:292-310
170. Wickham RE, Reed DE, Williamson RE. 2015. Establishing the psychometric properties of the Self and Perceived-Partner Authenticity in Relationships Scale-Short Form (AIRS-SF): measurement invariance, reliability, and incremental validity. Personal. Individ. Differ. 77:62-67
171. Wiens TK, Walker LJ. 2015. The chronic disease concept of addiction: helpful or harmful?Addict. Res. Theory 23:309-21
172. Winer ES, Veilleux JC, Ginger EJ. 2014. Development and validation of the Specific Loss of Interest and Pleasure Scale (SLIPS). J. Affect. Disord. 152:193-201
173. Wong CM. 2015. 50 Percent of millennials believe gender is a spectrum, Fusion's Massive Millennial Poll finds. The Huffington Post. http://www.huffingtonpost.com/2015/02/05/fusion-millennial-pollgender-n-6624200.html
174. Wurtele SK, Simons D, Moreno T. 2014. Sexual interest in children among an online sample of men and women: prevalence and correlates. Sex. Abuse 26:546-68
175. Wymbs BT, Dawson AE. 2015. Screening Amazon's Mechanical Turk for adults with ADHD. J. Attention Disord. doi: 10.1177/1087054715597471
176. Yang K, Friedman-Wheeler DG, Pronin E. 2014. Thought acceleration boosts positive mood among individuals with minimal to moderate depressive symptoms. Cogn. Ther. Res. 38:261-69