Many Labs 3: Evaluating participant pool quality across the academic semester via replication

Journal of Experimental Social Psychology

Volumn 67, Issue , 2016, Pages 68-82

  Ebersole, Charles R ^a   Atherton, Olivia E ^b   Belanger, Aimee L ^c   Skulborstad, Hayley M ^c   Allen, Jill M ^d   Banks, Jonathan B ^e   Baranski, Erica ^f   Bernstein, Michael J ^g   Bonfiglio, Diane B V ^h   Boucher, Leanne ^e   Brown, Elizabeth R ⁱ   Budiman, Nancy I ^b   Cairo, Athena H ^j   Capaldi, Colin A ^k   Chartier, Christopher R ^h   Chung, Joanne M ^b   Cicero, David C ^l   Coleman, Jennifer A ^j   Conway, John G ^m   Davis, William E ⁿ   Devos, Thierry ^o   Fletcher, Melody M ^p   German, Komi ^b   Grahe, Jon E ^q   Hermann, Anthony D ^r   Hicks, Joshua A ⁿ   Honeycutt, Nathan ^o   Humphrey, Brandon ^c   Janus, Matthew ^a   Johnson, David J ^s   Joy Gaba, Jennifer A ^j   Juzeler, Hannah ^q   Keres, Ashley ^h   Kinney, Diana ^a   Kirshenbaum, Jacqeline ^r   Klein, Richard A ^m   Lucas, Richard E ^s   Lustgraaf, Christopher J N ^p   Martin, Daniel ^a   Menon, Madhavi ^e   Metzger, Mitchell ^h   Moloney, Jaclyn M ^j   Morse, Patrick J ^f   Prislin, Radmila ^o   Razza, Timothy ^e   Re, Daniel E ^t   Rule, Nicholas O ^t   Sacco, Donald F ^p   Sauerberger, Kyle ^f   Shrider, Emily ^h   Shultz, Megan ^q   Siemsen, Courtney ^r   Sobocko, Karin ^k   Weylin Sternglanz, R ^e   Summerville, Amy ^c   Tskhay, Konstantin O ^t   van Allen, Zack ^k   Vaughn, Leigh Ann ^u   Walker, Ryan J ^c   Weinberg, Ashley ^o   Wilson, John Paul ^t   Wirth, James H ^v   Wortman, Jessica ^s   Nosek, Brian A ^a   more..

^a UNIVERSITY OF VIRGINIA   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c MIAMI UNIVERSITY   (United States)

^d MONTANA STATE UNIVERSITY   (United States)

^e NOVA SOUTHEASTERN UNIVERSITY   (United States)

^f UNIVERSITY OF CALIFORNIA   (United States)

^g PENNSYLVANIA STATE UNIVERSITY   (United States)

^h ASHLAND UNIVERSITY   (United States)

ⁱ UNIVERSITY OF NORTH FLORIDA   (United States)

^j VIRGINIA COMMONWEALTH UNIVERSITY   (United States)

^k CARLETON UNIVERSITY   (Canada)

^l UNIVERSITY OF HAWAII   (United States)

^m UNIVERSITY OF FLORIDA   (United States)

ⁿ TEXAS A AND M UNIVERSITY   (United States)

^o SAN DIEGO STATE UNIVERSITY   (United States)

^p UNIVERSITY OF SOUTHERN MISSISSIPPI   (United States)

^q PACIFIC LUTHERAN UNIVERSITY   (United States)

^r BRADLEY UNIVERSITY   (United States)

^s MICHIGAN STATE UNIVERSITY   (United States)

^t UNIVERSITY OF TORONTO   (Canada)

^u ITHACA COLLEGE   (United States)

^v OHIO STATE UNIVERSITY   (United States)

more..

Author keywords

Cognitive psychology; Individual differences; Participant pool; Replication; Sampling effects; Situational effects; Social psychology

Indexed keywords

BEHAVIORAL RESEARCH; HUMAN; MAJOR CLINICAL STUDY; MOOD; PARTICIPANT OBSERVATION; PSYCHOLOGY; SAMPLING; STRESS; UNIVERSITY;

EID: 84989936477     PISSN: 00221031     EISSN: 10960465     Source Type: Journal    
DOI: 10.1016/j.jesp.2015.10.012     Document Type: Article

References (48)

1. Amabile, T.M., Hill, K.G., Hennessey, B.A., Tighe, E.M., The Work Preference Inventory: assessing intrinsic and extrinsic motivational orientations. Journal of Personality and Social Psychology, 66(5), 1994, 950.
2. Aspinwall, L.G., Richter, L., Optimism and self-mastery predict more rapid disengagement from unsolvable tasks in the presence of alternatives. Motivation and Emotion 23 (1999), 221–245.
3. Aviv, A.L., Zelenski, J.M., Rallo, L., Larsen, R.J., Who comes when: Personality differences in early and later participation in a university subject pool. Personality and Individual Differences 33 (2002), 487–496.
4. Boroditsky, L., Metaphoric structuring: Understanding time through spatial metaphors. Cognition 75 (2000), 1–28.
5. Cacioppo, J.T., Petty, R.E., The need for cognition. Journal of Personality and Social Psychology 42 (1982), 116–131.
6. Cacioppo, J.T., Petty, R.E., Morris, K.J., Effects of need for cognition on message evaluation, recall, and persuasion. Journal of Personality and Social Psychology 45 (1983), 805–818.
7. Cesario, J., Priming, replication, and the hardest science. Perspectives on Psychological Science 9:1 (2014), 40–48.
8. Cloninger, C.R., A systematic method for clinical description and classification of personality variants: A proposal. Archives of General Psychiatry 44 (1987), 573–588.
9. Cohen, S., Kamarck, T., Mermelstein, R., A global measure of perceived stress. Journal of Health and Social Behavior 24 (1983), 385–396.
10. De Fruyt, F., Van De Wiele, L., Van Heeringen, C., Cloninger's psychobiological model of temperament and character and the five-factor model of personality. Personality and Individual Differences 29:3 (2000), 441–452.
11. Ehrhart, M.G., Ehrhart, K.H., Roesch, S.C., Chung-Herrera, B.G., Nadler, K., Bradshaw, K., Testing the latent factor structure and construct validity of the Ten-Item Personality Inventory. Personality and Individual Differences 47 (2009), 900–905.
12. Galinsky, A.D., Magee, J.C., Inesi, M.E., Gruenfeld, D.H., Power and perspectives not taken. Psychological Science 17 (2006), 1068–1074.
13. Gnambs, T., A meta-analysis of dependability coefficients (test–retest reliabilities) for measures of the Big Five. Journal of Research in Personality 52 (2014), 20–28.
14. Goldberg, L.R., Language and individual differences: The search for universals in personality lexicons. Review of Personality and Social Psychology 2 (1981), 141–165.
15. Gosling, S.D., Rentfrow, P.J., Swann, W.B. Jr., A very brief measure of the Big-Five personality domains. Journal of Research in Personality 37 (2003), 504–528.
16. Greenwald, A.G., Nosek, B.A., Banaji, M.R., Understanding and using the implicit association test: I. An improved scoring algorithm. Journal of Personality and Social Psychology 85 (2003), 197–216.
17. Grissom, R.J., Kim, J.J., Effect sizes for research: univariate and multivariate applications. 2012, Routledge, New York, NY.
18. Higgins, J.P., Thompson, S.G., Deeks, J.J., Altman, D.G., Measuring inconsistency in meta-analyses. BMJ [British Medical Journal] 327:7414 (2003), 557–560.
19. Hom, H.L., A methodological note: Time of participation effects on intrinsic motivation. Personality and Social Psychology Bulletin 13 (1987), 210–215.
20. Jostmann, N.B., Lakens, D., Schubert, T.W., Weight as an embodiment of importance. Psychological Science 20 (2009), 1169–1174.
21. Kane, M.J., Engle, R.W., Working-memory capacity and the control of attention: The contributions of goal neglect, response competition, and task set to Stroop interference. Journal of Experimental Psychology: General 132 (2003), 47–70.
22. Klein, R.A., Ratliff, K.A., Vianello, M., Adams, R.B. Jr., Bahník, Š., Bernstein, M.J.,. Nosek, B.A., Investigating variation in replicability: A “many labs” replication project. Social Psychology 45 (2014), 142–152.
23. Klein, R.A., Vianello, M., Hasselman, F., Alper, S., Aveyard, M., Axt, J.R.,. Nosek, B.A., Many labs 2: investigating variation in replicability across sample and setting. 2015, Manuscript in preparation.
24. Lai, V.T., Boroditsky, L., The immediate and chronic influence of spatio-temporal metaphors on the mental representations of time in English, Mandarin, and Mandarin–English speakers. Frontiers in Psychology, 4, 2013, 142.
25. Lambdin, C., Shaffer, V.A., Are within-subjects designs transparent?. Judgment and Decision Making 4:7 (2009), 554–556.
26. MacLeod, C.M., Half a century of research on the Stroop effect: An integrative review. Psychological Bulletin 109 (1991), 163–203.
27. Monin, B., Miller, D.T., Moral credentials and the expression of prejudice. Journal of Personality and Social Psychology 81 (2001), 33–43.
28. Nicholls, M.E., Loveless, K.M., Thomas, N.A., Loetscher, T., Churches, O., Some participants may be better than others: Sustained attention and motivation are higher early in semester. The Quarterly Journal of Experimental Psychology 68 (2015), 10–18.
29. Nosek, B.A., Sriram, N., Faulty assumptions: A comment on Blanton, Jaccard, Gonzales, and Christie (2006). Journal of Experimental Social Psychology 43 (2007), 393–398.
30. Open Science Collaboration. Estimating the reproducibility of psychological science. Science, 349(6251), 2015, 10.1126/science.aac4716.
31. Oppenheimer, D.M., Meyvis, T., Davidenko, N., Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology 45:4 (2009), 867–872.
32. Petty, R.E., Briñol, P., The elaboration likelihood model. Van Lange, P.A.M., Kruglanski, A.W., Higgins, E.T., (eds.) Handbook of theories of social psychology, 2012, Sage, London, 224–245.
33. Petty, R.E., Cacioppo, J.T., Kao, C.F., The efficient assessment of need for cognition. Journal of Personality Assessment 48:3 (1984), 306–307.
34. Robertson, I.H., Manly, T., Andrade, J., Baddeley, B.T., Yiend, J., Oops!’: Performance correlates of everyday attentional failures in traumatic brain injured and normal subjects. Neuropsychologia 35 (1997), 747–758.
35. Robins, R.W., Hendin, H.M., Trzesniewski, K.H., Measuring global self-esteem: Construct validation of a single-item measure and the Rosenberg Self-Esteem scale. Personality and Social Psychology Bulletin 27 (2001), 151–161.
36. Rojas, S.L., Widiger, T.A., Convergent and discriminant validity of the Five Factor Form. Assessment 21 (2014), 143–157.
37. Ross, M., Wilson, A.E., It feels like yesterday: Self-esteem, valence of personal past experiences, and judgments of subjective distance. Journal of Personality and Social Psychology 82 (2002), 792–803.
38. Schwarz, N., Clore, G.L., Mood, misattribution, and judgments of well-being: Informative and directive functions of affective states. Journal of Personality and Social Psychology 45 (1983), 513–523.
39. Sommer, K.L., Baumeister, R.F., Self-evaluation, persistence, and performance following implicit rejection: The role of trait self-esteem. Personality and Social Psychology Bulletin 28 (2002), 926–938.
40. Sriram, N., Greenwald, A.G., Nosek, B.A., Correlational biases in mean response latency differences. Statistical Methodology 7:3 (2010), 277–291.
41. Stroop, J.R., Studies of interference in serial verbal reactions. Journal of Experimental Psychology 18 (1935), 643–662.
42. Szymkow, A., Chandler, J., IJzerman, H., Parzuchowski, M., Wojciszke, B., Warmer hearts, warmer rooms. Social Psychology 44 (2013), 167–176.
43. Tversky, A., Kahneman, D., Availability: A heuristic for judging frequency and probability. Cognitive Psychology 5 (1973), 207–232.
44. Verplanken, B., Persuasive communication of risk information: A test of cue versus message processing effects in a field experiment. Personality and Social Psychology Bulletin 17:2 (1991), 188–193.
45. Verplanken, B., Hazenberg, P.T., Palenewen, G.R., Need for cognition and external information search effort. Journal of Research in Personality 26:2 (1992), 128–136.
46. Verhaeghen, P., De Meersman, L., Aging and the Stroop effect: A meta-analysis. Psychology and Aging, 13(1), 1998, 120.
47. Wang, A.Y., Jentsch, F.G., Point-of-time effects across the semester: Is there a sampling bias?. The Journal of Psychology 132 (1998), 211–219.
48. Witt, E.A., Donnellan, M.B., Orlando, M.J., Timing and selection effects within a psychology subject pool: Personality and sex matter. Personality and Individual Differences 50 (2011), 355–359.