Many labs 2: Investigating variation in replicability across samples and settings

Advances in Methods and Practices in Psychological Science

Volumn 1, Issue 4, 2018, Pages 443-490

  Klein, Richard A ^a   Vianello, Michelangelo ^b   Hasselman, Fred ^c   Adams, Byron G ^d,e   Adams, Reginald B ^f   Alper, Sinan ^g   Aveyard, Mark ^h   Axt, Jordan R ⁱ   Babalola, Mayowa T ^j   Bahník, Štěpán ^k   Batra, Rishtee ^l   Berkics, Mihály ^m   Bernstein, Michael J ⁿ   Berry, Daniel R ^o   Bialobrzeska, Olga ^p   Binan, Evans Dami ^q   Bocian, Konrad ^p   Brandt, Mark J ^d   Busching, Robert ^r   Rédei, Anna Cabak ^s   Cai, Huajian ^t   Cambier, Fanny ^u   Cantarero, Katarzyna ^p   Carmichael, Cheryl L ^v   Ceric, Francisco ^w   Chandler, Jesse ^x,y   Chang, Jen Ho ^z,aa   Chatard, Armand ^ab,ac   Chen, Eva E ^ad   Cheong, Winnee ^ae   Cicero, David C ^af   Coen, Sharon ^ag   Coleman, Jennifer A ^ah   Collisson, Brian ^ai   Conway, Morgan A ^aj   Corker, Katherine S ^ak   Curran, Paul G ^ak   Cushman, Fiery ^al   Dagona, Zubairu K ^q   Dalgar, Ilker ^am   Dalla Rosa, Anna ^b   Davis, William E ^an   de Bruijn, Maaike ^d   De Schutter, Leander ^ao   Devos, Thierry ^ap   de Vries, Marieke ^c,aq   Doğulu, Canay ^ar   Dozo, Nerisa ^as   Dukes, Kristin Nicole ^at   Dunham, Yarrow ^au   Durrheim, Kevin ^av   Ebersole, Charles R ^aw   Edlund, John E ^ax   Eller, Anja ^ay   English, Alexander Scott ^az   Finck, Carolyn ^ba   Frankowska, Natalia ^p   Freyre, Miguel Ángel ^ay   Friedman, Mike ^u   Galliani, Elisa Maria ^b   Gandi, Joshua C ^q   Ghoshal, Tanuka ^bb   Giessner, Steffen R ^bc   Gill, Tripat ^bd   Gnambs, Timo ^be,bf   Gómez, Ángel ^bg   González, Roberto ^bh   Graham, Jesse ^bi   Grahe, Jon E ^bj   Grahek, Ivan ^bk   Green, Eva G T ^bl   Hai, Kakul ^bm   Haigh, Matthew ^bn   Haines, Elizabeth L ^bo   Hall, Michael P ^x   Heffernan, Marie E ^bp   Hicks, Joshua A ^bq   Houdek, Petr ^br   Huntsinger, Jeffrey R ^bs   Huynh, Ho Phi ^bt   Ijzerman, Hans ^a   Inbar, Yoel ^bu   Innes Ker, Åse H ^bv   Jiménez Leal, William ^ba   John, Melissa Sue ^bw   Joy Gaba, Jennifer A ^ah   Kamiloğlu, Roza G ^bx   Kappes, Heather Barry ^by   Karabati, Serdar ^bz   Karick, Haruna ^p,q   Keller, Victor N ^ca   Kende, Anna ^m   Kervyn, Nicolas ^u   Knežević, Goran ^cb   Kovacs, Carrie ^bf   Krueger, Lacy E ^cc   Kurapov, German ^cd   Kurtz, Jamie ^ce   Lakens, Daniël ^cf   Lazarević, Ljiljana B ^cb   Levitan, Carmel A ^cg   Lewis, Neil A ^ch   Lins, Samuel ^ci   Lipsey, Nikolette P ^aj   Losee, Joy E ^aj   Maassen, Esther ^d   Maitner, Angela T ^h   Malingumu, Winfrida ^cj   Mallett, Robyn K ^bs   Marotta, Satia A ^ck   Međedović, Janko ^cl,cm   Mena Pacheco, Fernando ^cn   Milfont, Taciano L ^co   Morris, Wendy L ^cp   Murphy, Sean C ^cq   Myachykov, Andriy ^bn   Neave, Nick ^bn   Neijenhuijs, Koen ^bx,cr   Nelson, Anthony J ^f   Neto, Félix ^ci   Nichols, Austin Lee ^cs   Ocampo, Aaron ^cn   O’donnell, Susan L ^ct   Oikawa, Haruka ^cu   Oikawa, Masanori ^cu   Ong, Elsie ^cv   Orosz, Gábor ^cw   Osowiecka, Malgorzata ^p   Packard, Grant ^bd   Pérez Sánchez, Rolando ^cx   Petrović, Boban ^cm   Pilati, Ronaldo ^ca   Pinter, Brad ^f   Podesta, Lysandra ^c   Pogge, Gabrielle ^aj   Pollmann, Monique M H ^d   Rutchick, Abraham M ^cy   Saavedra, Patricio ^cz   Saeri, Alexander K ^da   Salomon, Erika ^db   Schmidt, Kathleen ^dc   Schönbrodt, Felix D ^dd   Sekerdej, Maciej B ^de   Sirlopú, David ^w   Skorinko, Jeanine L M ^bw   Smith, Michael A ^bn   Smith Castro, Vanessa ^cx   Smolders, Karin C H J ^cf   Sobkow, Agata ^p   Sowden, Walter ^df   Spachtholz, Philipp ^dd   Srivastava, Manini ^dg   Steiner, Troy G ^f   Stouten, Jeroen ^dh   Street, Chris N H ^di   Sundfelt, Oskar K ^bv   Szeto, Stephanie ^ag   Szumowska, Ewa ^de   Tang, Andrew C W ^cv   Tanzer, Norbert ^dj   Tear, Morgan J ^da   Theriault, Jordan ^dk   Thomae, Manuela ^dl   Torres, David ^dm   Traczyk, Jakub ^p   Tybur, Joshua M ^cr   Ujhelyi, Adrienn ^m   van Aert, Robbie C M ^d   van Assen, Marcel A L M ^d   van der Hulst, Marije ^dn   van Lange, Paul A M ^cr   van ’t Veer, Anna Elisabeth ^do   Vásquez Echeverría, Alejandro ^dp   Vaughn, Leigh Ann ^dq   Vázquez, Alexandra ^bg   Vega, Luis Diego ^cn   Verniers, Catherine ^dr   Verschoor, Mark ^ds   Voermans, Ingrid P J ^c   Vranka, Marek A ^dt   Welch, Cheryl ^ce   Wichman, Aaron L ^du   Williams, Lisa A ^dv   Wood, Michael ^dl   Woodzicka, Julie A ^dw   Wronska, Marta K ^p   Young, Liane ^dx   Zelenski, John M ^dy   Zhijia, Zeng ^dz   Nosek, Brian A ^aw,ea   more..

^a UNIVERSITÉ GRENOBLE ALPES   (France)

^b UNIVERSITY OF PADOVA   (Italy)

^c RADBOUD UNIVERSITY NIJMEGEN   (Netherlands)

^d TILBURG UNIVERSITY   (Netherlands)

^e UNIVERSITY OF JOHANNESBURG   (South Africa)

^f PENNSYLVANIA STATE UNIVERSITY   (United States)

^g YASAR UNIVERSITY   (Turkey)

^h AMERICAN UNIVERSITY OF SHARJAH   (United Arab Emirates)

ⁱ DUKE UNIVERSITY   (United States)

^j UNITED ARAB EMIRATES UNIVERSITY   (United Arab Emirates)

^k UNIVERSITY OF ECONOMICS   (Czech Republic)

^l SAINT JOSEPH S UNIVERSITY   (United States)

^m EÖTVÖS LORÁND UNIVERSITY   (Hungary)

ⁿ PENNSYLVANIA STATE UNIVERSITY   (United States)

^o CALIFORNIA STATE UNIVERSITY   (United States)

^p SWPS UNIVERSITY OF SOCIAL SCIENCES AND HUMANITIES   (Poland)

^q UNIVERSITY OF JOS   (Nigeria)

^r UNIVERSITY OF POTSDAM   (Germany)

^s LUND UNIVERSITY   (Sweden)

^t INSTITUTE OF PSYCHOLOGY   (China)

^u UNIVERSITÉ CATHOLIQUE DE LOUVAIN   (Belgium)

^v CITY UNIVERSITY OF NEW YORK   (United States)

^w UNIVERSIDAD DEL DESARROLLO   (Chile)

^x UNIVERSITY OF MICHIGAN   (United States)

^y MATHEMATICA POLICY RESEARCH   (United States)

^z INSTITUTE OF ETHNOLOGY   (Taiwan)

^aa NATIONAL TAIWAN UNIVERSITY   (Taiwan)

^ab UNIVERSITÉ DE POITIERS   (France)

^ac CNRS Unité Mixte de Recherche 7295   (France)

^ad HONG KONG UNIVERSITY OF SCIENCE AND TECHNOLOGY   (Hong Kong)

^ae HELP UNIVERSITY   (Malaysia)

^af UNIVERSITY OF HAWAII   (United States)

^ag UNIVERSITY OF SALFORD   (United Kingdom)

^ah VIRGINIA COMMONWEALTH UNIVERSITY   (United States)

^ai AZUSA PACIFIC UNIVERSITY   (United States)

^aj UNIVERSITY OF FLORIDA   (United States)

^ak GRAND VALLEY STATE UNIVERSITY   (United States)

^al HARVARD UNIVERSITY   (United States)

^am MIDDLE EAST TECHNICAL UNIVERSITY   (Turkey)

^an WITTENBERG UNIVERSITY   (United States)

^ao WHU OTTO BEISHEIM SCHOOL OF MANAGEMENT   (Germany)

^ap SAN DIEGO STATE UNIVERSITY   (United States)

^aq TILBURG UNIVERSITY   (Netherlands)

^ar BASKENT UNIVERSITY   (Turkey)

^as UNIVERSITY OF QUEENSLAND   (Australia)

^at ALLEGHENY COLLEGE   (United States)

^au YALE UNIVERSITY   (United States)

^av UNIVERSITY OF KWAZULU NATAL   (South Africa)

^aw UNIVERSITY OF VIRGINIA   (United States)

^ax ROCHESTER INSTITUTE OF TECHNOLOGY   (United States)

^ay UNIVERSIDAD NACIONAL AUTÓNOMA DE MÉXICO   (Mexico)

^az SHANGHAI INTERNATIONAL STUDIES UNIVERSITY   (China)

^ba UNIVERSIDAD DE LOS ANDES   (Colombia)

^bb CITY UNIVERSITY OF NEW YORK   (United States)

^bc ERASMUS UNIVERSITY   (Netherlands)

^bd WILFRID LAURIER UNIVERSITY   (Canada)

^be Leibniz Institute for Educational Trajectories   (Germany)

^bf JOHANNES KEPLER UNIVERSITY LINZ   (Austria)

^bg UNIVERSIDAD NACIONAL DE EDUCACIÓN A DISTANCIA   (Spain)

^bh PONTIFICIA UNIVERSIDAD CATÓLICA DE CHILE   (Chile)

^bi UNIVERSITY OF UTAH   (United States)

^bj PACIFIC LUTHERAN UNIVERSITY   (United States)

^bk GHENT UNIVERSITY   (Belgium)

^bl UNIVERSITY OF LAUSANNE   (Switzerland)

^bm Amity Universities   (India)

^bn NORTHUMBRIA UNIVERSITY   (United Kingdom)

^bo WILLIAM PATERSON UNIVERSITY   (United States)

^bp CHILDREN'S MEMORIAL HOSPITAL   (United States)

^bq TEXAS A AND M UNIVERSITY   (United States)

^br J E PURKINJE UNIVERSITY   (Czech Republic)

^bs LOYOLA UNIVERSITY CHICAGO   (United States)

^bt TEXAS A M UNIVERSITY SAN ANTONIO   (United States)

^bu UNIVERSITY OF TORONTO   (Canada)

^bv LUND UNIVERSITY   (Sweden)

^bw WORCESTER POLYTECHNIC INSTITUTE   (United States)

^bx UNIVERSITY OF AMSTERDAM   (Netherlands)

^by LONDON SCHOOL OF ECONOMICS   (United Kingdom)

^bz ISTANBUL BILGI UNIVERSITY   (Turkey)

^ca UNIVERSITY OF BRASILIA   (Brazil)

^cb UNIVERSITY OF BELGRADE   (Serbia)

^cc TEXAS A AND M UNIVERSITY COMMERCE   (United States)

^cd TILBURG UNIVERSITY   (Netherlands)

^ce JAMES MADISON UNIVERSITY   (United States)

^cf EINDHOVEN UNIVERSITY OF TECHNOLOGY   (Netherlands)

^cg OCCIDENTAL COLLEGE   (United States)

^ch Department of Obstetrics Gynecology   (United States)

^ci UNIVERSITY OF PORTO   (Portugal)

^cj OPEN UNIVERSITY OF TANZANIA   (Tanzania)

^ck TUFTS UNIVERSITY   (United States)

^cl SINGIDUNUM UNIVERSITY   (Serbia)

^cm Institute of Criminological and Sociological Research   (Serbia)

^cn Universidad Latina de Costa Rica   (Costa Rica)

^co VICTORIA UNIVERSITY OF WELLINGTON   (New Zealand)

^cp MCDANIEL COLLEGE   (United States)

^cq UNIVERSITY OF MELBOURNE   (Australia)

^cr VU UNIVERSITY AMSTERDAM   (Netherlands)

^cs UNIVERSITY OF CENTRAL FLORIDA   (United States)

^ct GEORGE FOX UNIVERSITY   (United States)

^cu DOSHISHA UNIVERSITY   (Japan)

^cv OPEN UNIVERSITY OF HONG KONG   (Hong Kong)

^cw STANFORD UNIVERSITY   (United States)

^cx UNIVERSIDAD DE COSTA RICA   (Costa Rica)

^cy CALIFORNIA STATE UNIVERSITY   (United States)

^cz UNIVERSITY OF SUSSEX   (United Kingdom)

^da MONASH UNIVERSITY   (Australia)

^db THE UNIVERSITY OF CHICAGO   (United States)

^dc SOUTHERN ILLINOIS UNIVERSITY   (United States)

^dd UNIVERSITY OF MUNICH   (Germany)

^de JAGIELLONIAN UNIVERSITY   (Poland)

^df WALTER REED ARMY INSTITUTE OF RESEARCH   (United States)

^dg UNIVERSITY OF REGENSBURG   (Germany)

^dh UNIVERSITY OF LEUVEN   (Belgium)

^di UNIVERSITY OF HUDDERSFIELD   (United Kingdom)

^dj UNIVERSITY OF GRAZ   (Austria)

^dk NORTHEASTERN UNIVERSITY   (United States)

^dl UNIVERSITY OF WINCHESTER   (United Kingdom)

^dm Universidad de Iberoamerica   (Costa Rica)

^dn ERASMUS MEDICAL CENTER   (Netherlands)

^do LEIDEN UNIVERSITY   (Netherlands)

^dp UNIVERSIDAD DE LA REPÚBLICA   (Uruguay)

^dq ITHACA COLLEGE   (United States)

^dr UNIVERSITÉ PARIS DESCARTES   (France)

^ds UNIVERSITY OF GRONINGEN   (Netherlands)

^dt Charles University   (Czech Republic)

^du WESTERN KENTUCKY UNIVERSITY   (United States)

^dv UNIVERSITY OF NEW SOUTH WALES   (Australia)

^dw WASHINGTON AND LEE UNIVERSITY   (United States)

^dx BOSTON COLLEGE   (United States)

^dy CARLETON UNIVERSITY   (Canada)

^dz ZHEJIANG UNIVERSITY OF FINANCE AND ECONOMICS   (China)

^ea CENTER FOR OPEN SCIENCE   (United States)

more..

Author keywords

Cognitive psychology; Culture; Individual differences; Meta analysis; Open data; Open materials; Preregistered; Registered Report; Replication; Sampling effects; Situational effects; Social psychology

Indexed keywords

EID: 85116789185     PISSN: 25152459     EISSN: 25152467     Source Type: Journal    
DOI: 10.1177/2515245918810225     Document Type: Article

References (117)

1. Adler, N. E., Boyce, T., Chesney, M. A., Cohen, S., Folkman, S., Kahn, R. L., & Syme, S. L. (1994). Socioeconomic status and health: The challenge of the gradient. American Psychologist, 49, 15–24. doi:10.1037/0003-066X.49.1.15
2. Adler, N. E., Epel, E. S., Castellazzo, G., & Ickovics, J. R. (2000). Relationship of subjective and objective social status with psychological and physiological functioning: Preliminary data in healthy White women. Health Psychology, 19, 586–592. doi:10.1037/0278-6133.19.6.586
3. Alter, A. L., Oppenheimer, D. M., Epley, N., & Eyre, R. N. (2007). Overcoming intuition: Metacognitive difficulty activates analytic reasoning. Journal of Experimental Psychology: General, 136, 569–576. doi:10.1037/00963445.136.4.569
4. Anderson, C., Kraus, M. W., Galinsky, A. D., & Keltner, D. (2012). The local-ladder effect: Social status and subjective well-being. Psychological Science, 23, 764–771. doi:10.1177/0956797611434537
5. Ashton-James, C. E., Maddux, W. W., Galinsky, A. D., & Chartrand, T. L. (2009). Who I am depends on how I feel: The role of affect in the expression of culture. Psychological Science, 20, 340–346.
6. Bakker, M., Hartgerink, C. H., Wicherts, J. M., & van der Maas, H. L. (2016). Researchers’ intuitions about power in psychological research. Psychological Science, 27, 1069–1077.
7. Bauer, M. A., Wilkie, J. E., Kim, J. K., & Bodenhausen, G. V. (2012). Cuing consumerism: Situational materialism undermines personal and social well-being. Psychological Science, 23, 517–523. doi:10.1177/0956797611429579
8. Baumeister, R. F., & Vohs, K. D. (2016). Misguided effort with elusive implications. Perspectives on Psychological Science, 11, 574–575.
9. Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J., Berk, R., … Johnson, V. E. (2018). Redefine statistical significance. Nature Human Behaviour, 2, 6–10. doi:10.1038/s41562-017-0189-z
10. Birnbaum, M. H. (2004). Human research and data collection via the Internet. Annual Review of Psychology, 55, 803–832. doi:10.1146/annurev.psych.55.090902.141601
11. Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. Chichester, England: John Wiley & Sons.
12. Brislin, R. W. (1970). Back-translation for cross-cultural research. Journal of Cross-Cultural Psychology, 1, 185– 216. doi:10.1177/135910457000100301
13. Button, K. S., Ioannidis, J. P., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S., & Munafò, M. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, 365–376.
14. Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., … Wu, H. (2016). Evaluating replicability of laboratory experiments in economics. Science, 351, 1433–1436. doi:10.1126/science.aaf0918
15. Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., … Wu, H. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 2, 637–644. doi:10.1038/s41562-018-0399-z
16. Campbell, D. F. J., Pölzlbauer, P., Barth, T. D., & Pölzlbauer, G. (2015). Democracy ranking 2015 (scores). Retrieved from http://democracyranking.org/ranking/2015/data/Scores_of_the_Democracy_Ranking_2015_A4.pdf
17. Chandler, J., Mueller, P., & Paolacci, G. (2014). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, 46, 112–130. doi:10.3758/s13428-013-0365-7
18. Cheung, F., & Lucas, R. E. (2014). Assessing the validity of single-item life satisfaction measures: Results from three large samples. Quality of Life Research, 10, 2809–2818. doi:10.1007/s11136-014-0726-4
19. Cohen, G. L., Sherman, D. K., Bastardi, A., Hsu, L., McGoey, M., & Ross, L. (2007). Bridging the partisan divide: Selfaffirmation reduces ideological closed-mindedness and inflexibility in negotiation. Journal of Personality and Social Psychology, 93, 415–430. doi:10.1037/00223514.93.3.415
20. Cohen, S., Alper, C. M., Doyle, W. J., Adler, N., Treanor, J. J., & Turner, R. B. (2008). Objective and subjective socioeconomic status and susceptibility to the common cold. Health Psychology, 27, 268–274. doi:10.1037/02786133.27.2.268
21. Coppock, A. (in press). Generalizing from survey experiments conducted on Mechanical Turk: A replication approach. Political Science Research Methods.
22. Cramer, A. O. J., van Ravenzwaaij, D., Matzke, D., Steingroever, H., Wetzels, R., Grasman, R. P. P. P., … Wagenmakers, E.-J. (2016). Hidden multiplicity in exploratory multiway ANOVA: Prevalence and remedies. Psychonomic Bulletin & Review, 23, 640–647.
23. Crisp, R. J., Miles, E., & Husnu, S. (2014). Support for the replicability of imagined contact effects. Social Psychology, 45, 303–304. doi:10.1027/1864-9335/a000202
24. Critcher, C. R., & Gilovich, T. (2008). Incidental environmental anchors. Journal of Behavioral Decision Making, 21, 241–251. doi:10.1002/bdm.586
25. Dandurand, F., Shultz, T. R., & Onishi, K. H. (2008). Comparing online and lab methods in a problem-solving experiment. Behavior Research Methods, 40, 428–434.
26. Doyen, S., Klein, O., Pichon, C. L., & Cleeremans, A. (2012). Behavioral priming: It’s all in the mind, but whose mind? PLOS ONE, 7(1), Article e29081. doi:10.1371/journal.pone.0029081
27. Dreber, A., Pfeiffer, T., Almenberg, J., Isaksson, S., Wilson, B., Chen, Y., … Johannesson, M. (2016). Using prediction markets to estimate the reproducibility of scientific research. Proceedings of the National Academy of Sciences, USA, 112, 15343–15347. doi:10.1073/pnas.15161 79112
28. Ebersole, C. R., Atherton, O. E., Belanger, A. L., Skulborstad, H. M., Allen, J. M., Banks, J. B., … Brown, E. R. (2016). Many Labs 3: Evaluating participant pool quality across the academic semester via replication. Journal of Experimental Social Psychology, 67, 68–82.
29. Education Index. (2017). Retrieved from https://en.wikipedia.org/wiki/Education_Index
30. Ehrhart, M. G., Ehrhart, K. H., Roesch, S. C., Chung-Herrera, B. G., Nadler, K., & Bradshaw, K. (2009). Testing the latent factor structure and construct validity of the TenItem Personality Inventory. Personality and Individual Differences, 47, 900–905. doi:10.1016/j.paid.2009.07.012
31. Ferguson, M. J., Carter, T. J., & Hassin, R. R. (2014). Commentary on the attempt to replicate the effect of the American flag on increased Republican attitudes. Social Psychology, 45, 301–302. doi:10.1027/1864-9335/a000202
32. Finucane, M. L., & Gullion, C. M. (2010). Developing a tool for measuring the decision-making competence of older adults. Psychology and Aging, 25, 271–288. doi:10.1037/a0019106
33. Forsell, E., Viganola, D., Pfeiffer, T., Almenberg, J., Wilson, B., Chen, Y., … Dreber, A. (in press). Predicting replication outcomes in the Many Labs 2 study. Journal of Economic Psychology.
34. Fraley, R. C., & Vazire, S. (2014). The N-pact factor: Evaluating the quality of empirical journals with respect to sample size and statistical power. PLOS ONE, 9(10), Article e109019. doi:10.1371/journal.pone.0109019
35. Frazier, P. A., Tix, A. P., & Barron, K. E. (2004). Testing moderator and mediator effects in counseling psychology research. Journal of Counseling Psychology, 51, 115–134.
36. Frederick, S. (2005). Cognitive reflection and decision making. Journal of Economic Perspectives, 19, 25–42. doi:10.1257/089533005775196732
37. Giessner, S. R., & Schubert, T. W. (2007). High in the hierarchy: How vertical location and judgments of leaders’ power are interrelated. Organizational Behavior and Human Decision Processes, 104, 30–44. doi:10.1016/j.obhdp.2006.10.001
38. Gilbert, D. T., King, G., Pettigrew, S., & Wilson, T. D. (2016). Comment on “Estimating the reproducibility of psychological science.” Science, 351, 1037. doi:10.1126/science.aad7243
39. Gilbert, D. T., & Malone, P. S. (1995). The correspondence bias. Psychological Bulletin, 117, 21–38. doi:10.1037/0033-2909.117.1.21
40. Gnambs, T. (2014). A meta-analysis of dependability coefficients (test-retest reliabilities) for measures of the Big Five. Journal of Research in Personality, 52, 20–28. doi:10.1016/j.jrp.2014.06.003
41. Goldberg, L. R. (1981). Language and individual differences: The search for universals in personality lexicons. In L. Wheeler (Ed.), Review of personality and social psychology (Vol. 2, pp. 141–165). Beverly Hills, CA: Sage.
42. Gosling, S. D., Rentfrow, P. J., & Swann, W. B., Jr. (2003). A very brief measure of the Big-Five personality domains. Journal of Research in Personality, 37, 504–528. doi:10.1016/S0092-6566(03)00046-1
43. Graham, J., Haidt, J., & Nosek, B. A. (2009). Liberals and conservatives rely on different sets of moral foundations. Journal of Personality and Social Psychology, 96, 1029– 1046. doi:10.1037/a0015141
44. Gray, K., & Wegner, D. M. (2009). Moral typecasting: Divergent perceptions of moral agents and moral patients. Journal of Personality and Social Psychology, 96, 505– 520. doi:10.1037/a0013748
45. Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82, 1–20.
46. Hagger, M. S., Chatzisarantis, N. L., Alberts, H., Anggono, C. O., Batailler, C., Birt, A. R., … Calvillo, D. P. (2016). A multilab preregistered replication of the ego-depletion effect. Perspectives on Psychological Science, 11, 546–573.
47. Haidt, J., McCauley, C., & Rozin, P. (1994). Individual differences in sensitivity to disgust: A scale sampling seven domains of disgust elicitors. Personality and Individual Differences, 16, 701–713. doi:10.1016/0191-8869(94)90212-7
48. Hauser, M. D., Cushman, F. A., Young, L., Jin, R., & Mikhail, J. M. (2007). A dissociation between moral judgments and justifications. Mind & Language, 22, 1–21. doi:10.1111/j.1468-0017.2006.00297.x
49. Hawkins, C. B., Fitzgerald, C., & Nosek, B. A. (2015). In search of an association between conception risk and prejudice. Psychological Science, 26, 249–252.
50. Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral & Brain Sciences, 33, 61–83.
51. Hilbig, B. E. (2016). Reaction time effects in lab-versus Web-based research: Experimental evidence. Behavior Research Methods, 48, 1718–1724.
52. Hsee, C. K. (1998). Less is better: When low-value options are valued more highly than high-value options. Journal of Behavioral Decision Making, 11, 107–121. doi:10.1002/(SICI)1099-0771(199806)11:2<107::AIDBDM292>3.0.CO;2-Y
53. Huang, Y., Tse, C. S., & Cho, K. W. (2014). Living in the north is not necessarily favorable: Different metaphoric associations between cardinal direction and valence in Hong Kong and in the United States. European Journal of Social Psychology, 44, 360–369. doi:10.1002/ejsp.2013
54. Hunter, J. E., & Schmidt, F. L. (1990). Methods of metaanalysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.
55. Inbar, Y., Pizarro, D., Knobe, J., & Bloom, P. (2009). Disgust sensitivity predicts intuitive disapproval of gays. Emotion, 9, 435–439. doi:10.1037/a0015960
56. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524–532.
57. Johnson, D. J., Cheung, F., & Donnellan, M. B. (2014). Does cleanliness influence moral judgments? A direct replication of Schnall, Benton, and Harvey (2008). Social Psychology, 45, 209–215. doi:10.1027/1864-9335/a000186
58. Jones, E. E., & Harris, V. A. (1967). The attribution of attitudes. Journal of Experimental Social Psychology, 3, 1–24. doi:10.1016/0022-1031(67)90034-0
59. Kahneman, D. (2016). Commentary on Ebersole et al. (2016). Journal of Experimental Social Psychology, 67, 97. doi:10.1016/j.jesp.2016.04.002
60. Kay, A. C., Laurin, K., Fitzsimons, G. M., & Landau, M. J. (2014). A functional basis for structure-seeking: Exposure to structure promotes willingness to engage in motivated action. Journal of Experimental Psychology: General, 143, 486–491. doi:10.1037/a0034462
61. Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Jr., Bahník, Š., Bernstein, M. J., … Nosek, B. A. (2014a). Investigating variation in replicability: A “Many Labs” replication project. Social Psychology, 45, 142–152. doi:10.1027/1864-9335/a000178
62. Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Jr., Bahník, Š., Bernstein, M. J., … Nosek, B. A. (2014b). Theory building through replication: Response to commentaries on the “Many Labs” replication project. Social Psychology, 45, 307–310.
63. Knobe, J. (2003). Intentional action and side effects in ordinary language. Analysis, 63, 190–193. doi:10.1111/14678284.00419
64. Knobe, J. (2006). The concept of intentional action: A case study in the uses of folk psychology. Philosophical Studies, 130, 203–231. doi:10.1007/s11098-004-4510-0
65. Krupnikov, Y., & Levine, A. S. (2014). Cross-sample comparisons and external validity. Journal of Experimental Political Science, 1, 59–80.
66. Lewin, K. (1936). Principles of topological psychology. New York, NY: McGraw-Hill.
67. Lucas, R. E., & Donnellan, M. B. (2012). Estimating the reliability of single-item life satisfaction measures: Results from four national panel studies. Social Indicators Research, 105, 323–331. doi:10.1007/s11205-011-9783-z
68. Markus, H. R., & Kitayama, S. (1991). Culture and the self: Implications for cognition, emotion, and motivation. Psychological Review, 98, 224–253. doi:10.1037/0033295X.98.2.224
69. McClelland, G. H. (1997). Optimal design in psychological research. Psychological Methods, 2, 3–19. doi:10.1037/1082-989X.2.1.3
70. Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17, 437–455. doi:10.1037/a0028085
71. Miguel, E., Camerer, C., Casey, K., Cohen, J., Esterling, K. M., Gerber, A., … Van der Laan, M. (2014). Promoting transparency in social science research. Science, 343, 30–31. doi:10.1126/science.1245317
72. Miyamoto, Y., & Kitayama, S. (2002). Cultural variation in correspondence bias: The critical role of attitude diagnosticity of socially constrained behavior. Journal of Personality and Social Psychology, 83, 1239–1248. doi:10.1037/0022-3514.83.5.1239
73. Morey, R. D., & Lakens, D. (2016). Why most of psychology is statistically unfalsifiable. doi:10.5281/zenodo.838685
74. Mullen, B., Atkins, J. L., Champion, D. S., Edwards, C., Hardy, D., Story, J. E., & Vanderklok, M. (1985). The false consensus effect: A meta-analysis of 115 hypothesis tests. Journal of Experimental Social Psychology, 21, 262–283. doi:10.1016/0022-1031(85)90020-4
75. Mullinix, K. J., Leeper, T. J., Druckman, J. N., & Freese, J. (2015). The generalizability of survey experiments. Journal of Experimental Political Science, 2, 109–138.
76. Murphy, R. O., Ackermann, K. A., & Handgraaf, M. J. (2011). Measuring social value orientation. Judgment and Decision Making, 6, 771–781. doi:10.2139/ssrn.1804189
77. Norenzayan, A., Smith, E. E., Kim, B. J., & Nisbett, R. E. (2002). Cultural preferences for formal versus intuitive reasoning. Cognitive Science, 26, 653–684. doi:10.1207/s15516709cog2605_4
78. Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., … Yarkoni, T. (2015). Promoting an open research culture. Science, 348, 1422–1425. doi:10.1126/science.aab2374
79. Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, USA, 115, 2600–2606. doi:10.1073/pnas.1708274114
80. Nosek, B. A., & Lakens, D. (2014). Registered Reports: A method to increase the credibility of published results. Social Psychology, 45, 137–141. doi:10.1027/1864-9335/a000192
81. Olatunji, B. O., Williams, N. L., Tolin, D. F., Abramowitz, J. S., Sawchuk, C. N., Lohr, J. M., & Elwood, L. S. (2007). The Disgust Scale: Item analysis, factor structure, and suggestions for refinement. Psychological Assessment, 19, 281–297. doi:10.1037/1040-3590.19.3.281
82. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, aac4716. doi:10.1126/science.aac4716
83. Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45, 867–872. doi:10.1016/j.jesp.2009.03.009
84. Oswald, A. J., & Wu, S. (2010). Objective confirmation of subjective measures of human well-being: Evidence from the U.S.A. Science, 327, 576–579. doi:10.1126/science.1180606
85. Ramscar, M., Shaoul, C., & Baayen, R. H. (2015). Why many priming results don’t (and won’t) replicate: A quantitative analysis. Unpublished manuscript, Department of Quantitative Linguistics, Eberhard Karls Universität, Tübingen.
86. Risen, J. L., & Gilovich, T. (2008). Why people are reluctant to tempt fate. Journal of Personality and Social Psychology, 95, 293–307. doi:10.1037/0022-3514.95.2.293
87. Robins, R. W., Hendin, H. M., & Trzesniewski, K. H. (2001). Measuring global self-esteem: Construct validation of a single-item measure and the Rosenberg Self-Esteem Scale. Personality and Social Psychology Bulletin, 27, 151–161. doi:10.1177/0146167201272002
88. Rojas, S. L., & Widiger, T. A. (2014). Convergent and discriminant validity of the Five Factor Form. Assessment, 21, 143–157. doi:10.1177/1073191113517260
89. Rosenberg, M. (1965). Society and the adolescent selfimage. Princeton, NJ: Princeton University Press.
90. Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86, 638–641.
91. Ross, L., Greene, D., & House, P. (1977). The “false consensus effect”: An egocentric bias in social perception and attribution processes. Journal of Experimental Social Psychology, 13, 279–301. doi:10.1016/0022-1031(77)90049-X
92. Ross, L., & Nisbett, R. E. (1991). The person and the situation: Perspectives of social psychology. New York, NY: McGraw-Hill.
93. Rottenstreich, Y., & Hsee, C. K. (2001). Money, kisses, and electric shocks: On the affective psychology of risk. Psychological Science, 12, 185–190. doi:10.1111/14679280.00334
94. 2 in assessing heterogeneity may mislead. BMC Medical Research Methodology, 8, Article 79. doi:10.1186/1471-2288-8-79
95. Sandvik, E., Diener, E., & Seidlitz, L. (1993). Subjective wellbeing: The convergence and stability of self-report and non-self-report measures. Journal of Personality, 61, 317–342. doi:10.1111/j.1467-6494.1993.tb00283.x
96. Savani, K., Markus, H. R., Naidu, N. V. R., Kumar, S., & Berlia, N. (2010). What counts as a choice? U.S. Americans are more likely than Indians to construe actions as choices. Psychological Science, 21, 391–398. doi:10.1177/0956797609359908
97. Schnall, S. (2014, November 18). Social media and the crowdsourcing of social psychology [Web log post]. Retrieved from https://web.archive.org/web/20170805031858/http://www.psychol.cam.ac.uk:80/cece/blog
98. Schwarz, N., & Clore, G. L. (2016). Evaluating psychological research requires more than attention to the N: A comment on Simonsohn’s (2015) “small telescopes.” Psychological Science, 27, 1407–1409. doi:10.1177/0956 797616653102
99. Schwarz, N., & Strack, F. (2014). Does merely going through the same moves make for a “direct” replication? Concepts, contexts, and operationalizations. Social Psychology, 45, 305–306. doi:10.1027/1864-9335/a000202
100. Schwarz, N., Strack, F., & Mai, H. P. (1991). Assimilation and contrast effects in part-whole question sequences: A conversational logic analysis. Public Opinion Quarterly, 55, 3–23. doi:10.1086/269239
101. Shafir, E. (1993). Choosing versus rejecting: Why some options are both better and worse than others. Memory & Cognition, 21, 546–556. doi:10.3758/BF03197186
102. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366. doi:10.1177/0956797611417632
103. Simonsohn, U. (2015). Small telescopes: Detectability and the evaluation of replication results. Psychological Science, 26, 559–569. doi:10.1177/0956797614567341
104. Srull, T. K., & Wyer, R. S. (1979). The role of category accessibility in the interpretation of information about persons: Some determinants and implications. Journal of Personality and Social Psychology, 37, 1660–1672. doi:10.1037/0022-3514.37.10.1660
105. Stroebe, W., & Strack, F. (2014). The alleged crisis and the illusion of exact replication. Perspectives on Psychological Science, 9, 59–71. doi:10.1177/1745691613514450
106. Todd, A. R., Hanko, K., Galinsky, A. D., & Mussweiler, T. (2011). When focusing on differences leads to similar perspectives. Psychological Science, 22, 134–141. doi:10.1177/0956797610392929
107. Tversky, A., & Gati, I. (1978). Studies of similarity. Cognition and Categorization, 1, 79–98.
108. Tversky, A., & Kahneman, D. (1981). The framing of decisions and the psychology of choice. Science, 211, 453–458. doi:10.1126/science.7455683
109. United Nations, Department of Economic and Social Affairs, Development Policy and Analysis Division. (2014). Country classification. Retrieved from http://www.un.org/en/development/desa/policy/wesp/wesp_ current/2014wesp_country_classification.pdf
110. United Nations Industrial Development Organization. (2015). Industrial Development Report 2016: The role of technology and innovation in inclusive and sustainable industrial development. Retrieved from https://www.unido.org/sites/default/files/2015-12/EBOOK_IDR2016_ FULLREPORT_0.pdf
111. Van Bavel, J. J., Mende-Siedlecki, P., Brady, W. J., & Reinero, D. A. (2016). Contextual sensitivity in scientific reproducibility. Proceedings of the National Academy of Sciences, USA, 113, 6454–6459. doi:10.1073/pnas.1521897113
112. Van Lange, P. A. M., Otten, W., De Bruin, E. M. N., & Joireman, J. A. (1997). Development of prosocial, individualistic, and competitive orientations: Theory and preliminary evidence. Journal of Personality and Social Psychology, 73, 733–746. doi:10.1037/0022-3514.73.4.733
113. Veenhoven, R. (2009). The international scale interval study: Improving the comparability of responses to survey questions about happiness. In V. Møller & D. Huschka (Eds.), Quality of life and the millennium challenge: Advances in quality-of-life studies, theory and research (pp. 45– 58). Dordrecht, The Netherlands: Springer.
114. Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3). doi:10.18637/jss.v036.i03
115. Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632–638.
116. Zaval, L., Keenan, E. A., Johnson, E. J., & Weber, E. U. (2014). How warm days increase belief in global warming. Nature Climate Change, 4, 143–147. doi:10.1038/nclimate2093
117. Zhong, C.-B., & Liljenquist, K. (2006). Washing away your sins: Threatened morality and physical cleansing. Science, 313, 1451–1452. doi:10.1126/science.1130726