Reconceptualizing replication as a sequence of different studies: A replication typology

Journal of Experimental Social Psychology

Volumn 66, Issue , 2016, Pages 81-92

  Hüffmeier, Joachim ^a   Mazei, Jens ^a   Schultze, Thomas ^b  

^a TU DORTMUND UNIVERSITY   (Germany)

^b UNIVERSITY OF GÖTTINGEN   (Germany)

Author keywords

Field research; Laboratory research; Real world relevance; Replication typology; Social psychology; Theory building

Indexed keywords

EDITOR; FIELD STUDY; FUNDING; HUMAN; PSYCHOLOGY; REPLICATION STUDY; SCIENTIST; THEORETICAL MODEL;

EID: 84977100590     PISSN: 00221031     EISSN: 10960465     Source Type: Journal    
DOI: 10.1016/j.jesp.2015.09.009     Document Type: Article

References (86)

1. Allport, G.H., The historical background of modern social psychology. Lindzey, G., (eds.) The handbook of social psychology, Vol. 1, 1954, Addison-Wesley, Cambridge, MA, 3–56.
2. Asendorpf, J.B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J.A., Fiedler, K. Wicherts, J.M., Recommendations for increasing replicability in psychology. European Journal of Personality 27 (2013), 108–119, 10.1002/per.1919.
3. Bakker, M., van Dijk, A., Wicherts, J.M., The rules of the game called psychological science. Perspectives on Psychological Science 7 (2012), 543–554, 10.1177/1745691612459060.
4. Bem, D., Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology 100:3 (2011), 407–425, 10.1037/a0021524.
5. Blaug, M., The methodology of economics: Or how economists explain. 1992, Cambridge University Press, New York.
6. Borenstein, M., Hedges, L.V., Higgins, J.P.T., Rothstein, H.R., Introduction to meta-analysis. 2009, Wiley, Chichester, England.
7. Brandt, M.J., IJzerman, H., Dijksterhuis, A., Farach, F.J., Geller, J., Giner-Sorolla, R. Van't Veer, A., The replication recipe: What makes for a convincing replication?. Journal of Experimental Social Psychology 50 (2014), 217–224, 10.1016/j.jesp.2013.10.005.
8. Burger, J.M., Replicating Milgram: Would people still obey today?. American Psychologist 64 (2009), 1–11, 10.1037/a0010932.
9. Cesario, J., Priming, replication, and the hardest science. Perspectives on Psychological Science 9 (2014), 40–48, 10.1177/1745691613513470.
10. Cohen, J., A power primer. Psychological Bulletin 112 (1992), 155–159, 10.1037/0033-2909.112.1.155.
11. Collins, H.M., Changing order. 1985, Sage, London.
12. Cortina, J.M., Folger, R.G., When is it acceptable to accept a null hypothesis: No way, Jose?. Organizational Research Methods 1 (1998), 334–350, 10.1177/109442819813004.
13. Dijksterhuis, A., Van Knippenberg, A., The relation between perception and behavior, or how to win a game of trivial pursuit. Journal of Personality and Social Psychology, 74, 1998, 865, 10.1037/0022-3514.74.4.865.
14. Dollard, J., Miller, N.E., Doob, L.W., Mowrer, O.H., Sears, R.R., Frustration and aggression. 1939, Yale University Press, New Haven, CT.
15. Doyen, S., Klein, O., Pichon, C.L., Cleeremans, A., Behavioral priming: It's all in the mind, but whose mind?. PloS One, 7, 2012, 10.1371/journal.pone.0029081.
16. Fabrigar, L.R., Wegener, D.T., Conceptualizing and evaluating the replication of research results. Journal of Experimental Social Psychology 66 (2016), 68–80.
17. Garner, W.R., The analysis of unanalyzed perceptions. Kubovy, M., Pomerantz, J.R., (eds.) Perceptual organization, 1981, Erlbaum, Hillsdale, NJ, 119–139.
18. Gelfand, M.J., Brett, J., Gunia, B.C., Imai, L., Huang, T.J., Hsu, B.F., Toward a culture-by-context perspective on negotiation: Negotiating teams in the United States and Taiwan. Journal of Applied Psychology, 98, 2013, 504, 10.1037/a0031908.
19. Giner-Sorolla, R., Science or art? How aesthetic standards grease the way through the publication bottleneck but undermine science. Perspectives on Psychological Science 7 (2012), 562–571, 10.1177/1745691612457576.
20. Good, C., Aronson, J., Inzlicht, M., Improving adolescents' standardized test performance: An intervention to reduce the effects of stereotype threat. Journal of Applied Developmental Psychology 24 (2003), 645–662, 10.1016/j.appdev.2003.09.00.
21. Greenwald, A.G., Consequences of prejudice against the null hypothesis. Kerens, G., Lewis, C., (eds.) A handbook for data analysis in the behavioral sciences, 1993, Erlbaum, Hillsdale, NJ, 419–448.
22. Henrich, J., Heine, S.J., Norenzayan, A., The weirdest people in the world?. Behavioral and Brain Sciences 33 (2010), 61–83, 10.1017/S0140525X0999152X.
23. Hertel, G., Kerr, N.L., Messé, L.A., Motivation gains in performance groups: Paradigmatic and theoretical developments on the Köhler effect. Journal of Personality and Social Psychology 79 (2000), 580–601, 10.1037//0022-3514.79.4.58.
24. Hüffmeier, J., Hertel, G., When the whole is more than the sum of its parts: Group motivation gains in the wild. Journal of Experimental Social Psychology 47 (2011), 455–459, 10.1016/j.jesp.2010.12.004.
25. Hüffmeier, J., Krumm, S., Kanthak, J., Hertel, G., “Don't let the group down”: Facets of instrumentality moderate the motivating effects of groups in a field experiment. European Journal of Social Psychology 42 (2012), 533–538, 10.1002/ejsp.1875.
26. Ioannidis, J.P., Why most published research findings are false. PLoS Medicine, 2, 2005, e124, 10.1371/journal.pmed.0020124.
27. Jacoby, J., Sassenberg, K., Interactions do not only tell us when, but can also tell us how: Testing process hypotheses by interaction. European Journal of Social Psychology 41 (2011), 180–190, 10.1002/ejsp.762.
28. John, L.K., Loewenstein, G., Prelec, D., Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science 23 (2012), 524–532, 10.1177/0956797611430953.
29. Klein, O., Doyen, S., Leys, C., Miller, S., Questienne, L., Cleeremans, A., Low hopes, high expectations expectancy effects and the replicability of behavioral experiments. Perspectives on Psychological Science 7 (2012), 572–584, 10.1177/1745691612463704.
30. Klein, R.A., Ratliff, K.A., Vianello, M., Adams, R.B. Jr., Bahník, Š., Bernstein, M.J. Woodzicka, J.A., Investigating variation in replicability: A “many labs” replication project. Social Psychology 45 (2014), 142–152, 10.1027/1864-9335/a0.
31. Koole, S.L., Lakens, D., Rewarding replications: A sure and simple way to improve psychological science. Perspectives on Psychological Science 7 (2012), 608–614, 10.1177/1745691612462586.
32. Kuhn, T.S., The structure of scientific revolutions. 1962, University of Chicago Press, Chicago, IL.
33. Larzelere, R.E., Cox, R.B., Swindle, T.M., Many replications do not causal inferences make: The need for critical replications to test competing explanations of nonrandomized studies. Perspectives on Psychological Science 10 (2015), 380–389, 10.1177/1745691614567904.
34. LeBel, E.P., Peters, K.R., Fearing the future of empirical psychology: Bem's (2011) evidence of psi as a case study of deficiencies in modal research practice. Review of General Psychology 15 (2011), 371–379, 10.1037/a0025172.
35. Locke, E.A., Theory building, replication, and behavioral priming: Where do we need to go from here?. Perspectives on Psychological Science 10 (2015), 408–414, 10.1177/1745691614567231.
36. Lykken, D.T., Statistical significance in psychological research. Psychological Bulletin 70 (1968), 151–159, 10.1037/h0026141.
37. Makel, M.C., Plucker, J.A., Facts are more important than novelty: Replication in the education sciences. Educational Researcher, 2014 Advance online publication. doi:0013189X14545513.
38. Makel, M.C., Plucker, J.A., Hegarty, B., Replications in psychology research: How often do they really occur?. Perspectives on Psychological Science 7 (2012), 537–542, 10.1177/1745691612460688.
39. Maner, J.K., Into the wild: Field research can increase both replicability and real-world impact. Journal of Experimental Social Psychology 66 (2016), 100–106.
40. Maxwell, S.E., The persistence of underpowered studies in psychological research: Causes, consequences, and remedies. Psychological Methods 9 (2004), 147–163, 10.1037/1082-989X.9.2.147.
41. Meyer, A., Frederick, S., Burnham, T.C., Guevara Pinto, J.D., Boyer, T.W., Ball, L.J. Schuldt, J.P., Disfluent fonts don't help people solve math problems. Journal of Experimental Psychology: General 144 (2015), e16–e30, 10.1037/xge0000049.
42. Milgram, S., Obedience to authority: An experimental view. 1974, Harper & Row, New York.
43. Mischel, W., Becoming a cumulative science. Observer, 22, 2009, 1.
44. Mummendey, A., Scientific misconduct in social psychology—Towards a currency reform in science. European Bulletin of Social Psychology 24 (2012), 4–7.
45. Murayama, K., Pekrun, R., Fiedler, K., Research practices that can prevent an inflation of false-positive rates. Personality and Social Psychology Review 18 (2014), 107–118, 10.1177/1088868313496330.
46. Neuliep, J.W., Crandall, R., Editorial bias against replication research. Journal of Social Behavior & Personality 5 (1990), 85–90.
47. Neuliep, J.W., Crandall, R., Reviewer bias against replication research. Journal of Social Behavior & Personality 8 (1993), 21–29.
48. Nosek, B.A., Lakens, D.E., Call for proposals: Special issue of social psychology on “Replications of important results in social psychology”. Social Psychology 44 (2013), 59–60, 10.1027/1864-9335/a000143.
49. Nosek, B.A., Simonsohn, U., Moore, D.A., Nelson, L.D., Simmons, J.P., Sallans, A., LeBel, E.P., Standard Reviewer Statement for Disclosure of Sample, Conditions, Measures, and Exclusions, 2014, February 4) Retrieved from osf.io/hadz3.
50. Nosek, B.A., Spies, J.R., Motyl, M., Scientific utopia II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science 7 (2012), 615–631, 10.1177/1745691612459058.
51. Open Science Collaboration. An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science 7 (2012), 657–660, 10.1177/1745691612462588.
52. Orne, M.T., On the social psychology of the psychology experiment: With particular reference to demand characteristics and their implications. American Psychologist 17 (1962), 776–783, 10.1037/h0043424.
53. Pashler, H., Harris, C.R., Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science 7 (2012), 531–536, 10.1177/1745691612463401.
54. Popper, K., The logic of scientific discovery. 1992, Routledge, New York, NY (Original work published 1934).
55. Proffit, D.R., Bertenthal, B.I., Converging operations revisited: Assessing what infants perceive using discrimination measures. Perception & Psychophysics 47 (1990), 1–11, 10.3758/BF03208159.
56. Quiñones-Vidal, E., Loźpez-García, J.J., Peñaranda-Ortega, M., Tortosa-Gil, F., The nature of social and personality psychology as reflected in JPSP, 1965–2000. Journal of Personality and Social Psychology 86 (2004), 435–452, 10.1037/0022-3514.86.3.435.
57. Radder, H., In and about the world philosophical studies of science and technology. 1996, SUNY Press, Albany, NY.
58. Richard, F.D., Bond, C.F., Stokes-Zoota, J.J., One hundred years of social psychology quantitatively described. Review of General Psychology 7 (2003), 331–363, 10.1037/1089-2680.7.4.331.
59. Ritchie, S.J., Wiseman, R., French, C.C., Failing the future: Three unsuccessful attempts to replicate Bem's ‘retroactive facilitation of recall’ effect. PloS One, 7, 2012, e33423, 10.1371/journal.pone.0033423.
60. Ritchie, S.J., Wiseman, R., French, C.C., Replication, replication, replication. Psychologist 25 (2012), 346–348.
61. Roberts, M.C., Ilardi, S.S., Research methodology and clinical psychology: An overview. Roberts, M.C., Ilardi, S.S., (eds.) Handbook of research methods in clinical psychology, 2003, Blackwell Publishing, Oxford, England, 3–12, 10.1002/9780470756980.ch1.
62. Rosenthal, R., Experimenter effects in behavioral research. Rosenthal, R., Rosnow, R.L., (eds.) Artifacts in behavioral research, 2nd ed., 2009, Oxford University Press, Oxford, England, 287–666 (Original work published 1975).
63. Rozin, P., What kind of empirical research should we publish, fund, and reward? A different perspective. Perspectives on Psychological Science 4 (2009), 435–439, 10.1111/j.1745-6924.2009.01151.x.
64. Schimmack, U., The ironic effect of significant results on the credibility of multiple-study articles. Psychological Methods 17 (2012), 551–566, 10.1037/a0029487.
65. Schmidt, S., Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology 13 (2009), 90–100, 10.1037/a0015108.
66. Sigall, H., Mills, J., Measures of independent variables and mediators are useful in social psychology experiments: But are they necessary?. Personality and Social Psychology Review 2 (1998), 218–226, 10.1207/s15327957pspr0203_5.
67. Simmons, J.P., Nelson, L.D., Simonsohn, U., False-positive psychology undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science 22 (2011), 1359–1366, 10.1177/0956797611417632.
68. Simons, D.J., The value of direct replication. Perspectives on Psychological Science 9 (2014), 76–80, 10.1177/1745691613514755.
69. Simonsohn, U., Small telescopes: Detectability and the evaluation of replication results. Psychological Science, 2015 (Advance online publication. doi:10.1177/0956797614567341).
70. Spellman, B.A., Introduction to the special section on advancing science. Perspectives on Psychological Science 8 (2013), 412–413, 10.1177/1745691611432124.
71. Stanley, D.J., Spence, J.R., Expectations for replications: Are yours realistic?. Perspectives on Psychological Science 9 (2014), 305–318, 10.1177/1745691614528518.
72. Steele, C.M., Aronson, J., Stereotype threat and the intellectual test performance of African Americans. Journal of Personality and Social Psychology 69 (1995), 797–811, 10.1037/0022-3514.69.5.797.
73. Steiner, I.D., Group process and productivity. 1972, Academic Press, New York, NY.
74. Stone, J., Lynch, C.I., Sjomeling, M., Darley, J.M., Stereotype threat effects on black and white athletic performance. Journal of Personality and Social Psychology 77:6 (1999), 1213–1227, 10.1037/0022-3514.77.6.1213.
75. Strack, F., The wow and how of research in social psychology. Causes and consequences. European Bulletin of Social Psychology 24 (2012), 4–8.
76. Stroebe, W., Strack, F., The alleged crisis and the illusion of exact replication. Perspectives on Psychological Science 9 (2014), 59–71, 10.1177/1745691613514450.
77. The GRADE working group, Grading quality of evidence and strength of recommendations. British Medical Journal 328 (2004), 1490–1494, 10.1136/bmj.328.7454.1490.
78. The Royal Netherlands Academy of Arts and Sciences, Standard evaluation protocol 2009–2015. Protocol for research assessments in The Netherlands., 2009 (Retrieved from www.knaw.nl/sep).
79. The Swedish Council on Health Technology Assessment, Catheter ablation for atrial fibrillation. Sbu alert report no. 2010–06. 2010 Retrieved from http://www.sbu.se/201006e.
80. Tsang, E.W., Kwan, K.M., Replication and theory development in organizational science: A critical realist perspective. Academy of Management Review 24 (1999), 759–780, 10.2307/259353.
81. Tversky, A., Kahneman, D., Belief in the law of small numbers. Psychological Bulletin 76 (1971), 105–110, 10.1037/h0031322.
82. Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H.L.J., Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011). Journal of Personality and Social Psychology 100 (2011), 426–432, 10.1037/a0022790.
83. Wittchen, M., Schlereth, D., Hertel, G., Social indispensability in spite of temporal and spatial separation: Motivation gains in a sequential task during anonymous cooperation on the internet. International Journal of Internet Science 2 (2007), 12–27.
84. Witte, E.H., Brandt, V., Social psychological research: The comparison of four journals. 2011 Retrieved from http://psydok.sulb.uni-saarland.de/volltexte/2011/2743/.
85. Yong, E., In the wake of high-profile controversies, psychologists are facing up to problems with replication. Nature 485 (2012), 298–300.
86. Zwaan, R.A., Zeelenberg, R., Replication attempts of important results in the study of cognition. Frontiers in Cognition, 2013 (Retrieved from http://www.frontiersin. org/Cognition/researchtopics/Replication_Attempts_of_Import/1461).