From discovery to justification: Outline of an ideal research program in empirical psychology

Frontiers in Psychology

Volumn 8, Issue OCT, 2017, Pages

  Witte, Erich H ^a   Zenker, Frank ^b,c,d,e  

^a UNIVERSITY OF HAMBURG   (Germany)

^b LUND UNIVERSITY   (Sweden)

^c INSTITUTE OF CHEMISTRY   (Slovakia)

^d UNIVERSITY OF KONSTANZ   (Germany)

^e SUN YAT SEN UNIVERSITY   (China)

Author keywords

Confirmation; Knowledge accumulation; Meta analysis; Psi hypothesis; Replicability crisis; Research programs; Significance test; Test power

Indexed keywords

EID: 85032586300     PISSN: None     EISSN: 16641078     Source Type: Journal    
DOI: 10.3389/fpsyg.2017.01847     Document Type: Article

References (85)

1. Azzalini, A. (1996). Statistical Inference. Based on the Likelihood. London: Chapman & Hall
2. Bakan, D. (1966). The test of significance in psychological research. Psychol. Bull. 66, 423-437. doi: 10.1037/h0020412
3. Baker, M. (2015). First results from psychology's largest reproducibility test. Nat. News. doi: 10.1038/nature.2015.17433
4. Bakker, M., van Dijk, A., and Wicherts, J. M. (2012). The rules of the game called psychological science. Perspect. Psychol. Sci. 7, 543-554. doi: 10.1177/1745691612459060
5. Bem, D., Tressoldi, P., Rabeyron, T. H., and Duggan, M. (2016). Feeling the future: a meta-analysis of 90 experiments on the anticipation of random future events [version 2; referees: 2 approved]. F1000 Res. 4, 1188. doi: 10.12688/f1000research.7177.2
6. Bem, D. J. (2011). Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect. J. Pers. Soc. Psychol. 100, 407-425. doi: 10.1037/a0021524
7. Bem, D. J., Utts, J., and Johnson, W. O. (2011). Reply. Must psychologists change the way they analyze their data? J. Pers. Soc. Psychol. 101, 716-719. doi: 10.1037/a0024777
8. Benjamin, D. J., Berger, J., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J., Berk, R., et al. (2017). Redefine Statistical Significance. Available at: psyarxiv.com/mky9j
9. Cafri, G., Kromrey, J. D., and Brannick, M. T. (2010). A meta-meta-analysis: empirical review of statistical power, type I error rates, effect sizes, and model selection of meta-analyses published in psychology. Multivariate Behav. Res. 45, 239-270. doi: 10.1080/00273171003680187
10. Chan, M.-L. E., and Arvey, R. D. (2012). Meta-analysis and the development of knowledge. Perspect. Psychol. Sci. 7, 79-92. doi: 10.1177/1745691611429355
11. Cohen, J. (1962). The statistical power analysis for the behavioral sciences: a review. J. Abnorm. Soc. Psychol. 65, 145-153
12. Cohen, J. (1977). Statistical Power Analysis for the Behavioral Sciences (Rev. ed.). London: Academic Press
13. Cowles, M. (1989). Statistics in Psychology. An Historical Perspective. Hillsdale: Erlbaum
14. Ellemers, N. (2013). Connecting the dots: mobilizing theory to reveal the big picture in social psychology (and why we should do this). Eur. J. Soc. Psychol. 43, 1-8. doi: 10.1002/ejsp.1932
15. Eriksson, K., and Simpson, B. (2013). Editorial decisions may perpetuate belief in invalid research findings. PLOS ONE 8:e73364. doi: 10.1371/journal.pone.0073364
16. Etz, A., and Vandekerckhove, J. (2016). A Bayesian perspective on the reproducibility project: psychology. PLOS ONE 11:0149794. doi: 10.1371/journal.pone.0149794
17. Fanelli, D., and Glänzel, W. (2013). Bibliometric evidence for a hierarchy of the sciences. PLOS ONE 8:e66938. doi: 10.1371/journal.pone.0066938
18. Faul, F., Erdfelder, E., Lang, A.-G., and Buchner, A. (2007). G*Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav. Res. Methods 39, 175-191. doi: 10.3758/BF03193146
19. Ferguson, C. J., and Heene, M. (2012). A vast graveyard of undead theories: publication bias and psychological science's aversion to the null. Perspect. Psychol. Sci. 7, 555-561. doi: 10.1177/1745691612459059
20. Fisher, R. A. (1956). Statistical Methods and Scientific Inference. New York, NY: Hafner
21. Francis, G. (2012). The psychology of replication and replication in psychology. Perspect. Psychol. Sci. 7, 585-594. doi: 10.1177/1745691612459520
22. Fuchs, H. M., Jenny, M., and Fiedler, S. (2012). Psychologists are open to change, yet wary of rules. Perspect. Psychol. Sci. 7, 639-642. doi: 10.1177/1745691612459521
23. Galak, J., LeBoeuf, R. A., Nelson, L. D., and Simmons, J. P. (2012). Correcting the past: failures to replicate Psi. J. Pers. Soc. Psychol. 103, 933-948. doi: 10.1037/a0029709
24. Gelman, A. (2011). Induction and deduction in Bayesian data analysis. Ration. Mark. Morals 2, 67-78
25. Gigerenzer, G. (2010). Personal reflections on theory and psychology. Theory Psychol. 20, 733-743. doi: 10.1177/0959354310378184
26. Gilbert, D. T., King, G., Pettigrew, S., and Wilson, T. D. (2016). Comment on "Estimating the reproducibility of psychological science. Science 351, 1037. doi: 10.1126/science.aad7243
27. Hacking, I. (1978). Imre Lakatos's philosophy of science. Br. J. Philos. Sci. 30, 381-410. doi: 10.1093/bjps/30.4.381
28. Harlow, L. L., Mulaik, S. A., and Steiger, J. H. (eds) (1997). What If There Were No Significance Tests? Mahwah: Erlbaum
29. Holcombe, A. O. (2016). Introduction to a registered replication report on ego depletion. Perspect. Psychol. Sci. 11, 545. doi: 10.1177/1745691616652871
30. Hoyningen-Huene, P. (2006). "Context of discovery vs. context of justification and Thomas Kuhn," in Revisiting Discovery and Justification, eds J. Schickore and F. Steinle (Dordrecht: Springer), 119-131
31. Ioannidis, J. P. A. (2012). Why science is not necessarily self-correcting. Perspect. Psychol. Sci. 7, 645-654. doi: 10.1177/1745691612464056
32. Ioannidis, J. P. A. (2014). How to make more published research true. PLOS Med. 11:e1001747. doi: 10.1371/journal.pmed.1001747
33. Jeffrey, H. (1961). The Theory of Probability. Oxford: Oxford University Press
34. Kerr, N. L. (1998). HARKing: hypothesizing after the results are known. Pers. Soc. Psychol. Rev. 2, 196-217. doi: 10.1207/s15327957pspr0203_4
35. Klein, S. B. (2014). What can recent replication failures tell us about the theoretical commitments of psychology? Theory Psychol. 24, 326-338. doi: 10.1177/0959354314529616
36. Kriegeskorte, N., Simmons, W. K., Bellgowan, P. S. F., and Baker, C. I. (2009). Circular analysis in systems neuroscience: the dangers of double dipping. Nat. Neurosci. 12, 535-540. doi: 10.1038/nn.2303
37. Kuhn, T. S. (1970). The Structure of Scientific Revolutions, 2nd Edn. Chicago: University of Chicago Press
38. Lakatos, I. (1978). The Methodology of Scientific Research Programs. Cambridge: Cambridge University Press
39. Larvor, B. (1998). Lakatos: An Introduction. London: Routledge
40. Ly, A., Verhagen, J., and Wagenmakers, E.-J. (2016). Harold Jeffrey's default Bayes factor hypothesis tests: explanation, extension, and application in psychology. J. Math. Psychol. 72, 19-32. doi: 10.1016/j.jmp.2015.06.004
41. Maxwell, S. E. (2004). The persistence of underpowered studies in psychological research: causes, consequences, and remedies. Psychol. Methods 9, 147-163. doi: 10.1037/1082-989X.9.2.147
42. Mayo, D. G. (1996). Error and the Growth of Experimental Knowledge. Chicago: University of Chicago Press. doi: 10.7208/chicago/9780226511993.001.0001
43. Mayo, D. G. (2011). Statistical science and philosophy of science: where do/should they meet in 2011 (and beyond)? Ration. Mark. Morals 2, 79-102
44. Meehl, P. E. (1967). Theory testing in psychology and physics: a methodological paradox. Philos. Sci. 34, 103-115. doi: 10.1086/288135
45. Mitchell, G. (2012). Revisiting truth or triviality: the external validity of research in the psychological laboratory. Perspect. Psychol. Sci. 7, 109-117. doi: 10.1177/1745691611432343
46. Motterlini, M. (2002). Reconstructing Lakatos: a reassessment of Lakatos' epistemological project in the light of the Lakatos Archive. Stud. Hist. Philos. Sci. 33, 487-509. doi: 10.1016/S0039-3681(02)00024-9
47. Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., et al. (2015). Promoting an open research culture. Science 348, 1422-1425. doi: 10.1126/science.aab2374
48. Nosek, B. A., and Bar-Anan, Y. (2012). Scientific utopia: I. Opening scientific communication. Psychol. Inq. 23, 217-243. doi: 10.1080/1047840X.2012.692215
49. Nosek, B. A., Spies, J. R., and Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspect. Psychol. Sci. 7, 615-631. doi: 10.1177/1745691612459058
50. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science 349:acc4716. doi: 10.1126/science.aac4716
51. Pashler, H., and Wagenmakers, E.-J. (2012). Editors' introduction to the special section on replicability in psychological science: a crisis of confidence? Perspect. Psychol. Sci. 7, 528-530. doi: 10.1177/1745691612465253
52. Reichenbach, H. (1938). Experience and Prediction. Chicago: The University of Chicago Press
53. Rosnow, R., and Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. Am. Psychol. 44, 1276-1284. doi: 10.1037/0003-066X.44.10.1276
54. Rouder, J. N. (2014). Optional stopping: no problem for Bayesians. Psychon. Bull. Rev. 21, 301-308. doi: 10.3758/s13423-014-0595-4
55. Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., and Iverson, G. (2009). Baysian t-tests for accepting and rejecting the null hypothesis. Psychon. Bull. Rev. 16, 225-237. doi: 10.3758/PBR.16.2.225
56. Royall, R. (1997). Statistical Evidence. A Likelihood Paradigm. London: Chapman & Hall
57. Scargle, J. D. (2000). Publication bias: the "file-drawer" problem in scientific inference. J. Sci. Explor. 14, 91-106
58. Schickore, J., and Steinle, F. (eds) (2006). Revisiting Discovery and Justification: Historical and Philosophical Perspectives on the Contest Distinction. Dordrecht: Springer
59. Schmidt, F. L., Oh, I., and Hayes, T. L. (2009). Fixed versus random-effect models in meta-analysis: model properties and an empirical comparison of differences in results. Br. J. Math. Stat. Psychol. 62, 97-128. doi: 10.1348/000711007X255327
60. Schonemann, P. H., and Scargle, J. D. (2008). A generalized publication bias model. Chin. J. Psychol. 50, 21-29
61. Sedlmeier, P., and Gigerenzer, G. (1989). Do studies of statistical power have an effect on the power of studies? Psychol. Bull. 105, 309-316. doi: 10.1037/0033-2909.105.2.309
62. Simmons, J. P., Nelson, L. D., and Simonsohn, U. (2011). False-positive psychology. Psychol. Sci. 22, 1359-1366. doi: 10.1177/0956797611417632
63. Spellman, B. A. (2012). Introduction to the special section on research practices. Perspect. Psychol. Sci. 7, 655-656. doi: 10.1177/1745691612465075
64. Stegenga, J. (2011). Is meta-analysis the platinum standard of evidence? Stud. Hist. Philos. Biol. Sci. 42, 497-507. doi: 10.1016/j.shpsc.2011.07.003
65. Sterling, T. D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance-or vice versa. J. Am. Stat. Assoc. 54, 30-34
66. Sturm, T. H., and Mülberger, A. (2012). Crisis discussions in psychology: new historical and philosophical perspectives. Stud. Hist. Philos. Biol. Sci. 43, 425-433. doi: 10.1016/j.shpsc.2011.11.001
67. van Lange, P. A. M., Kruglanski, A. W., and Higgins, E. T. (2012). Handbook of Theories of Social Psychology, Vol. 1+2. London: Sage Publications
68. Verhagen, A. J., and Wagenmakers, E.-J. (2014). Bayesian tests to quantify the result of a replication attempt. J. Exp. Psychol. Gen. 143, 1457-1475. doi: 10.1037/a0036731
69. Wagenmakers, E.-J., Wetzels, R., Borsboom, D., and van der Maas, H. L. J. (2011). Why psychologists must change the way they analyze their data: the psi case: comment on Bem (2011). J. Pers. Soc. Psychol. 100, 426-432. doi: 10.1037/a0022790
70. Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., and Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspect. Psychol. Sci. 7, 632-638. doi: 10.1177/1745691612463078
71. Wald, A. (1947). Sequential Analysis. New York: Wiley
72. Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., and Wagenmakers, E.-J. (2011). Statistical evidence in experimental psychology: an empirical comparison using 855 t-tests. Perspect. Psychol. Sci. 6, 291-298. doi: 10.1177/1745691611406923
73. Willy, R. (1889). Die Krisis in der Psychologie. Leipzig: Reisland
74. Witte, E. H. (1980). Signifikanztest und statistische Inferenz. Analysen, Probleme, Alternativen [Significance test and statistical inference: Analyses, problems, alternatives]. Stuttgart: Enke
75. Witte, E. H. (1994). "Minority influences and innovations: the search for an integrated explanation of psychological and sociological models," in Minority Influence, eds S. Moscovici, A. Mucchi-Faina, and A. Maass (Chicago: Nelson-Hall), 67-93
76. Witte, E. H. (1996a). "Small-group research and the crisis of social psychology: an introduction," in Understanding Group Behavior, Vol. 2, eds E. H. Witte and J. H. Davis (Mahwah: Erlbaum), 1-8
77. Witte, E. H. (1996b). "The extended group situation theory (EGST): explaining the amount of change," in Understanding Group Behavior, Vol. 1, eds E. H. Witte and J. H. Davis (Mahwah: Erlbaum), 253-291
78. Witte, E. H. (2005). "Theorienentwicklung und-konstruktion in der Sozialpsychologie [Theory development and theory construction in social psychology]," in Entwicklungsperspektiven der Sozialpsychologie [Developmental Perspectives of Social Psychology], ed. E. H. Witte (Lengerich: Pabst), 172-188
79. Witte, E. H., and Davis, J. H. (eds) (1996). Understanding Group Behavior, Vol. 1 and 2. Mahwah: Erlbaum
80. Witte, E. H., and Heitkamp, I. (2006). Quantitative rekonstruktionen (retrognosen) als instrument der theorienbildung und theorienprüfung in der sozialpsychologie [Quantitative reconstructions (retrognoses) as an instrument for theory construction and theory assessment in social psychology]. Z. Sozialpsychol. 37, 205-214. doi: 10.1024/0044-3514.37.3.205
81. Witte, E. H., and Kaufman, J. (1997). The Stepwise Hybrid Statistical Inference Strategy: FOSTIS. HAFOS, 18. Available at: http://psydok.sulb.uni-saarland.de/frontdoor.php?source:opus=2286&la=de [accessed February 22, 2017]
82. Witte, E. H., and Strohmeier, C. E. (2013). Forschung in der psychologie. Ihre disziplinäre matrix im vergleich zu physik, biologie und sozialwissenschaft [Research in psychology. Its disciplinary matrix as compared to physics, biology, and the social sciences]. Psychol. Rundsch. 64, 16-24. doi: 10.1026/0033-3042/a0000145
83. Witte, E. H., and Zenker, F. (2016a). Beyond schools-reply to Marsman, Ly & Wagenmakers. Basic Appl. Soc. Psychol. 38, 313-317. doi: 10.1080/01973533.2016.1227710
84. Witte, E. H., and Zenker, F. (2016b). Reconstructing recent work on macro-social stress as a research program. Basic Appl. Soc. Psychol. 38, 301-307. doi: 10.1080/01973533.2016.1207077
85. Witte, E. H., and Zenker, F. (2017). Extending a multilab preregistered replication of the ego-depletion effect to a research program. Basic Appl. Soc. Psychol. 39, 74-80. doi: 10.1080/01973533.2016.1269286