Inflationary Incidence of False-Positive Results in Psychological Research;Inflation von falsch-positiven Befunden in der psychologischen Forschung: Mögliche Ursachen und Gegenmaßnahmen

Psychologische Rundschau

Volumn 67, Issue 3, 2016, Pages 163-174

  Ulrich, Rolf ^a   Erdfelder, Edgar ^b   Deutsch, Roland ^b   Strauß, Bernhard ^b   Brüggemann, Anne ^b   Hannover, Bettina ^b   Tuschen Caffier, Brunna ^b   Kirschbaum, Clemens ^b   Blickle, Gerhard ^b   Möller, Jens ^b   Rief, Winfried ^b  

^a UNIVERSITY OF TÜBINGEN   (Germany)

^b NONE

Author keywords

Clinical Studies; DFG Research Funding; Direct And Conceptual Replications; Ethics Of Science; False Positives; Importance Of Negative Results; Journal Policy; Psychology Review Board Of The German Research Foundation (DFG); Questionable Research Practices; Replicability; Replication Crisis

Indexed keywords

EID: 84978767415     PISSN: 00333042     EISSN: None     Source Type: Journal    
DOI: 10.1026/0033-3042/a000296     Document Type: Article

References (59)

1. Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denißen, J. J. A., Fiedler, K. et al. (2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27, 108-119.
2. Bakker, M., Dijk, A. van & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543-554.
3. Begley, C. G. & Ellis, L. M. (2012). Raise standards for preclinical cancer research. Nature, 483, 531-533.
4. Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R. et al. (2014). The Replication Recipe: What makes for a convincing replication? Journal of Experimental Social Psychology, 50, 217-224.
5. Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J. et al. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews. Neuroscience, 14, 365-376.
6. Chan, A. W., Song, F., Vickers, A., Jefferson, T., Dickersin, K., Gøtzsche, P. C. et al. (2014). Increasing value and reducing waste: Addreßing inacceßible research. The Lancet, 383, 257-266.
7. Cohen, J. (1992). A Power Primer. Psychological Bulletin, 112, 155-159.
8. Colquhoun, D. & London, C. (2014). An investigation of the false discovery rate and the misinterpretation of p-values. Royal Society Open Science, 1, 1-15. doi: 10.1098/rsos.140216
9. Egger, M., Davey Smith, G., Schneider, M. & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315, 629-634.
10. Fanelli, D. (2009). How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS ONE, 4 (5), e5738.
11. Fanelli, D. (2010a). Do preßures to publish increase scientists' bias? An empirical support from US states data. PLoS ONE, 5 (4), e10271.
12. Fanelli, D. (2010b). "Positive" results increase down the hierarchy of the sciences. PLoS ONE, 5 (4), e10068.
13. Faul, F., Erdfelder, E., Lang, A.-G. & Buchner, A. (2007). G∗Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175-191.
14. Fiedler, K. (2011). Voodoo correlations are everywhere-not only in neuroscience. Perspectives on Psychological Science, 6, 163-171.
15. Fiedler, K., Kutzner, F. & Krueger, J. (2012). The long way from α-error control to validity proper problems with a short-sighted false-positive debate. Perspectives on Psychological Science, 7, 661-669.
16. Fiedler, K. & Schwarz, N. (2015). Questionable research practices revisited. Social Psychology and Personality Sciences, 1-8.
17. Finkel, E. J., Eastwick, P. W. & Reis, H. T. (2015). Best research practices in psychology : Illustrating epistemological and pragmatic considerations with the case of relationship science. Journal of Personality and Social Psychology, 108, 275-297.
18. Francis, G. (2012a). Publication bias and the failure of replication in experimental psychology. Psychonomic Bulletin & Review, 19, 975-991.
19. Francis, G. (2012b). Too good to be true: Publication bias in two prominent studies from experimental psychology. Psychonomic Bulletin & Review, 19, 151-156.
20. Francis, G., Tanzman, J. & Matthews, W. J. (2014). Exceß succeß for psychology articles in the journal Science. PLoS ONE, 9, e114255.
21. Glasziou, P., Altman, D. G., Boßuyt, P., Boutron, I., Clarke, M., Julious, S. et al. (2014). Reducing waste from incomplete or unusable reports of biomedical research. The Lancet, 383, 267-276.
22. Greve, W., Bröder, A. & Erdfelder, E. (2013). Result-blind peer reviews and editorial decisions: A mißing pillar of scientific culture. European Psychologist, 18, 286-294.
23. Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2, e124.
24. Ioannidis, J. P. A., Greenland, S., Hlatky, M. a., Khoury, M. J., Macleod, M. R., Moher, D. et al. (2014). Increasing value and reducing waste in research design, conduct, and analysis. The Lancet, 383, 166-175.
25. Ioannidis, J. P. A. & Trikalinos, T. A. (2007). An exploratory test for an exceß of significant findings. Clinical Trials (London, England), 4, 245-253.
26. Iverson, G. J., Wagenmakers, E.-J. & Lee, M. D. (2010). A modelaveraging approach to replication: the case of p rep. Psychological Methods, 15, 172-181.
27. Jena, S. (2015, 3./4. März). Öffentlicher Zugang zu allen Daten klinischer Studien in Deutschland: Wunsch und Wirklichkeit. Beitrag vorgestellt auf dem DFG Workshop "Qualitätskriterien patientenorientierter Forschung", Bonn.
28. John, L. K., Loewenstein, G. & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524-32.
29. Johnson, V. E. (2013). Revised standards for statistical evidence. Proceedings of the National Academy of Sciences of the United States of America, 110, 19313-19317.
30. Kerr, N. L. (1998). HARKing: Hypotheszing after the results are known. Personality and Social Psychology Review, 2, 196-217.
31. Killeen, P. R. (2005). An alternative to null-hypothesis significance tests. Psychological Science, 16, 345-353.
32. Lakens, D. & Evers, E. R. K. (2014). Sailing from the seas of chaos into the corridor of stability: Practical recommendations to increase the informational value of studies. Perspectives on Psychological Science, 9, 278-292.
33. Lane, D. & Dunlap, W. (1978). Estimating effect size: Bias resulting from the significance criterion in editorial decisions. British Journal of Mathematical and Statistical Psychology, 31 (2), 107-112.
34. Lau, J., Ioannidis, J. P. A., Terrin, N., Schmid, C. H. & Olkin, I. (2006). Evidence based medicine: The case of the misleading funnel plot. British Medical Journal, 333, 597-600.
35. Leek, J. T. & Peng, R. D. (2015). P values are just the tip of the iceberg. Nature, 520, 612.
36. Macleod, M. R., Michie, S., Roberts, I., Dirnagl, U., Chalmers, I., Ioannidis, J. P. A. et al. (2014). Biomedical research: Increasing value, reducing waste. The Lancet, 383, 101-104.
37. Makel, M. C., Plucker, J. A. & Hegarty, B. (2012). Replications in psychology research: How often do they really occur? Perspectives on Psychological Science, 7, 537-542.
38. Maraun, M. & Gabriel, S. (2010). Killeen's (2005) Prep coefficient: Logical and mathematical problems. Psychological Methods, 15, 182-191.
39. Margraf, J. (2015). Zur Lage der Psychologie. Psychologische Rundschau, 66, 1-30.
40. Miller, J. (2009). What is the probability of replicating a statistically significant effect? Psychonomic Bulletin & Review, 16, 617-640. doi:10.3758/PBR.16.4.617
41. Miller, J. & Schwarz, W. (2011). Aggregate and individual replication probability within an explicit model of the research proceß. Psychological Methods, 16, 337-360.
42. Moshagen, M. & Erdfelder, E. (2015). A new strategy for testing structural equation models. Structural Equation Modeling: A Multidisciplinary Journal, 5511 (April), 1-7.
43. Neuliep, J. W. & Crandall, R. (1993). Reviewer bias against replication research. Journal of Social Behavior and Personality, 8, 21-29.
44. Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J. et al. (2015). Promoting an open research culture. Science, 348 (6242), 1422-1425.
45. Pashler, H. & Harris, C. R. (2012). Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science, 7, 531-536.
46. Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86, 638-641.
47. Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D. & Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16, 225-237.
48. Schimmack, U. (2012). The ironic effect of significant results on the credibility of multiple-study articles. Psychological Methods, 17, 551-566.
49. Simmons, J. P., Nelson, L. D. & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-1366.
50. Simonsohn, U. (2013). It really just does not follow, comments on Francis (2013). Journal of Mathematical Psychology, 57 (5), 174-176.
51. Simonsohn, U., Nelson, L. D. & Simmons, J. P. (2014). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143, 534-547.
52. Smart, R. G. (1964). The imporantce of negative results in psychological research. The Canadian Psychologist, 5, 225-232.
53. Sterling, T. D. (1959). Publication decisions and their poßible effects on inferences drawn from tests of significance-or vice versa. Journal of the American Statistical Aßociation, 54, 30-34.
54. Sterling, T. D., Rosenbaum, W. L. & Weinkam, J. J. (1995). Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. The American Statistician, 49, 108-112.
55. Trafimow, D. & Marks, M. (2015). Editorial. Basic and Applied Social Psychology, 37, 1-2.
56. Ulrich, R. & Miller, J. (2015). p-hacking by post-hoc selection with multiple opportunities: Detectability by skewneß test?-Comment on Simonsohn, Nelson, and Simmons (2014). Journal of Experimental Psychology: General, 144, 1137-1145.
57. Vul, E., Harris, C., Winkielman, P. & Pashler, H. (2009). Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition. Perspectives on Psychological Science, 4, 274-290.
58. Wagenmakers, E.-J., Wetzels, R., Borsboom, D. & Maas, H. L. J. Van der (2011). Why psychologists must change the way they analyze their data: the case of psi: comment on Bem (2011). Journal of Personality and Social Psychology, 100, 426-432.
59. Wißing, F. (2015, 3./4. März). Klinische Forschung in Deutschland: Problemfelder aus der Sicht der DFG. Beitrag vorgestellt auf dem DFG Workshop "Qualitätskriterien patientenorientierter Forschung", Bonn.