P-curve: A key to the file-drawer

Journal of Experimental Psychology: General

Volumn 143, Issue 2, 2014, Pages 534-547

  Simonsohn, Uri ^a   Nelson, Leif D ^b   Simmons, Joseph P ^a  

^a UNIVERSITY OF PENNSYLVANIA   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

False positive psychology; Hypothesis testing; P hacking; Publication bias; Selective reporting

Indexed keywords

HUMAN; PSYCHOLOGY; PUBLISHING; REPRODUCIBILITY; STATISTICAL ANALYSIS; STATISTICAL MODEL; STATISTICS;

DATA INTERPRETATION, STATISTICAL; HUMANS; MODELS, STATISTICAL; PSYCHOLOGY, EXPERIMENTAL; PUBLICATION BIAS; REPRODUCIBILITY OF RESULTS; STATISTICS AS TOPIC;

EID: 84896876198     PISSN: 00963445     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0033242     Document Type: Article

References (51)

1. Bakker, M.,&Wicherts, J. M. (2011). The (mis)reporting of statistical results in psychology journals. Behavior Research Methods, 43, 666-678.
2. Bross, I. D. J. (1971). Critical levels, statistical language, and scientific inference. Foundations of Statistical Inference, 500-513.
3. Campbell, D. T.,&Stanley, J. C. (1966). Experimental and quasiexperimental designs for research. Chicago, IL: Rand-McNally.
4. Card, D.,&Krueger, A. B. (1995). Time-series minimum-wage studies: A meta-analysis. The American Economic Review, 85, 238-243.
5. Clarkson, J. J., Hirt, E. R., Jia, L.,&Alexander, M. B. (2010). When perception is more than reality: The effects of perceived versus actual resource depletion on self-regulatory behavior. Journal of Personality and Social Psychology, 98, 29-46.
6. Cohen, J. (1962). The statistical power of abnormal-social psychological research: A review. Journal of Abnormal and Social Psychology, 65, 145-153.
7. Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum.
8. Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155-159.
9. Cohen, J. (1994). The earth is round (p <. 05). American Psychologist, 49, 997-1003.
10. Cole, L. C. (1957). Biological clock in the unicorn. Science, 125, 874-876.
11. Cumming, G. (2008). Replication and p intervals: P values predict the future only vaguely, but confidence intervals do much better. Perspectives on Psychological Science, 3, 286-300.
12. Duval, S.,&Tweedie, R. (2000). Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in metaanalysis. Biometrics, 56, 455-463.
13. Egger, M., Smith, G. D., Schneider, M.,&Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315, 629-634.
14. Fanelli, D. (2012). Negative results are disappearing from most disciplines and countries. Scientometrics, 90, 891-904.
15. Gadbury, G. L.,&Allison, D. B. (2012). Inappropriate fiddling with statistical analyses to obtain a desirable p-value: Tests to detect its presence in published literature. PLoS ONE, 7, e46363.
16. Gelman, A.,&Stern, H. (2006). The difference between "significant" and "not significant" is not itself statistically significant. The American Statistician, 60, 328-331.
17. Gerber, A. S.,&Malhotra, N. (2008a). Do statistical reporting standards affect what is published? Publication bias in two leading political science journals. Quarterly Journal of Political Science, 3, 313-326.
18. Gerber, A. S.,&Malhotra, N. (2008b). Publication bias in empirical sociological research: Do arbitrary significance levels distort published results? Sociological Methods&Research, 37, 3-30.
19. Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82, 1-20.
20. Hodges, J.,&Lehmann, E. L. (1954). Testing the approximate validity of statistical hypotheses. Journal of the Royal Statistical Society: Series B. Methodological, 16, 261-268.
21. Hung, H. M. J., O'Neill, R. T., Bauer, P.,&Kohne, K. (1997). The behavior of the p-value when the alternative hypothesis is true. Biometrics, 11-22.
22. Ioannidis, J. P. A. (2005). Why most published research findings are false [Editorial material]. PLoS Medicine, 2, 696-701.
23. Ioannidis, J. P. A. (2008). Why most discovered true associations are inflated. Epidemiology, 19, 640-648.
24. Ioannidis, J. P. A.,&Trikalinos, T. A. (2007). An exploratory test for an excess of significant findings. Clinical Trials, 4, 245-253.
25. Jacoby, L. L., Kelley, C. M., Brown, J.,&Jasechko, J. (1989). Becoming famous overnight: Limits on the ability to avoid unconscious influences of the past. Journal of Personality and Social Psychology, 56, 326-338.
26. Kerr, N. L. (1998). Harking: Hypothesizing after the results are known. Personality and Social Psychology Review, 2, 196-217.
27. Kunda, Z. (1990). The case for motivated reasoning. Psychological Bulletin, 108, 480-498.
28. Lau, J., Ioannidis, J., Terrin, N., Schmid, C. H.,&Olkin, I. (2006). The case of the misleading funnel plot. British Medical Journal, 333, 597-600.
29. Lehman, E. L. (1986). Testing statistical hypotheses (2nd ed.). New York, NY: Wiley.
30. Masicampo, E. J.,&Lalande, D. R. (2012). A peculiar prevalence of p values just below .05. The Quarterly Journal of Experimental Psychology, 65, 2271-2279.
31. Nelson, L. D., Simonsohn, U.,&Simmons, J. P. (2013). P-curve estimates publication bias free effect sizes. Manuscript in preparation.
32. Nieuwenhuis, S., Forstmann, B. U.,&Wagenmakers, E. J. (2011). Erroneous analyses of interactions in neuroscience: A problem of significance. Nature Neuroscience, 14, 1105-1107.
33. Orwin, R. G. (1983). A fail-safe N for effect size in meta-analysis. Journal of Educational Statistics, 157-159.
34. Pashler, H.,&Harris, C. R. (2012). Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science, 7, 531-536.
35. Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R.,&Rushton, L. (2007). Performance of the trim and fill method in the presence of publication bias and between-study heterogeneity. Statistics in Medicine, 26, 4544-4562.
36. Phillips, C. V. (2004). Publication bias in situ. BMC Medical Research Methodology, 4, 20.
37. Ridley, J., Kolm, N., Freckelton, R.,&Gage, M. (2007). An unexpected influence of widely used significance thresholds on the distribution of reported p-values. Journal of Evolutionary Biology, 20, 1082-1089.
38. Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86, 638-641.
39. Serlin, R.,&Lapsley, D. (1985). Rationality in psychological research: The Good-enough principle. American Psychologist, 40, 73-83.
40. Simmons, J. P., Nelson, L. D.,&Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-1366.
41. Simmons, J. P., Nelson, L. D.,&Simonsohn, U. (2012). A 21 word solution. Dialogue: The Official Newsletter of the Society for Personality and Social Psychology, 26, 4-7.
42. Simonsohn, U. (2012). It does not follow: Evaluating the one-off publication bias critiques by Francis (2012a, 2012b, 2012c, 2012d, 2012e, in press). Perspectives on Psychological Science, 7, 597-599.
43. Simonsohn, U. (2013). It really just does not follow: Comments on Francis (2013). Journal of Mathematical Psychology. Advance online publication.
44. Sterling, T. D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance- or vice versa. Journal of the American Statistical Association, 54, 30-34.
45. Sterling, T. D., Rosenbaum, W.,&Weinkam, J. (1995). Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. American Statistician, 49, 108-112.
46. Tang, J. L.,&Liu, J. L. Y. (2000). Misleading funnel plot for detection of bias in meta-analysis. Journal of Clinical Epidemiology, 53, 477-484.
47. Topolinski, S.,&Strack, F. (2010). False fame prevented: Avoiding fluency effects without judgmental correction. Journal of Personality and Social Psychology, 98, 721-733.
48. Van Boven, L., Kane, J., McGraw, A. P.,&Dale, J. (2010). Feeling close: Emotional intensity reduces perceived psychological distance. Journal of Personality and Social Psychology, 98, 872-885.
49. Wallis, W. A. (1942). Compounding probabilities from independent significance tests. Econometrica, 44, 265-274.
50. White, H. (2000). A reality check for data snooping. Econometrica, 68, 1097-1126.
51. Wohl, M. J.,&Branscombe, N. R. (2005). Forgiveness and collective guilt assignment to historical perpetrator groups depend on level of social category inclusiveness. Journal of Personality and Social Psychology, 88, 288-303.