Conducting Meta-Analyses Based on p Values: Reservations and Recommendations for Applying p-Uniform and p-Curve

Perspectives on Psychological Science

Volumn 11, Issue 5, 2016, Pages 713-729

  van Aert, Robbie C M ^a   Wicherts, Jelte M ^a   van Assen, Marcel A L M ^a,b  

^a TILBURG UNIVERSITY   (Netherlands)

^b UTRECHT UNIVERSITY   (Netherlands)

Author keywords

heterogeneity; meta analysis; p curve; p hacking; p uniform

Indexed keywords

HUMAN; INTERNET; META ANALYSIS (TOPIC); PROCEDURES; PSYCHOLOGY; PUBLISHING; SOFTWARE; STATISTICAL ANALYSIS; STATISTICAL MODEL;

DATA INTERPRETATION, STATISTICAL; HUMANS; INTERNET; META-ANALYSIS AS TOPIC; MODELS, STATISTICAL; PSYCHOLOGY; PUBLICATION BIAS; SOFTWARE;

EID: 84989225639     PISSN: 17456916     EISSN: 17456924     Source Type: Journal    
DOI: 10.1177/1745691616650874     Document Type: Article

References (56)

1. Ackerman J. M., Nocera C. C., Bargh J. A., (2010, June 27). Incidental haptic sensations influence social judgments and decisions. Science, 328, 1712–1715. doi:10.1126/science.1189993
2. American Psychological Association. (2010). Publication manual (6th ed.). Washington, DC: Author.
3. Augusteijn H. E. M., (2015). The effect of publication bias on the Q test and assessment of heterogeneity (Unpublished master’s thesis). Tilburg University, Tilburg, the Netherlands.
4. Bakker M., Van Dijk A., Wicherts J. M., (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543–554. doi:10.1177/1745691612459060
5. Bakker M., Wicherts J. M., (2014a). Outlier removal and the relation with reporting errors and quality of psychological research. PLoS ONE, 9(7), e103360. doi:10.1371/journal.pone.0103360
6. Bakker M., Wicherts J. M., (2014b). Outlier removal, sum scores, and the inflation of the Type I error rate in independent samples t tests: The power of alternatives and recommendations. Psychological Methods, 19, 409–427. doi:10.1037/met0000014
7. Bland M., (2013). Do baseline p values follow a uniform distribution in randomised trials? PLoS ONE, 8(10). doi: 10.1371/journal.pone.0076010
8. Borenstein M., Hedges L. V., Higgins J. P. T., Rothstein H. R., (2009). Introduction to meta-analysis. Chichester, England: Wiley.
9. Bruns S. B., Ioannidis J. P., (2016). P-curve and p-hacking in observational research. PLoS ONE, 11(2), e0149144. doi:10.1371/journal.pone.0149144
10. Casella G., Berger R. L., (2002). Statistical inference. Belmont, CA: Duxbury.
11. Chandler J. J., Reinhard D., Schwarz N., (2012). To judge a book by its weight you need to know its content: Knowledge moderates the use of embodied cues. Journal of Experimental Social Psychology, 48, 948–953. doi:10.1016/j.jesp.2012.03.003
12. Fanelli D., (2012). Negative results are disappearing from most disciplines and countries. Scientometrics, 90, 891–904. doi:10.1007/s11192-011-0494-7
13. Ferguson C. J., Brannick M. T., (2012). Publication bias in psychological science: Prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychological Methods, 17, 120–128. doi:10.1037/a0024445
14. Field A. P., Gillett R., (2010). How to do a meta-analysis. British Journal of Mathematical and Statistical Psychology, 63, 665–694. doi:10.1348/000711010X502733
15. Fisher R. A., (1925). Statistical methods for research workers (1st ed.). London, England: Oliver & Boyd.
16. Francis G., (2013). Replication, statistical consistency, and publication bias. Journal of Mathematical Psychology, 57, 153–169. doi:10.1016/j.jmp.2013.02.003
17. Gentle J. E., (2004). Random number generation and Monte Carlo methods (2nd ed.). New York, NY: Springer.
18. Greenhouse J. B., Iyengar S., (2009). Sensitivity analysis and diagnostics. In Cooper H., Hedges L. V., Valentine J. C., (Eds.), The handbook of research synthesis and meta-analysis (pp. 417–433). New York, NY: Russell Sage Foundation.
19. Guan M., Vandekerckhove J., (2015). A Bayesian approach to mitigation of publication bias. Psychonomic Bulletin & Review. Advance online publication. doi:10.3758/s13423-015-0868-6
20. Häfner M., (2013). When body and mind are talking: Interoception moderates embodied cognition. Experimental Psychology, 60, 255–259. doi:10.1027/1618-3169/a000194
21. Hartgerink C. H. J., van Aert R. C. M., Nuijten M. B., Wicherts J. M., van Assen M. A. L. M. (2015). Distributions of p values smaller than.05 in psychology: What is going on? PeerJ, 4, e1935. doi:10.7717/peerj.1935
22. Hedges L. V., (1984). Estimation of effect size under nonrandom sampling: The effects of censoring studies yielding statistically insignificant mean differences. Journal of Educational Statistics, 9, 61–85.
23. Hedges L. V., Vevea J. L., (2005). Selection method approaches. In Rothstein H. R., Sutton A. J., Borenstein M., (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 145–174). Chichester, England: Wiley.
24. Higgins J. P. T., Thompson S. G., (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21, 1539–1558. doi:10.1002/sim.1186
25. Hung H. M., O’Neill R. T., Bauer P., Köhne K., (1997). The behavior of the p value when the alternative hypothesis is true. Biometrics, 53, 11–22.
26. IJzerman H., Padiotis N., Koole S. L., (2013). Replicability of social-cognitive priming: The case of weight as an embodiment of importance. Retrieved from http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2255726
27. Ioannidis J. P. A., (2008). Why most discovered true associations are inflated. Epidemiology, 19, 640–648. doi:10.1097/EDE.0b013e31818131e7
28. Ioannidis J. P. A., Trikalinos T. A., Zintzaras E., (2006). Extreme between-study homogeneity in meta-analyses could offer useful insights. Journal of Clinical Epidemiology, 59, 1023–1032. doi:10.1016/j.jclinepi.2006.02.013
29. John L. K., Loewenstein G., Prelec D., (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524–532. doi:10.1177/0956797611430953
30. Jostmann N. B., Lakens D., Schubert T. W., (2009). Weight as an embodiment of importance. Psychological Science, 20, 1169–1174.
31. Kaspar K., (2013). A weighty matter: Heaviness influences the evaluation of disease severity, drug effectiveness, and side effects. PLoS ONE, 8(11), e78307. doi:10.1371/journal.pone.0078307
32. Kaspar K., Krull J., (2013). Incidental haptic stimulation in the context of flirt behavior. Journal of Nonverbal Behavior, 37, 165–173. doi:10.1007/s10919-013-0154-0
33. Klein R. A., Ratliff K. A., Brooks B., Vianello M., Galliani E. M., Adams R. B., Jr.. Hasselman F., (2014). Investigating variation in replicability: A “many labs” replication project. Social Psychology, 45(3), 142–152. doi: http://dx.doi.org/10.1027/1864-9335/a000178
34. Kouchaki M., Gino F., Jami A., (2014). The burden of guilt: Heavy backpacks, light snacks, and enhanced morality. Journal of Experimental Psychology: General, 143, 414–424. doi:10.1037/a0031769
35. Lakens D., (2014). What p-hacking really looks like: A comment on Masicampo and LaLande (2012). Quarterly Journal of Experimental Psychology, 68, 829–832. doi:10.1080/17470218.2014.982664
36. Lane D. M., Dunlap W. P., (1978). Estimating effect size: Bias resulting from the significance criterion in editorial decisions. British Journal of Mathematical and Statistical Psychology, 31, 107–112.
37. Maglio S. J., Trope Y., (2012). Disembodiment: Abstract construal attenuates the influence of contextual bodily state in judgment. Journal of Experimental Psychology: General, 141, 211–216. doi:10.1037/a0024520
38. Moreno S. G., Sutton A. J., Abrams K. R., Cooper N. J., Turner E. H., Palmer T. M., Ades A. E., (2009). Novel methods to deal with publication biases: Secondary analysis of antidepressant trials in the FDA trial registry database and related journal publications. BMJ, 339, Article b2981. doi:10.1136/bmj.b2981
39. Moreno S. G., Sutton A. J., Ades A. E., Stanley T. D., Abrams K. R., Peters J. L., Cooper N. J., (2009). Assessment of regression-based methods to adjust for publication bias through a comprehensive simulation study. BMC Medical Research Methodology, 9(2). doi:10.1186/1471-2288-9-2
40. Nuijten M. B., Hartgerink C. H. J., van Assen M. A. L. M., Epskamp S., Wicherts J. M., (2015). The prevalence of statistical reporting errors in psychology (1985–2013). Behavior Research Methods. 1–22. doi:10.3758/s13428-015-0664-2
41. Rabelo A. L. A., Keller V. N., Pilati R., Wicherts J. M., (2015). No effect of weight on judgments of importance in the moral domain and evidence of publication bias from a meta-analysis. PLoS ONE, 10(8), e0134808. doi:10.1371/journal.pone.0134808
42. Raudenbush S. W., (2009). Analyzing effect sizes: Random-effects models. In Cooper H., Hedges L. V., Valentine J. C., (Eds.), The handbook of research synthesis and meta-analysis (pp. 295–315). New York, NY: Russell Sage Foundation.
43. R Core Team. (2015). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.R-project.org/
44. Schmidt F. L., Oh I. S., Hayes T. L., (2009). Fixed- versus random-effects models in meta-analysis: Model properties and an empirical comparison of differences in results. British Journal of Mathematical and Statistical Psychology, 62, 97–128. doi:10.1348/000711007X255327
45. Simmons J. P., Nelson L. D., Simonsohn U., (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366. doi:10.1177/0956797611417632
46. Simonsohn U., (2015, February 9). “The” effect size does not exist [Web log post]. Retrieved from http://datacolada.org/2015/02/09/33-the-effect-size-does-not-exist/
47. Simonsohn U., Nelson L. D., Simmons J. P., (2014a). P-curve and effect size: Correcting for publication bias using only significant results. Perspectives on Psychological Science, 9, 666–681. doi:10.1177/1745691614553988
48. Simonsohn U., Nelson L. D., Simmons J. P., (2014b). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143, 534–547. doi:10.1037/a0033242
49. Stanley T. D., Doucouliagos H., (2014). Meta-regression approximations to reduce publication selection bias. Research Synthesis Methods, 5, 60–78.
50. Sterling T. D., Rosenbaum W. L., Weinkam J. J., (1995). Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. The American Statistician, 49, 108–112. doi:10.2307/2684823
51. Sterne J. A. C., Egger M., (2005). Regression methods to detect publication and other bias in meta-analysis. In Rothstein H. R., Sutton A. J., Borenstein M., (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 99–110). Chichester, England: Wiley.
52. Ulrich R., Miller J., (2015). P-hacking by post hoc selection with multiple opportunities: Detectability by skewness test? Comment on Simonsohn, Nelson, and Simmons (2014). Journal of Experimental Psychology: General, 144, 1137–1145. doi:10.1037/xge0000086
53. van Aert R. C. M., Maassen E., Wicherts J. M., van Assen M. A. L. M. (2016). Estimating effect size in meta-analysis in the presence of publication bias and questionable research practices. Manuscript in preparation.
54. van Assen M. A. L. M., van Aert R. C. M., Wicherts J. M. (2015). Meta-analysis using effect size distributions of only statistically significant studies. Psychological Methods, 20, 293–309. doi:10.1037/met0000025
55. Witkovsky V., (2013). A note on computing extreme tail probabilities of the noncentral t distribution with large noncentrality parameter. Acta Universitatis Palackianae Olomucensis. Facultas Rerum Naturalium. Mathematica, 52, 131–143. Retrieved from http://dml.cz//handle/10338.dmlcz/143545
56. Zhang M., Li X., (2012). From physical weight to psychological significance: The contribution of semantic activations. Journal of Consumer Research, 38, 1063–1075. doi:10.1086/661768