Possible Solution to Publication Bias Through Bayesian Statistics, Including Proper Null Hypothesis Testing

Communication Methods and Measures

Volumn 9, Issue 4, 2015, Pages 280-302

  Konijn, Elly A ^a   van de Schoot, Rens ^b   Winter, Sonja D ^b   Ferguson, Christopher J ^c  

^a VU UNIVERSITY AMSTERDAM   (Netherlands)

^b UTRECHT UNIVERSITY   (Netherlands)

^c STETSON UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84948689435     PISSN: 19312458     EISSN: 19312466     Source Type: Journal    
DOI: 10.1080/19312458.2015.1096332     Document Type: Article

References (68)

1. C.Anderson,, & K.Dill, (2000). Video games and aggressive thoughts, feelings and behavior in the laboratory and in life. Journal of Personality and Social Psychology, 78, 772–790. doi:10.1037/0022-3514.78.4.772
2. J.B.Asendorpf,, M.Conner,, F.De Fruyt,, J.De Houwer,, J.J.A.Denissen,, K.Fiedler, … J.M.Wicherts, (2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27, 108–119. doi:10.1002/per.1919
3. M.Bakker,, A.van Dijk,, & J.M.Wicherts, (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543–554. doi:10.1177/1745691612459060
4. M.Bakker,, & J.M.Wicherts, (2011). The (mis)reporting of statistical results in psychology journals. Behavior Research Methods, 43(3), 666–678. doi:10.3758/s13428-011-0089-5
5. J.D.Brown,, & P.S.Bobkowski, (2011). Older and newer media: Patterns of use and effects on adolescents’ health and Well‐Being. Journal of Research on Adolescence, 21(1), 95–113. doi:10.1111/j.1532-7795.2010.00717.x
6. J.Cohen, (1994). The earth is round (p <.05). American Psychologist, 49, 997–1003.
7. D.Fanelli, (2010). ‘Positive’ results increase down the hierarchy of the sciences. PLoS ONE, 5(4), e10068. doi:10.1371/journal.pone.0010068
8. F.C.Fang,, R.G.Steen,, & A.Casadevall, (2012). Misconduct accounts for the majority of retracted scientific publications. Proceedings of the National Academy of Sciences, doi:10.1073/pnas.1212247109
9. C.J.Ferguson,, & M.T.Brannick, (2012). Publication bias in psychological science: Prevalence, methods for identifying and controlling and implications for the use of meta-analyses. Psychological Methods, 17(1), 120–128. doi:10.1037/a0024445
10. C.J.Ferguson,, A.M.Cruz,, & S.M.Rueda, (2008). Gender, video game playing habits and visual memory tasks. Sex Roles, 58, 279–286. doi:10.1007/s11199-007-9332-z
11. C.J.Ferguson,, & M.Heene, (2012). A vast graveyard of undead theories: Publication bias and psychological science’s aversion to the null. Perspectives on Psychological Science, 7(6), 555–561. doi:10.1177/1745691612459059
12. A.Fritz,, T.Scherndl,, & A.Kühberger, (2012, April). Correlation between effect size and sample size in psychological research: Sources, consequences, and remedies. Presented at the 10th Conference of the Austrian Psychological Society, Graz, Austria.
13. C.R.Gallistel, (2009). The importance of proving the null. Psychological Review, 116(2), 439–453. doi:10.1037/a0015251
14. M.Gollwitzer,, & A.Melzer, (2012). Macbeth and the joystick: Evidence for moral cleansing after playing a violent video game. Journal of Experimental Social Psychology, 48, 1356–1360. doi:10.1016/j.jesp.2012.07.001
15. Hare, E. (2014, May 23). D&AD Winners (Sweetie by Lemz and Terre des Hommes won in the White Pencil category of D&AD award competition for social adverts) [web log post]. Retrieved from http://www.contagious.com/blogs/news-and-views/14215281-d-ad-winnners
16. J.S.Harrison,, G.Banks,, J.M.Pollack,, E.H.O’Boyle,, & J.Short, (2014). Publication bias in strategic management research. Journal of Management. doi:10.1177/0149206314535438
17. H.Hoijtink, (2011). Informative hypotheses: Theory and practice for behavioral and social scientists. Boca Raton, FL: Chapman and Hall/CRC.
18. J.F.Hoorn,, E.A.Konijn,, D.M.Germans,, S.Burger,, & A.Munneke, (2015). The in-between machine: The unique value proposition of a robot or why we are modelling the wrong things. In S.Loiseau, J.Filipe, B.Duval, & J.van den Herik (Eds.), Proceedings of the 7th International Conference on Agents and Artificial Intelligence (ICAART) Jan. 10-12, 2015. Lisbon, Portugal (pp. 464–469). Lisbon, PT: SCITEPRESS.
19. J.Ioannidis, (2012). Scientific inbreeding and same-team replication: Type D personality as an example. Journal of Psychosomatic Research, 73, 408–410. doi:10.1016/j.jpsychores.2012.09.014
20. H.Jeffreys, (1961). Theory of probability (3rd ed.). Oxford, UK: Oxford University Press.
21. L.K.John,, G.Loewenstein,, & D.Prelec, (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532. doi:10.1177/0956797611430953
22. V.E.Johnson, (2013). Revised standards for statistical evidence. Proceedings of the National Academy of Sciences, 110(48), 19313–19317. doi:10.1073/pnas.1313476110
23. D.J.Johnstone, (1990). Interpreting statistical insignificance: A Bayesian perspective. Psychological Reports, 66(1), 115–121. doi:10.2466/pr0.1990.66.1.115
24. Jovanovic, D. (2013, November 5). ‘Sweetie’ sting lures thousands of alleged pedophiles. ABC News. Retrieved from http://abcnews.go.com/WNT/video/meet-sweetie-virtual-girl-identified-1000-pedophiles-world-20797023
25. D.Kaplan,, & S.Depaoli, (2013). Bayesian statistical methods. In T.D.Little (Ed.), Oxford handbook of quantitative methods (pp. 407–437). Oxford, UK: Oxford University Press.
26. R.E.Kass,, & A.E.Raftery, (1995). Bayes factors. Journal of the American Statistical Association, 90, 773–795. doi:10.1080/01621459.1995.10476572
27. S.Kepes,, & M.A.McDaniel, (2013). How trustworthy is the scientific literature in industrial and organizational psychology? Industrial and Organizational Psychology: Perspectives on Science and Practice, 6(3), 252–268. doi:10.1111/iops.12045
28. P.R.Killeen, (2005). An alternative to null-hypothesis significance tests. Psychological Science, 16, 345–353. doi:10.1111/j.0956-7976.2005.01538.x
29. I.Klugkist,, O.Laudy,, & H.Hoijtink, (2005). Inequality constrained analysis of variance: A Bayesian approach. Psychological Methods, 10, 477–493. doi:10.1037/1082-989X.10.4.477
30. E.A.Konijn, (2015). Moved by media and emotion. Amsterdam, The Netherlands: Breckfield Midway.
31. J.I.Krueger, (2001). Null hypothesis significance testing: On the survival of a flawed method. American Psychologist, 56, 16–26. doi:10.1037/0003-066X.56.1.16
32. S.-Y.Lee,, & X.-Y.Song, (2004). Evaluation of the Bayesian and maximum likelihood approaches in analyzing structural equation models with small sample sizes. Multivariate Behavioral Research, 39(4), 653–686. doi:10.1207/s15327906mbr3904_4
33. T.R.Levine,, R.Weber,, C.R.Hullett,, H.S.Park,, & L.L.M.Massi Lindsey, (2008). A critical assessment of null hypothesis significance testing in quantitative communication research. Human Communication Research, 34, 171–187. doi:10.1111/j.1468-2958.2008.00317.x
34. T.R.Levine,, R.Weber,, H.S.Park,, & C.R.Hullett, (2008). A communication researchers’ guide to null hypothesis significance testing and alternatives. Human Communication Research, 34, 188–209. doi:10.1111/j.1468-2958.2008.00318.x
35. J.Love,, R.Selker,, M.Marsman,, T.Jamil,, A.J.Verhagen,, A.Ly, … E.-J.Wagenmakers, (2015). JASP (Version 0.6.6) [Computer software]. Retrieved from https://jasp-stats.org/
36. J.Meyers,, & K.Meyers, (1995). Rey complex figure test under four different administration procedures. The Clinical Neuropsychologist, 9, 63–67. doi:10.1080/13854049508402059
37. R.D.Morey,, & J.N.Rouder, (2012) Bayes Factor: An R package for computing Bayes factor for a variety of psychological research designs. Web-based: available on the Comprehensive R Archive Network.
38. J.Mulder,, H.Hoijtink,, & C.De Leeuw, (2012). BIEMS: A Fortran 90 program for calculating Bayes factor for inequality and equality constrained models. Journal of Statistical Software, 46, 1–39.
39. J.Mulder,, H.Hoijtink,, & I.Klugkist, (2010). Equality and inequality constrained multivariate linear models: Objective model selection using constrained posterior priors. Journal of Statistical Planning and Inference, 140, 887–906. doi:10.1016/j.jspi.2009.09.022
40. J.Mulder,, I.Klugkist,, R.Van de Schoot,, W.Meeus,, M.Selfhout,, & H.Hoijtink, (2009). Bayesian model selection of informative hypotheses for repeated measurements. Journal of Mathematical Psychology, 53, 530–546. doi:10.1016/j.jmp.2009.09.003
41. R.S.Nickerson, (2000). Null hypothesis significance testing: A review of an old and continuing controversy. Psychological Methods, 5(2), 241–301. doi:10.1037/1082-989X.5.2.241
42. B.A.Nosek,, J.R.Spies,, & M.Motyl, (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7(6), 615–631. doi:10.1177/1745691612459058
43. G.Oppy,, & D.Dowe, (2011). The Turing test. In E.N.Zalta (Ed.), The Stanford encyclopedia of philosophy. Stanford, CA: The Metaphysics Research Lab, Stanford University. ISSN 1095-5054.
44. L.Pericchi, (2015, March). The chronicle of a death foretold rejection rule. Invited Contribution, ISBA Bulletin, 22(1), 7.
45. M.Pontier,, J.F.Hoorn,, E.A.Konijn,, & R.vd Schoot, (under review). Dating a Robot is Like Dating a Man.
46. J.W.Romeijn,, & R.Van de Schoot, (2008). A philosopher’s view on Bayesian evaluation of informative hypotheses. In H.Hoijtink, I.Klugkist, & P.Boelen (Eds.), Bayesian evaluation of informative hypotheses (pp. 329–358). New York, NY: Springer.
47. R.Rosenthal, (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638–641.
48. R.Rosenthal,, R.L.Rosnow,, & D.B.Rubin, (2000). Contrasts and effect sizes in behavioral research: A correlational approach. Cambridge, UK: Cambridge University Press.
49. A.M.Rosenthal-Von Der Pütten,, F.P.Schulte,, S.C.Eimler,, S.Sobieraj,, L.Hoffmann,, S.Maderwald, … N.C.Krämer, (2014). Investigations on empathy towards humans and robots using fMRI. Computers in Human Behavior, 33, 201–212. doi:10.1016/j.chb.2014.01.004
50. R.L.Rosnow,, & R.Rosenthal, (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276–1284. doi:10.1037/0003-066X.44.10.1276
51. J.N.Rouder,, P.L.Speckman,, D.Sun,, R.D.Morey,, & G.Iverson, (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16(2), 225–237. doi:10.3758/PBR.16.2.225
52. J.Schooler, (2011). Unpublished results hide the decline effect. Nature, 470(7335), 437. doi:10.1038/470437a
53. T.Sellke,, M.J.Bayarri,, & J.O.Berger, (2001). Calibration of p values for testing precise null hypotheses. The American Statistician, 55, 62–71. doi:10.1198/000313001300339950
54. M.J.Silvapulle,, & P.K.Sen, (2004). Constrained statistical inference: Order, inequality, and shape constraints. London, UK: Wiley.
55. J.P.Simmons,, L.D.Nelson,, & U.Simonsohn, (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366. doi:10.1177/0956797611417632
56. Simonsohn, U. (2014, January 2), Posterior-Hacking: Selective reporting invalidates Bayesian results also. University of Pennsylvania, The Wharton School. Retrieved from http://papers.ssrn.com/sol3/Papers.cfm?abstract_id=2374040
57. A.J.Sutton,, & K.R.Abrams, (2001). Bayesian methods in meta-analysis and evidence synthesis. Statistical Methods in Medical Research, 10, 277–303. doi:10.1177/096228020101000404
58. A.M.Turing, (1950). Computing machinery and intelligence. Mind, LIX(236), 433–460. doi:10.1093/mind/LIX.236.433
59. R.van de Schoot,, & S.Depaoli, (2014). Bayesian analyses: Where to start and what to report. European Health Psychologist, 16(2), 75–84.
60. R.van de Schoot,, H.Hoijtink,, & M.Deković, (2010). Testing inequality constrained hypotheses in SEM Models. Structural Equation Modeling, 17, 443–463. doi:10.1080/10705511.2010.489010
61. R.van de Schoot,, H.Hoijtink,, J.Mulder,, M.A.G.Van Aken,, B.Orobio de Castro,, W.Meeus,, & J.-W.Romeijn, (2011). Evaluating expectations about negative emotional states of aggressive boys using Bayesian model selection. Developmental Psychology, 47, 203–212. doi:10.1037/a0020957
62. R.van de Schoot,, D.Kaplan,, J.Denissen,, J.B.Asendorpf,, F.J.Neyer,, & M.A.G.Van Aken, (2014). A gentle introduction to bayesian analysis: Applications to developmental research. Child Development, 85(3), 842–860. doi:10.1111/cdev.2014.85.issue-3
63. M.A.M.van Kemenade,, E.A.Konijn,, & J.F.Hoorn, (2015). Robots humanize care: Moral concerns versus witnessed benefits for the elderly. In C.Verdier, M.Bienkiewicz, A.Fred, H.Gamboa, & D.Elias (Eds.), Health Informatics (pp. 648–653). Lisbon, Portugal: SCITEPRESS.
64. E.Wagenmakers, (2007). A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review, 14, 779–804. doi:10.3758/BF03194105
65. E.J.Wagenmakers,, J.Verhagen,, A.Ly,, M.Bakker,, M.D.Lee,, D.Matzke,, … & R.D.Morey, (2014). A power fallacy. Behavior Research Methods, 1–5.
66. E.-J.Wagenmakers,, A.J.Verhagen,, A.Ly,, D.Matzke,, H.Steingroever,, J.N.Rouder,, & R.D.Morey, (in press). The need for Bayesian hypothesis testing in psychological science. In S.O.Lilienfeld & I.Waldman (Eds.), Psychological science under scrutiny: Recent challenges and proposed solutions. Hoboken, NJ: John Wiley and Sons.
67. R.Wetzels,, J.G.Raaijmakers,, E.Jakab,, & E.J.Wagenmakers, (2009). How to quantify support for and against the null hypothesis: A flexible WinBUGS implementation of a default Bayesian t test. Psychonomic Bulletin & Review, 16(4), 752–760. doi:10.3758/PBR.16.4.752
68. J.M.Wicherts,, D.Borsboom,, J.Kats,, & D.Molenaar, (2006). The poor availability of psychological research data for reanalysis. American Psychologist, 61, 726–728. doi:10.1037/0003-066X.61.7.726