Publication bias: Nderstanding the myths concerning threats to the advancement of science

More Statistical and Methodological Myths and Urban Legends

Volumn , Issue , 2014, Pages 36-64

  Banks, George C ^a   Kepes, Sven ^b   McDaniel, Michael A ^b  

^a LONGWOOD UNIVERSITY   (United States)

^b VIRGINIA COMMONWEALTH UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84941235615     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.4324/9780203775851-8     Document Type: Chapter

References (107)

1. American Psychological Association. (2008). Reporting standards for research in psychology: Why do we need them? What might they be? American Psychologist, 63, 839–851.
2. American Psychological Association. (2010). Publication manual of the American Psychological Association (6th ed.). Washington, DC: American Psychological Association.
3. Ariely, D. (2012). The (honest) truth about dishonesty: How we lie to everyone—especially ourselves. New York, NY: HarperCollins.
4. Banks, G. C., Batchelor, J. H., & McDaniel, M. A. (2010). Smarter people are (a bit) more symmetrical: A meta-analysis of the relationship between intelligence and fl uctuating asymmetry. Intelligence, 38, 393–401.
5. Banks, G. C., Batchelor, J. H., Seers, A., O’Boyle, E. H., Pollack, J. M., & Gower, K. (2014). What does team-member exchange bring to the party? A meta-analytic review of team and leader social exchange. Journal of Organizational Behavior, 3, 273–295.
6. Banks, G. C., Kepes, S., & Banks, K. P. (2012). Publication bias: The antagonist of metaanalytic reviews and effective policy making. Educational Evaluation and Policy Analysis, 34, 259–277.
7. Banks, G. C., Kepes, S., & McDaniel, M. A. (2012). Publication bias: A call for improved meta-analytic practice in the organizational sciences. International Journal of Selection and Assessment, 20, 182–196.
8. Banks, G. C., & McDaniel, M. A. (2011). The kryptonite of evidence-based I-O psychology. Industrial and Organizational Psychology: Perspectives on Science and Practice, 4, 40–44.
9. Banks, G. C., & O’Boyle, E. H. (2013). Why we need I-O psychology to fi x I-O psychology. Industrial and Organizational Psychology: Perspectives on Science and Practice, 6, 291–294.
10. Becker, B. J. (1994). Combining significance levels. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 215–230). New York, NY: Russell Sage Foundation.
11. Becker, B. J. (2005). The failsafe N or file-drawer number. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta analysis: Prevention, assessment, and adjustments (pp. 111–126). West Sussex, UK: Wiley.
12. Begg, C. B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 50, 1088–1101.
13. Bennett, A. A., & Miao, C. (2013). How do we know truth? Extensions and examples from similar academic fields. Industrial and Organizational Psychology: Perspectives on Science and Practice, 6, 276–278.
14. Berlin, J. A., & Ghersi, D. (2005). Preventing publication bias: Registries and prospective meta-analysis. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta analysis: Prevention, assessment, and adjustments (pp. 35–48). West Sussex, UK: Wiley.
15. Biemann, T. (2013). What if we were Texas sharpshooters? Predictor reporting bias in regression analysis. Organizational Research Methods, 16, 335–363.
16. Blackwell, S. C., Thompson, L., & Refuerzo, J. (2009). Full publication of clinical trials presented at a national maternal-fetal medicine meeting: Is there a publication bias? American Journal of Perinatology, 26, 679–682.
17. Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2009). Introduction to metaanalysis. West Sussex, UK: Wiley.
18. Bosco, F. A., Field, J. G., & Pierce, C. A. (2012, August). Accomodational plasticity in organizational science: Impact of hypothesis framing on effect size. Paper presented at the annual meeting of the Academy of Management, Boston, MA.
19. Briner, R. B., & Rousseau, D. M. (2011). Evidence-based I-O psychology: Not there yet. Industrial and Organizational Psychology: Perspectives on Science and Practice, 4, 3–22.
20. Chalmers, I., & Dickersin, K. (2013). Biased under-reporting of research reflects biased under-submission more than biased editorial rejection. F1000Research, 2, 1–6.
21. Chan, A.-W. (2008). Bias, spin, and misreporting: Time for full access to trial protocols and results. PLoS Medicine, 5, e230.
22. Chan, A.-W., & Altman, D. G. (2005). Identifying outcome reporting bias in randomised trials on PubMed: Review of publications and survey of authors. British Medical Journal, 330, 753–756.
23. Chiaburu, D. S., Peng, A. C., Oh, I.-S., Banks, G. C., & Lomeli, L. C. (2013). Employee organizational cynicism antecedents and outcomes: A meta-analysis. Journal of Vocational Behavior, 83, 181–197.
24. Cooper, H., DeNeve, K., & Charlton, K. (1997). Finding the missing science: The fate of studies submitted for review by a human subjects committee. Psychological Methods, 2, 447–452.
25. Curfman, G. D., Morrissey, S., & Drazen, J. M. (2006). Expression of concern reaffirmed. New England Journal of Medicine, 354, 1193.
26. Dalton, D. R., Aguinis, H., Dalton, C. M., Bosco, F. A., & Pierce, C. A. (2012). Revisiting the file drawer problem in a meta-analysis: An assessment of published and nonpublished correlation matrices. Personnel Psychology, 65, 221–249.
27. De Angelis, C., Drazen, J. M., Frizelle, F. A., Haung, C., Hoey, J., Horton, R., Kotzin, S. VanDer Weyden, M. B. (2004). Clinical trial registration: A statement from the International Committee of Medical Journal Editors. New England Journal of Medicine, 351, 1250–1251.
28. Dickersin, K. (2005). Publication bias: Recognizing the problem, understandings its origins and scope, and preventing harm. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in metaanalysis: Prevention, assessment, and adjustments (pp. 11–34). West Sussex, UK: Wiley.
29. Duval, S. J. (2005). The “trim and fill” method. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta analysis: Prevention, assessment, and adjustments (pp. 127–144). West Sussex, UK: Wiley.
30. Duval, S. J., & Tweedie, R. L. (2000a). A nonparametric “trim and fill” method of accounting for publication bias in meta-analysis. Journal of the American Statistical Association, 95, 89–98.
31. Duval, S. J., & Tweedie, R. L. (2000b). Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56, 455–463.
32. Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Williamson, P. R. (2008). Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PloS One, 3, e3081.
33. Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991). Publication bias in clinical research. The Lancet, 337, 867–872.
34. Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315, 629–634.
35. Emerson, G. B., Warme, W. J., Wolf, F. M., Heckman, J., Brand, R. A., & Leopold, S. S. (2010). Testing for the presence of positive-outcome bias in peer review: A randomized controlled trial. Archives of Internal Medicine, 170, 1934–1939.
36. Epstein, W. M. (1990). Confi rmational response bias among social work journals. Science, Technology, and Human Values, 15, 9–37.
37. Etter, J. F., & Stapleton, J. (2009). Citations to trials of nicotine replacement therapy were biased toward positive results and high-impact-factor journals. Journal of Clinical Epidemiology, 62, 831–837.
38. Evans, S. (1996). Statistician’s comment (to Misleading meta-analysis: “Fail safe N” is a useful mathematical measure of the stability of results by R. Persaud). British Medical Journal, 312, 125.
39. Fanelli, D. (2012). Negative results are disappearing from most disciplines and countries. Scientometrics, 90, 891–904.
40. Ferguson, C. J., & Brannick, M. T. (2012). Publication bias in psychological science: Prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychological Methods, 17, 120–128.
41. Gerber, A., & Malhotra, N. (2008a). Do statistical reporting standards affect what is published? Publication bias in two leading political science journals. Quarterly Journal of Political Science, 3, 313–326.
42. Gerber, A. S., & Malhotra, N. (2008b). Publication bias in empirical sociological research: Do arbitrary significance levels distort published results? Sociological Methods & Research, 37, 3–30.
43. Greenhouse, J. B., & Iyengar, S. (2009). Sensitivity analysis and diagnostics. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 417–433). New York, NY: Russell Sage Foundation.
44. Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82, 1–20.
45. Hahn, S., Williamson, P. R., Hutton, J. L., Garner, P., & Flynn, E. V. (2000). Assessing the potential for bias in meta-analysis due to selective reporting of subgroup analyses within studies. Statistics In Medicine, 19, 3325–3336.
46. Harrison, J. S., Banks, G. C., Pollack, J. M., O’Boyle Jr., E. H., & Short, J. C. (in press). Publication bias in strategic management research. Journal of Management.
47. Hartshorne, J., & Schachner, A. (2012). Tracking replicability as a method of post-publication open evaluation. Frontiers in Computational Neuroscience, 6, 8.
48. Hedges, L. V., & Vevea, J. L. (2005). Selection methods approaches. In H. R. Rothstein, A. Sutton, & M. Borenstein (Eds.), Publication bias in meta analysis: Prevention, assessment, and adjustments (pp. 145–174). West Sussex, UK: Wiley.
49. Higgins, J. P., & Green, S. (Eds.). (2011). Cochrane handbook for systematic reviews of interventions; Version 5.1.0 [updated March 2011]: The Cochrane Collaboration. Available from www.cochrane-handbook.org
50. Hopewell, S., Clarke, M., & Mallett, S. (2005). Grey literature and systematic reviews. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta analysis: Prevention, assessment, and adjustments (pp. 48–72). West Sussex, UK: Wiley.
51. Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.
52. Ioannidis, J. (2008). Interpretation of tests of heterogeneity and bias in meta-analysis. Journal of Evaluation in Clinical Practice, 14, 951–957.
53. Ioannidis, J., & Doucouliagos, H. (2013). What’s to know about the credibility of empirical economics? Journal of Economic Surveys, 27, 997–1004.
54. Jick, T. D. (1979). Mixing qualitative and quantitative methods: Triangulation in action. Administrative Science Quarterly, 24, 602–611.
55. Kepes, S., Banks, G. C., McDaniel, M. A., & Whetzel, D. L. (2012). Publication bias in the organizational sciences. Organizational Research Methods, 15, 624–662.
56. Kepes, S., Banks, G. C., & Oh, I. S. (2014). Avoiding bias in publication bias research: The value of “null” findings. Journal of Business and Psychology, 29, 183–203.
57. Kepes, S., Bennett, A., & McDaniel, M. A. (2014). Evidence-based management and the trustworthiness of our cumulative scientific knowledge: Implications for research, teaching, and practice. Academy of Management Learning & Education, 13, 446–466.
58. Kepes, S., & McDaniel, M. A. (2013). How trustworthy is the scientific literature in I-O psychology? Industrial and Organizational Psychology: Perspectives on Science and Practice, 6, 252–268.
59. Kepes, S., McDaniel, M. A., Banks, G. C., Hurtz, G. M., & Donovan, J. J. (2011, April). Big five validity and publication bias: Conscientiousness’ validity is lower than assumed. Paper presented at the annual meeting of the Society for Industrial and Organizational Psychology, Chicago, IL.
60. Kepes, S., McDaniel, M. A., Brannick, M. T., & Banks, G. C. (2013). Meta-analytic reviews in the organizational sciences: Two meta-analytic schools on the way to MARS. Journal of Business and Psychology, 28, 123–143.
61. Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2, 196–217.
62. Kicinski, M. (2013). Publication bias in recent meta-analyses. PLoS One, 8, e81823.
63. Kirkham, J. J., Riley, R. D., & Williams, P. R. (2011). A multivariate meta-analysis approach for reducing the impact of outcome reporting bias in systematic reviews. Statistics in Medicine, 31, 2179–2195.
64. Leavitt, K., Mitchell, T. R., & Peterson, J. (2010). Theory pruning: Strategies to reduce our dense theoretical landscape. Organizational Research Methods, 13, 644–667.
65. Lipsey, M. W., & Wilson, D. B. (1993). The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis. American Psychologist, 48, 1181–1209.
66. Mahoney, M. J. (1977). Publication prejudices: An experimental study of confi rmatory bias in the peer review system. Cognitive Therapy & Research, 1, 161–175.
67. Mazar, N., Amir, O., & Ariely, D. (2008). The dishonesty of honest people: A theory of self-concept maintenance. Journal of Marketing Research, 45, 633–644.
68. McDaniel, M. A. (2009, April). Cumulative meta-analysis as a publication bias method. Paper presented at the annual meeting of the Society for Industrial and Organizational Psychology, New Orleans, LA.
69. McDaniel, M. A., & Kepes, S. (in press). An evaluation of Spearman’s hypothesis by manipulating g-saturation. International Journal of Selection and Assessment.
70. McDaniel, M. A., McKay, P., & Rothstein, H. (2006, April). Publication bias and racial effects on job performance: The elephant in the room. Paper presented at the annual meeting of the Society for Industrial and Organizational Psychology, Dallas, TX.
71. McDaniel, M. A., Rothstein, H. R., & Whetzel, D. L. (2006). Publication bias: A case study of four test vendors. Personnel Psychology, 59, 927–953.
72. McGauran, N., Wieseler, B., Kreis, J., Schuler, Y. B., Kolsch, H., & Kaiser, T. (2010). Review reporting bias in medical research—a narrative. Trials, 11, 1–15.
73. McKay, P. F., & McDaniel, M. A. (2006). A reexamination of black–white mean differences in work performance: More data, more moderators. Journal of Applied Psychology, 91, 538–554.
74. Murtaugh, P. A. (2002). Journal quality, effect size, and publication bias in meta-analysis. Ecology, 83, 1162–1166.
75. O’Boyle, E. H., Banks, G. C., & Gonzalez-Mule, E. (in press). The chrysalis effect: How ugly data metamorphosize into beautiful articles. Journal of Management
76. O’Boyle, E. H., Rutherford, M., & Banks, G. C. (in press). Publication bias in entrepreneurship research: An examination of dominant relations to performance. Journal of Business Venturing.
77. Olson, C. M., et al. (2002). Publication bias in editorial decision making. Journal of the American Medical Association, 287, 2825–2828.
78. Orwin, R. G. (1983). A fail-safe N for effect size in meta-analysis. Journal of Educational Statistics, 8, 157–159.
79. Palmer, T. M., Peters, J. L., Sutton, A. J., & Moreno, S. G. (2008). Contour-enhanced funnel plots for meta-analysis. Stata Journal, 8, 242–254.
80. Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2008). Contourenhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry. Journal of Clinical Epidemiology, 61, 991–996.
81. Rosenthal, R. (1979). The fi le drawer problem and tolerance for null results. Psychological Bulletin, 86, 638–641.
82. Rothstein, H. (2012). Accessing relevant literature. In H. M. Cooper (Ed.), APA handbook of research methods in psychology: Vol. 1. Foundations, planning, measures, and psychometrics (pp. 133–144). Washington, DC: American Psychological Association.
83. Rothstein, H. R., & Hopewell, S. (2009). Grey literature. In H. M. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 103–126). New York, NY: Russell Sage Foundation.
84. Rothstein, H. R., Sutton, A. J., & Borenstein, M. (2005). Publication bias in meta-analyses. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta analysis: Prevention, assessment, and adjustments (pp. 1–7). West Sussex, UK: Wiley.
85. Sackett, P. R., & Larson, J. R. (1990). Research strategies and tactics in industrial and organizational psychology. In M. D. Dunnette & L. M. Hough (Eds.), Handbook of industrial and organizational psychology (Vol. 1, pp. 419–489). Palo Alto, CA: Consulting Psychologists Press.
86. Schmidt, F., & Oh, I. S. (2013). Methods for second-order meta-analysis and illustrative applications. Organizational Behavior and Human Decision Processes, 121, 204–218.
87. Schmidt, F. L., & Hunter, J. E. (2014). Methods of meta-analysis: Correcting error and bias in research findings (3rd ed.). Newbury Park, CA: Sage.
88. Schmidt, F. L., Hunter, J. E., Pearlman, K., Hirsh, H. R. (1985). Forty questions about validity generalization and meta-analysis. Personnel Psychology, 38, 697–798.
89. Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve: A key to the file drawer. Journal of Experimental Psychology: General, 143, 534–547.
90. Song, F., et al. (2010). Dissemination and publication of research findings: An updated review of related biases. Health Technology Assessment, 14, 1–220.
91. Sterling, T. D., & Rosenbaum, W. L. (1995). Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. American Statistician, 49, 108–112.
92. Sterne, J. A., Gavaghan, D., & Egger, M. (2005). The funnel plot. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta analysis: Prevention, assessment, and adjustments (pp. 75–98). West Sussex, UK: Wiley.
93. Sterne, J.A.C., Sutton, A. J., Ioannidis, J. P., Terrin, N., Jones, D. R., Lau, J., Carpenter, J., Higgins, J. P. (2011). Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. British Medical Journal, 343, d4002.
94. Strech, D. (2011). The ethics of a restrictive regulation of trial registration. Ethik in Der Medizin, 23, 177–189.
95. Sutton, A. J. (2009). Publication bias. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 435–452). New York, NY: Russell Sage Foundation.
96. Sutton, A. J., & Pigott, T. D. (2005). Bias in meta-analysis induced by incompletely reported studies. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta analysis: Prevention, assessment, and adjustments (pp. 223–240). West Sussex, UK: Wiley.
97. Tate, B. W., & McDaniel, M. A. (2008, August). Race differences in personality: An evaluation of moderators and publication bias. Paper presented at the annual meeting of the Academy of Management, Anaheim, CA.
98. Terrin, N., Schmid, C. H., Lau, J., & Olkin, I. (2003). Adjusting for publication bias in the presence of heterogeneity. Statistics in Medicine, 22, 2113–2126.
99. Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008). Selective publication of antidepressant trials and its influence on apparent efficacy. New England Journal of Medicine, 358, 252–260.
100. Vevea, J. L., Clements, N. C., & Hedges, L. V. (1993). Assessing the effects of selection bias on validity data for the General Aptitude Test Battery. Journal of Applied Psychology, 78, 981–987.
101. Vevea, J. L., & Woods, C. M. (2005). Publication bias in research synthesis: Sensitivity analysis using a priori weight functions. Psychological Methods, 10, 428–443.
102. Weinhandl, E. D., & Duval, S. (2012). Generalization of trim and fi ll for application in meta-regression. Research Synthesis Methods, 3, 51–67.
103. Wertil, M. M., Schob, M., Brunner, F., & Steurer, J. (2013). Incomplete reporting of baseline characteristics in clinical trials: An analysis of randomized controlled trials and systematic reviews involving patients with chronic low back pain. PLos One, 8, e58512.
104. Whetzel, D. L. (2006, April). Publication bias of studies on the validity of customer service measures: A review of Frei and McDaniel’s (1998) Meta-analytic results. Paper presented at the annual meeting of the Society for Industrial and Organizational Psychology, Dallas, TX.
105. Williamson, P. R., & Gamble, C. (2007). Application and investigation of a bound for outcome reporting bias. Trials, 8, 9.
106. Zarin, D. A., Tse, T., & Ide, N. C. (2005). Trial registration at ClinicalTrials.gov between May and October 2005. New England Journal of Medicine, 353, 2779–2787.
107. Zarin, D. A., Tse, T., Williams, R. J., Califf, R. M., & Ide, N. C. (2011). The ClinicalTrials. Gov results database: Update and key issues. New England Journal of Medicine, 364, 852–860.