Avoiding Bias in Publication Bias Research: The Value of "Null" Findings

Journal of Business and Psychology

Volumn 29, Issue 2, 2014, Pages 183-203

  Kepes, Sven ^a   Banks, George C ^b   Oh, In Sue ^c  

^a VIRGINIA COMMONWEALTH UNIVERSITY   (United States)

^b LONGWOOD UNIVERSITY   (United States)

^c TEMPLE UNIVERSITY   (United States)

Author keywords

Dissemination bias; Meta analysis; Null results; Publication bias; Systematic reviews

Indexed keywords

EID: 84901052992     PISSN: 08893268     EISSN: None     Source Type: Journal    
DOI: 10.1007/s10869-012-9279-0     Document Type: Article

References (87)

1. Aguinis, H., Pierce, C. A., Bosco, F. A., Dalton, D. R., & Dalton, C. M. (2011). Debunking myths and urban legends about meta-analysis. Organizational Research Methods, 14, 306-331. doi: 10. 1177/1094428110375720.
2. American Psychological Association. (2010). Publication manual of the American Psychological Association (6th ed.). Washington, DC: American Psychological Association.
3. Banks, G. C., Batchelor, J. H., & McDaniel, M. A. (2010). Smarter people are (a bit) more symmetrical: A meta-analysis of the relationship between intelligence and fluctuating asymmetry. Intelligence, 38, 393-401. doi: 10. 1016/j. intell. 2010. 04. 003.
4. Banks, G. C., Kepes, S., & Banks, K. P. (2012a). Publication bias: The antagonist of meta-analytic reviews and effective policy making. Educational Evaluation and Policy Analysis, 34, 259-277. doi: 10. 3102/0162373712446144.
5. Banks, G. C., Kepes, S., & McDaniel, M. A. (2012b). Publication bias: A call for improved meta-analytic practice in the organizational sciences. International Journal of Selection and Assessment, 20, 182-196. doi: 10. 1111/j. 1468-2389. 2012. 00591. x.
6. Banks, G. C., & McDaniel, M. A. (2011). The kryptonite of evidence-based I-O psychology. Industrial and Organizational Psychology: Perspectives on Science and Practice, 4, 40-44. doi: 10. 1111/j. 1754-9434. 2010. 01292. x.
7. Becker, B. J. (1994). Combining significance levels. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 215-230). New York: Russell Sage Foundation.
8. Becker, B. J. (2005). The failsafe N or file-drawer number. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 111-126). West Sussex: Wiley.
9. Begg, C. B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 50, 1088-1101. doi: 10. 2307/2533446.
10. Berlin, J. A., & Ghersi, D. (2005). Preventing publication bias: Registries and prospective meta-analysis. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 35-48). West Sussex: Wiley.
11. Bettencourt, B. A., & Miller, N. (1996). Gender differences in aggression as a function of provocation: A meta-analysis. Psychological Bulletin, 119, 422-447. doi: 10. 1037/0033-2909. 119. 3. 422.
12. Blackwell, S. C., Thompson, L., & Refuerzo, J. (2009). Full publication of clinical trials presented at a national maternal-fetal medicine meeting: Is there a publication bias? American Journal of Perinatology, 26, 679-682. doi: 10. 1055/s-0029-1220786.
13. Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2005). Comprehensive meta-analysis (Version 2). Englewood: Biostat.
14. Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2009). Introduction to meta-analysis. Chichester: Wiley.
15. Briner, R. B., & Rousseau, D. M. (2011). Evidence-based I-O psychology: Not there yet. Industrial and Organizational Psychology: Perspectives on Science and Practice, 4, 3-22. doi: 10. 1111/j. 1754-9434. 2010. 01287. x.
16. Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004). Empirical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles. Journal of the American Medical Association, 291, 2457-2465. doi: 10. 1001/jama. 291. 20. 2457.
17. Cooper, H. (2003). Editorial. Psychological Bulletin, 129, 3-9. doi: 10. 1037/0033-2909. 129. 1. 3.
18. Curfman, G. D., Morrissey, S., & Drazen, J. M. (2006). Expression of concern reaffirmed. New England Journal of Medicine, 354, 1193. doi: 10. 1056/NEJMe068054.
19. Davis, M. S. (1971). That's interesting! Towards a phenomenology of sociology and a sociology of phenomenology. Philosophy of the Social Sciences, 1, 309-344. doi: 10. 1177/004839317100100211.
20. Dickersin, K. (1990). The existence of publication bias and risk factors for its occurrence. Journal of the American Medical Association, 263, 1385-1389. doi: 10. 1001/jama. 263. 10. 1385.
21. Dickersin, K. (2005). Publication bias: Recognizing the problem, understandings its origins and scope, and preventing harm. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 11-34). West Sussex: Wiley.
22. Duval, S. J. (2005). The "trim and fill" method. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 127-144). West Sussex: Wiley.
23. Duval, S. J., & Tweedie, R. L. (2000a). A nonparametric "Trim and Fill" method of accounting for publication bias in meta-analysis. Journal of the American Statistical Association, 95, 89-98. doi: 10. 2307/2669529.
24. Duval, S. J., & Tweedie, R. L. (2000b). Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56, 455-463. doi: 10. 1111/j. 0006-341X. 2000. 00455. x.
25. Dwan, K., et al. (2008). Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS ONE, 3, e3081. doi: 10. 1371/journal. pone. 00030.
26. Eagly, A. H., Johannesen-Schmidt, M. C., & van Engen, M. L. (2003). Transformational, transactional, and laissez-faire leadership styles: A meta-analysis comparing women and men. Psychological Bulletin, 129, 569-591. doi: 10. 1037/0033-2909. 129. 4. 569.
27. Egger, M., & Smith, G. D. (1998). Bias in location and selection of studies. British Medical Journal, 316, 61-66. doi: 10. 1136/bmj. 316. 7124. 61.
28. Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315, 629-634. doi: 10. 1136/bmj. 315. 7109. 629.
29. Evangelou, E., Trikalinos, T. A., & Ioannidis, J. P. (2005). Unavailability of online supplementary scientific information from articles published in major journals. The FASEB Journal, 19, 1943-1944. doi: 10. 1096/fj. 05-4784lsf.
30. Evans, S. (1996). Statistician's comment (to Misleading meta-analysis: "Fail safe N" is a useful mathematical measure of the stability of results by R. Persaud). British Medical Journal, 312, 125.
31. Ferguson, C. J., & Brannick, M. T. (2011). Publication bias in psychological science: Prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychological Methods, 17, 120-128. doi: 10. 1037/a0024445.
32. Field, A. P., & Gillett, R. (2010). How to do a meta-analysis. British Journal of Mathematical and Statistical Psychology, 63, 665-694. doi: 10. 1348/000711010X502733.
33. Gigerenzer, G. (2004). Mindless statistics. Journal of Socio-Economics, 33, 587-606. doi: 10. 1016/j. socec. 2004. 09. 033.
34. Greenhouse, J. B., & Iyengar, S. (2009). Sensitivity analysis and diagnostics. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 417-433). New York: Russell Sage Foundation.
35. Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82, 1-20. doi: 10. 1037/h0076157.
36. Hedges, L. V. (1992). Modeling publication selection effects in meta-analysis. Statistical Science, 7, 246-255. doi: 10. 1214/ss/1177011364.
37. Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. New York: Academic Press.
38. Hedges, L. V., & Vevea, J. L. (2005). Selection methods approaches. In H. R. Rothstein, A. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 145-174). West Sussex: Wiley.
39. Hermelin, E., Lievens, F., & Robertson, I. T. (2007). The validity of assessment centers for the prediction of supervisory performance ratings: A meta-analysis. International Journal of Selection and Assessment, 15, 405-411. doi: 10. 1111/j. 1468-2389. 2007. 00399. x.
40. Higgins, J. P. T., & Green, S. (Eds.). (2009). Cochrane handbook for systematic reviews of interventions; Version 5. 0. 2 [updated September 2009]: The Cochrane Collaboration. www. cochrane-handbook. org.
41. Higgins, J. P. T., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21, 1539-1558. doi: 10. 1002/sim. 1186.
42. Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring inconsistency in meta-analyses. British Medical Journal, 327, 557-560. doi: 10. 1136/bmj. 327. 7414. 557.
43. Hopewell, S., Clarke, M., & Mallett, S. (2005). Grey literature and systematic reviews. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 48-72). West Sussex: Wiley.
44. Hopewell, S., Loudon, K., Clarke, M. J., Oxman, A. D., & Dickersin, K. (2009). Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database of Systematic Reviews. doi: 10. 1002/14651858. MR000006. pub3.
45. Hopewell, S., McDonald, S., Clarke, M. J., & Egger, M. (2007). Grey literature in meta-analyses of randomized trials of health care interventions. Cochrane Database of Systematic Reviews. doi: 10. 1002/14651858. MR000006. pub3.
46. Hubbard, R., & Armstrong, J. S. (1997). Publication bias against null results. Psychological Reports, 80, 337-338. doi: 10. 2466/PR0. 80. 1. 337-338.
47. Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park: Sage.
48. Ioannidis, J. P. (2005). Why most published research findings are false. PLoS Medicine, 2, e124. doi: 10. 1371/journal. pmed. 0020124.
49. Jick, T. D. (1979). Mixing qualitative and quantitative methods: Triangulation in action. Administrative Science Quarterly, 24, 602-611. doi: 10. 2307/2392366.
50. Kepes, S., Banks, G. C., McDaniel, M., & Whetzel, D. L. (2012). Publication bias in the organizational sciences. Organizational Research Methods,. doi: 10. 1177/1094428112452760.
51. Kisamore, J. L., & Brannick, M. T. (2008). An illustration of the consequences of meta-analysis model choice. Organizational Research Methods, 11, 35-53. doi: 10. 1177/1094428106287393.
52. Le, H., Oh, I.-S., Shaffer, J., & Schmidt, F. L. (2007). Implications of methodological advances for the practice of personnel selection: How practitioners benefit from meta-analysis. Academy of Management Perspectives, 21, 6-15.
53. McDaniel, M. A. (2009, April). Cumulative meta-analysis as a publication bias method. Paper presented at the annual meeting of the Society for Industrial and Organizational Psychology, New Orleans, LA.
54. McDaniel, M. A., Rothstein, H. R., & Whetzel, D. L. (2006). Publication bias: A case study of four test vendors. Personnel Psychology, 59, 927-953. doi: 10. 1111/j. 1744-6570. 2006. 00059. x.
55. McNatt, D. B. (2000). Ancient Pygmalion joins contemporary management: A meta-analysis of the result. Journal of Applied Psychology, 85, 314-322. doi: 10. 1037/0021-9010. 85. 2. 314.
56. Moreno, S. G., Sutton, A. J., Turner, E. H., Abrams, K. R., Cooper, N. J., Palmer, T. M., et al. (2009). Novel methods to deal with publication biases: Secondary analysis of antidepressant trials in the FDA trial registry database and related journal publications. British Medical Journal, 339, b2981. doi: 10. 1136/bmj. b2981.
57. Palmer, T. M., Peters, J. L., Sutton, A. J., & Moreno, S. G. (2008). Contour-enhanced funnel plots for meta-analysis. Stata Journal, 8, 242-254.
58. Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2008). Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry. Journal of Clinical Epidemiology, 61, 991-996. doi: 10. 1016/j. jclinepi. 2007. 11. 010.
59. Peters, J., Sutton, A., Jones, D. R., Abrams, K. R., Rushton, L., & Moreno, S. G. (2010). Assessing publication bias in meta-analyses in the presence of between-study heterogeneity. Journal of the Royal Statistical Society (Series A), 173, 575-591. doi: 10. 1111/j. 1467-985X. 2009. 00629. x.
60. Quiñones, M. A., Ford, J. K., & Teachout, M. S. (1995). The relationship between work experience and job performance: A conceptual and meta-analytic review. Personnel Psychology, 48, 887-910. doi: 10. 1111/j. 1744-6570. 1995. tb01785. x.
61. Renkewitz, F., Fuchs, H. M., & Fiedler, S. (2011). Is there evidence of publication biases in JDM research? Judgment and Decision Making, 6, 870-881.
62. Robbins, S., Oh, I.-S., Le, H., & Button, C. (2009). Intervention effects on college performance and retention as mediated by motivational, emotional, and social control factors: Integrated meta-analytic path-analysis. Journal of Applied Psychology, 94, 1163-1184. doi: 10. 1037/a0015738.
63. Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86, 638-641. doi: 10. 1037/0033-2909. 86. 3. 638.
64. Rothstein, H. R. (2012). Accessing relevant literature. In H. Cooper (Ed.), APA handbook of research methods in psychology: Vol. 1. Foundations, planning, measures, and psychometrics (pp. 133-144). Washington, DC: American Psychological Association.
65. Rothstein, H. R., & Hopewell, S. (2009). Grey literature. In H. M. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 103-126). New York: Russell Sage Foundation.
66. Rothstein, H. R., Sutton, A. J., & Borenstein, M. (2005a). Publication bias in meta-analyses. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 1-7). West Sussex: Wiley.
67. Rothstein, H. R., Sutton, A. J., & Borenstein, M. (2005b). Publication bias in meta-analysis: Prevention, assessment, and adjustments. West Sussex: Wiley.
68. Sackett, P. R., & Larson, J. R. (1990). Research strategies and tactics in industrial and organizational psychology. In M. D. Dunnette & L. M. Hough (Eds.), Handbook of industrial and organizational psychology (Vol. 1, pp. 419-489). Palo Alto, CA: Consulting Psychologists Press.
69. Scandura, T. A., & Williams, E. A. (2000). Research methodology in management: Current practices, trends, and implications for future research. The Academy of Management Journal, 43, 1248-1264. doi: 10. 2307/1556348.
70. Schmidt, F. L., & Le, H. (2005). Hunter & Schmidt's Meta-analysis programs (Version 1. 1). The University of Iowa, IA.
71. Schmidt, F. L., Oh, I.-S., & Hayes, T. (2009). Fixed versus random effects models in meta-analysis: Model properties and an empirical comparison of differences in results. British Journal of Mathematical and Statistical Psychology, 62, 97-128. doi: 10. 1348/000711007X255327.
72. Schopfel, J. (2006). Observations on the future of grey literature. Grey Journal, 2, 67-76.
73. Schulze, R. (2004). Meta-analysis: A comparison of approaches. Cambridge: Hogrefe & Huber.
74. Smith, G. D., & Egger, M. (1994). Who benefits from medical interventions? British Medical Journal, 308, 72-74. doi: 10. 1136/bmj. 308. 6921. 72.
75. Song, F., Easterwood, A., Gilbody, S., Duley, L., & Sutton, A. J. (2000). Publication and other selection biases in systematic reviews. Health Technology, 4, 1-115. doi: 10. 3310/hta4100.
76. Song, F., et al. (2010). Dissemination and publication of research findings: An updated review of related biases. Health Technology Assessment, 14, 1-220. doi: 10. 3310/hta14080.
77. Sterling, T. D., & Rosenbaum, W. L. (1995). Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. American Statistician, 49, 108-112. doi: http://www. tandfonline. com/doi/abs/10. 1080/00031305. 1995. 10476125.
78. Sterne, J. A., & Egger, M. (2005). Regression methods to detect publication bias and other bias in meta-analysis. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 99-110). West Sussex: Wiley.
79. Sterne, J. A., Gavaghan, D., & Egger, M. (2005). The funnel plot. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 75-98). West Sussex: Wiley.
80. Sterne, J. A., et al. (2011). Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. British Medical Journal, 342, d4002-d4010. doi: 10. 1136/bmj. d4002.
81. Sutton, A. J. (2005). Evidence concerning the consequences of publication and related biases. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 175-192). West Sussex: Wiley.
82. Sutton, A. J. (2009). Publication bias. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 435-452). New York: Russell Sage Foundation.
83. Terrin, N., Schmid, C. H., Lau, J., & Olkin, I. (2003). Adjusting for publication bias in the presence of heterogeneity. Statistics in Medicine, 22, 2113-2126. doi: 10. 1002/sim. 1461.
84. Torgerson, C. J. (2006). Publication bias: The Achilles' heel of systematic reviews? British Journal of Educational Studies, 54, 89-102. doi: http://onlinelibrary. wiley. com/doi/10. 1111/j. 1467-8527. 2006. 00332. x/abstract.
85. Trikalinos, T. A., & Ioannidis, J. P. A. (2005). Assessing the evolution of effect sizes over time. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 241-259). West Sussex: Wiley.
86. Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008). Selective publication of antidepressant trials and its influence on apparent efficacy. New England Journal of Medicine, 358, 252-260. doi: 10. 1056/NEJMsa065779.
87. Vevea, J. L., & Woods, C. M. (2005). Publication bias in research synthesis: Sensitivity analysis using a priori weight functions. Psychological Methods, 10, 428-443. doi: 10. 1037/1082-989X. 10. 4. 428.