Publication Bias: A call for improved meta-analytic practice in the organizational sciences

International Journal of Selection and Assessment

Volumn 20, Issue 2, 2012, Pages 182-196

  Banks, George C ^a   Kepes, Sven ^a   Mcdaniel, Michael A ^a  

^a VIRGINIA COMMONWEALTH UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84861209038     PISSN: 0965075X     EISSN: 14682389     Source Type: Journal    
DOI: 10.1111/j.1468-2389.2012.00591.x     Document Type: Article

References (82)

1. Aguinis, H., Dalton, D. R., Bosco, F. A., Pierce, C. A., & Dalton, C. M. (2011a). Meta-analytic choices and judgment calls: Implications for theory building, testing, obtained effect sizes, and scholarly impact. Journal of Management, 37, 5-38.
2. Aguinis, H., Pierce, C. A., Bosco, F. A., Dalton, D. R., & Dalton, C. M. (2011b). Debunking myths and urban legends about meta-analysis. Organizational Research Methods, 14, 306-331.
3. Aguinis, H., Pierce, C. A., Bosco, F. A., & Muslin, I. S. (2009). First decade of organizational research methods: Trends in design, measurement, and data-analysis topics. Organizational Research Methods, 12, 69-112.
4. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing (2nd ed.). Washington, DC: American Educational Research Association.
5. American Psychological Association. (2010). Publication manual of the American Psychological Association (6th ed.). Washington, DC: Author.
6. Banks, G. C., Batchelor, J. H., & McDaniel, M. A. (2010). Smarter people are (a bit) more symmetrical: A meta-analysis of the relationship between intelligence and fluctuating asymmetry. Intelligence, 38, 393-401.
7. Banks, G. C., & McDaniel, M. A. (2011). The kryptonite of evidence-based I-O psychology. Industrial and Organizational Psychology: Perspectives on Science and Practice, 4, 40-44.
8. Beal, D. J., Corey, D. M., & Dunlap, W. P. (2002). On the bias of Huffcutt and Arthur's (1995) procedure for identifying outliers in the meta-analysis of correlations. Journal of Applied Psychology, 87, 583-589.
9. Becker, B. J. (2005). The failsafe N or file-drawer number. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 111-126). Chichester, UK: Wiley.
10. Begg, C. B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 50, 1088-1101.
11. Berlin, J. A., & Ghersi, D. (2005). Preventing publication bias: Registries and prospective meta-analysis. In H. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 35-48). Chichester, UK: Wiley.
12. Berry, C. M., Sackett, P. R., & Tobares, V. (2010). A meta-analysis of conditional reasoning tests of aggression. Personnel Psychology, 63, 361-384.
13. Bing, M. N., Stewart, S. M., Davison, H. K., Green, P. D., McIntyre, M. D., & James, L. R. (2007). An integrative typology of personality assessment for aggression: Implications for predicting counterproductive workplace behavior. Journal of Applied Psychology, 92, 722-744.
14. Borenstein, M., Hedges, L., Higgins, J., & Rothstein, H. (2005). Comprehensive meta-analysis version 2. Englewood, NJ: Biostat.
15. Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. Chichester, UK: Wiley.
16. Briner, R. B., & Rousseau, D. M. (2011). Evidence-based I-O psychology: Not there yet. Industrial and Organizational Psychology: Perspectives on Science and Practice, 4, 3-22.
17. Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.
18. Davis, M. S. (1971). That's interesting! Towards a phenomenology of sociology and a sociology of phenomenology. Philosophy of the Social Sciences, 1, 309-344.
19. Dickersin, K. (2005). Recognizing the problem, understanding its origins and scope, and preventing harm. In H. R. Rothstein & A. J. Sutton (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 11-34). Chichester, UK: Wiley.
20. Duval, S. J. (2005). The trim and fill method. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 125-144). Chichester, UK: Wiley.
21. Duval, S. J., & Tweedie, R. L. (2000a). A non-parametric 'trim and fill' method of accounting for publication bias in meta-analysis. Journal of the American Statistical Association, 95, 89-98.
22. Duval, S. J., & Tweedie, R. L. (2000b). Trim and fill: A simple funnel plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56, 276-284.
23. Egger, M., Davey Smith, G., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315, 629-634.
24. Evangelou, E., Trikalinos, T. A., & Ioannidis, J. P. (2005). Unavailability of online supplementary scientific information from articles published in major journals. FASEB Journal, 19, 1943-1944.
25. Evans, S. (1996). Statistician's comments (to Misleading meta-analysis: 'Fail safe N' is a useful mathematical measure of the stability of the results by R. Persaud). British Medical Journal, 312, 125.
26. Ferguson, C. J., & Brannick, M. T. (2012). Publication bias in psychological science: Prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychological Methods, 17, 120-128. doi: 10.1037/a0024445.
27. Field, A. P. (2001). Meta-analysis of correlation coefficients: A Monte Carlo comparison of fixed- and random-effects methods. Psychological Methods, 6, 161-180.
28. Field, A. P. (2005). Is the meta-analysis of correlation coefficients accurate when population correlations vary? Psychological Methods, 10, 444-467.
29. Frost, B. C. (2002). Implicit versus explicit aggression: Who's ready to rumble? Unpublished manuscript, Knoxville, TN: University of Tennessee.
30. Frost, B. C. (2005). Implicit and explicit personality: An integrative approach to predicting aggressive behavior in a field setting Doctoral dissertation, University of Tennessee. ProQuest Dissertations, AAT 3188875.
31. Geyskens, I., Krishnan, R., Steenkamp, J., & Cunha, P. V. (2009). A review and evaluation of meta-analysis practices in management research. Journal of Management, 35, 393-419.
32. Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82, 1-20.
33. Hafdahl, A. R. (2009). Improved Fisher z estimators for univariate random-effects meta-analysis of correlations. British Journal of Mathematical and Statistical Psychology, 62, 233-261.
34. Hafdahl, A. R. (2010). Random-effects meta-analysis of correlations: Monte Carlo evaluation of mean estimators. British Journal of Mathematical and Statistical Psychology, 63, 227-254.
35. Hafdahl, A. R., & Williams, M. A. (2009). Meta-analysis of correlations revisited: Attempted replication and extension of Field's (2001) simulation studies. Psychological Methods, 14, 24-42.
36. Hawes, S. R. (2000). A comparison of biodata, ability, and a conditional reasoning test as predictors of reliable behavior in the workplace Doctoral dissertation, University of Tennessee. ProQuest Dissertations, AAT 9996356.
37. Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. New York: Academic Press.
38. Higgins, J. P., & Green, S. (2009). Cochrane handbook for systematic reviews of interventions Version 5.0.2 [updated September 2009]: The Cochrane Collaboration. Available at
39. Hopewell, S., Clarke, M., & Mallett, S. (2005). Grey literature and systematic reviews. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 48-72). Chichester, UK: Wiley.
40. Hornick, C. W., Fox, K. A., Axton, T. R., & Wyatt, B. S. (1999). The relative contribution of conditional reasoning and multiple intelligence measures in predicting firefighter and law enforcement officer job performance Unpublished manuscript.
41. Huffcutt, A. I., & Arthur, W. (1995). Development of a new outlier statistic for meta-analytic data. Journal of Applied Psychology, 80, 327-334.
42. Hunter, J. E., & Schmidt, F. L. (1990). Dichotomization of continuous variables: The implications for meta-analysis. Journal of Applied Psychology, 75, 334-349.
43. Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.
44. Ioannidis, J. P. (1998). Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. Journal of the American Medical Association, 279, 281-286.
45. Ioannidis, J. P. (2005). Differentiating biases from genuine heterogeneity: Distinguishing artifactual from substantive effects. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 287-302). Chichester, UK: Wiley.
46. Ioannidis, J. P. (2006). Evolution and translation of research findings: From bench to where? PLoS Clinical Trials, 2, 1-5.
47. James, L. R. (1998). Measurement of personality via conditional reasoning. Organizational Research Methods, 1, 131-163.
48. James, L. R., & Mazerolle, M. D. (2002). Personality at work. Thousand Oaks, CA: Sage.
49. James, L. R., & McIntyre, M. D. (2000). Conditional reasoning test of aggression test manual. Knoxville, TN: Innovative Assessment Technology, LLC.
50. James, L. R., McIntyre, M. D., Glisson, C. A., Green, P. D., Patton, T. W., LeBreton, J. M. etal. (2005). A conditional reasoning measure for aggression. Organizational Research Methods, 8, 69-99.
51. Jick, T. D. (1979). Mixing qualitative and quantitative methods: Triangulation in action. Administrative Science Quarterly, 24, 602-611.
52. Journal Citation Reports. (2010). Published by Thompson Reuters. Accessed via the ISI Web of Knowledge database.
53. Lau, J., & Antman, E. M. (1992). Cumulative meta-analysis of therapeutic trials for myocardial infarction. New England Journal of Medicine, 327, 248-254.
54. Lau, J., Schmid, C. H., & Chalmers, T. C. (1995). Cumulative meta-analysis of clinical trials builds evidence for exemplary medical care. Journal of Clinical Epidemiology, 48, 45-57.
55. LeBreton, J. M. (2002). Use of differential framing to measure implicit social cognitions associated with aggression Doctoral dissertation, University of Tennessee. ProQuest Dissertations, AAT 3054126.
56. Light, R. J., & Pillemer, D. B. (1984). Summing up. Boston, MA: Harvard University Press.
57. McDaniel, M. A. (2009, April). Cumulative meta-analysis as a publication bias method Paper presented at the annual meeting of the Society for Industrial and Organizational Psychology, New Orleans, LA.
58. McDaniel, M. A., Rothstein, H. R., & Whetzel, D. L. (2006). Publication bias: A case study of four test vendors. Personnel Psychology, 59, 927-953.
59. McKay, P., & McDaniel, M. A. (2006). A re-examination of Black-White mean differences in work performance: More data, more moderators. Journal of Applied Psychology, 91, 531-554.
60. Moreno, S. G., Sutton, A. J., Turner, E. H., Abrams, K. R., Cooper, N. J., Palmer, T. M. etal. (2009). Novel methods to deal with publication biases: Secondary analysis of antidepressant trials in the FDA trial registry database and related journal publications. British Medical Journal, 339, b2981.
61. Orwin, R. F. (1983). A fail-safe N for effect size in meta-analysis. Journal of Educational Statistics, 8, 157-159.
62. Patton, T. W. (1998). Measuring personal reliability via conditional reasoning: Identifying people who will work reliably Doctoral dissertation, University of Tennessee. ProQuest Dissertations, AAT 9936290.
63. Podsakoff, P. M., MacKenzie, S. B., Podsakoff, N. P., & Bachrach, D. G. (2008). Scholarly influence in the field of management: A bibliometric analysis of the determinants of university and author impact in the management literature in the past quarter century. Journal of Management, 34, 641-720.
64. Rosenthal, R. (1979). The 'file drawer problem' and tolerance for null results. Psychological Bulletin, 86, 638-641.
65. Rothstein, H. R., & Hopewell, S. (2009). Grey literature. In H. Cooper, L. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 103-126). New York, NY: Russell Sage Foundation.
66. Rothstein, H. R., Sutton, A. J., & Borenstein, M. (2005a). Publication bias in meta-analysis: Prevention, assessment and adjustments. Chichester, UK: Wiley.
67. Rothstein, H. R., Sutton, A. J., & Borenstein, M. (2005b). Publication bias in meta-analysis. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 1-7). Chichester, UK: Wiley.
68. Russell, S. M., James, L. R., & McIntyre, M. D. (2004, April). Predicting lying, cheating, and defiance in an internet-based environment Paper presented at the annual meeting of the Society for Industrial and Organizational Psychology, Chicago, IL.
69. Sablynski, C. J., & Mitchell, T. R. (2006, May). Post hoc analyses of interactions between the aggression questionnaire and conditional reasoning test of aggression Paper presented at the annual meeting of the Society for Industrial and Organizational Psychology, Dallas, TX.
70. Sackett, P. R., & Larson, J. R. (1990). Research strategies and tactics in industrial and organizational psychology. In M. D. Dunnette & L. M. Hough (Eds.), Handbook of industrial and organizational psychology (Vol. 1, pp. 419-489). Palo Alto, CA: Consulting Psychologists Press.
71. Schmidt, F. L., & Le, H. (2005). Hunter & Schmidt's meta-analysis programs (version 1.1). Iowa City, IA: The University of Iowa.
72. Society for Industrial and Organizational Psychology. (2003). Principles for the validation and use of personnel selection procedures (4th ed.). Bowling Green, OH: The Society for Industrial and Organizational Psychology.
73. Song, F., Parekh, S., Hooper, L., Loke, Y. K., Ryder, J., Sutton, A. J. etal. (2010). Dissemination and publication of research findings: An updated review of related biases. Health Technology Assessment, 14, 1-220.
74. Stern, J. M., & Simes, R. J. (1997). Publication bias: Evidence of delayed publication in a cohort study of clinical research projects. British Medical Journal, 315, 640-645.
75. Sterne, J. A. C., & Egger, M. (2005). Regression methods to detect publication and other bias in meta-analysis. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 99-110). Chichester, UK: Wiley.
76. Sutton, A. J. (2005). Evidence concerning the consequences of publication and related biases. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 175-192). Chichester, UK: Wiley.
77. Sutton, A. J. (2009). Publication bias. In H. Cooper, L. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 435-472). New York, NY: Russell Sage Foundation.
78. Tate, B. W., & McDaniel, M. A. (2008). Race differences in personality: An evaluation of moderators and publication bias. Paper presented at the Annual meeting of the Academy of Management, Anaheim CA.
79. Terrin, N., Schmid, C. H., Lau, J., & Olkin, I. (2003). Adjusting for publication bias in the presence of heterogeneity. Statistics in Medicine, 22, 2113-2126.
80. Trikalinos, T. A., & Ioannidis, J. P. A. (2005). Assessing the evolution of effect sizes over time. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 241-259). Chichester, UK: Wiley.
81. Uitterlinden, A. G., Ralston, S. H., Brandi, M. L., Carey, A. H., Grinberg, D., Langdahl, B. L. etal. (2006). The association between common vitamin d receptor gene variations and osteoporosis: A participant-level meta-analysis. Annals of Internal Medicine, 145, 255-264.
82. Walton, W. R. (2004). Justification of antisocial behavior Doctoral dissertation, University of Tennessee. ProQuest Dissertations, AAT 3148313.