Meta-analytic choices and judgment calls: Implications for theory building and testing, obtained effect sizes, and scholarly impact

Journal of Management

Volumn 37, Issue 1, 2011, Pages 5-38

  Aguinis, Herman ^a   Dalton, Dan R ^a   Bosco, Frank A ^b   Pierce, Charles A ^b   Dalton, Catherine M ^a  

^a INDIANA UNIVERSITY   (United States)

^b UNIVERSITY OF MEMPHIS   (United States)

Author keywords

Literature review; Meta analysis; Quantitative review; Research synthesis

Indexed keywords

EID: 78650357473     PISSN: 01492063     EISSN: 15571211     Source Type: Journal    
DOI: 10.1177/0149206310377113     Document Type: Article

References (83)

1. Aguinis, H. 2001. Estimation of sampling variance of correlations in meta-analysis. Personnel Psychology, 54: 569-590.
2. Aguinis, H., Beaty, J.C., Boik, R.J., & Pierce, C.A. 2005. Effect size and power in assessing moderating effects of categorical variables using multiple regression: A 30-year review. Journal of Applied Psychology, 90: 94-107.
3. Aguinis, H., Boik, R.J., & Pierce, C.A. 2001. A generalized solution for approximating the power to detect effects of categorical moderator variables using multiple regression. Organizational Research Methods, 4: 291-323.
4. Aguinis, H., Culpepper, S.A., & Pierce, C.A. in press. Revival of test bias research in preemployment testing. Journal of Applied Psychology.
5. Aguinis, H., Dalton, D.A., Bosco, F.A., Pierce, C.A., & Dalton, C.M. 2009. Meta-analytic choices and judgment calls: Implications for theory and scholarly impact. Paper presented at the meeting of the Academy of Management, Chicago, IL, August.
6. Aguinis, H., Gottfredson, R.K., & Wright, T.A. in press. Best-practice recommendations for estimating interaction effects using meta-analysis. Journal of Organizational Behavior.
7. Aguinis, H., & Harden, E.E. 2009. Cautionary note on conveniently dismissing χ2 goodness-of-fit test results: Implications for strategic management research. In D. D. Bergh & D. J. Ketchen (Eds.), Research methodology in strategy and management (Vol. 5): 111-120. Howard House, UK: Emerald Group.
8. Aguinis, H., & Pierce, C.A. 1998. Testing moderator variable hypotheses meta-analytically. Journal of Management, 24: 577-592.
9. Aguinis, H., Pierce, C.A., Bosco, F.A., Dalton, D.R., & Dalton, C.M. in press. Debunking myths and urban legends about meta-analysis. Organizational Research Methods.
10. Aguinis, H., Pierce, C.A., Bosco, F.A., & Muslin, I.S. 2009. First decade of Organizational Research Methods: Trends in design, measurement, and data-analysis topics. Organizational Research Methods, 12: 69-112.
11. Aguinis, H., Sturman, M.C., & Pierce, C.A. 2008. Comparison of three meta-analytic procedures for estimating moderating effects of categorical variables. Organizational Research Methods, 11: 9-34.
12. Aguinis, H., & Whitehead, R. 1997. Sampling variance in the correlation coefficient under indirect range restriction: Implications for validity generalization. Journal of Applied Psychology, 82: 528-538.
13. Aguinis, H., Werner, S., Abbott, J.L., Angert, C., Park, J.H., & Kohlhausen, D. 2010. Customer-centric science: Reporting research results with rigor, relevance, and practical impact in mind. Organizational Research Methods, 13: 515-539.
14. Algera, J.A., Jansen, P.G., Roe, R.A., & Vijn, P. 1984. Validity generalization: Some critical remarks on the Schmidt-Hunter procedure. Journal of Occupational Psychology, 57: 197-210.
15. Bacharach, S.B. 1989. Organizational theories: Some criteria for evaluation. Academy of Management Review, 14: 496-515.
16. Bandalos, D.L., & Boehm-Kaufman, M.R. 2009. Four common misconceptions in exploratory factor analysis. In C. E. Lance & R. J. Vandenberg (Eds.), Statistical and methodological myths and urban legends: Doctrine, verity and fable in the organizational and social sciences: 61-87. New York: Routledge.
17. Bays, C.L. 2001. Quality of life of stroke survivors: A research synthesis. Journal of Neuroscience Nursing, 33: 310-316.
18. Beal, B.J., Cohen, R.R., Burke, M.J., & McLendon, C.L. 2003. Cohesion and performance in groups: A meta-analytic clarification of construct relationships. Journal of Applied Psychology, 88: 989-1004.
19. Bierstedt, R. 1959.Nominal and real definitions in sociological theory. In L. Gross (Ed.), Symposium on sociological theory: 121-144. New York: Harper & Row.
20. Bobko, P., & Roth, P.L. 2003. Meta-analysis and validity generalization as research tools: Issues of sample bias and degrees of mis-specification. In K. R. Murphy (Ed.), Validity generalization: A critical review: 67-90. Mahwah, NJ: Lawrence Erlbaum.
21. Bobko, P., & Roth, P.L. 2008. Psychometric accuracy and (the continuing need for) quality thinking in meta-analysis. Organizational Research Methods, 11: 114-126.
22. Bobko, P., & Stone-Romero, E.F. 1998. Meta-analysis may be another useful research tool, but it is not a panacea. Research in personnel and human resources management (Vol. 16): 359-397. Stamford, CT: JAI.
23. Boyd, B.K., Finkelstein, S., & Gove, S. 2005. How advanced is the strategy paradigm? The role of particularism and universalism in shaping research outcomes. Strategic Management Journal, 26: 841-854.
24. Bullock, R.J., & Svyantek, D.J. 1985. Analyzing meta-analysis: Potential problems, an unsuccessful replication, and evaluation criteria. Journal of Applied Psychology, 70: 108-115.
25. Callender, J.C., & Osburn, H.G. 1988. Unbiased estimation of sampling variance of correlations. Journal of Applied Psychology, 73: 312-315.
26. Cascio, W.F., & Aguinis, H. 2005. Test development and use: New twists on old questions. Human Resource Management, 44: 219-235.
27. Cascio, W.F., & Aguinis, H. 2008. Research in industrial and organizational psychology from 1963 to 2007: Changes, choices, and trends. Journal of Applied Psychology, 93: 1062-1081.
28. Colquitt, J.A., LePine, J.A., & Noe, R.A. 2000. Toward an integrative theory of training motivation: A meta-analytic path analysis of 20 years of research. Journal of Applied Psychology, 85: 678-707.
29. Colquitt, J.A., & Zapata-Phelan, C.P. 2007. Trends in theory building and theory testing: A five-decade study of Academy of Management Journal. Academy of Management Journal, 50: 1281-1303.
30. Combs, J.G. 2010. Big samples and small effects: Let's not trade relevance and rigor for power. Academy of Management Journal, 53: 9-13.
31. Conway, J.M., & Huffcutt, A.I. 2003. A review and evaluation of exploratory factor analysis practices in organizational research. Organizational Research Methods, 6: 147-168.
32. Cooper, H.M. 1982. Scientific guidelines for conducting integrative research reviews. Review of Educational Research, 52: 291-302.
33. Cortina, J.M., & Landis, R.S. 2009. When small effect sizes tell a big story, and when large effect sizes don't. In C. E. Lance & R. J. Vandenberg (Eds.), Statistical and methodological myths and urban legends: Doctrine, verity and fable in the organizational and social sciences: 287-308. New York: Routledge.
34. Dalton, D.R., & Dalton, C.M. 2008. Meta-analyses: Some very good steps toward a bit longer journey. Organizational Research Methods, 11: 127-147.
35. Duriau, V.J., Reger, R.K., & Pfarrer, M.D. 2007. A content analysis of the content analysis literature in organization studies: Research themes, data sources, and methodological refinements. Organizational Research Methods, 10: 5-34.
36. Erez, A., Bloom, M.C., & Wells, M.T. 1996. Using random rather than fixed effects models in meta-analysis: Implications for situational specificity and validity generalization. Personnel Psychology, 49: 275-306.
37. Geyskens, I., Krishnan, R., Steenkamp, J.E.M., & Cunha, P.V. 2009. A review and evaluation of meta-analysis practices in management research. Journal of Management, 35: 393-419.
38. Guzzo, R.A., Jackson, S.E., & Katzell, R.A. 1987. Meta-analysis analysis. Research in Organizational Behavior, 9: 407-442.
39. Hedges, L.V. 1982. Estimation of effect sizes from a series of experiments. Psychological Bulletin, 92: 490-499.
40. Hedges, L.V., & Olkin, I. 1985. Statistical methods for meta-analysis. Orlando, FL: Academic Press.
41. Hunter, J.E., & Schmidt, F.L. 1990. Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.
42. Hunter, J.E., & Schmidt, F.L. 2004. Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks, CA: Sage.
43. Hunter, J.E., Schmidt, F.L., & Jackson, G.B. 1982. Meta-analysis: Cumulating research findings across studies. Beverly Hills, CA: Sage.
44. Hunter, J.E., Schmidt, F.L., & Le, H. 2006. Implications of direct and indirect range restriction for meta-analysis methods and findings. Journal of Applied Psychology, 91: 594-612.
45. James, L.R., Demaree, R.G., & Mulaik, S.A. 1986. A note on validity generalization procedures. Journal of Applied Psychology, 71: 440-450.
46. James, L.R., Demaree, R.G., Mulaik, S.A., & Ladd, R.T. 1992. Validity generalization in the context of situational models. Journal of Applied Psychology, 77: 3-14.
47. Johnson, B.T., Mullen, B., & Salas, E. 1995. Comparison of three major meta-analytic approaches. Journal of Applied Psychology, 80: 94-106.
48. Judge, T.A., Jackson, C.L., Shaw, J.C., Scott, B.A., & Rich, B.L. 2007. Self-efficacy and work-related performance: The integral role of individual differences. Journal of Applied Psychology, 92: 107-127.
49. Kisamore, J.L., & Brannick, M.T. 2008. An illustration of the consequences of meta-analysis model choice. Organizational Research Methods, 11: 35-53.
50. Kuhn, T.S. 1962. The structure of scientific revolutions. Chicago: University of Chicago Press.
51. Kulinskaya, E., Morgenthaler, S., & Staudte, R.G. 2008. Meta analysis: A guide to calibrating and combining statistical evidence. West Sussex, UK: John Wiley.
52. Law, K.S., Schmidt, F.L., & Hunter, J.E. 1994. A test of two refinements in procedures for meta-analysis. Journal of Applied Psychology, 79: 978-986.
53. Long, J.S., & Fox, M.F. 1995. Scientific careers: Universalism and particularism. Annual Review of Sociology, 21: 45-71.
54. Matt, G.E. 1989. Decision rules for selecting effect sizes in meta-analysis: A review and reanalysis of psychotherapy outcome studies. Psychological Bulletin, 105: 106-115.
55. McDaniel, M.A., Rothstein, H.R. & Whetzel, D.L. 2006. Publication bias: A case study of four test vendors. Personnel Psychology, 59: 927-953.
56. Mill, J.S. 1843/2002. A system of logic. Honolulu, HI: University Press of the Pacific.
57. Murphy, K.R. 2003. The logic of validity generalization. In K. R. Murphy (Ed.), Validity generalization: A critical review: 1-29. Mahwah, NJ: Lawrence Erlbaum.
58. Murphy, K.R., & Newman, D.A. 2003. The past, the present, and future of validity generalization. In K. R. Murphy (Ed.), Validity generalization: A critical review: 403-424. Mahwah, NJ: Lawrence Erlbaum.
59. Newman, D.A., Jacobs, R.R., & Bartram, D. 2007. Choosing the best method for local validity estimation: Relative accuracy of meta-analysis versus a local study versus Bayes-analysis. Journal of Applied Psychology, 92: 1394-1413.
60. Oh, I., Schmidt, F.L., Shaffer, J.A., & Le, H. 2008. The Graduate Management Admission Test (GMAT) is even more valid than we thought: A new development in meta-analysis and its implications for the validity of the GMAT. Academy of Management Learning and Education, 7: 563-570.
61. Osburn, H.G., & Callender, J. 1992. A note on the sampling variance of the mean uncorrected correlation in meta-analysis and validity generalization. Journal of Applied Psychology, 77: 115-122.
62. Oswald, F.L., & Johnson, J.W. 1998. On the robustness, bias, and stability of statistics from meta-analysis of correlation coefficients: Some initial Monte Carlo findings. Journal of Applied Psychology, 83: 164-178.
63. Pierce, C.A., Block, R.A., & Aguinis, H. 2004. Cautionary note on reporting eta-squared values from multifactor ANOVA designs. Educational and Psychological Measurement, 64: 916-924.
64. Podsakoff, P.M., & Dalton, D.R. 1987. Research methodology in organizational studies. Journal of Management, 13: 419-441.
65. Podsakoff, P.M., MacKenzie, S.B., Podsakoff, N.P., & Bachrach, D.G. 2008. Scholarly influence in the field of management: A bibliometric analysis of university and author impact on the management literature during the past quarter century. Journal of Management, 34: 641-720.
66. Raju, N.S., Burke, M.J., Normand, J., & Langlois, G.M. 1991. A new meta-analytic approach. Journal of Applied Psychology, 76: 432-446.
67. Rosenthal, R. 1979. The file drawer problem and tolerance for null results. Psychological Bulletin, 86: 638-641.
68. Rosenthal, R., & Rubin, D.B. 1982. Comparing effect sizes of independent studies. Psychological Bulletin, 92: 500-504.
69. Rothstein, H.R. 2003. Progress is our most important product: Contributions of validity generalization and meta-analysis to the development and communication of knowledge in I/O psychology. In K. R. Murphy (Ed.), Validity generalization: A critical review: 115-154. Mahwah, NJ: Lawrence Erlbaum.
70. Schmidt, F.L. 1992. What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist, 47: 1173-1181.
71. Schmidt, F.L. 1996. Statistical significance testing and cumulative knowledge in psychology: Implications for the training of researchers. Psychological Methods, 1: 115-129.
72. Schmidt, F.L. 2008. Meta-analysis: A Constantly evolving research integration tool. Organizational Research Methods, 11: 96-113.
73. Schmidt, F.L., & Hunter, J.E. 2003. History, development, evolution, and impact of validity generalization and meta-analysis methods, 1975-2001. In K. R. Murphy (Ed.), Validity generalization: A critical review: 31-66. Mahwah, NJ: Lawrence Erlbaum.
74. Schmidt, F.L., Oh, I., & Hayes, T.L. 2009. Fixed- versus random-effects models in meta-analysis: Model properties and an empirical comparison of differences in results. British Journal of Mathematical and Statistical Psychology, 62: 97-128.
75. Schmidt, F.L., Pearlman, K., & Hunter, J.E. 1980. The validity and fairness of employment and educational tests for Hispanic Americans: A review and analysis. Journal of Applied Psychology, 33: 705-724.
76. Schmidt, F.L., & Raju, N. 2007. Updating meta-analytic research findings: Bayesian approaches versus the medical model. Journal of Applied Psychology, 92: 297-308.
77. Schmidt, F.L., Shaffer, J.A., & Oh, I. 2008. Increased accuracy for range restriction corrections: Implications for the role of personality and general mental ability in job and training performance. Personnel Psychology, 61: 827-868.
78. Schulze, R., Holling, H., & Bohning, D. 2003. Meta-analysis: New developments and applications in medical and social sciences. Cambridge, MA: Hogreffe & Huber.
79. Steel, P.D., & Kammeyer-Mueller, J.D. 2002. Comparing meta-analytic moderator estimation techniques under realistic conditions. Journal of Applied Psychology, 87: 96-111.
80. Steiner, D.D., Lane, I.M., Dobbins, G.H., Schnur, A., & McConnell, S. 1991. A review of meta-analyses in organizational behavior and human resources management: An empirical assessment. Educational and Psychological Measurement, 51: 609-626.
81. Tait, M., Padgett, M.Y., & Baldwin, T. 1989. Job and life satisfaction: A reevaluation of the strength of the relationship and gender effects as a function of the date of the study. Journal of Applied Psychology, 74: 502-507.
82. Van Maanen, J., Sarensen, J.B., & Mitchell, T.R. 2007. Introduction to special topic forum: The interplay between theory and method. Academy of Management Review, 32: 1145-1154.
83. Wanous, J.P., Sullivan, S.E., & Malinak, J. 1989. The role of judgment calls in meta-analysis. Journal of Applied Psychology, 74: 259-264.