On the robustness, bias, and stability of statistics from meta-analysis of correlation coefficients: Some initial Monte Carlo findings

Journal of Applied Psychology

Volumn 83, Issue 2, 1998, Pages 164-178

  Oswald, Frederick L ^b   Johnson, Jeff W ^a  

^a Pers Decisions Research Institutes   (United States)

^b UNIVERSITY OF MINNESOTA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0032364480     PISSN: 00219010     EISSN: None     Source Type: Journal    
DOI: 10.1037/0021-9010.83.2.164     Document Type: Article

References (57)

1. Aguinis, H., & Whitehead, R. (1997). Sampling variance in the correlation coefficient under indirect range restriction: Implications for validity generalization. Journal of Applied Psychology, 82, 528-538.
2. Bobko, P. (1983). An analysis of correlations corrected for attenuation and range restriction. Journal of Applied Psychology, 68, 584-589.
3. Bock, D. R., & Petersen, A. C. (1975). A multivariate correction for attenuation. Biometrika, 62, 673-678.
4. Bradley, J. V. (1978). Robustness? British Journal of Mathematical and Statistical Psychology, 31, 144-152.
5. Brown, S. H., Stout, J. D., Dalessio, A. T., & Crosby, M. M. (1988). Stability of validity indices through test score ranges. Journal of Applied Psychology, 73, 736-742.
6. Brown, S. P., & Peterson, R. A. (1993). Antecedents and consequences of salesperson job satisfaction: Meta-analysis and assessment of causal effects. Journal of Marketing Research, 30, 63-77.
7. Callender, J. C., Osburn, H. G., Greener, J. M., & Ashworth, S. (1982). Multiplicative validity generalization model: Accuracy of estimates as a function of sample size and mean, variance, and shape of the distribution of true validities. Journal of Applied Psychology, 67, 859-867.
8. Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49, 997-1003.
9. Cornwell, J. M., & Ladd, R. T. (1993). Power and accuracy of the Schmidt and Hunter meta-analytic procedures. Educational and Psychological Measurement, 53, 877-895.
10. Coward, W. M., & Sackett, P. R. (1990). Linearity of ability-performance relationships: A reconfirmation. Journal of Applied Psychology, 75, 297-300.
11. Fisher, J. (1959). The twisted pear and the prediction of behavior. Journal of Consulting Psychology, 23, 400-405.
12. Ghiselli, E. E. (1966). The validity of occupational aptitude tests. New York: Wiley.
13. Greener, J. M., & Osburn, H. G. (1980). Accuracy of corrections for restriction in range due to explicit selection in heteroscedastic and nonlinear distributions. Educational and Psychological Measurement, 40, 337-346.
14. Gross, A. L. (1982). Relaxing the assumptions underlying corrections for restriction of range. Educational and Psychological Measurement, 42, 795-801.
15. Gross, A. L., & Fleischman, L. E. (1983). Restriction of range corrections when both distribution and selection assumptions are violated. Applied Psychological Measurement, 7, 227-237.
16. Gross, A. L., & Fleischman, L. E. (1987). The correction for restriction of range and nonlinear regressions: An analytic study. Applied Psychological Measurement, 11, 211-217.
17. Guion, R. M. (1965). Personnel testing. New York: McGraw-Hill.
18. Harris, M. J., & Rosenthal, R. (1985). Mediation of interpersonal expectancy effects: 31 meta-analyses. Psychological Bulletin, 97, 363-386.
19. Hom, P. W., Caranikas-Walker, F., Prussia, G. E., & Griffeth, R. W. (1992). A meta-analytical structural equations analysis of a model of employee turnover. Journal of Applied Psychology, 77, 890-909.
20. Hunter, J. E., & Schmidt, F. L. (1990a). Dichotomization of continuous variables: The implications for meta-analysis. Journal of Applied Psychology, 75, 334-349.
21. Hunter, J. E., & Schmidt, F. L. (1990b). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.
22. Hunter, J. E., & Schmidt, F. L. (1994) Estimation of sampling error variance in the meta-analysis of correlations: Use of average correlation in the homogeneous case. Journal of Applied Psychology, 79, 171-177.
23. Johnson, J. W., & Sager, C. E. (1991, April). The robustness of range restriction correction due to incidental selection. Poster presented at the Sixth Annual Conference of the Society for Industrial and Organizational Psychology, St. Louis, MO.
24. Kemery, E. R., Mossholder, K. W., & Roth, L. (1987). The power of the Schmidt and Hunter additive model of validity generalization. Journal of Applied Psychology, 72, 30-37.
25. Koslowsky, M., & Sagie, A. (1993). Detecting moderators with meta-analysis: An evaluation and comparison of techniques. Personnel Psychology, 46, 629-639.
26. Koslowsky, M., & Sagie, A. (1994). Components of artifactual variance in meta-analytic research. Personnel Psychology, 47, 561-574.
27. Law, K. S., Schmidt, F. L., & Hunter, J. E. (1994). A test of two refinements in procedures for meta-analysis. Journal of Applied Psychology, 79, 978-986.
28. Lee, R., & Foley, P. P. (1986). Is the validity of a test constant throughout the test score range? Journal of Applied Psychology, 71, 641-644.
29. Lent, R. H., Aurbach, H. A., & Levin, L. S. (1971). Research design and validity assessment. Personnel Psychology, 24, 247-274.
30. Lindgren, B. W. (1993). Statistical theory (4th ed.). New York: Chapman & Hall.
31. Linn, R. L. (1968). Range restriction problems in the use of self-selected groups for test validation. Psychological Bulletin, 69, 69-73.
32. Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
33. Mendoza, J. L., & Reinhardt, R. N. (1991). Validity generalization procedures using sample-based estimates: A comparison of six procedures. Journal of Applied Psychology, 110, 596-610.
34. Micceri, T. (1989). The unicorn, the normal curve, and other improbable creatures. Psychological Bulletin, 105, 156-166.
35. Mone, M. A., Mueller, G. C., & Mauland, W. (1996). The perceptions and usage of statistical power in applied psychology and management research. Personnel Psychology, 49, 103-120.
36. Murphy, K. R. (1996). Individual differences and behavior in organizations: Much more than g. In K. R. Murphy (Ed.), Individual differences and behavior in organizations (pp. 3-30). San Francisco: Jossey-Bass.
37. Osburn, H. G., & Callender, J. (1992). A note on the sampling variance of the mean uncorrected correlation in meta-analysis and validity generalization. Journal of Applied Psychology, 77, 115-122.
38. Osburn, H. G., Callender, J. C., Greener, J. M., & Ashworth, S. (1983). Statistical power of tests of the situational specificity hypothesis in validity generalization studies: A cautionary note. Journal of Applied Psychology, 68, 115-122.
39. Paese, P. W., & Switzer, F. S. (1988). Validity generalization and hypothetical reliability distributions: A test of the Schmidt-Hunter procedure. Journal of Applied Psychology, 73, 267-274.
40. Pearlman, K., Schmidt, F. L., & Hunter, J. E. (1980). Validity generalization results for tests used to predict job proficiency and training success in clerical occupations. Journal of Applied Psychology, 65, 373-406.
41. Peters, L. H., Hartke, D. D., & Pohlmann, J. T. (1985). Fiedler's contingency theory of leadership: An application of the meta-analysis procedures of Schmidt and Hunter. Psychological Bulletin, 97 274-285.
42. Premack, S. L., & Hunter, J. E. (1988). Individual unionization decisions. Psychological Bulletin, 103, 223-234.
43. Raju, N. S., Burke, M. J., Normand, J., & Langlois, G. M. (1991). A new meta-analytic approach. Journal of Applied Psychology, 76, 432-446.
44. Raju, N. S., Pappas, S., & Williams, C. P. (1989). An empirical Monte Carlo test of the accuracy of the correlation, covariance, and regression slope models for assessing validity generalization. Journal of Applied Psychology, 74, 901-911.
45. Roth, L., & Sackett, P. R. (1991). Development and Monte Carlo evaluation of meta-analytic estimators for correlated data. Journal of Applied Psychology, 110, 318-327.
46. Sackett, P. R., Harris, M. M., & Orr, J. M. (1986). On seeking moderator variables in the meta-analysis of correlational data: A Monte Carlo investigation of statistical power and resistance to Type I error. Journal of Applied Psychology, 71, 302-310.
47. Sagie, A., & Koslowsky, M. (1993). Detecting moderators with meta-analysis: An evaluation and comparison of techniques. Personnel Psychology, 46, 629-640.
48. SAS Institute, Inc. (1995). SAS language, Release 6.10 [computer software]. Cary, NC: Author.
49. Schmidt, F. L., Hunter, J. E., & Outerbridge, A. N. (1986). Impact of job experience and ability on job knowledge, work sample performance, and supervisory ratings of job performance. Journal of Applied Psychology, 71, 432-439.
50. Shadish, W. R. (1996). Meta-analysis and the exploration of causal mediating processes: A primer of examples, methods, and issues. Psychological Methods, 1, 47-65.
51. Spector, P. E., & Levine, E. L. (1987). Meta-analysis for integrating study outcomes: A Monte-Carlo study of its susceptibility to Type I and Type II errors. Journal of Applied Psychology, 72, 3-9.
52. Switzer, F. S., Paese, P. W., & Drasgow, F. (1992). Bootstrap estimates of standard errors in validity generalization. Journal of Applied Psychology, 77, 123-129.
53. Tukey, J. W. (1977). Exploratory data analysis. Reading, MA: Addison-Wesley.
54. U.S. Department of Labor. (1979). Manual for the USES General Aptitude Test Battery: Section III: Development. Washington, DC: U.S. Government Printing Office.
55. Viswesvaran, C., & Ones, D. (1995). Theory testing: Combining psychometric meta-analysis and structural equations modeling. Personnel Psychology, 48, 865-885.
56. Wainer, H. (1982). Robust statistics: A survey and some prescriptions. In G. Keren (Ed.), Statistical and methodological issues in psychology and social sciences research (pp. 187-214). Hillsdale, NJ: Erlbaum.
57. Whitener, E. (1990). Confusion of confidence intervals and credibility intervals in meta-analysis. Journal of Applied Psychology, 75, 315-321.