Criterion-related validity in multiple-hurdle designs: Estimation and bias

Organizational Research Methods

Volumn 7, Issue 4, 2004, Pages 418-441

  Mendoza, Jorge L ^a   Bard, David E ^a   Mumford, Michael D ^a   Ang, Siew C ^a  

^a UNIVERSITY OF OKLAHOMA   (United States)

Author keywords

Corrections; Missing data; Multiple hurdle design; Range restriction; Selection

Indexed keywords

EID: 4944262787     PISSN: 10944281     EISSN: None     Source Type: Journal    
DOI: 10.1177/1094428104268752     Document Type: Review

References (53)

1. Allison, P. D. (2001). Missing data. Thousand Oaks, CA: Sage.
2. Amemiya, T. (1984). Tobit models: A survey. Journal of Econometrics, 24, 3-61.
3. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
4. Bobko, P. (1995). Correlation and regression: Principles and applications for industrial/ organizational psychology and management. New York: McGraw-Hill.
5. Chasteen, C. S., & Mendoza, J. L. (2003, April). Restriction in range issues in validation designs: Modern tools for old problems. Paper presented at the national meetings of the American Educational Research Association, Chicago, IL.
6. Collins, L. M., Schafer, J. L., & Kam, C. M. (2001). A comparison of inclusive and restrictive strategies in modern missing-data procedures. Psychological Methods, 6, 330-351.
7. Efron, B. (1994). Missing data, imputation, and the bootstrap. Journal of the American Statistical Association, 89, 463-475.
8. Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, 39(series B), 1-38.
9. Fan, X., & Thompson, B. (2001). Confidence intervals about score reliability coefficients, please: An EPM guidelines editorial. Educational and Psychological Measurement, 61, 517-531.
10. Ghiselli, E. E. (1964). Theory of psychological measurement. New York: McGraw-Hill.
11. Ghiselli, E. E. (1966). The validity of occupational aptitude tests. New York: John Wiley.
12. Graham, J. W., & Schafer, J. L. (1999). On the performance of multiple imputation for multivariate data with small sample size. In R. Hoyle (Ed.), Statistical strategies for small sample research (pp. 1-29). Thousand Oaks, CA: Sage.
13. Greener, J. M., & Osburn, H. G. (1979). An empirical study of the accuracy of corrections for restriction in range due to explicit selection. Applied Psychological Measurements, 5, 31-41.
14. Greener, J. M., & Osburn, H. G. (1980). Accuracy of corrections for restriction in range due to explicit in heteroscedastic and nonlinear distributions. Educational and Psychological Measurement, 40, 337-346.
15. Gross, A. L. (1990). A maximum likelihood approach to test validation with missing data and censored dependent variables. Psychometrika, 55, 533-549.
16. Gross, A. L., & McGanney, M. L. (1987). The restriction of range problem and nonignorable selection process. Journal of Applied Psychology, 72, 604-610.
17. Heckman, J. (1976). The common structure of statistical models of truncation, sample selection and limited dependent variables, and a simple estimator for such models. Annals of Economics and Social Measurement, 5, 475-492.
18. Hoffman, C. (1995).Applying range restriction corrections using published norms: Three case studies. Personnel Psychology, 48, 913-923.
19. Horton, N. J., & Lipsitz, S. R. (2001). Multiple imputation in practice: Comparison of software packages for regression models with missing variables. American Statistician, 55, 244-254.
20. James, L. R., Demaree, R., Muliak, S. A., & Mumford, M. D. (1988). Validity generalization: A rejoinder to Schmidt, Hunter, and Raju. Journal of Applied Psychology, 73, 443-452.
21. Kolmstetter, E. (2003). I-Os making an impact: TSA transportation security screener skill standards, selection system, and hiring process. Industrial-Organizational Psychologist, 40(4), 39-46.
22. Lawley, D. N. (1943). A note on Karl Pearson's selection formulae. Proceedings of the Royal Society of Edinburgh, 62, 28-30.
23. Li, K. H., Raghunathan, T. E., & Rubin, D. B. (1991). Large-sample significance levels from multiply imputed data using moment-based statistics and an F reference distribution. Journal of the American Statistical Association, 86, 1065-1073.
24. Linn, R. L., Harnisch, D. L., & Dunbar, S. B. (1981). Correction for range restriction: An empirical investigation of conditions resulting in conservative corrections. Journal of Applied Psychology, 66, 655-663.
25. Linn, R. T. (1968). Range restriction problems in the use of self-selected groups for test validation. Psychological Bulletin, 69, 69-73.
26. Little, R. J. A. (1995). Modeling the dropout mechanism in repeated-measures studies. Journal of the American Statistical Association, 90, 1112-1121.
27. Little, R. J. A., & Rubin, D. B (1987). Statistical analysis with missing data. New York: John Wiley.
28. Little, R. J. A., & Schenker, N. (1995). Missing data. In Arminger, G., Clogg, C. C., & Sobel, M. E. (Eds.), Handbook of statistical modeling for the social and behavioral sciences (pp. 39-75). New York: Plenum.
29. Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
30. McLachlan, G. J., & Krishnan, T. (1997). The EM algorithm and extensions. New York: John Wiley.
31. Mendoza, J. L. (1993). Fisher transformations for correlations corrected for selection and missing data. Psychometrika, 58, 601-615.
32. Mendoza, J. L., & Mumford, M. D. (1987). Corrections for attenuation and range restriction. Journal of Educational Statistics, 12, 282-293.
33. Mendoza, J. L., Stafford, K. L., & Stauffer, J. M. (2000). Large-sample confidence intervals for validity and reliability coefficients. Psychological Methods, 5, 356-369.
34. Meng, X. L., & Rubin, D. B. (1991). Using the EM to obtain asymptotic variance-covariance matrices: The SEM algorithm. Journal of the American Statistical Association, 86, 899-909.
35. Messick, S. J. (1998). Alternative models of assessment, uniform standards of validity. In M. D. Hakel (Ed.), Beyond multiple choice: Evaluating alternatives to traditional testing for selection (pp. 59-74). Mahwah, NJ: Lawrence Erlbaum.
36. Olkin, I., & Finn, J. D. (1995). Correlation redux. Psychological Bulletin, 118, 155-164.
37. Ones, D. S., & Viswesvaran, C. (2003). Job-specific applicant pools and national norms for personality scales implications for range-restriction corrections invalidation research. Journal of Applied Psychology, 88, 570-577.
38. Pearson, K. (1903). Mathematical contributions to the theory of evolution XI: On this influence of natural selection on the variability and correlation of organs. Philosophical Transactions of the Royal Society, London, Series A, 200, 1-66.
39. Rao, J. N. K., & Shao, J. (1992). Jackknife variance estimation with survey data under hot-deck imputation. Biometrika, 79, 811-822.
40. Ree, M. J., Carretta, T. R., Earles, J. A., & Albert, W. (1994). Sign changes when correcting for range restriction: A note on Pearson's and Lawley's selection formulas. Journal of Applied Psychology, 79, 298-301.
41. Rubin, D. B. (1976). Inference and missing data. Biometrika, 63, 581-592.
42. Rubin, D. B. (1978). Multiple imputations in sample surveys-A phenomenological Bayesian approach to nonresponse. Proceedings of the Survey Research Methods Section American Statistical Association, 20-34.
43. Rubin, D. B. (1987). Multiple imputation for nonresponse in surveys. New York: John Wiley.
44. Rubin, D. B. (1996). Multiple imputation after 18+ years. Journal of the American Statistical Association, 91, 473-489.
45. Sackett, P. R., Laczo, R. M., & Arvey, R. D. (2002). The effects of range restriction on estimates of criterion interrater reliability: Implications for validation research. Personnel Psychology, 55, 807-825.
46. Sackett, P. R., & Ostgard, D. L. (1994). Job-specific applicant pools and national norms for cognitive ability tests: Implications for range restriction corrections in validation research. Journal of Applied Psychology, 79, 680-684.
47. Sackett, P. R., & Yang, H. (2000). Correction for range restriction: An expanded typology. Journal of Applied Psychology, 85, 112-118.
48. Schafer, J. L. (1997). Analysis of incomplete multivariate data. London: Chapman & Hall.
49. Schafer, J. L., & Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7, 147-177.
50. Schafer, J. L., & Olsen, M. K. (1998). Multiple imputation for multivariate missing-data problems: A data analyst's perspective. Multivariate Behavioral Research, 33, 545-571.
51. Schafer, J. L., & Schenker, N. (2000). Inferences with imputed conditional means. Journal of the American Statistical Association, 95, 144-154.
52. Switzer, F. S., Roth, P. L., & Switzer, D.M. (1998). Systematic data loss in HRM settings: A Monte Carlo analysis. Journal of Management, 24, 763-777.
53. Tanner, M. A., & Wong, W.H. (1987). The calculation of the posterior distributions by data augmentation. Journal of the American Statistical Association, 82, 528-550.