Test bias, differential prediction, and a revised approach for determining the suitability of a predictor in a selection context

Organizational Research Methods

Volumn 12, Issue 4, 2009, Pages 738-761

  Meade, Adam W ^a   Fetzer, Michael ^b  

^a NORTH CAROLINA STATE UNIVERSITY   (United States)

^b PreVisor ^*  (United Kingdom)

Author keywords

Differential prediction; Gender bias; Group bias; Racial bias; Selection bias; Test bias

Indexed keywords

EID: 70350065520     PISSN: 10944281     EISSN: 15527425     Source Type: Journal    
DOI: 10.1177/1094428109331487     Document Type: Article

References (69)

1. Aguinis, H. (1995). Statistical power problems with moderated multiple regression in management research. Journal of Management, 21, 1141-1158.
2. Aguinis, H., & Smith, M.A. (2007). Understanding the impact of test validity and bias on selection errors and adverse impact in human resource selection. Personnel Psychology, 60, 165-199.
3. American Educational Research Association/American Psychological Association/ National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
4. Austin, J.T., & Crespin, T.R. (2006). Problems of criteria in industrial and organizational psychology: progress, pitfalls, and prospects. In W. J. Bennett, C. E. Lance, & D. J. Woehr (Eds.). Performance measurement: Current perspectives and future challenges (pp. 9-48). Mahwah, NJ: Lawrence Erlbaum Associates Publishers.
5. Bartlett, C.J., Bobko, P., Mosier, S.B., & Hannan, R. (1978). Testing for fairness with a moderated multiples regression strategy. Personnel Psychology, 31, 233-242.
6. Bartlett, C.J., & O'Leary, B. (1969). A differential prediction model to moderate the effects of heterogeneous groups in personnel selection and classification. Personnel Psychology, 22, 1-19.
7. Bennett, W.J., Lance, C.E., & Woehr, D.J. (2006). Performance measurement: Current perspectives and future challenges. Mahwah, NJ: Lawrence Erlbaum Associates Publishers.
8. Bingham, W.V. (1926). Measures of occupational success. Harvard Business Review, 5, 1-10.
9. Binning, J.F., & Barrett, G.V. (1989). Validity of personnel decisions: A conceptual analysis of the inferential and evidential bases. Journal of Applied Psychology, 74, 478-494.
10. Camilli, G., & Shepard, L.A. (1994). Methods for identifying biased test items. Thousand Oaks, CA: Sage.
11. Cleary, T.A. (1968). Test bias: Prediction of grades of Negro and White students in integrated colleges. Journal of Educational Measurement, 5, 115-124.
12. Cohen, J., & Cohen, P. (1975). Applied multiple regression/correlation analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum.
13. Crocker, L.M., & Algina, J. (1986). Introduction to classical and modern test theory. New York: Harcourt Brace Jovanovich College Publishers.
14. Darlington, R.B. (1971). Another look at "cultural fairness". Journal of Educational Measurement, 8, 71-82.
15. Dunnette, M.D. (1963). A note on "the" criterion. Journal of Applied Psychology, 47, 251-254.
16. Eagly, A.H., Makhijani, M.G., & Klonsky, B.G. (1992). Gender and the evaluation of leaders: A meta-analysis. Psychological Bulletin, 111, 3-22.
17. Erwin, F. (2001a). Customer Service and Clerical Potential Index (CSCPI) Technical Manual. Developed by Richardson, Bellows, Henry & Company (RBH). Minneapolis, MN: PreVisor, Incorporated.
18. Erwin, F. (2001b). Test of Learning Ability (TLA) Technical Manual. Developed by Richardson, Bellows, Henry & Company (RBH). Minneapolis, MN: PreVisor, Incorporated.
19. Ford, J.K., Kraiger, K., & Schechtman, S.L. (1986). Study of race effects in objective indices and subjective evaluations of performance: A meta-analysis of performance criteria. Psychological Bulletin, 99, 330-337.
20. French, B.F., & Finch, W.H. (2008). Multigroup confirmatory factor analysis: Locating the invariant referent sets. Structural Equation Modeling, 15, 96-113.
21. Guion, R.M. (1998). Assessment, measurement, and prediction for personnel decisions. Mahwah, NJ: Lawrence Erlbaum Associates, Publishers.
22. Hartnett, R.T., & Willingham, W.W. (1980). The criterion problem: What measure of success in graduate education? Applied Psychological Measurement, 4, 281-291.
23. Hulin, C.L., Drasgow, F., & Parsons, C. (1983). Item response theory: Application to psychological measurement. Homewood, IL: Dow Jones-Irwin.
24. Humphreys, L.G. (1986). An analysis and evaluation of test and item bias in the prediction context. Journal of Applied Psychology, 71, 327-333.
25. Hunter, J.E., & Schmidt, F.L. (1976). Critical analysis of the statistical and ethical implications of various definitions of test bias. Psychological Bulletin, 83, 1053-1071.
26. James, L.R. (1980). The unmeasured variables problem in path analysis. Journal of Applied Psychology, 65, 415-421.
27. Johnson, E.C., Meade, A.W., & DuVernet, A.M. (in press). The role of referent indicators in tests of measurement invariance. Structural Equation Modeling.
28. Kraiger, K., & Ford, J.K. (1985). A meta-analysis of ratee race effects in performance ratings. Journal of Applied Psychology, 70, 56-65.
29. Kraiger, K., & Ford, J.K. (1990). The relation of job knowledge, job performance, and supervisory ratings as a function of ratee race. Human Performance, 3, 269-279.
30. Landy, F.J., Shankster, L.J., & Kohler, S.S. (1994). Personnel selection and placement. Annual Review of Psychology, 45, 261-296.
31. Latham, G.P., & Wexley, K.N. (1994). Increasing productivity through performance appraisal. Reading, MA: Addison Wesley.
32. Lautenschlager, G.J., & Mendoza, J.L. (1986). A step-down hierarchical multiple regression analysis for examining hypotheses about test bias in prediction. Applied Psychological Measurement, 10, 133-139.
33. Ledvinka, J. (1979). The statistical definition of fairness in the federal selection guidelines and its implications for minority employment. Personnel Psychology, 32, 551-562.
34. Linn, R.L. (1973). Fair test use in selection. Review of Educational Research, 43, 139-161.
35. Linn, R.L., & Werts, C.E. (1971). Considerations for studies of test bias. Journal of Educational Measurement, 8, 1-4.
36. Lord, F.M. (1980). Applications of item response theory to practical testing problems (). Hillsdale, NJ: Lawrence Erlbaum Associates.
37. Lord, F.M., & Novick, M.R. (1968). Statistical theories of mental test scores. Oxford, England: Addison-Wesley.
38. Lyness, K.S., & Heilman, M.E. (2006). When fit is fundamental: Performance evaluations and promotions of upper-level female and male managers. Journal of Applied Psychology, 91, 777-785.
39. Mauro, R. (1990). Understanding L.O.V.E. (left out variables error): A method for estimating the effects of omitted variables. Psychological Bulletin, 108, 314-329.
40. McKay, P.F., & McDaniel, M.A. (2006). A reexamination of black-white mean differences in work performance: More data, more moderators. Journal of Applied Psychology, 91, 538-554.
41. McNemar, Q. (1975). On so-called test bias. American Psychologist, 30, 848-851.
42. Meade, A.W., & Lautenschlager, G.J. (2004). A comparison of item response theory and confirmatory factor analytic methodologies for establishing measurement equivalence/invariance. Organizational Research Methods, 7, 361-388.
43. Motowidlo, S.J. (2003). Job performance. In W. C. Borman, D. R. Ilgen, & R. J. Klimoski (Eds.). Handbook of psychology: Vol. 12, Industrial and organizational psychology (pp. 39-53). Hoboken, NJ: John Wiley & Sons.
44. Murphy, K.R., & Davidshofer, C.O. (2001). Psychological testing: Principles and applications (5th ed.). Upper Saddle River, NJ: Pearson Prentice Hall.
45. Navas-Ara, M.J., & Gomez-Benito, J. (2002). Effects of ability scale purification on the identification of DIF. European Journal of Psychological Assessment, 18, 9-15.
46. Neisser, U., Boodoo, G., Bouchard, T.J.J., Boykin, A.W., Brody, N., & Ceci, S.J.,et al.(1996). Intelligence: Knowns and unknowns. American Psychologist, 51, 77-101.
47. Pazy, A., & Oron, I. (2001). Sex proportion and performance evaluation among high-ranking military officers. Journal of Organizational Behavior, 22, 689-702.
48. Pulakos, E.D., White, L.A., Oppler, S.H., & Borman, W.C. (1989). Examination of race and sex effects on performance ratings. Journal of Applied Psychology, 74, 770-780.
49. Rensvold, R.B., & Cheung, G.W. (2001). Testing for metric invariance using structural equation models: Solving the standardization problem. In C. A. Schriesheim, & L. L. Neider (Eds.). Research in management (Vol. 1): Equivalence in measurement (pp. 21-50). Greenwich, CT: Information Age.
50. Roth, P.L., Bevier, C.A., Bobko, P., Switzer, F.S.I., & Tyler, P. (2001). Ethnic group differences in cognitive ability in employment and educational settings: A meta-analysis. Personnel Psychology, 54, 297-330.
51. Roth, P.L., Huffcutt, A.I., & Bobko, P. (2003). Ethnic group differences in measures of job performance: A new meta-analysis. Journal of Applied Psychology, 88, 694-706.
52. Rotundo, M., & Sackett, P.R. (1999). Effect of rater race on conclusions regarding differential predication in cognitive ability tests. Journal of Applied Psychology, 84, 815-822.
53. Sackett, P.R., & DuBois, C.L. (1991). Rate-ratee race effects on performance evaluation: Challenging meta-analytic conclusions. Journal of Applied Psychology, 76, 873-877.
54. Sackett, P.R., Laczo, R.M., & Lippe, Z.P. (2003). Differential prediction and the use of multiple predictors: The omitted variables problem. Journal of Applied Psychology, 88, 1046-1056.
55. Sackett, P.R., & Wilk, S.L. (1994). Within-group norming and other forms of score adjustment in preemployment testing. American Psychologist, 49, 929-954.
56. Schmidt, F.L., & Hunter, J.E. (1981). Employment testing: Old theories and new research findings. American Psychologist, 36, 1128-1137.
57. Schmidt, F.L., & Hunter, J.E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124, 262-274.
58. Schmitt, N., Clause, C.S., & Pulakos, E.D. (1996). Subgroup differences associated with different measures of some common job-relevant constructs. In C. L. Cooper, & I. T. Robertson (Eds.). International review of industrial and organizational psychology (Vol. 11, pp. 115-139). New York: Wiley.
59. Society for Industrial and Organizational Psychology. (2003). Principles for the validation and use of personnel selection procedures. Bowling Green, OH: Society for Industrial and Organizational Psychology.
60. Stauffer, J.M., & Buckley, M.R. (2005). The existence and nature of racial bias in supervisory ratings. Journal of Applied Psychology, 90, 586-591.
61. Swim, J., Borgida, E., Maruyama, G., & Myers, D.G. (1989). Joan McKay versus John McKay: Do gender stereotypes bias evaluations? Psychological Bulletin, 105, 409-429.
62. Terris, W. (1997). The traditional regression model for measuring test bias is incorrect and biased against minorities. Journal of Business and Psychology, 12, 25-37.
63. Thayer, P.W. (1992). Construct validation: Do we understand our criteria? Human Performance, 5, 97-108.
64. Thorndike, R.L. (1971). Concepts of culture-fairness. Journal of Educational Measurement, 8, 63-70.
65. U.S. Equal Employment Opportunity Commission, Civil Service Commission, Department of Labor, and Department of Justice. (1978). Uniform guidelines on employee selection procedures. Federal Register, 43, 38290-38315.
66. Vandenberg, R.J., & Lance, C.E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4-69.
67. Waldman, D.A., & Avolio, B.J. (1991). Race effects in performance evaluations: Controlling for ability, education, and experience. Journal of Applied Psychology, 76, 897-901.
68. Wang, W.-C., & Yeh, Y.-L. (2003). Effects of anchor item methods on differential item functioning detection with the likelihood ratio test. Applied Psychological Measurement, 27, 479-498.
69. Wendelken, D.J., & Inn, A. (1981). Nonperformance influences on performance evaluations: A laboratory phenomenon? Journal of Applied Psychology, 66, 149-158.