High-stakes testing in employment, credentialing, and higher education: Prospects in a post-affirmative-action world

American Psychologist

Volumn 56, Issue 4, 2001, Pages 302-318

  Sackett, Paul R ^a   Schmitt, Neal ^b   Ellingson, Jill E ^b   Kabin, Melissa B ^b  

^a UNIVERSITY OF MINNESOTA   (United States)

^b NONE

Author keywords

[No Author keywords available]

Indexed keywords

ACCREDITATION; APTITUDE TEST; ARTICLE; CULTURAL ANTHROPOLOGY; EDUCATION; EMPLOYMENT; HUMAN; STANDARD;

APTITUDE TESTS; CREDENTIALING; CULTURAL DIVERSITY; EDUCATION; EMPLOYMENT; HUMANS;

EID: 84978049716     PISSN: 0003066X     EISSN: None     Source Type: Journal    
DOI: 10.1037/0003-066X.56.4.302     Document Type: Article

References (117)

1. Adarand Constructors, Inc. v. Pena, 115 S. Ct. 2097, 2113 (1995).
2. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Psychological Association.
3. Applebee, A. N., Langer, J. A., Jenkins, L. B., Mullis, I. V. S., & Foertsch, M. A. (1990). Learning to write in our nation's schools: Instruction and achievement in 1988 at grades 4, 8, and 12 (NAEP Rep. No. 19-W-02). Princeton, NJ: Educational Testing Service.
4. Asher, J. J., & Sciarrino, J. A. (1974). Realistic work sample tests: A review. Personnel Psychology, 27, 519-533.
5. Bangert-Downs, R. L., Kulik, J. A., & Kulik, C-L. C. (1983). Effects of coaching programs on achievement test scores. Review of Educational Research, 53, 571-585.
6. Barrick, M. R., & Mount, M. K. (1991). The Big-Five personality dimensions in job performance: A meta-analysis. Personnel Psychology, 44, 1-26.
7. Berk, R. A. (1982). Handbook of methods for detecting test bias. Baltimore: Johns Hopkins University Press.
8. Bobko, P., Roth, P. L., & Potosky, D. (1999). Derivation and implications of a meta-analytic matrix incorporating cognitive ability, alternative predictors, and job performance. Personnel Psychology, 52, 561-590.
9. Bond, L. (1995). Unintended consequences of performance assessment: Issues of bias and fairness. Educational Measurement: Issues and Practice, 14, 21-24.
10. Chan, D., & Schmitt, N. (1997). Video-based versus paper-and-pencil method of assessment in situational judgment tests: Subgroup differences in test performance and face validity perceptions. Journal of Applied Psychology, 82, 143-159.
11. Chan, D., Schmitt, N., DeShon, R. P., Clause, C. C., & Delbridge, K. (1997). Reactions to cognitive ability tests: The relationships between race, test performance, face validity perceptions, and test-taking motivation. Journal of Applied Psychology, 82, 300-310.
12. Chan, D., Schmitt, N., Sacco, J. M., & DeShon, R. P. (1998). Understanding pretest and posttest reactions to cognitive ability and personality measures. Journal of Applied Psychology, 83, 471-485.
13. City of Richmond v. J. A. Croson Co., 488 U.S. 469 (1989).
14. Cole, N. S. (1981). Bias in testing. American Psychologist, 36, 1067-1077.
15. D'Costa, A. G. (1993). The impact of courts on teacher competence testing. Theory into Practice: Assessing Tomorrow's Teachers, 32, 104-112.
16. De Corte, W. (1999). Weighing job performance predictors to both maximize the quality of the selected workforce and control the level of adverse impact. Journal of Applied Psychology, 84, 695-702.
17. DerSimonian, R., & Laird, N. (1983). Evaluating the effect of coaching on SAT scores: A meta-analysis. Harvard Educational Review, 18, 694-734.
18. DeShon, R. P., Smith, M., Chan, D., & Schmitt, N. (1998). Can adverse impact on cognitive ability and personality tests be reduced by presenting problems in a social context? Journal of Applied Psychology, 83, 438-451.
19. Dunbar, S. B., Koretz, D. M., & Hoover, H. D. (1991). Quality control in the development and use of performance assessments. Applied Measurement in Education, 4, 289-303.
20. Dutt-Doner, K., & Gilman, D. A. (1998). Students react to portfolio assessment. Contemporary Education, 69, 159-165.
21. Dwyer, C. A., & Ramsey, P. A. (1995). Equity issues in teacher assessment. In M. T. Nettles & A. L. Nettles (Eds.), Equity and excellence in educational testing and assessment (pp. 327-342). Boston: Kluwer Academic.
22. Evans, F. R., & Reilly, R. R. (1973). A study of test speededness as a potential source of bias in the quantitative score of the admission test for graduate study in business. Research in Higher Education, 1, 173-183.
23. Ford, J. K., Kraiger, K., & Schechtman, S. L. (1986). Study of race effects in objective indices and subjective evaluations of performance: A meta-analysis of performance criteria. Psychological Bulletin, 99, 330-337.
24. Freedle, T., & Kostin, I. (1990). Item difficulty of four verbal item types and an index of differential item functioning for Black and White examinees. Journal of Educational Measurement, 27, 329-343.
25. Freedle, R., & Kostin, I. (1997). Predicting Black and White differential item functioning in verbal analogy performance. Intelligence, 24, 417-444.
26. Frierson, H. T. (1986). Enhancing minority college students' performance on educational tests. Journal of Negro Education, 55, 38-45.
27. Goldstein, H. W., Yusko, K. P., Braverman, E. P., Smith, D. B., & Chung, B. (1998). The role of cognitive ability in the subgroup differences and incremental validity of assessment center exercises. Personnel Psychology, 51, 357-374.
28. Harmon, M. (1991). Fairness in testing: Are science education assessments biased? In G. Kulm & S. M. Malcom (Eds.), Science assessment in the service of reform (pp. 31-54). Washington, DC: American Association for the Advancement of Science.
29. Hartigan, J. A., & Wigdor, A. K. (1989). Fairness in employment testing. Washington. DC: National Academy Press.
30. Hattrup, K., Rock, J., & Scalia, C. (1997). The effects of varying conceptualizations of job performance on adverse impact, minority hiring, and predicted performance. Journal of Applied Psychology, 82, 656-664.
31. Helms, J. E. (1992). Why is there no study of cultural equivalence in standardized cognitive ability testing? American Psychologist, 47, 1083-1101.
32. Hopwood v. State of Texas, 78 F. 3d 932, 948 (5th Cir. 1996).
33. Hough, L. M. (1984). Development and evaluation of the "accomplishment record" method of selecting and promoting professionals. Journal of Applied Psychology, 69, 135-146.
34. Hough, L. M., Keyes, M. A., & Dunnette, M. D. (1983). An evaluation of three "alternative" selection procedures. Personnel Psychology, 36, 261-276.
35. Hunter, J. E., & Hunter, R. F. (1984). Validity and utility of alternative predictors of job performance. Psychological Bulletin, 96, 72-88.
36. Ironson, G. H., & Subkoviak, M. J. (1979). A comparison of several methods of assessing item bias. Journal of Educational Measurement, 16, 209-225.
37. Jaeger, R. M. (1996a). Conclusions on the technical measurement quality of the 1995-1996 operational version of the National Board for Professional Teaching Standards' Early Childhood Generalist Assessment. Center for Educational Research and Evaluation, University of North Carolina at Greensboro.
38. Jaeger, R. M. (1996b). Conclusions on the technical measurement quality of the 1995-1996 operational version of the National Board for Professional Teaching Standards' Middle Childhood Generalist Assessment. Center for Educational Research and Evaluation, University of North Carolina at Greensboro.
39. Jensen, A. R. (1980). Bias in mental testing. New York: Free Press.
40. Kier, F. J., & Davenport, D. S. (1997). Ramifications of Hopwood v. Texas on the process of applicant selection in APA-accredited professional psychology programs. Professional Psychology: Research and Practice, 28, 486-491.
41. Klein, S. P. (1983). An analysis of the relationship between trial practice skills and bar examination results. Unpublished manuscript.
42. Klein, S. P., & Bolus, R. E. (1982). An analysis of the relationship between clinical legal skills and bar examination results. Unpublished manuscript.
43. Klein, S. P., Jovanovic, J., Stecher, B. M., McCaffrey, D., Shavelson, R. J., Haertel, E., Solano-Flores, G., & Comfort, K. (1997). Gender and racial/ethnic differences on performance assessments in science. Educational Evaluation and Policy Analysis, 19, 83-97.
44. Koenig, J. A., & Leger, K. F. (1997). A comparison of retest performances and test-preparation methods for MCAT examinees grouped by gender and race-ethnicity. Academic Medicine, 72, S100-S102.
45. Koenig, J. A., & Mitchell, K. J. (1988). An interim report on the MCAT essay pilot project. Journal of Medical Education, 63, 21-29.
46. Lee, O. (1999). Equity implications based on the conceptions of science achievement in major reform documents. Review of Educational Research, 69, 83-115.
47. Linn, R. L. (1993). Educational assessment: Expanded expectations and challenges. Educational Evaluation and Policy Analysis, 15, 1-16.
48. LeMahieu, P. G., Gitomer, D. H., & Eresh, J. T. (1995). Portfolios in large-scale assessment: Difficult but not impossible. Educational Measurement: Issues and Practice, 14, 11-16, 25-28.
49. Lynn, R. (1996). Racial and ethnic differences in intelligence in the U.S. on the Differential Ability Scale. Personality and Individual Differences, 20, 271-273.
50. McCauley, C. D., & Mendoza, J. (1985). A simulation study of item bias using a two-parameter item response model. Applied Psychological Measurement, 9, 389-400.
51. McDaniel, M. A., Schmidt, F. L., & Hunter, J. E. (1988). A meta-analysis of the validity of methods for rating training and experience in personnel selection. Personnel Psychology, 41, 283-309.
52. Medley, D. M., & Quirk, T. J. (1974). The application of a factorial design to the study of cultural bias in general culture items on the National Teacher Examination. Journal of Educational Measurement, 11, 235-245.
53. Mehrens, W. A. (1989). Using test scores for decision making. In B. R. Gifford (Ed.), Test policy and text performance: Education, language, and culture (pp. 93-99). Boston: Kluwer Academic.
54. Mehrens, W. A. (1999). The CBEST saga: Implications for licensure and employment testing. The Bar Examiner, 68, 23-32.
55. Messick, S. M., & Jungeblut, A. (1981). Time and method in coaching for the SAT. Psychological Bulletin, 89, 191-216.
56. Miller-Jones, D. (1989). Culture and testing. American Psychologist, 44, 360-366.
57. Millman, J., Mehrens, W. A., & Sackett, P. R. (1993). An evaluation of the New York State Bar Examination. Unpublished manuscript.
58. Mishkin, P. J. (1996). Foreword: The making of a turning point - Metro and Adarand. California Law Review, 84, 875-886.
59. Motowidlo, S. J., Dunnette, M. D., & Carter, G. W. (1990). An alternative selection procedure: The low-fidelity simulation. Journal of Applied Psychology, 75, 640-647.
60. Mount, M. K., & Barrick, M. R. (1995). The Big Five personality dimensions: Implications for research and practice in human resources management. In G. Ferris (Ed.), Research in personnel and human resources management (Vol. 13, pp. 153-200). Greenwich, CT: JAI Press.
61. National Academy of Sciences. (1982). Ability testing: Uses, consequences, and controversies (Vol. 1). Washington, DC: National Academy Press.
62. Neill, M. (1995). Some prerequisites for the establishment of equitable, inclusive multicultural assessment systems. In M. T. Nettles & A. L. Nettles (Eds.), Equity and excellence in educational testing and assessment (pp. 115-157). Boston: Kluwer Academic.
63. Neisser, U., Boodoo, G., Bouchard, T. J., Jr., Boykin, A. W., Brody, N., Ceci, S. J., Halpern, D. F., Loehlin, J. C., Perloff, R., Sternberg, R. J., & Urbina, S. (1996). Intelligence: Knowns and unknowns. American Psychologist, 51, 77-101.
64. O'Connor, M. C. (1989). Aspects of differential performance by minorities on standardized tests: Linguistic and sociocultural factors. In B. R. Gifford (Ed.), Test policy and test performance: Education, language, and culture (pp. 129-181). Boston: Kluwer Academic.
65. O'Neil, H. F., & Brown, R. S. (1997). Differential effects of question formats in math assessment on metacognition and effect (Tech. Rep. No. 449). Los Angeles: University of California. National Center for Research on Evaluation, Standards, and Student Testing.
66. Ones, D. S., Viswesvaran, C., & Schmidt, F. L. (1993). Comprehensive meta-analysis of integrity test validities: Findings and implications for personnel selection and theories of job performance. Journal of Applied Psychology, 78, 656-664.
67. Pear, R. (1996, November 6). The 1996 elections: The nation - the states. The New York Times, p. B7.
68. Powers, D. E. (1987). Who benefits most from preparing for a "coachable" admissions test? Journal of Educational Measurement, 24, 247-262.
69. Powers, D. E., & Swinton, S. S. (1982). The effects of self-study of test familiarization materials for the analytical section of the GRE Aptitude Test (GRE Board Research Report GREB No. 79-9). Princeton, NJ: Educational Testing Service.
70. Powers, D. E., & Swinton, S. S. (1984). Effects of self-study for coachable test item types. Journal of Educational Psychology, 76, 266-278.
71. Pulakos, E. D., & Schmitt, N. (1996). An evaluation of two strategies for reducing adverse impact and their effects on criterion-related validity. Human Performance, 9, 241-258.
72. Robertson, I. T., & Kandola, R. S. (1982). Work sample tests: Validity, adverse impact, and applicant reaction. Journal of Occupational Psychology, 55, 171-183.
73. Ryan, A. M., Ployhart, R. E., & Friedel, L. A. (1998). Using personality testing to reduce adverse impact: A cautionary note. Journal of Applied Psychology, 83, 298-307.
74. Ryan, A. M., Ployhart, R. E., Greguras, G. J., & Schmit, M. J. (1998). Test preparation programs in selection contexts: Self-selection and program effectiveness. Personnel Psychology, 51, 599-622.
75. Ryer, J. A., Schmidt, D. B., & Schmitt, N. (1999, April). Candidate orientation programs: Effects on test scores and adverse impact. Paper presented at the annual conference of the Society for Industrial and Organizational Psychology, Atlanta, GA.
76. Sacco, J. M., Scheu, C. R., Ryan, A. M., Schmitt, N., Schmidt, D. B., & Rogg, K. L. (2000). Reading level and verbal test scores as predictors of subgroup differences and validities of situational judgment tests. Unpublished manuscript.
77. Sackett, P. R. (1998). Performance assessment in education and professional certification: Lessons for personnel selection. In M. D. Hakel (Ed.), Beyond multiple-choice: Evaluating alternatives to traditional testing for selection (pp. 113-129). Mahwah, NJ: Erlbaum.
78. Sackett, P. R., Burris, L. R., & Ryan, A. M. (1989). Coaching and practice effects in personnel selection. In C. L. Cooper & I. Robertson (Eds.), International review of industrial and organizational psychology 1989. London: Wiley.
79. Sackett, P. R., & Ellingson, J. E. (1997). The effects of forming multipredictor composites on group differences and adverse impact. Personnel Psychology, 50, 707-722.
80. Sackett, P. R., & Wilk, S. L. (1994). Within-group norming and other forms of score adjustment in preemployment testing. American Psychologist, 49, 929-954.
81. Salgado, J. F. (1997). The five factor model of personality and job performance in the European community. Journal of Applied Psychology, 82, 30-43.
82. Scarr, S. (1981). Race, social class, and individual differences in I.Q. Hillsdale. NJ: Erlbaum.
83. Scheuneman, J. (1987). An experimental, exploratory study of causes of bias in test items. Journal of Educational Measurement, 24, 97-118.
84. Scheuneman, J., & Gerritz, K. (1990). Using differential item functioning procedures to explore sources of item difficulty and group performance characteristics. Journal of Educational Measurement, 27, 109-131.
85. Scheuneman, J., & Grima, A. (1997). Characteristics of quantitative word items associated with differential performance for female and Black examinees. Applied Measurement in Education, 10, 299-319.
86. Schmeiser, C. B., & Ferguson, R. L. (1978). Performance of Black and White students on test materials containing content based on Black and White cultures. Journal of Educational Measurement, 15, 193-200.
87. Schmidt, F. L. (1988). The problem of group differences in ability test scores in employment selection. Journal of Vocational Behavior, 33, 272-292.
88. Schmidt, F. L., Greenthal, A. L., Hunter, J. E., Berner, J. G., & Seaton, F. W. (1977). Job sample vs. paper-and-pencil trades and technical tests: Adverse impact and examinee attitudes. Personnel Psychology, 30, 187-196.
89. Schmidt, F. L., & Hunter, J. E. (1981). Employment testing: Old theories and new research findings. American Psychologist, 36, 1128-1137.
90. Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124, 262-274.
91. Schmidt, F. L., Kaplan, J. R., Bemis, S. E., Decuir, R., Dunn, L., & Antone, L. (1979). The behavioral consistency method of unassembled examining (TM-79-21). Washington, DC: U.S. Office of Personnel Management, Personnel Research and Development Center.
92. Schmidt, F. L., Mack, M. J., & Hunter, J. E. (1984). Selection utility in the occupation of U.S. park ranger for three modes of test use. Journal of Applied Psychology, 69, 490-497.
93. Schmit, M. J. (1994). Pre-employment processes and outcomes, applicant belief systems, and minority-majority group differences. Unpublished doctoral dissertation, Bowling Green State University.
94. Schmit, M. J., & Ryan, A. M. (1997). Applicant withdrawal: The role of test-taking attitudes and racial differences. Personnel Psychology, 50, 855-876.
95. Schmitt, A. P., & Dorans, N. J. (1990). Differential item functioning for minority examinees on the SAT. Journal of Educational Measurement, 27, 67-81.
96. Schmitt, N., Clause, C. S., & Pulakos, E. D. (1996). Subgroup differences associated with different measures of some job-relevant constructs. In C. R. Cooper & I. T. Robertson (Eds.), International review of industrial and organizational psychology (Vol. 11, pp. 115-140). New York: Wiley.
97. Schmitt, N., Gooding, R. Z., Noe, R. A., & Kirsch, M. P. (1984). Meta-analyses of validity studies published between 1964 and 1982 and the investigation of study characteristics. Personnel Psychology, 37, 407-422.
98. Schmitt, N., & Mills, A. E. (in press). Traditional tests and simulations: Minority and majority performance and test validities. Journal of Applied Psychology.
99. Schmitt, N., Rogers, W., Chan, D., Sheppard, L., & Jennings, D. (1997). Adverse impact and predictive efficiency of various predictor combinations. Journal of Applied Psychology, 82, 719-730.
100. Smith, F. D. (1991). Work samples as measures of performance. In A. K. Wigdor & B. G. Green Jr. (Eds.), Performance assessment for the workplace (pp. 27-52). Washington: DC: National Academy Press.
101. Stecher, B. M., & Klein, S. P. (1997). The cost of science performance assessments in large-scale testing programs. Educational Evaluation and Policy Analysis, 19, 1-14.
102. Steele, C. M. (1997). A threat in the air: How stereotypes shape intellectual identity and performance. American Psychologist, 52, 613-629.
103. Steele, C. M., (1999, August). Thin ice:" Stereotype threat" and Black college students. Atlantic Monthly, 284(3), 44-54.
104. Steele, C. M., & Aronson, J. (1995). Stereotype threat and the intellectual test performance of African Americans. Journal of Personality and Social Psychology, 69, 797-811.
105. Stokes, G. S., Mumford, M. D., & Owens, W. A. (1994). Biodata handbook. Palo Alto, CA: Consulting Psychologists Press.
106. Supovitz, J. A., & Brennan, R. T. (1997). Mirror, mirror on the wall, which is the fairest test of all? An examination of the equitability of portfolio assessment relative to standardized test. Harvard Educational Review, 67, 472-506.
107. Thornton, G. C., & Byham, W. C. (1982). Assessment centers and managerial performance. New York: Academic Press.
108. Verhovek, S. H., & Ayres, B. D., Jr. (1998, November 4). The 1998 elections: The nation - referendums. The New York Times, p. B2.
109. Weekly, J. A., & Jones, C. (1997). Video-based situational testing. Personnel Psychology, 50, 25-49.
110. Welch, C., Doolittle, A., & McLarty, J. (1989). Differential performance on a direct measure of writing skills for Black and White college freshmen (ACT Research Reports, Tech. Rep. No. 89-8). Iowa City, IA: ACT.
111. Whaley, A. L. (1998). Issues of validity in empirical tests of stereotype threat theory. American Psychologist, 53, 679-680.
112. White, E. M., & Thomas, L. L. (1981). Racial minorities and writing skills assessment in the California State University and colleges. College English, 43, 276-283.
113. Whitney, D. J., & Schmitt, N. (1997). Relationship between culture and responses to biodata employment items. Journal of Applied Psychology, 82, 113-129.
114. Wightman, L. F. (1997). The threat to diversity in legal education: An empirical analysis of the consequences of abandoning race as a factor in law school admission decisions. New York University Law Review, 72, 1-53.
115. Wild, C. L., Durso, R., & Rubin, D. B. (1982). Effects of increased test-taking time on test scores by ethnic group, years out of school, and sex. Journal of Educational Measurement, 19, 19-28.
116. Wilson, K. M. (1981). Analyzing the long-term performance of minority and nonminority students: A tale of two studies. Research in Higher Education, 15, 351-375.
117. Wolfe, R. N., & Johnson, S. D. (1995). Personality as a predictor of college performance. Educational and Psychological Measurement, 55, 177-185.