An Examination of Psychometric Bias Due to Retesting on Cognitive Ability Tests in Selection Settings

Journal of Applied Psychology

Volumn 92, Issue 6, 2007, Pages 1672-1682

  Lievens, Filip ^a   Reeve, Charlie L ^b   Heggestad, Eric D ^b  

^a GHENT UNIVERSITY   (Belgium)

^b UNIVERSITY OF NORTH CAROLINA AT CHARLOTTE   (United States)

Author keywords

cognitive ability tests; measurement bias; psychometric bias; retesting

Indexed keywords

ACHIEVEMENT; ADULT; ARTICLE; COGNITION; EPIDEMIOLOGY; FEMALE; HUMAN; MALE; PSYCHOLOGIC TEST; PSYCHOMETRY; STATISTICS;

ACHIEVEMENT; ADULT; BIAS (EPIDEMIOLOGY); COGNITION; FEMALE; HUMANS; MALE; PSYCHOLOGICAL TESTS; PSYCHOMETRICS;

EID: 36349000691     PISSN: 00219010     EISSN: None     Source Type: Journal    
DOI: 10.1037/0021-9010.92.6.1672     Document Type: Article

References (48)

1. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. Standards for educational and psychological testing (1999), American Psychological Association, Washington, DC
2. Anastasi A. Coaching, test sophistication and developed abilities. American Psychologist 36 (1981) 1086-1093
3. Arbuckle J.L., and Wothke W. Amos 4.0 user's guide (1996), SmallWaters Corporation, Chicago
4. Bagozzi R.P., and Edwards J.R. A general approach for representing constructs in organizational research. Organizational Research Methods 1 (1998) 45-87
5. Bentler P.M. Comparative fit indexes in structural models. Psychological Bulletin 107 (1990) 238-246
6. Briggs D.C. The effect of admissions test preparation: Evidence from NELS: 88. Chance 14 (2001) 10-18
7. Browne M.W., and Cudeck R. Alternative ways of assessing model fit. Sociological Methods and Research 21 (1992) 230-258
8. Carroll J.B. Human cognitive abilities: A survey of factor-analytic studies (1993), Cambridge University Press, New York
9. Cheung G.W., and Rensvold R.B. Evaluating goodness-of-fit indexes for testing measurement equivalence. Structural Equation Modeling 9 2 (2002) 233-255
10. Cole D.A., and Maxwell S.E. Multitrait-multimethod comparisons across populations: A confirmatory factor analytic approach. Multivariate Behavioral Research 20 (1985) 389-417
11. Coyle T.R. Test-retest changes on scholastic aptitude tests are not related to g. Intelligence 34 (2006) 15-27
12. Crocker L., and Algina J. Introduction to classical and modern test theory (1986), Harcourt, Orlando, FL
13. Hamilton J.C., and Shuminsky T.R. Self-awareness mediates the relationship between serial position and item reliability. Journal of Personality and Social Psychology 59 (1990) 1301-1307
14. Hausknecht J.P., Halpert J.A., Di Paolo N.T., and Gerrard M.O.M. Retesting in selection: A meta-analysis of coaching and practice effects for tests of cognitive ability. Journal of Applied Psychology 92 (2007) 373-385
15. Hausknecht J.P., Trevor C.O., and Farr J.L. Retaking ability tests in a selection setting: Implications for practice effects, training performance, and turnover. Journal of Applied Psychology 87 (2002) 243-254
16. Hu L., and Bentler P.M. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling 6 (1999) 1-55
17. Jensen A.R. Bias in mental testing (1980), Free Press, New York
18. Jensen A.R. The g factor: The science of mental ability (1998), Praeger, Westport, CT
19. Knowles E.S. Item context effects on personality scales: Measuring changes the measure. Journal of Personality and Social Psychology 55 (1988) 312-320
20. Kulik J.A., Bangert-Drowns R.L., and Kulik C.C. Effectiveness of coaching for aptitude tests. Psychological Bulletin 95 (1984) 179-188
21. Kulik J.A., Kulik C.C., and Bangert R.L. Effects of practice on aptitude and achievement test scores. American Educational Research Journal 21 (1984) 435-447
22. Kuncel N.R., Hezlett S.A., and Ones D.S. A comprehensive meta-analysis of the predictive validity of the Graduate Record Examinations: Implications for graduate student selection and performance. Psychological Bulletin 127 (2001) 162-181
23. Lievens F. Longitudinal study of the validity of different cognitive ability tests in a student admission context. Applied HRM Research 9 (2004) 27-30
24. Lievens F., Buyse T., and Sackett P.R. Retest effects in operational selection settings: Development and test of a framework. Personnel Psychology 58 (2005) 981-1007
25. Lipsey M.W., and Wilson D.B. The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis. American Psychologist 48 (1993) 1181-1209
26. Little T.D. Mean and covariance (MACS) analysis of cross-cultural data: Practical and theoretical issues. Multivariate Behavioural Research 32 (1997) 53-76
27. Lubinski D. Scientific and social significance of assessing individual differences: "Sinking shafts at a few critical points.". Annual Review of Psychology 51 (2000) 405-444
28. McArdle J.J. Contemporary statistical models for examining test bias. In: McArdle J.J., and Woodcock R.W. (Eds). Human cognitive abilities in theory and practice (1998), Erlbaum, Mahwah, NJ 157-195
29. Messick S., and Jungeblut A. Time and method in coaching for the SAT. Psychological Bulletin 89 (1981) 191-216
30. Minnaert, A. (1996). Academic performance, cognition, metacognition and motivation: Assessing freshmen characteristics on task: A validation and replication study in higher education. Unpublished doctoral dissertation, University of Louvain, Louvain, Belgium.
31. Muchinsky P.M. When the psychometrics of test development meets organizational realities: A conceptual framework for organizational change, examples and recommendations. Personnel Psychology 57 (2004) 175-209
32. Raykov T. Behavioral scale reliability and measurement invariance evaluation using latent variable modeling. Behavior Therapy 35 (2004) 299-331
33. Reeve C.L. Differential ability antecedents of general and specific dimensions of declarative knowledge: More than g. Intelligence 32 (2004) 621-652
34. Reeve C.L., and Lam H. The psychometric paradox of practice effects due to retesting: Measurement invariance and stable ability estimates in the face of observed score changes. Intelligence 33 (2005) 535-549
35. Reeve C.L., and Lam H. The relation between practice effects, test-taker characteristics, and degree of g-saturation. International Journal of Testing 7 (2007) 225-242
36. Roediger III H.L., and Karpicke J.D. The power of testing memory: Basic research and implications for educational practice. Perspectives on Psychological Science 1 (2006) 181-210
37. Sackett P.R., Burris L.R., and Ryan A.M. Coaching and practice effects in personnel selection. In: Cooper C.L., and Robertson I.T. (Eds). International review of industrial and organizational psychology (1989), Wiley, New York 145-183
38. Salthouse T.A., Schroeder D.H., and Ferrer E. Estimating retest effects in longitudinal assessments of cognitive functioning in adults between 18 and 60 years of age. Developmental Psychology 40 (2004) 813-822
39. Schmidt F.L., and Hunter J.E. The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin 124 (1998) 262-274
40. Spitz H.H. The raising of intelligence: A selected history of attempts to raise retarded intelligence (1986), Erlbaum, Hillsdale, NJ
41. Steiger, J. H., & Lind, J. M. (1980, May). Statistically based tests for the number of common factors. Paper presented at the meeting of the Psychometric Society, Iowa City, IA.
42. Steinberg L. Context and serial order effects in personality measurement: Limits on the generality of "measuring changes the measure.". Journal of Personality and Social Psychology 66 (1994) 341-349
43. Te Nijenhuis J., van Vianen A.E.M., and van der Flier H. Score gains on g-loaded tests: No g. Intelligence 35 (2007) 283-300
44. Thorndike R.L. Personnel selection: Test and measurement techniques (1949), Wiley, New York
45. U.S. Department of Labor. Uniform guidelines on employee selection procedures. Federal Register 43 (1978) 38290-38309
46. Vandenberg R.J., and Lance C.E. A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods 3 (2000) 4-70
47. Vey, M. A., Ones, D. S., Hezlett, S. A., Kuncel, N. R., Vannelli, J. R., Briggs, K. H., & Campbell, J. P. (2003, April). Relationships among college grade indices: A meta-analysis examining temporal influences. Paper presented at the meeting of the Society for Industrial and Organizational Psychology, Orlando, FL.
48. Wheeler, J. K. (2004, April). Practical implications of selection retesters on testing development and policy. Practitioner forum presented at the meeting of the Society for Industrial and Organizational Psychology, Chicago.