The formalization of fairness: Issues in testing for measurement invariance using subtest scores

Educational Research and Evaluation

Volumn 19, Issue 2-3, 2013, Pages 223-244

  Molenaar, Dylan ^a   Borsboom, Denny ^a  

^a UNIVERSITY OF AMSTERDAM   (Netherlands)

Author keywords

differential item functioning; factor analysis; item response theory; measurement invariance; non normality; sum scores

Indexed keywords

EID: 84875818659     PISSN: 13803611     EISSN: 17444187     Source Type: Journal    
DOI: 10.1080/13803611.2013.767628     Document Type: Article

References (69)

1. Aguinis, H. and Smith, M. E. 2007. Understanding the impact of test validity and bias on selection errors and adverse impact in human resource selection. Personnel Psychology, 60: 165-199. (doi:10.1111/j.1744-6570.2007.00069.x)
2. Amthauer, R., Brocke, B., Liepmann, D. and Beauducel, A. 2001. I-S-T 2000R. Intelligenz-Struktur-Test 2000R., Göttingen, Germany: Hogrefe.
3. Andersen, E. B. 1973. Asymptotic properties of conditional maximum-likelihood estimators. Journal of the Royal Statistical Society, Series B, 32: 283-301.
4. Bartholomew, D. J. 1987. Latent variable models and factor analysis, London, UK: Griffin.
5. Bauer, D. J. 2005. The role of nonlinear factor-to-indicator relationships in tests of measurement equivalence. Psychological Methods, 10: 305-316. (doi:10.1037/1082-989X.10.3.305)
6. Birnbaum, A. 1968. "Some latent trait models and their use in inferring an examinee's ability". In Statistical theories of mental test scores, Edited by: Lord, E. M. and Novick, M. R. 397-479. Reading, MA: Addison Wesley.
7. Borsboom, D. 2005. Measuring the mind: Conceptual issues in contemporary psychometrics, Cambridge, UK: Cambridge University Press.
8. Borsboom, D. 2006. When does measurement invariance matter?. Medical Care, 44: S176-S181. (doi:10.1097/01.mlr.0000245143.08679.cc)
9. Borsboom, D. 2008. Latent variable theory. Measurement, 6: 25-53.
10. Borsboom, D., Romeijn, J. W. and Wicherts, J. M. 2008. Measurement invariance versus selection invariance: Is fair selection possible?. Psychological Methods, 13: 75-98. (doi:10.1037/1082-989X.13.2.75)
11. Cleary, T. A. 1968. Test bias: Prediction of grades of Negro and White students in integrated colleges. Journal of Educational Measurement, 5: 115-124. (doi:10.1111/j.1745-3984.1968.tb00613.x)
12. Curran, P. J., West, S. G. and Finch, J. F. 1996. The robustness of test statistics to nonnormality and specification error in confirmatory factor analysis. Psychological Methods, 1: 16-29. (doi:10.1037/1082-989X.1.1.16)
13. Dolan, C. V. 2000. Investigating Spearman's hypothesis by means of multi-group confirmatory factor analysis. Multivariate Behavioral Research, 35: 21-50. (doi:10.1207/S15327906MBR3501_2)
14. Dolan, C. V., Colom, R., Abad, F. J., Wicherts, J. M., Hessen, D. J. and Van der Sluis, S. 2006. Multi-group covariance and mean structure modeling of the relationship between WAIS-III common factors and gender and educational attainment in Spain. Intelligence, 34: 193-210. (doi:10.1016/j.intell.2005.09.003)
15. Dorans, N. J. and Kulick, E. 1986. Demonstrating the utility of the standardization approach to assessing unexpected differential item performance on the Scholastic Aptitude Test. Journal of Educational Measurement, 23: 355-368. (doi:10.1111/j.1745-3984.1986.tb00255.x)
16. Edwards, J. R. and Bagozzi, R. P. 2000. On the nature and direction of relationships between constructs and measures. Psychological Methods, 5: 155-174. (doi:10.1037/1082-989X.5.2.155)
17. Ellis, J. L. and Junker, B. W. 1997. Tail-measurability in monotone latent variable models. Psychometrika, 62: 495-523. (doi:10.1007/BF02294640)
18. Evers, A., Te Nijenhuis, J. and Van der Flier, H. 2005. "Ethnic bias and fairness in personnel selection: Evidence and consequences". In The Blackwell handbook of personnel selection, Edited by: Evers, A., Anderson, N. and Voskuijl, O. F. 306-328. Oxford, UK: Blackwell.
19. Gamliel, E. and Cahan, S. 2007. Mind the gap: Between-group differences and fair test use. International Journal of Selection and Assessment, 15: 273-282. (doi:10.1111/j.1468-2389.2007.00387.x)
20. Grayson, D. 1988. Two-group classification in latent trait theory: Scores with monotone likelihood ratio. Psychometrika, 53: 383-392. (doi:10.1007/BF02294219)
21. Horn, J. L. and McArdle, J. J. 1992. A practical and theoretical guide to measurement invariance in aging research. Experimental Aging Research, 18: 117-144. (doi:10.1080/03610739208253916)
22. Hunter, J. E. and Schmidt, F. L. 2000. Racial and gender bias in ability and achievement tests. Psychology, Public Policy, and Law, 6: 151-158. (doi:10.1037/1076-8971.6.1.151)
23. Jensen, A. R. and Reynolds, C. R. 1983. Sex differences on the WISC-R. Personality and Individual Differences, 4: 223-226. (doi:10.1016/0191-8869(83)90029-6)
24. Jöreskog, K. G. 1971. Statistical analysis of sets of congeneric tests. Psychometrika, 36: 109-133. (doi:10.1007/BF02291393)
25. Lubke, G. H. and Dolan, C. V. 2003. Can unequal residual variances across subpopulations mask differences in residual means in the common factor model?. Structural Equation Modeling, 10: 175-192. (doi:10.1207/S15328007SEM1002_1)
26. Lubke, G. H. and Muthén, B. O. 2004. Applying multigroup confirmatory factor models for continuous outcomes to Likert scale data complicates meaningful group comparisons. Structural Equation Modeling, 11: 514-534. (doi:10.1207/s15328007sem1104_2)
27. Lord, F. M. 1980. Applications of item response theory to practical testing problems, Hillsdale, NJ: Lawrence Erlbaum Associates.
28. Magis, D., Beland, S., Tuerlinckx, F. and De Boeck, P. 2010. A general framework and an R package for the detection of dichotomous differential item functioning. Behavior Research Methods, 42: 847-862. (doi:10.3758/BRM.42.3.847)
29. Mantel, N. and Haenszel, W. 1959. Statistical aspects of the analysis of data from retrospective studies of disease. Journal of the National Cancer Institute, 22: 719-748.
30. Mellenbergh, G. J. 1989. Item bias and item response theory. International Journal of Educational Research, 13: 127-143. (doi:10.1016/0883-0355(89)90002-5)
31. Mellenbergh, G. J. 1994a. Generalized linear item response theory. Psychological Bulletin, 115: 300-307. (doi:10.1037/0033-2909.115.2.300)
32. Mellenbergh, G. J. 1994b. A unidimensional latent trait model for continuous item responses. Multivariate Behavioral Research, 29: 223-236. (doi:10.1207/s15327906mbr2903_2)
33. Meredith, W. 1964. Notes on factorial invariance. Psychometrika, 29: 177-185. (doi:10.1007/BF02289699)
34. Meredith, W. 1993. Measurement invariance, factor analysis, and factorial invariance. Psychometrika, 58: 525-543. (doi:10.1007/BF02294825)
35. Messick, S. 1989. "Validity". In Educational measurement, Edited by: Linn, R. L. 13-103. Washington, DC: American Council on Education and National Council on Measurement in Education.
36. Millsap, R. E. 1997. Invariance in measurement and prediction: Their relationship in the single-factor case. Psychological Methods, 2: 248-260. (doi:10.1037/1082-989X.2.3.248)
37. Millsap, R. E. 1998. Group differences in regression intercepts: Implications for factorial invariance. Multivariate Behavioral Research, 33: 403-424. (doi:10.1207/s15327906mbr3303_5)
38. Millsap, R. E. 2008. Invariance in measurement and prediction revisited. Psychometrika, 72: 461-473. (doi:10.1007/s11336-007-9039-7)
39. Millsap, R. E. 2011. Statistical approaches to measurement invariance, New York, NY: Routledge.
40. Molenaar, D., & Dolan, C. V. (2012). Substantively motivated extensions of the traditional latent trait model. Netherlands Journal of Psychology, 67, 48-57.
41. Muthén, B. O. 1989. Latent variable modeling in heterogeneous populations. Psychometrika, 54: 557-585. (doi:10.1007/BF02296397)
42. Muthén, L. K. and Muthén, B. O. 2007. Mplus User's Guide, 5, Los Angeles, CA: Authors.
43. Neisser, U., Boodoo, G., Bourchard, T. J., Boykin, A. W., Brody, N., Ceci, S. J. and Urbina, S. 1996. Intelligence: Knowns and unknowns. American Psychologist, 51: 77-101. (doi:10.1037/0003-066X.51.2.77)
44. Osterlind, S. J. and Everson, H. T. 2009. Differential item functioning, 2, Thousand Oaks, CA: Sage.
45. Penfield, R. D. and Camilli, G. 2007. "Differential item functioning and item bias". In Handbook of statistics: Vol. 26. Psychometrics, Edited by: Rao, C. R. and Sinharay, S. 125-167. Amsterdam, The Netherlands: Elsevier.
46. Raju, N. S. 1988. The area between two item characteristic curves. Psychometrika, 53: 495-502. (doi:10.1007/BF02294403)
47. Raju, N. S. 1990. Determining the significance of estimated signed and unsigned areas between two item response functions. Applied Psychological Measurement, 14: 197-207. (doi:10.1177/014662169001400208)
48. Rasch, G. 1960. Probabilistic models for some intelligence and attainment tests, Copenhagen, Denmark: Danmarks Paedogogiske Institut.
49. Rushton, J. P. and Jensen, A. R. 2005. Thirty years of research on race differences in cognitive ability. Psychology, Public Policy, and Law, 11: 235-294. (doi:10.1037/1076-8971.11.2.235)
50. Sackett, P. R., Borneman, M. J. and Connelly, B. S. 2008. High-stakes testing in higher education and employment: Appraising the evidence for validity and fairness. American Psychologist, 63: 215-227. (doi:10.1037/0003-066X.63.4.215)
51. Sackett, P. R., Schmitt, N., Ellington, J. E. and Kabin, M. B. 2001. High-stakes testing in employment, credentialing, and higher education: Prospects in a post-affi{ligature}rmative action world. American Psychologist, 56: 302-318. (doi:10.1037/0003-066X.56.4.302)
52. Samejima, F. 1969. Estimation of latent ability using a response pattern of graded scores (Psychometric Monograph, No. 17, Richmond, VA: Psychometric Society. Retrieved from http://www.psychometrika.org/journal/online/MN17.pdf
53. Schermelleh-Engel, K., Moosbrugger, H. and Müller, H. 2003. Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of fit measures. Methods of Psychological Research, 8: 23-74.
54. Shealy, R. and Stout, W. 1993. A model-based standardization approach that separates true bias/DIF from group ability differences and detects test bias/DIF as well as item bias/DIF. Psychometrika, 58: 159-194. (doi:10.1007/BF02294572)
55. Sörbom, D. 1974. A general method for studying differences in factor means and factor structure between groups. British Journal of Mathematical and Statistical Psychology, 27: 229-239. (doi:10.1111/j.2044-8317.1974.tb00543.x)
56. Spearman, C. 1904. "General intelligence" objectively determined and measured. American Journal of Psychology, 15: 201-293. (doi:10.2307/1412107)
57. Spearman, C. E. 1927. The abilities of man: Their nature and measurement, New York, NY: Macmillan.
58. Swaminathan, H. and Rogers, H. J. 1990. Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27: 361-370. (doi:10.1111/j.1745-3984.1990.tb00754.x)
59. Thomson, G. H. and Lederman, W. 1939. The influence of multivariate selection on the factorial analysis of ability. British journal of psychology, 29: 288-305.
60. Thurstone, L. L. 1947. Multiple factor analysis, Chicago, IL: university of Chicago Press.
61. Tobin, J. 1958. Estimation of relationships for limited dependent variables. Econometrica, 26: 24-36. (doi:10.2307/1907382)
62. Tucker-Drob, E. M. 2009. Differentiation of cognitive abilities across the life span. Developmental Psychology, 45: 1097-1118. (doi:10.1037/a0015864)
63. Van der Sluis, S., Dolan, C. V., Neale, M. C., Boomsma, D. I. and Posthuma, D. 2006. Detecting genotype-environment interaction in monozygotic twin data: Comparing the Jinks & Fulker test and a new test based on marginal maximum likelihood estimation. Twin research and Human Genetics, 9: 377-392.
64. Van der Sluis, S., Posthuma, D., Dolan, C. V., De Geus, E. J.C., Colom, R. and Boomsma, D. I. 2006. Gender differences on the Dutch WAIS-III. Intelligence, 34: 273-289. (doi:10.1016/j.intell.2005.08.002)
65. Verhelst, N. D. and Glas, C. A.W. 1995. "The one-parameter logistic model". In Rasch models: Foundations, recent developments, and applications., Edited by: Fischer, G. H. and Molenaar, I. W. 215-237. New York, NY: Springer.
66. Wechsler, D. 1997. Wechsler Adult Intelligence Scale- III (WAISIII)., San Antonio, TX: Psychological Corporation.
67. Wicherts, J. M., Dolan, C. V. and Hessen, D. J. 2005. Stereotype threat and group differences in test performance: A question of measurement invariance. Journal of Personality and Social Psychology, 89: 696-716. (doi:10.1037/0022-3514.89.5.696)
68. Wicherts, J. M. and Millsap, R. E. 2009. The absence of underprediction does not imply the absence of measurement bias. American Psychologist, 64: 281-283. (doi:10.1037/a0014992)
69. Widaman, K. F. and Reise, S. P. 1997. "Exploring the measurement invariance of psychological instruments: Applications in the substance use domain". In The science of prevention: Methodological advances from alcohol and substance abuse research, Edited by: Bryant, K. J. and Windle, M. 281-324. Washington, DC: American Psychological Association.