Effect Size Indices for Analyses of Measurement Equivalence: Understanding the Practical Importance of Differences Between Groups

Journal of Applied Psychology

Volumn 96, Issue 5, 2011, Pages 966-980

  Nye, Christopher D ^a   Drasgow, Fritz ^a  

^a UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

Author keywords

Differential item functioning; Effect size; Employee selection; Measurement equivalence

Indexed keywords

ARTICLE; COMPARATIVE STUDY; HUMAN; METHODOLOGY; PERSONNEL MANAGEMENT; PREDICTIVE VALUE; PSYCHOLOGY; PSYCHOMETRY; STATISTICAL MODEL;

HUMANS; MODELS, STATISTICAL; PERSONNEL SELECTION; PREDICTIVE VALUE OF TESTS; PSYCHOLOGY, INDUSTRIAL; PSYCHOMETRICS;

EID: 80052827627     PISSN: 00219010     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0022955     Document Type: Article

References (73)

1. Bentler P.M. EQS: Structural equations program manual 1995, Multivariate Software, Encino, CA.
2. Bobko P., Roth P.L. An analysis of two methods for assessing and indexing adverse impact: A disconnect between the academic literature and some practice. Adverse impact: Implications for organizational staffing and high stakes selection 2009, 271-288. Erlbaum, Mahwah, NJ. J.L. Outtz (Ed.).
3. Bock R.D., Muraki E., Pfeiffenberger W. Item pool maintenance in the presence of item parameter drift. Journal of Educational Measurement 1988, 25:275-285. 10.1111/j.1745-3984.1988.tb00308.x.
4. Bollen K.A. Structural equations with latent variables 1989, Wiley, New York, NY.
5. Boomsma A. Robustness of LISREL against small sample sizes in factor analysis models. Systems under indirect observation: Causality, structure, prediction, Part I 1982, 149-173. North Holland, Amsterdam, the Netherlands. K.G. Jöreskog, H. Wold (Eds.).
6. Byrne B.M. Structural equation modeling in LISREL, PRELIS, and SIMPLIS: Basic concepts, applications, and programming 1998, Erlbaum, Mahwah, NJ.
7. Camilli G., Shepard L.A. The inadequacy of ANOVA for detecting test bias. Journal of Educational Statistics 1987, 12:87-99. 10.2307/1164630.
8. Candell G.L., Drasgow F. An iterative procedure for linking metrics and assessing item bias in item response theory. Applied Psychological Measurement 1988, 12:253-260. 10.1177/014662168801200304.
9. Cascio W.F., Jacobs R., Silva J. Validity, utility, and adverse impact: Practical implications from 30 years of data. Adverse impact: Implications for organizational staffing and high stakes selection 2009, 271-288. Erlbaum, Mahwah, NJ. J.L. Outtz (Ed.).
10. Chan D. Racial subgroup differences in predictive validity perceptions on personality and cognitive ability tests. Journal of Applied Psychology 1997, 82:311-320. 10.1037/0021-9010.82.2.311.
11. Chan D. Detection of differential item functioning on the Kirton Adaptation-Innovation Inventory using multiple-group mean and covariance structure analyses. Multivariate Behavioral Research 2000, 35:169-199. 10.1207/S15327906MBR3502_2.
12. Chan K.-Y., Drasgow F., Sawin L.L. What is the shelf life of a test? The effect of time on the psychometrics of a cognitive ability test battery over 16 years. Journal of Applied Psychology 1999, 84:610-619. 10.1037/0021-9010.84.4.610.
13. Cheung G.W., Rensvold R.B. Testing factorial invariance across groups: A reconceptualization and proposed new method. Journal of Management 1999, 25:1-27. 10.1177/014920639902500101.
14. Cheung G.W., Rensvold R.B. Assessing extreme and acquiescence response sets in cross-cultural research using structural equations modeling. Journal of Cross-Cultural Psychology 2000, 31:187-212. 10.1177/0022022100031002003.
15. Cheung G.W., Rensvold R.B. Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling 2002, 9:233-255. 10.1207/S15328007SEM0902_5.
16. Cohen J. Statistical power analysis for the behavioral sciences 1988, Erlbaum, Hillsdale, NJ. 2nd ed.
17. Cohen J. Things I have learned (so far). American Psychologist 1990, 45:1304-1312. 10.1037/0003-066X.45.12.1304.
18. Cohen J. The earth is round (p < .05). American Psychologist 1994, 49:997-1003. 10.1037/0003-066X.49.12.997.
19. Cole D.A., Ciesla J.A., Steiger J.H. The insidious effects of failing to include design-driven correlated residuals in latent-variable covariance structure analysis. Psychological Methods 2007, 12:381-398. 10.1037/1082-989X.12.4.381.
20. Cole M.S., Bedeian A.G., Field H.S. The measurement equivalence of web-based and paper-and-pencil measures of transformational leadership: A multinational test. Organizational Research Methods 2006, 9:339-368. 10.1177/1094428106287434.
21. Colquitt J.A., Scott B.A., Judge T.A., Shaw J.C. Justice and personality: Using integrative theories to derive moderators of justice effects. Organizational Behavior and Human Decision Processes 2006, 100:110-127. 10.1016/j.obhdp.2005.09.001.
22. Drasgow F. Scrutinizing psychological tests: Measurement equivalence and equivalent relations with external variables are the central issues. Psychological Bulletin 1984, 95:134-135. 10.1037/0033-2909.95.1.134.
23. Drasgow F., Kanfer R. Equivalence of psychological measurement in heterogeneous populations. Journal of Applied Psychology 1985, 70:662-680. 10.1037/0021-9010.70.4.662.
24. Drasgow F., Nye C.D., Guo J. The stability of item parameters over time for the NCLEX-RN exam: A report to the National Council of State Boards of Nursing 2008, The National Council of State Boards of Nursing, Chicago, IL.
25. Gatewood R., Field H.S., Barrick M. Human resource selection 2007, South-Western College, Cincinnati, OH. 6th ed.
26. Ghorpade J., Hattrup K., Lackritz J.R. The use of personality measures in cross-cultural research: A test of three personality scales across two countries. Journal of Applied Psychology 1999, 84:670-679. 10.1037/0021-9010.84.5.670.
27. Glass G.V. Primary, secondary, and meta-analysis of research. Educational Researcher 1976, 5:3-8.
28. González-Romá V., Hernández A., Gómez-Benito J. Power and Type I error of the mean and covariance structure analysis model for detecting differential item functioning in graded response items. Multivariate Behavioral Research 2006, 41:29-53. 10.1207/s15327906mbr4101_3.
29. Harlow L.L., Mulaik S.A., Steiger J.H. What if there were no significance tests? 1997, Erlbaum, Mahwah, NJ.
30. Hays W.L. Statistics for psychologists 1963, Holt, Rinehart and Winston, New York, NY.
31. Hedges L.V. Distributional theory for Glass's estimator of effect size and related estimators. Journal of Educational Statistics 1981, 6:107-128. 10.2307/1164588.
32. Hernandez, A., Chernyshenko, O., Stark, S., & Drasgow, F. (2008, April). DIF detection with MACS: Effectiveness and efficiency of two approaches. Paper presented at the 23rd annual conference of the Society for Industrial and Organizational Psychology, San Francisco, CA.
33. Hoogland J.J., Boomsma A. Robustness studies in covariance structure modeling: An overview and a meta-analysis. Sociological Methods & Research 1998, 26:329-367. 10.1177/0049124198026003003.
34. Hopwood C.J., Donnellan M.B. How should the internal structure of personality inventories be evaluated?. Personality and Social Psychology Review 2010, 14:332-346. 10.1177/1088868310361240.
35. Hu L., Bentler P.M. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling 1999, 6:1-55.
36. Hulin C.L., Drasgow F., Parsons C.K. Item response theory: Applications to psychological measurement 1983, Dow Jones Irwin, Homewood, IL.
37. Johnson E.C., Meade A.W., DuVernet A.M. The role of referent indicators in tests of measurement invariance. Structural Equation Modeling 2009, 16:642-657. 10.1080/10705510903206014.
38. Jöreskog K. Simultaneous factor analysis in several populations. Psychometrika 1971, 36:409-426. 10.1007/BF02291366.
39. Judge T.A., Bono J.E., Ilies R., Gerhardt M.W. Personality and leadership: A qualitative and quantitative review. Journal of Applied Psychology 2002, 87:765-780. 10.1037/0021-9010.87.4.765.
40. Judge T.A., Heller D., Mount M.K. Five-factor model of personality and job satisfaction. Journal of Applied Psychology 2002, 87:530-541. 10.1037/0021-9010.87.3.530.
41. Kanfer R., Heggestad E.D. Motivational traits and skills: A person-centered approach to work motivation. Research in Organizational Behavior 1997, 19:1-56.
42. Kehoe J.F. Cut scores and adverse impact. Adverse impact: Implications for organizational staffing and high stakes selection 2009, 289-322. Erlbaum, Mahwah, NJ. J.L. Outtz (Ed.).
43. Kirk R.E. Practical significance: A concept whose time has come. Educational and Psychological Measurement 1996, 56:746-759. 10.1177/0013164496056005002.
44. Kirk R.E. Effect magnitude: A different focus. Journal of Statistical Planning and Inference 2006, 137:1634-1646. 10.1016/j.jspi.2006.09.011.
45. Kuncel N.R., Hezlett S.A. Standardized tests predict graduate students' success. Science 2007, February 23, 315:1080-1081. 10.1126/science.1136618.
46. Lam S.S.K., Hui C., Law K.S. Organizational citizenship behavior: Comparing perspectives of supervisors and subordinates across four international samples. Journal of Applied Psychology 1999, 84:594-601. 10.1037/0021-9010.84.4.594.
47. Lawrence I.M., Curley W.E., McHale F.J. Differential item functioning for males and females on SAT-Verbal Reading subscore items 1988, College Board, New York, NY, (College Board Report No. 88-4).
48. Lei M., Lomax R.G. The effects of varying degrees of nonnormality in structural equation modeling. Structural Equation Modeling 2005, 12:1-27. 10.1207/s15328007sem1201_1.
49. Lievens F., Reeve C.L., Heggestad E.D. An examination of psychometric bias due to retesting on cognitive ability tests in selection settings. Journal of Applied Psychology 2007, 92:1672-1682. 10.1037/0021-9010.92.6.1672.
50. Little T.D. Mean and covariance structures (MACS) analyses of cross-cultural data: Practical and theoretical issues. Multivariate Behavioral Research 1997, 32:53-76. 10.1207/s15327906mbr3201_3.
51. Maxwell S.E., Arvey R.D. The search for predictors with high validity and low adverse impact: Compatible or incompatible goals?. Journal of Applied Psychology 1993, 78:433-437. 10.1037/0021-9010.78.3.433.
52. Meade A.W. A taxonomy of effect size measures for the differential functioning of items and scales. Journal of Applied Psychology 2010, 95:728-743. 10.1037/a0018966.
53. Meade A.W., Johnson E.C., Braddy P.W. Power and sensitivity of alternative fit indices in tests of measurement invariance. Journal of Applied Psychology 2008, 93:568-592. 10.1037/0021-9010.93.3.568.
54. Morris S.B., DeShon R.P. Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychological Methods 2002, 7:105-125. 10.1037/1082-989X.7.1.105.
55. Newman D.A., Jacobs R.R., Bartram D. Choosing the best method for local validity estimation: Relative accuracy of meta-analysis versus a local study versus Bayes-analyses. Journal of Applied Psychology 2007, 92:1394-1413. 10.1037/0021-9010.92.5.1394.
56. Nye C.D., Roberts B.W., Saucier G., Zhou X. Testing the measurement equivalence of personality traits across cultures. Journal of Research in Personality 2008, 42:1524-1536. 10.1016/j.jrp.2008.07.004.
57. Oishi S. The concept of life satisfaction across cultures: An IRT analysis. Journal of Research in Personality 2006, 40:411-423. 10.1016/j.jrp.2005.02.002.
58. Parker C.P., Baltes B.B., Christiansen N.D. Support for affirmative action, justice perceptions, and work attitudes: A study of gender and racial-ethnic group differences. Journal of Applied Psychology 1997, 82:376-389. 10.1037/0021-9010.82.3.376.
59. Raju N.S., van der Linden W.J., Fleer P.F. IRT-based internal measure of differential functioning of items and tests. Applied Psychological Measurement 1995, 19:353-368. 10.1177/014662169501900405.
60. Ryan A.M., West B.J., Carr J.Z. Effects of the terrorist attacks of 9/11/01 on employee attitudes. Journal of Applied Psychology 2003, 88:647-659. 10.1037/0021-9010.88.4.647.
61. Sagie A., Elizur D., Yamauchi H. The structure and strength of achievement motivation: A cross-cultural comparison. Journal of Organizational Behavior 1996, 17:431-444.
62. Salgado, J. F. (2000). The Big Five personality dimensions as predictors of alternative job performance criteria. Paper presented at the 15th annual convention of the Society for Industrial and Organizational Psychology, New Orleans, LA.
63. Saucier G. Mini-Markers: A brief version of Goldberg's unipolar Big-Five markers. Journal of Personality Assessment 1994, 63:506-516. 10.1207/s15327752jpa6303_8.
64. Saucier G., Goldberg L.R. Lexical studies of indigenous personality factors: Premises, products, and prospects. Journal of Personality 2001, 69:847-879. 10.1111/1467-6494.696167.
65. Schmidt F.L. Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers. Psychological Methods 1996, 1:115-129. 10.1037/1082-989X.1.2.115.
66. Schmitt A.P., Dorans N.J. Differential item functioning for minority examinees on the SAT 1988, Educational Testing Service, Princeton, NJ, (Educational Testing Service Report No. 88-32).
67. Stark S., Chernyshenko O.S., Chan K.-Y., Lee W.C., Drasgow F. Effects of the testing situation on item responding: Cause for concern. Journal of Applied Psychology 2001, 86:943-953. 10.1037/0021-9010.86.5.943.
68. Stark S., Chernyshenko O.S., Drasgow F. Examining the effects of differential item (functioning and differential) test functioning on selection decisions: When are statistically significant effects practically important?. Journal of Applied Psychology 2004, 89:497-508. 10.1037/0021-9010.89.3.497.
69. Stark S., Chernyshenko O.S., Drasgow F. Detecting differential item functioning with CFA and IRT: Toward a unified strategy. Journal of Applied Psychology 2006, 91:1292-1306. 10.1037/0021-9010.91.6.1292.
70. Steenkamp J.E.M., Baumgartner H. Assessing measurement invariance in cross-national consumer research. Journal of Consumer Research 1998, 25:78-107. 10.1086/209528.
71. Vandenberg R.J., Lance C.E. A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods 2000, 3:4-70. 10.1177/109442810031002.
72. Wasti S.A., Bergman M.E., Glomb T.M., Drasgow F. Test of the cross-cultural generalizability of a model of sexual harassment. Journal of Applied Psychology 2000, 85:766-778. 10.1037/0021-9010.85.5.766.
73. Woehr D.J., Sheehan M.K., Bennett W. Assessing measurement equivalence across rating sources: A multitrait-multirater approach. Journal of Applied Psychology 2005, 90:592-600. 10.1037/0021-9010.90.3.592.