A taxonomy of effect size measures for the differential functioning of items and scales

Journal of Applied Psychology

Volumn 95, Issue 4, 2010, Pages 728-743

  Meade, Adam W ^a  

^a NORTH CAROLINA STATE UNIVERSITY   (United States)

Author keywords

DIF; Differential functioning; Invariance testing; Measurement invariance

Indexed keywords

ARTICLE; COMPUTER PROGRAM; CULTURAL FACTOR; HUMAN; INFORMATION PROCESSING; PSYCHOLOGIC TEST; PSYCHOMETRY; REFERENCE VALUE; SAMPLE SIZE; STANDARD; STATISTICAL ANALYSIS; STATISTICAL MODEL; STATISTICS; STANDARDS; STATISTICS AND NUMERICAL DATA;

CROSS-CULTURAL COMPARISON; DATA COLLECTION; DATA INTERPRETATION, STATISTICAL; HUMANS; MODELS, STATISTICAL; PSYCHOLOGICAL TESTS; PSYCHOMETRICS; REFERENCE VALUES; SAMPLE SIZE; SOFTWARE;

EID: 77954634517     PISSN: 00219010     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0018966     Document Type: Article

References (48)

1. Baker F.B. Equate 2.1: Computer program for equating two metrics in item response theory [Software] 1995, University of Wisconsin, Laboratory of Experimental Design, Madison.
2. Baranik L.E., Meade A.W., Lakey C.E., Lance C.E., Hu C., Hua W., Michalos A. Examining the differential item functioning of the Rosenberg Self-Esteem Scale across eight countries. Journal of Applied Social Psychology 2008, 38:1867-1904.
3. Bartram D., Brown A. Online testing: Mode of administration and the stability of OPQ32i scores. International Journal of Selection and Assessment 2004, 12:278-284.
4. Bolt D.M. A Monte Carlo comparison of parametric and nonparametric polytomous DIF detection methods. Applied Measurement in Education 2002, 15:113-141.
5. Camilli G., Shepard L.A. Methods for identifying biased test items 1994, Sage, Thousand Oaks, CA.
6. Chang H.H., Mazzeo J., Roussos L. Detecting DIF for polytomously scored items: An adaption of the SIBTEST procedure. Journal of Educational Measurement 1996, 33:333-353.
7. Chen F.F. What happens if we compare chopsticks with forks? The impact of making inappropriate comparisons in cross-cultural research. Journal of Personality and Social Psychology 2008, 95:1005-1018.
8. Cohen J. Statistical power analysis for the behavioral sciences 1988, Earlbaum, Hillsdale, NJ. 2nd ed.
9. Dorans N.J., Schmitt A.P. Constructed response and differential item functioning: A pragmatic approach (Research Report No. RR-91-47) 1991, Educational Testing Service, Princeton, NJ.
10. Facteau J.D., Craig S.B. Are performance appraisal ratings from different rating sources comparable?. Journal of Applied Psychology 2001, 86:215-227.
11. Flowers C.P., Oshima T.C., Raju N.S. A description and demonstration of the polytomous-DFIT framework. Applied Psychological Measurement 1999, 23:309-326.
12. Hambleton R.K., Swaminathan H. Item response theory: Principles and applications 1985, Kluwer Academic, Boston, MA.
13. Han K.T. WinGen: Windows software that generates IRT parameters and item responses. Applied Psychological Measurement 2007, 31:457-459.
14. Han K.T., Hambleton R.K. User's manual: WinGen (Center for Educational Assessment Report No. 642) 2007, University of Massachusetts, School of Education, Amherst.
15. Horn J.L., McArdle J.J. A practical and theoretical guide to measurement invariance in aging research. Experimental Aging Research 1992, 18:117-144.
16. Kim, S.-H. (2000, April). An investigation of the likelihood ratio test, the Mantel test, and the generalized Mantel-Haenszel test of DIF. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
17. Kim S.-H., Cohen A.S. A comparison of two area measures for detecting differential item functioning. Applied Psychological Measurement 1991, 15:269-278.
18. Kim S.-H., Cohen A.S., Alagoz C., Kim S. DIF detection and effect size measures for polytomously scored items. Journal of Educational Measurement 2007, 44:93-116.
19. Meade A.W., Lautenschlager G.J. A comparison of item response theory and confirmatory factor analytic methodologies for establishing measurement equivalence/invariance. Organizational Research Methods 2004, 7:361-388.
20. Meade A.W., Lautenschlager G.J., Johnson E.C. A Monte Carlo examination of the sensitivity of the differential functioning of items and tests framework for tests of measurement invariance with Likert data. Applied Psychological Measurement 2007, 31:430-455.
21. Meade A.W., Michels L.C., Lautenschlager G.J. Are Internet and paper-and-pencil personality tests truly comparable? An experimental design measurement invariance study. Organizational Research Methods 2007, 10:322-345.
22. Ployhart R.E., Weekley J.A., Holtz B.C., Kemp C. Web-based and paper-and-pencil testing of applicants in a proctored setting: Are personality, biodata and situational judgment tests comparable?. Personnel Psychology 2003, 56:733-752.
23. Potosky D., Bobko P. Selection testing via the Internet: Practical considerations and exploratory empirical findings. Personnel Psychology 2004, 57:1003-1034.
24. Raju N.S. The area between two item characteristic curves. Psychometrika 1988, 53:495-502.
25. Raju N.S. Determining the significance of estimated signed and unsigned areas between two item response functions. Applied Psychological Measurement 1990, 14:197-207.
26. Raju N.S., Fortmann-Johnson K.A., Kim W., Morris S.B., Nering M.L., Oshima T.C. The item parameter replication method for detecting differential functioning in the polytomous DFIT framework. Applied Psychological Measurement 2009, 33:133-147.
27. Raju N.S., Laffitte L.J., Byrne B.M. Measurement equivalence: A comparison of methods based on confirmatory factor analysis and item response theory. Journal of Applied Psychology 2002, 87:517-529.
28. Raju N.S., van der Linden W.J., Fleer P.F. IRT-based internal measures of differential functioning of items and tests. Applied Psychological Measurement 1995, 19:353-368.
29. Reise S.P., Yu J. Parameter recovery in the graded response model using MULTILOG. Journal of Educational Measurement 1990, 27:133-144.
30. Rosenberg M. Society and the adolescent child 1965, Princeton University Press, Princeton, NJ.
31. Rosenfeld P., Doherty L.M., Vicino S.M., Kantor J., Greaves J. Attitude assessment in organizations: Testing three microcomputer-based survey systems. Journal of General Psychology 1989, 116:145-154.
32. Ryan A.M., Horvath M., Ployhart R.E., Schmitt N., Slade L.A. Hypothesizing differential item functioning in global employee opinion surveys. Personnel Psychology 2000, 53:531-562.
33. Shealy R., Stout W. A model-based standardization approach that separates true bias/DIF from group ability differences and detects test bias/DTF as well as item bias/DIF. Psychometrika 1993, 58:159-194.
34. OPQ32 technical manual 2000, SHL, Author, Boulder, CO.
35. Principles for the validation and use of personnel selection procedures 2003, Society for Industrial and Organizational Psychology, Author, Bowling Green, OH.
36. Stark S., Chernyshenko O.S., Drasgow F. Examining the effects of differential item functioning and differential test functioning on selection decisions: When are statistically significant effects practically important?. Journal of Applied Psychology 2004, 89:497-508.
37. Stark S., Chernyshenko O.S., Drasgow F. Detecting differential item functioning with confirmatory factor analysis and item response theory: Toward a unified strategy. Journal of Applied Psychology 2006, 91:1292-1306.
38. Steinberg L., Thissen D. Using effect sizes for research reporting: Examples using item response theory to analyze differential item functioning. Psychological Methods 2006, 11:402-415.
39. Stocking M.L., Lord F.M. Developing a common metric in item response theory. Applied Psychological Measurement 1983, 7:201-210.
40. Thissen D. Software for the computation of the statistics involved in item response theory likelihood-ratio tests for differential item functioning [Computer software] 2001, University of North Carolina at Chapel Hill, Chapel Hill.
41. Thissen D., Steinberg L., Wainer H. Use of item response theory in the study of group differences in trace lines. Test validity 1988, 147-172. Erlbaum, Hillsdale, NJ. H. Wainer, H.I. Braun (Eds.).
42. Thissen D., Steinberg L., Wainer H. Detection of differential item functioning using the parameters of item response models. Differential item functioning 1993, 67-113. Erlbaum, Hillsdale, NJ. P.W. Holland, H. Wainer (Eds.).
43. Vandenberg R.J. Toward a further understanding of an improvement in measurement invariance methods and procedures. Organizational Research Methods 2002, 5:139-158.
44. Vandenberg R.J., Lance C.E. A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods 2000, 3:4-69.
45. Wainer H. Model-based standardized measurement of an item's differential impact. Differential item functioning 1993, 123-135. Erlbaum, Hillsdale, NJ. P.W. Holland, H. Wainer (Eds.).
46. Whitaker B.G., McKinney J.L. Assessing the measurement invariance of latent job satisfaction ratings across survey administration modes for respondent subgroups: A MIMIC modeling approach. Behavior Research Methods 2007, 39:502-509.
47. Wu C.-H., Chen L.H., Tsai Y.-M. Longitudinal invariance analysis of the Satisfaction with Life Scale. Personality and Individual Differences 2009, 46:396-401.
48. Zumbo B.D. A handbook on the theory and methods of differential item functioning (DIF) 1999, Directorate of Human Resources Research and Evaluation, Department of National Defense, Ottawa, Ontario, Canada.