A comparison of selected empirical methods for assessing the structure of responses to test items

Applied Psychological Measurement

Volumn 27, Issue 3, 2003, Pages 159-203

  Tate, Richard ^a  

^a FLORIDA STATE UNIVERSITY   (United States)

Author keywords

Local item dependencies; Test dimensionality; Test factors; Test structure

Indexed keywords

EID: 0038470042     PISSN: 01466216     EISSN: None     Source Type: Journal    
DOI: 10.1177/0146621603027003001     Document Type: Article

References (95)

1. Ackerman, T. A. (1994). Using multidimensional item response theory to understand what items are measuring. Applied Measurement in Education, 7, 255-278.
2. Ackerman, T. A. (1996). Graphical representation of multidimensional item response theory. Applied Psychological Measurement, 20, 311-329.
3. Adams, R. J., Wilson, M., & Wang, W. (1997). The multidimensional random coefficients multinomial logit model. Applied Psychological Measurement, 21, 1-23.
4. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
5. Ansley, T. N., & Forsyth, R. A. (1985). An examination of the characteristics of unidimensional IRT parameter estimates derived from two-dimensional data. Applied Psychological Measurement, 9, 37-48.
6. Beland, A., & Mislevy, R. J. (1996). Probability-based inferences in a domain of proportional reasoning tasks. Journal of Educational Measurement, 33, 3-28.
7. Bock, R. D., Gibbons, R., & Muraki, E. (1988). Full information item factor analysis. Applied Psychological Measurement, 12, 261-280.
8. Bock, R. D., Gibbons, R., Schilling, S. G., Muraki, E., Wilson, D. T., & Wood, R. (1999). TESTFACT 3: Test scoring, items statistics, and full-information item factor analysis. Chicago: Scientific Software International.
9. Bollen, K. A. (1989). Structural equations with latent variables. New York: Wiley.
10. Bolt, D. M. (2001). Conditional covariance-based representation of multidimensional test structure. Applied Psychological Measurement, 25, 244-257.
11. Bradlow, E. T., Wainer, H., & Wang, X. (1999). A Bayesian random effects model for testlets. Psychometrika, 64, 153-168.
12. Breithaupt, K., & Gessaroli, M. E. (1996, April). A comparison of DIMTEST and the approximate chi-square statistic in conditions of correlated factors and pseudo-guessing. Paper presented at the annual meeting of the National Council on Measurement in Education, New York.
13. Carlson, J. E., & Jirele, T. (1992, April). Dimensionality of 1990 NAEP mathematics data. Paper presented at the annual meeting of the American Educational Research Association, San Francisco.
14. Chen, W. H. (1993). IRT-LD: A computer program for the detection of pairwise local dependence between test items (Research Memorandum 93-2). Chapel Hill: University of North Carolina at Chapel Hill, LL. Thurstone Psychometric Laboratory.
15. Chen, W. H., & Thissen, D. (1997). Local dependence indexes for item pairs using item response theory. Journal of Educational and Behavioral Statistics, 22, 265-289.
16. De Champlain, A. F. (1996). The effect of multidimensionality on IRT true-score equating for sub-groups of examinees. Journal of Educational Measurement, 33, 181-201.
17. De Champlain, A. F., & Tang, K. L. (1997). CHIDIM: A FORTRAN program for assessing the dimensionality of binary item responses based on McDonald's nonlinear factor analytic model. Educational and Psychological Measurement, 57, 174-178.
18. Douglas, J., & Cohen, A. (2001). Nonparametric item response function estimation for assessing parametric model fit. Applied Psychological Measurement, 25, 234-243.
19. Douglas, J. A., Roussos, L. A., & Stout, W. (1996). Item-bundle DIF hypothesis testing: Identifying suspect bundles and assessing their differential functioning. Journal of Educational Measurement, 33, 465-484.
20. Drasgow, F., & Parsons, C. K. (1983). Application of unidimensional item response theory models to multidimensional data. Applied Psychological Measurement, 7,189-200.
21. Elias, S., Hattie, J., & Douglas, G. (1998, April). An assessment of various IRT and SEM fit indices to detect unidimensionality. Paper presented at the annual meeting of the National Council on Measurement in Education, San Diego, CA.
22. Fraser, C., & McDonald, R. P. (1988). NOHARM: Least squares item factor analysis. Multivariate Behavioral Research, 23, 267-269.
23. Froelich, A. M., & Habing, B. (2001, April). Refinements of the DIMTEST methodology for testing unidimensionality and local independence. Paper presented at the annual meeting of the National Council on Measurement in Education, Seattle, WA.
24. Froelich, A. M., & Stout, W. F. (2003), A new bias correction method for the DIMTEST procedure. Manuscript submitted for publication.
25. 2 statistic to test the number of dimensions underlying the responses to a set of items. Journal of Educational Measurement, 33, 157-179.
26. Gessaroli, M. E., De Champlain, A., & Folske, J. C. (1997, March). Assessing dimensionality using a likelihood-ratio chi-square test based on a non-linear factor analysis of item response data. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago.
27. Hambleton, R. K., & Rovinelli, R. J. (1986). Assessing the dimensionality of a set of test items. Applied Psychological Measurement, 10, 287-302.
28. Hambleton, R. K., & Swaminathan, H., (1985). Item response theory: Principles and applications. Boston: Kluwer-Nijhoff.
29. Hattie, J. (1984). An empirical study of various indices for determining unidimensionality. Multivariate Behavioral Research, 19, 49-78.
30. Hattie, J. (1985). Methodology review: Assessing unidimensionality of tests and items. Applied Psychological Measurement, 9, 139-164.
31. Hattie, J., Krakowski, K., Rogers, H. J., & Swaminathan, H. (1996). An assessment of Stout's index of essential dimensionality. Applied Psychological Measurement, 20, 1-14.
32. Janssen, R., & DeBoeck, P. (1999). Confirmatory analyses of componential test structure using multidimensional item response theory. Multivariate Behavioral Research, 34, 245-268.
33. Jöreskog, K. G. (1990). New developments in LISREL: Analysis of ordinal variables using polychoric correlations and weighted least squares. Quality & Quantity, 24, 387-404.
34. Jöreskog, K. G., & Sorborn, D., (1988). PRELIS: A program for multivariate data screening and data summarization [Computer program], Chicago: Scientific Software, Inc.
35. Jöreskog, K. G., & Sörbom, D. (1989). LISREL 7: User's reference guide [Computer program]. Chicago: Scientific Software, Inc.
36. Junker, B. W. (1991). Essential independence and likelihood-based ability estimation for polytomous items. Psychometrika, 56, 255-278.
37. Junker, B. W., & Sijtsma, K. (2001). Nonparametric item response theory in action: An overview of the special issue. Applied Psychological Measurement, 25, 211-220.
38. Junker, B. W., & Stout, W. F. (1994). Robustness of ability estimation when multiple traits are present with one trait dominant. In D. Laveault, B. D. Zumbo, M. E. Gessaroli, & M. W. Boss (Eds.), Modern theories of measurement: Problems and issues. Ottawa, Canada: Edumetrics Research Group.
39. Kim, H. R. (1994). New techniques for the dimensionality assessment of standardized test data (Doctoral dissertation, University of Illinois at Urbana-Champaign). Dissertation Abstracts International, 55-12B, 5598.
40. Kim, H. R., Zhang, J., & Stout, W. (1995). A new index of dimensionality - DETECT. Unpublished manuscript.
41. Knol, D. L., & Berger, M. P. F. (1991). Empirical comparison between factor analysis and multidimensional item response models. Multivariate Behavioral Research, 26, 457-477.
42. McDonald, R. P. (1962). A general approach to non-linear factor analysis. Psychometrika, 27, 398-415.
43. McDonald, R. P. (1967). Nonlinear factor analysis (Psychometric Monographs, No. 15). The Psychometric Society.
44. McDonald, R. P. (1981). The dimensionality of tests and items. British Journal of Mathematical and Statistical Psychology, 34, 100-117.
45. McDonald, R. P. (1985). Factor analysis and related methods. Hillsdale, NJ: Lawrence Erlbaum.
46. McDonald, R. P. (1997). Normal-ogive multidimensional model. In W. J. van der Linden & R. K. Hambleton (Eds.), Handbook of modern item response theory. New York: Springer.
47. McDonald, R. P. (1999). Test theory: A unified treatment. Mahwah, NJ: Lawrence Erlbaum.
48. McDonald, R. P., & Mok, M. M.-C. (1995). Goodness of fit in item response models. Multivariate Behavioral Research, 30, 23-40.
49. Mislevy, R. J. (1986). Recent developments in the factor analysis of categorical variables. Journal of Educational Statistics, 11, 3-31.
50. Mislevy, R. J. (1996). Test theory reconceived. Journal of Educational Measurement, 33, 379-416.
51. Mislevy, R. J., & Bock, R. (1993). PC-BILOG 3: Item analysis and test scoring with binary logistic models. Chicago: Scientific Software.
52. Mislevy, R. J., & Verhelst, N. (1990). Modeling item responses when different subjects employ different solution strategies. Psychometrika, 55, 195-216.
53. Muraki, E. (1996). RESGEN: Item response generator (Program manual). Princeton, NJ: Educational Testing Service.
54. Muraki, E., & Engelhard, G., Jr. (1985). Full-information item factor analysis: Applications of EAP scores. Applied Psychological Measurement, 9, 417-430.
55. Muthen, B. (1984). A general structural equation model with dichotomous, ordered, and continuous latent variable indicators. Psychometrika, 49, 115-132.
56. Muthen, B. (1993). Goodness of fit with categorical and other non-normal variables. In K. A. Bollen, & J. S. Long (Eds.), Testing structural equation models (pp. 205-243). Newbury Park, CA: Sage.
57. Muthen, B., du Toit, S. H. C., & Spisic, D. (1999). Robust inference using weighted least squares and quadratic estimating equations in latent variable modeling with categorical and continuous outcomes. Manuscript submitted for publication.
58. Muthen, L. K., & Muthen, B. (1998). Mplus: The comprehensive modeling program for applied researchers: User's guide. Los Angeles: Muthen & Muthen.
59. Nandakumar, R. (1991). Traditional dimensionality versus essential dimensionality. Journal of Educational Measurement, 28, 99-117.
60. Nandakumar, R. (1993). Assessing essential unidimensionality of real data. Applied Psychological Measurement, 17, 29-38.
61. Nandakumar, R. (1994). Assessing the dimensionality of a set of item responses-Comparison of different approaches. Journal of Educational Measurement, 31,17-35.
62. Nandakumar, R., & Stout, W. (1993). Refinements of Stout's procedure for assessing latent trait unidimensionality. Journal of Educational Statistics, 18, 41-68.
63. Nandakumar, R., & Yu, F. (1996). Empirical validation of DIMTEST on nonnormal ability distributions. Journal of Educational Measurement, 33, 355-368.
64. Reckase, M. D. (1979). Unifactor latent trait models applied to multifactor tests: Results and implications. Journal of Educational Statistics, 4, 207-230.
65. Reckase, M. D. (1985). The difficulty of test items that measure more than one ability. Applied Psychological Measurement, 9, 401-412.
66. Reckase, M. D. (1997). A linear logistic multidimensional model for dichotomous items response data. In W. J. van der Linden & R. K. Hambleton (Eds.), Handbook of modern item response theory. New York: Springer.
67. Reckase, M. D., & McKinley, R. L. (1991). The discriminating power of items that measure more than one dimension. Applied Psychological Measurement, 15, 361-373.
68. Roussos, L. (1995). A new dimensionality estimation tool for multiple-item tests and a new DIF analysis paradigm based on multidimensionality and construct validity (Doctoral dissertation, University of Illinois at Urbana-Champaign). Dissertation Abstracts International, 57-04B, 2956.
69. Roussos, L. A., Reese, L., & Harris, V. (1997, March). Formulation of DETECT's conditional covariance parameter and evaluation of estimator bias. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago.
70. Roussos, L. A., Stout, W. F., & Marden, J. I. (1993, April). Dimensional and structural analysis of standardized tests using DIMTEST with hierarchical cluster analysis. Paper presented at the annual meeting of the National Council on Measurement in Education, Atlanta, GA.
71. Roussos, L., Stout, W. F., & Marden, J. I. (1998). Using new proximity measures with hierarchical cluster analysis to detect dimensionality. Journal of Educational Measurement, 35, 1-30.
72. Seraphine, A.E. (2000). The performance of DIMTEST when latent trait and item difficulty distributions differ. Applied Psychological Measurement, 24, 82-94.
73. Shealy, R., & Stout, W. F. (1993a). An item response theory model for test bias. In P. W. Holland & H. Wainer (Eds.), Differential item functioning (pp. 197-238). Hillsdale, NJ: Lawrence Erlbaum.
74. Shealy, R., & Stout, W. (1993b). A model-based standardization approach that separates true bias/DIF from group ability differences and detects test bias/DIF as well as item bias/DIF. Psychometrika, 58, 159-194.
75. Sireci, S. G., Thissen, D., & Wainer, H. (1991). On the reliability of testlet-based tests. Journal of Educational Measurement, 28, 237-247.
76. Stout, W. (1987). A nonparametric approach for assessing latent trait unidimensionality. Psychometrika, 52, 589-617.
77. Stout, W. (1990). A new item response theory modeling approach with applications to unidimensionality assessment and ability estimation. Psychometrika, 55, 293-325.
78. Stout, W. (2001). Nonparametric item response theory: A maturing and applicable measurement modeling approach. Applied Psychological Measurement, 25, 300-306.
79. Stout, W., Douglas, J., Junker, B., & Roussos, L., (1993). DIMTEST manual. Urbana-Champaign: University of Illinois at Urbana-Champaign, Department of Statistics.
80. Stout, W., Froelich, A. G., & Gao, F. (2001). Using resampling to produce an improved DIMTEST procedure. In A. Boomsma, M. A. J. van Duijn, & T. A. B. Snijders (Eds.), Essays on item response theory (pp. 357-376). New York: Springer-Verlag.
81. Stout, W., Habing, B., Douglas, J., Kim, H. R., Roussos, L., & Zhang, J. (1996). Conditional covariance-based nonparametric multidimensionality assessment. Applied Psychological Measurement, 20, 331-354.
82. Stout, W., Nandakumar, R., Junker, B., Chang, H., & Steidinger, D. (1992). DIMTEST: A Fortran program for assessing dimensionality of binary item responses. Applied Psychological Measurement, 16, 236.
83. Tanaka, J. S. (1993). Multifaceted conceptions of fit in structural equation models. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models. Newbury Park, CA: Sage.
84. Tate, R. L. (2002). Test dimensionality. In J. Tindal & T. M. Haladyna (Eds.), Large scale assessment programs for all students: Development, implementation, and analysis. Mahwah, NJ: Lawrence Erlbaum.
85. Takane, Y., & de Leeuw, J. (1987). On the relationship between item response theory and factor analysis of discretized variables. Psychometrika, 52, 393-408.
86. van der Linden, W. J., & Hambleton, R. K. (1997). Handbook of modern item response theory. New York: Springer.
87. Wang, M. (1986, April). Fitting a unidimensional model to multidimensional item response data. Paper presented at the Conference of the Office of Navel Research contractors, Gatlinburg, TN.
88. Wang, M. (1987, April). Estimation of ability parameters from response data to items that are precalibrated with a unidimensional model. Paper presented at the annual meeting of the American Educational Research Association, Washington, DC.
89. Wang, M. (1988, April). Measurement bias in the application of a unidimensional model to multidimensional item-response data. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
90. Wu, M. L., Adams, R. J., & Wilson, M. R. (1998). ACER ConQuest: Generalized item response modelling software. Melbourne, Australia: Australian Council for Educational Research.
91. Yen, W. M. (1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8, 125-145.
92. Yen, W.M. (1993). Scaling performance assessments: Strategies for managing local item dependence. Journal of Educational Measurement, 30, 187-214.
93. Zhang, J., & Stout, W. (1999a). Conditional covariance structure of generalized compensatory multidimensional items. Psychometrika, 64, 129-152
94. Zhang, J., & Stout, W. (1999b). The theoretical DETECT index of dimensionality and its application to approximate simple structure. Psychometrika, 64, 213-249.
95. Zwick, R. (1987). Assessing the dimensionality of NAEP reading data. Journal of Educational Measurement, 24, 293-308.