A method for analyzing sparse data matrices in the generalizability theory framework

Applied Psychological Measurement

Volumn 26, Issue 3, 2002, Pages 321-338

  Chiu, Christopher W T ^a   Wolfe, Edward W ^b  

^a Law School Admission Council   (United States)

^b MICHIGAN STATE UNIVERSITY   (United States)

Author keywords

Analysis of variance (ANOVA); Crossed nested MBIB design; Generalizability theory and method; Interrater reliability; Large scale analysis; Missing data; Performance assessment; Rater design; Variance component

Indexed keywords

EID: 0036375218     PISSN: 01466216     EISSN: None     Source Type: Journal    
DOI: 10.1177/0146621602026003006     Document Type: Article

References (39)

1. Babb, J. S. (1986). Pooling maximum likelihood estimates of variance components obtained from subsets of unbalanced data. Unpublished master's thesis, Cornell University.
2. Bell, J. F. (1985). Generalizability theory: The software problem. Journal of Educational Statistics, 10(1), 19-29.
3. Brennan, R. L. (1992a). Elements of generalizability theory. Iowa City, IA: American College Testing.
4. Brennan, R. L. (1992b). An NCME instructional module on generalizability theory. Educational Measurement: Issues and Practice, 11(4), 27-34.
5. Brennan, R. L., Gao, X., & Colton, D. A. (1995). Generalizability analyses of work keys listening and writing tests. Educational and Psychological Measurement, 55(2), 157-176.
6. Brennan, R. L., Harris, D. J., & Hanson, B. A. (1987). The bootstrap and other procedures for examining the variability of estimated variance components in testing contexts. Paper presented at the annual meeting of the National Council on Measurement in Education, Washington, DC.
7. Brennan, R. L., Jarjoura, D., & Deaton, E. L. (1980). Some issues concerning the estimation and interpretation of variance components in generalizability theory. ACT Technical Bulletin (Report No. 36). Iowa City, IA: American College Testing.
8. CAAP (2001). Documentation of the Collegiate Assessment of Academic Proficiency [Online]. Available: http://www.act.org/caap/index.html
9. Chiu, C. W. T. (1999). Scoring performance assessments based on judgements: Utilizing metaanalysis to estimate variance components in generalizability theory for unbalanced situations. Unpublished doctoral dissertation, Michigan State University.
10. Chiu, C. W. T. (2001a). Generalizability theory: Scoring large-scale assessments based on judgements. Boston, MA: Kluwer Academic.
11. Chiu, C. W. T. (2001b). An SPSS module: Generalizability analysis for balanced and unbalanced designs [Software]. Law School Admission Council (LSAC), Newtown, PA.
12. Crick, J. E., & Brennan, R. L. (1982). GENOVA: A generalized analysis of variance system (FORTRAN IV computer program and manual). Dorchester, MA: Computer Facilities, University of Massachusetts at Boston.
13. Cronbach, L. J., Bradburn, N. M., & Horvitz, D. G. (1994). Sampling and statistical procedures used in the California Learning Assessment System. Report of the Select Committee, Palo Alto, CA.
14. Cronbach, L. J., Gleser, G. C., & Rajaratnam, N. (1963). Theory of generalizability. A liberalization of reliability theory. British Journal of Mathematical and Statistical Psychology, 16, 137-173.
15. Donoghue, J. R., & Hombo, C. M. (2000). A comparison of different model assumptions bout rater effects. Paper presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA.
16. Engelhard, G., Jr. (1997). Constructing rater and task banks for performance assessments. Journal of Outcome Measurement, 1(1), 19-93.
17. Fleiss, J. L. (1981). Balanced incomplete block designs for inter-rater reliability studies. Applied Psychological Measurement, 5, 105-112.
18. Harwell, M., Stone, C. A., Hsu, T. C., & Kirisci, L. (1996). Monte Carlo studies in item response theory. Applied Psychology Measurement, 20(2), 101-125.
19. Henderson, C. R. (1953). Estimation of variance and covariance components. Biometrics, 9, 227-252.
20. Hoyt, W. T. (2000). Rater bias in psychological research: When is it a problem and what can we do about it? Psychological Methods, 5(1), 64-86.
21. Koretz, D., Stecher, B., & McCaffrey, D. (1994). The Vermont Portfolio Assessment Program: Findings and implications. Educational Measurement: Issues and Practice, 13, 5-6.
22. Lane, S., Liu, M., Ankenmann, R. D., & Stone, C. A. (1996). Generalizability and validity of a mathematics performance assessment. Journal of Educational Measurement, 33(1), 71-92.
23. Longford, N. T. (1995). Models for uncertainty in educational testing. New York: Springer-Verlag.
24. Luecht, R. M., & Smith, P. L. (1989). The effects of bootstrapping strategies on the estimation of variance components. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.
25. Messick, S. (1989). Meaning and values in test validation: The science and ethics of assessment. Educational Researcher, 18, 5-11.
26. Myford, C. M., Marr, D. B., & Linacre, J. M. (1995). Reader calibration and its potential role in equating for the Test of Written English (Report prepared by the Center for Performance Assessment, MS # 95-02). Princeton, NJ: Educational Testing Service.
27. Nitko, A. J. (1996). Educational assessment of students (2nd ed.). Englewood Cliffs, NJ: Prentice Hall.
28. Othman, A. R. (1995). Examining task sampling variability in science performance assessments. Unpublished doctoral dissertation, University of California, Santa Barbara.
29. Patz, R. (1996). Markov chain Monte Carlo methods for item response theory models. Unpublished doctoral dissertation, Carnegie Mellon University, Pittsburgh, PA.
30. Saal, F. E., Downey, R. G., & Lahey, M. A. (1980). Rating the rating: Assessing the psychometric quality of rating data. Psychological Bulletin, 88, 413-428.
31. SAS Institute, Inc. (1985). SAS user's guide: Statistics (Version 5). Cary, NC: Author.
32. Schroeder, M. L. (1986). Inferential procedures for multifaceted coefficients of generalizability. Unpublished doctoral dissertation, University of British Columbia.
33. Searle, S. R. (1987). Linear models for unbalanced data. New York: John Wiley.
34. Searle, S. R., Casella, G., & McCulloch, C. E. (1992). Variance components. New York: John Wiley.
35. Shavelson, R. J., Baxter, G. P., & Gao, X. (1993). Sampling variability of performance assessments. Journal of Educational Measurement, 30(3), 215-232.
36. Smith, P. L. (1982). A confidence interval approach for variance component estimates in the context of generalizability theory. Educational and Psychological Measurement, 42, 459-466.
37. Texas Education Agency. (1999). Student Assessment Division report: Writing assessment (Chap. 11). Retrieved from http://www.tea.state.tx.us/ student.assessment/TECHDIG/chapt11.ht
38. Wainer, H. (1993). Measurement problems. Journal of Educational Measurement, 30(1), 1-21.
39. Wolfe, E. W., Chiu, C. W. T., & Myford, C. M. (2000). Detecting rater effects with a multi-faceted Rasch rating scale model. In M. Wilson & G. Engelhard (Eds.), Objective measurement: Theory into practice (Vol. 5, pp. 147-164). Princeton, NJ: Educational Testing Service.