The Concept of Reliability in the Context of Automated Essay Scoring

Automated Essay Scoring: A Cross-Disciplinary Perspective

Volumn , Issue , 2003, Pages 118-138

  Cizek, Gregory J ^a   Page, Bethany A ^a  

^a UNIVERSITY OF NORTH CAROLINA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 85142597268     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.4324/9781410606860-18     Document Type: Chapter

References (29)

1. American Educational Research Association, American Psychological Association, National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
2. Brennan, R.L. (1983). Elements of generalizability theory. Iowa City, IA: American College Testing.
3. Brennan, R.L. (1998). Raw-score conditional standard errors of measurement in generalizability theory. Applied Psychological Measurement, 22, 307-331.
4. Burry-Stock, J.A., Shaw, D.G., Laurie, C., and Chissom, B.S. (1996). Rater agreement indexes for performance assessment. Educational and Psychological Measurement, 56, 251-262.
5. Cherry, R.D., and Meyer, P.R. (1993). Reliability issues in holistic assessment In M.W. Williamson and B.A.Huot (Eds.), Validating holistic scoring for writing assessment: Theoretical and empirical foundations (pp. 109-141). Cresskill, NJ: Hampton.
6. Cizek, G.J. (1996). Standard-setting guidelines. Educational Measurement: Issues and Practice, 15(1) 13-21.
7. Cizek, G.J. (2001). Setting performance standards: Concepts, methods, and perspectives. Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
8. Clauser, B.E., Harik, P., and Clyman, S.G. (2000). The generalizability of scores for a performance assessment scored with a computer-automated scoring system. Journal of Educational Measurement, 37, 245-261.
9. Crick, J.E., and Brennan, R.L. (1984). GENOVA; A general purpose analysis of variance system [Computer software]. Iowa City, IA: American College Testing.
10. Cronbach, L.J., Gleser, G.C., Nanda, H., and Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: Wiley.
11. Ebel, R.L. (1961). Must all tests be valid? American Psychologst, 16, 640-647.
12. Feldt, L.S., and Brennan, R.L. (1989). Reliability. In R.L.Linn (Ed), Educational measurement, third edition (pp. 105-146). New York: Macmillan.
13. Feldt, L.S., Steffen, M., and Gupta, N.C. (1985). A comparison of five methods for estimating the standard error of measurement at specific score levels. Applied Psychological Measurement, 9, 351-361.
14. Feldt, L.S., and Qualls, A.L. (1998). Approximating scale score standard error of measurement from the raw score standard error. Applied Measurement in Education, 11, 159-177.
15. Gulliksen, H. (1950). Theory of mental tests. New York: Wiley.
16. Hambleton, R.K., Swaminathan, H., and Rogers, H.J. (1991). Fundamentals of item response theory. Newbury Park, CA: Sage.
17. Keats, J.A. (1957). Estimation of error variances of test scores. Psychometrika, 22, 29-41.
18. Kolen, M.J., Hanson, B.A., and Brennan, R.L. (1992). Conditional standard errors of measurement for scale scores. Journal of Educational Measurement, 29, 285-307.
19. Kolen, M.J., Zeng, L., and Hanson, B.A. (1996). Conditional standard errors of measurement for scale scores using IRT. Journal of Educational Measurement, 33, 129-140.
20. Livingston, S.A., and Lewis, C. (1995). Estimating the consistency and accuracy of classifications based on test scores. Journal of Educational Measurement, 32, 179-197.
21. Lord, F.M. (1980). Applications of item response theory to practical testing problems, Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
22. Lord, F.M. (1984). Standard errors of measurement at different ability levels. Journal of Educational Measurement, 21, 239-243.
23. Page, E.B., and Petersen, N.S. (1995). The computer moves into essay grading. Phi Delta Kappan 76, 561-565.
24. Shavelson, R.J., and Webb, N.M. (1991). Generalizability theory: A primer. Newbury Park, CA:Sage.
25. Spearman, C. (1904). The proof and measurement of association between two things. Journal of Psychology, 15, 72-101.
26. Stanley, J.C. (1998, April). In Ellis B.Page (chair), Qualitative and quantitative essay grading by computer. Symposium conducted at the annual meeting of the American Educational Research Association, San Diego, CA.
27. Subkoviak, M.J. (1976). Estimating reliability from a single administration of a criterion-referenced test Journal of Educational Measurement, 13, 265-276.
28. Traub, R.E. (1997). Classical test theory in historical perspective. Educational Measurement:Issues and Practise, 16, 8-14.
29. Wang, T., Kolen, M.J., and Harris, D.J. (2000). Psychometric properties of scale scores and performance levels for performance assessments using polytomous IRT. Journal of Educational Measurement, 37, 141-162.