Issues in the Reliability and Validity of Automated Scoring of Constructed Responses

Automated Essay Scoring: A Cross-Disciplinary Perspective

Volumn , Issue , 2003, Pages 21-36

  Chung, Gregory K W K ^a   Baker, Eva L ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 85142566672     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.4324/9781410606860-10     Document Type: Chapter

References (57)

1. Almond, R., Steinberg, L., and Mislevy, R. (2001). A sample assessment using the four process framework (CSE Tech. Rep. No. 543). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
2. American Educational Research Association, American Psychological Association, and National Council for Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
3. Baker, E.L. (1995). Learning based assessments of history understanding. Educational Psychologist, 29, 97-106.
4. Baker, E.L., Freeman, M., and Clayton, S. (1991). Cognitive assessment of history for large-scale testing. In M.C.Wittrock and E.L.Baker (Eds.), Testing and cognition (pp. 131-153). Englewood Cliffs, NJ: Prentice-Hall.
5. Baker, E.L., and Herman, J.L. (in press). Technology and evaluation. In G.Haertel and B, Means (Eds.), Approaches to evaluating the impact of educational technology. New York: Teachers College Press.
6. Baker, E.L., Linn, R.L., Abedi, J., and Niemi, D. (1995). Dimensionality and generalizability of domain-independent performance assessments. Journal of Educational Research, 89, 197-205.
7. Baker, E.L., and Mayer, R.E. (1999). Computer-based assessment of problem solving. Computers in Human Behavior, 15, 269-282.
8. Baker, E.L., and O’Neil, H. F., Jr. (1996). Performance assessment and equity. In M.B.Kane and R.Mitchell (Eds.), Implementing performance assessment: Promises, problems, and challenges (pp. 183-199). Mahwah, NJ: Erlbaum.
9. Baker, E.L., O’Neil, H.F., Jr., and Linn, R.L. (1993). Policy and validity prospects for performancebased assessment. American Psychologist, 48, 1210-1218.
10. Bennett, R.E. (1993a). On the meaning of constructed response. In R.E.Bennett and W.C. Ward (Eds.), Construction versus choice in cognitive measurement: Issues in constructed response, performance testing and portfolio assessment (pp. 1-27). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
11. Bennett, R.E. (1993b). Toward intelligent assessment: An integration of constructed-response testing, artificial intelligence, and model-based measurement. In N. Frederiksen, R.J.Mislevy, and I.I.Bejar (Eds.), Test theory for a new generation of tests (pp. 99-123). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
12. Bennett, R.E., and Bejar, I.I. (1998). Validity and automated scoring: It’s not only the scoring, Educational Measurement: Issues and Practice, 17(4), 9-17.
13. Brennan, R.L. and Johnson, E.G. (1995). Generalizability of performance assessments, Educational Measurement: Issues and Practice, 14(4), 9-12.
14. Burstein, J. (2001, April). Automated essay evaluation with natural language processing. Paper presented at the annual meeting of the National Council on Measurement in Education, Seattle, WA.
15. Burstein, J.C., and Chodorow, M. (1999, June). Automated essay scoring for nonnative English speakers. In Computer-mediate language assessment and evaluation of natural language processing Joint symposium of the Association of Computational Linguistics and the International Association of Language Learning Technologies, College Park, MD.
16. Burstein, J., Kukich, K., Braden-Harder, L., Chodorow, M., Hua, S., Kaplan, B., et al (1998). Computer analysis of essay content for automatic score prediction: A prototype automated scoring system for GMAT analytical writing assement (RR-98-15). Princeton, NJ: Educational Testing Service.
17. Burstein, J., Kukich, K., Wolff, S., and Lu, C. (1998, April). Computer analysis of essay content for automated score prediction. Paper presented at the annual meeting of the National Council on Measurement in Education, San Diego, CA.
18. Burstein, J., Wolff, S., and Lu, C. (1999). Using lexical semantic techniques to classify freeresponses. In E.Viegas (Ed.), Breadth and depth of semantic lexicons (pp. 227-246). New York:Kluwer.
19. Chi, M.T.H., Glaser, R., and Farr, M.J. (Eds.). (1988). The nature of expertise, Hillsdale, NJ:Lawrence Erlbaum Associates, Inc.
20. Chung, G.K.W.K., de Vries, L.F., Cheak, A.M., Stevens, R.H., and Bewley, W.L. (in press). Process measures of problem solving. Computers in Human Behavior.
21. Chung, G.K.W.K., Harmon, T.C., and Baker, E.L. (2001). The impact of a simulation-based learning design project on student learning. IEEE transactions on Education, 44, 390-398.
22. Clauser, B. E, (2000). Recurrent issues and recent advances in scoring performance assessments. Applied Psychological Measurement, 24, 310-324.
23. Clauser, B.E., Margolis, M.J., Clyman, S.G., and Ross, L.P. (1997). Development of automated scoring algorithms for complex performance assessments: A comparison of two approaches. Journal Educational Measurement, 34, 141-161.
24. Clauser, B.E., Subhiyah, R G., Nungester, R.J., Ripkey, D.R., Clyman, S.G., and McKinley, D. (1995). Scoring a performance-based assessment by modeling the judgments of experts. Journal of Educational Measurement, 32, 397-415.
25. Clauser, B.E., Swanson, D.B., and Clyman, S.G. (1999). A comparison of the generalizability of scores produced by expert raters and automated scoring systems. Applied Measurement in Education, 12, 281-299.
26. Cohen, P.R. (1995). Empirical methods for artificial intellect, Cambridge, MA: MIT Press.
27. Embretson, S.E. (1998). A cognitive design system approach to generating valid tests: Application to abstract reasoning. Psychological Methods, 5, 380-396.
28. Herl, H.E., Niemi, D., and Baker, E.L. (1996). Construct validation of an approach to modeling cognitive structure of U.S. history knowledge. Journal of Educational Research, 89, 206-218.
29. Herl, H.E., O’Neil, H.F., Jr., Chung, G.K.W.K., and Schacter, J. (1999). Reliability and validity of a computer-based knowledge mapping system to measure content understanding. Computers in Human Behavior, 15, 315-334.
30. Herrmann, D.S. (1999). Software safety and reliability: Techniques, approaches, and standards of key industrial sectors. Piscataway, NJ: IEEE Computer Society.
31. Jaeger, R.M., and Craig, N. (2001). An integrated judgment procedure for setting standards on complex, large-scale assessments. In G.J.Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 313-338). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
32. Kane, M., Crooks, T., and Cohen, A. (1999). Validating measures of performance. Educational Measurement: Issues and Practice, 18(2), 5-17.
33. Klein, D.C. D., Chung, G.K. W.K., Osmundson, E., Herl, H.E., and O’Neil, H.F., Jr. (2001). Examining the validity of knowledge mapping as a measaure of elementary students’ Scientific understanding (Final deliverable to OERI). Los Angelses: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
34. Klein, D.C.D., Yarnall, L., and Glaubke, C. (2001). Using technology to assess students’ Web expertise (CSE Tech. Rep. No. 544). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
35. Landauer, T.K., Foltz, P.W., and Laham, D. (1998). An introduction to latent semantic analysis. Discourse Processes, 25, 259-284.
36. Landauer, T.K., Laham, D., Render, B., and Schreiner, M.E. (1997). How well can passage meaning be derived without using word order? A comparison of latent semantic analysis and humans. Proceedings of the 19th annual meeting of the Cognitive Science Society, USA, 412-417.
37. Linn, R.L., Baker, E.L., and Dunbar, S.B. (1991). Complex, performance-based assessment Expectations and validation criteria. Educational Researcher, 20(8), 15-21.
38. Messick, S. (1989). Validity. In R.L.Linn (Ed), Educational measurement (3rd ed., pp. 13-103). New York: Macmillan.
39. Mislevy, R.J., Steinberg, L.S., Breyer, F.J., Almond, R.G., and Johnson, L. (1999). A cognitive task analysis with implications for designing simulation-based performance assessment. Computers in Human Behavior, 15, 335-3374.
40. Mislevy, R, Steinberg, L., Almond, R., Breyer, F.J., and Johnson, L. (2001). Making sense of data from complex assessments (CSE Tech. Rep. No. 538). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
41. Nichols, P.D., and Kuehl, B.J. (1999). Prophesying the reliability of cognitively complex assessments. Applied Measurement in Education, 12, 73-94.
42. Nichols, P.D., and Smith, P.L. (1998). Contextualmng the interpretation of reliability data. Educational Measurement; Issues and Practice, 17(3), 24-36.
43. Nichols, P.D., and Sugrue, B. (1999). Contextualizing the interpretation of reliability data. Educational Measurement: Issues and Practice, 18(2), 18-29.
44. O’Neil, H.F., Jr., and Baker, E.L. (1991). Issues in intelligent computer-assisted instruction:Evaluation and measurement In T.B.Gutkin and S.L.Wise (Eds.), The computer and the decisionmaking process (pp. 199-224). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
45. Osmundson, E., Chung, G.K. W.K., Herl, H.E., and Klein, D.C.D. (1999). Concept mapping in the classroom: A tool for examining the development of students’ conceptual understandings (CSE Tech. Rep. No. 507). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
46. Page, E.B. (1966). The imminence of grading essays by computer. Phi Delta Kappan, 47, 238-243.
47. Page, E.B. (1994). New computer grading of student prose, using modem concepts and software. Journal of Experimental Education, 62, 127-142.
48. Page, E.B., and Petersen, N.S. (1995). The computer moves into essay grading: Updating the ancient test. Phi Delta Kappan, 76, 561-565.
49. Pfleeger, S.L. (1998). Software engineering: Theory and practice. Upper Saddle River, NJ:Prentice Hall.
50. Rogosa, D. (1999a). Accuracy of individual scores expressed in percentile ranks: Classical test theory calculations. (CSE Tech. Rep. No. 509). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
51. Rogosa, D. (1999b). Accuracy of year-1, year-2 comparisons using individual percentile rank scores: Classical test theory calculations, (CSE Tech. Rep. No. 510). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
52. Rogosa, D. (1999c). How accurate are the STAR national percentile rank scores for individual students? An interpretive guide. Palo Alto, CA: Stanford University Press.
53. Schacter, J., Chung, G.K.W.K., and Dorr, A. (1998). Children’s Internet searching on complex problems: Performance and process analyses. Journal of the American Society for Information Science, 49, 840-849.
54. Schacter, J., Herl, H.E., Chung, G.K.W.K., Dennis, R.A., and O’Neil, H.F., Jr. (1999). Computerbased performance assessments: A solution to the narrow measurement and reporting of problem solving. Computers in Human Behavior, 15, 403-118.
55. Shavelson, R.J., and Webb, N.M. (1991). Generalizablity theory: A primer. Newbury Park, CA:Sage.
56. Stevens, R., Ikeda, J., Casillas, A., Palacio-Cayetano, J., and Clyman, S. (1999). Artificial neural network-based performance assessments. Computers in Human Behavior, 15, 295-313.
57. Williamson, D.M., Bejar, I.I., and Hone, A.S. (1999). ‘Mental model’ comparison of automated and human scoring. Journal of Educational Measurement, 36, 158-184.