'Mental model' comparison of automated and human scoring

Journal of Educational Measurement

Volumn 36, Issue 2, 1999, Pages 158-184

  Williamson, David M ^a   Bejar, Isaac I ^b   Hone, Anne S ^a  

^a Chauncey Group International   (United States)

^b EDUCATIONAL TESTING SERVICE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0033147856     PISSN: 00220655     EISSN: None     Source Type: Journal    
DOI: 10.1111/j.1745-3984.1999.tb00552.x     Document Type: Article

References (26)

1. Almond, R. G., & Mislevy, R. J. (January, 1997). Graphical Models and Computerized Adaptive Testing. Paper presented at the Sixth International Workshop on Artificial Intelligence and Statistics, Fort Lauderdale, Florida.
2. Bejar, I. I. (1991). A methodology for scoring open-ended architectural design problems. Journal of Applied Psychology, 76, 522-532.
3. Bejar, I. I., & Braun, H. (1994). On the synergy between assessment and instruction: Early lessons from computer-based simulations. Machine-Mediated Learning, 4, 5-25.
4. Bejar, I. I., Yepes-Baraya, M., & Miller, S. (1997). Characterization of complex performance: From qualitative features to scores and decisions. Paper presented at the Annual Meeting of the National Council on Educational Measurement, Chicago, IL.
5. Bejar, I. I., & Bennett, R. E. (1998) La puntuaciōn de las respeutas como un parámetro del diseño de exámenes: Implicaciones en la validez [Scoring as a test design parameter: Implications for validity]. En Olea, J., Posoda, V. y Prieto, G. (Eds.), Tests informatizados: Fundamentos, estrategias y aplicaciones. Madrid: Pirámide.
6. Bennett, R. E., & Bejar, I. I. (1998). Validity and automated scoring: It's not only the scoring Educational Measurement: Issues and Practice, 17(4), 9-17.
7. Benett, R. E., & Sebrechts, M. M. (1996). The accuracy of expert-system diagnoses of mathematical problem solutions. Applied Measurement in Education, 9, 133-150.
8. Benett, R. E., Steffen, M., Singley, M. K., Morley, M. & Jacquemin, D. (1997). Evaluationg and automatically scorable, open-ended response type for measuring mathematical reasoning in computerized-adaptive tests. Journal of Educational Measurement, 34, 162-176.
9. Braun, H. I., Benett, R. E., Frye, D., & Soloway, E. (1990). Scoring constructed responses using expert systems. Journal of Educational Measurement, 27, 93-108.
10. Brennan, R. L., & Prediger, D. J. (1981). Coefficient kappa: Some uses, misuses, and alternatives. Educational and Psychological Measurement, 41, 687-699.
11. Burstein, J. Kukich, K., Wolff, S., & Lu, C. (1998). Computer analysis of essay content for automated score prediction. Presented at the Annual Meeting of the National Council on Measurement in Education. San Diego.
12. Clauser, B. E., Subhiyah, R. G., Nungester, R. J., Ripley, D. R., Clyman, D. R., & McKinley, D. (1995). Scoring a performance-based assessment by modeling the judgment process of experts. Journal of Educational Measurement, 32, 397-415.
13. Clauser, B. E., Margolis, M. J., Clyman, S. G., & Ross, L. P. (1997). Development of automated scoring algorithms for complex performance assessments: A Comparison of two approaches. Journal of Educational Measurement, 34, 141-161.
14. Clyman, S. G., Melnick, D. E., & Clauser, B. E. (1995). Computer-based simulations. In E. L. Mancall & E. G. Bashook (Eds.), Assessing clinical reasoning: The oral examination and alternative methods (pp. 139-149). Evanston, IL: American Board of Medical Specialties.
15. Cramer, H. (1946). Mathematical methods of statistics. Princeton, NJ: Princeton University Press.
16. Embretson, S. E. (1983). Construct validity: Construct representation versus nomothetic span. Psychological Bulletin, 93, 179-197.
17. Kenney, J. F. (1997). New testing methodologies for the Architect Registration Examination, CLEAR Exam Review, 8(2), 23-28.
18. Lewis, C. & Sheehan, K. (1990). Using Bayesian decision theory to design a computerized mastery test. Applied Psychological Measurement, 14, 367-380.
19. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (pp. 13-103). New York: American Council on Education.
20. Mislevy, R. J. (1994). Evidence and inference in educational assessment. Psychometika, 59(4), 439-483.
21. Mislevy, R. J. (1996). Test theory reconceived. Journal of Educational Measurement, 33, 379-416.
22. Mislevy, R. J., Steinberg, L. S., Breyer, F. J., Almond, R. G., & Johnson, L. (in press). A cognitive task analysis, with implications for designing a simulation-based assessment system. Computers in Human Behavior.
23. Nichols, P. D., Chipman, S. F., & Brennan, R. L. (Eds.) (1995). Cognitively diagnostic assessment. Hillsdale, New Jersey: Lawrence Erlbaum Associates.
24. Oltman, P. K., Bejar, I. I., & Kim, S. H. (1993). An approach to automated scoring of architectural designs. In U. Flemming & S. van Wyk (Eds.), CAAD Futures 93 (pp. 215-224). Pittsburgh, PA: Elsevier Science Publishers B.V.
25. Sebrechts, M. M., Bennett, R. E., & Rock, D. A. (1991). Agreement between expert system and human raters' scores on complex constructed-response quantitative items. Journal of Applied Psychology, 76, 856-862.
26. Williamson, D. M., Hone, A. S., Miller, S., & Bejar, I. I. (April, 1998). Classification trees for quality control processes in automated constructed response scoring. Presented at the Annual Meeting of the National Council on Measurement in Education. San Diego.