Automated tools for subject matter expert evaluation of automated scoring

Applied Measurement in Education

Volumn 17, Issue 4, 2004, Pages 323-357

  Williamson, David M ^a   Bejar, Isaac I ^a   Sax, Anne ^b  

^a EDUCATIONAL TESTING SERVICE   (United States)

^b Chauncey Group International   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 8744280233     PISSN: 08957347     EISSN: None     Source Type: Journal    
DOI: 10.1207/s15324818ame1704_1     Document Type: Review

References (41)

1. Almond, R. G., Steinberg, L. S., & Mislevy, R. J. (2002). Enhancing the design and delivery of assessment systems: A four process architecture. The Journal of Technology, Learning and Assessment, 1(5). Retrieved January 12, 2003 from http://www.jtla.org
2. Balakrishan, P. V., Cooper, M. C., Jacob, V. S., & Lewis, P. A. (1994). A study of the classification capabilities of neural networks using unsupervised learning: A comparison with K-means clustering. Psychometrika, 59, 509-525.
3. Bauer, M., Williamson, D. M., Steinberg, L. S., Mislevy, R. J., & Behrens, J. T. (2001, April). How to create complex measurement models: A case study of Principled Assessment Design. Paper presented at the annual meeting of the National Council on Measurement in Education, Seattle, WA.
4. Bejar I. I. (1991). A methodology for scoring open-ended architectural design problems. Journal of Applied Psychology, 76, 522-532.
5. Bejar, I. I., & Braun, H. (1994). On the synergy between assessment and instruction: Early lessons from computer-based simulations. Machine-Mediated Learning, 4, 5-25.
6. Bejar, I. I., & Braun, H. I. (1999). Architectural simulations: From research to implementation: Final report to the National Council of Architectural Registration Boards (RM-99-2). Princeton, NJ: Educational Testing Service.
7. Bejar, I. I., & Whalen, S. J. (1997, March). A system for interactive standard setting. Paper presented at the annual conference of the National Council on Measurement in Education, Chicago, IL.
8. Bejar, I. I., Yepes-Baraya, M., & Miller, S. (1997, March). Characterization of complex performance: From qualitative features to scores and decisions. Paper presented at the annual conference of the National Council on Measurement in Education, Chicago, IL.
9. Bennett, R. E., & Bejar, I. I. (1998). Validity and automated scoring: It's not only the scoring. Educational Measurement: Issues and Practice, 17(4), 9-17.
10. Bennett, R. E., & Sebrechts, M. M. (1996). The accuracy of expert-system diagnoses of mathematical problem solutions. Applied Measurement in Education, 9, 133-150.
11. Bennett, R. E., Steffen, M., Singley, M. K., Morley, M, & Jacquemin, D. (1997). Evaluating an automatically scorable, open-ended response type for measuring mathematical reasoning in computerized-adaptive tests. Journal of Educational Measurement, 34, 162-176.
12. Braun, H. I., Bennett, R. E., Frye, D, & Soloway, E. (1990). Scoring constructed responses using expert systems. Journal of Educational Measurement, 27, 93-108.
13. Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984). Classification and regression trees. Belmont, CA: Wadsworth.
14. Burstein, J., Kukich, K., Wolff, S., & Lu, C. (1998, April). Computer analysis of essay content for automated score prediction. Paper presented at the annual meeting of the National Council of Measurement in Education, San Diego, CA.
15. Clauser, B. E., Margolis, M. J., Clyman, S. G., & Ross, L. P. (1997). Development of automated scoring algorithms for complex performance assessments: A comparison of two approaches. Journal of Educational Measurement, 34, 141-161.
16. Clauser, B. E., Subhiyah, R. G., Nungenster, R. J., Ripkey, D. R., Clyman, D. R., & McKinley, D. (1995). Scoring a performance-based assessment by modeling the judgement process of experts. Journal of Educational Measurement, 32, 397-415.
17. Clyman, S. G., Melnick, D. E., & Clauser, B. E. (1995). Computer-based simulations. In E. L. Mancall & E. G. Bashook (Eds.), Assessing clinical reasoning: The oral examination and alternative methods (pp. 139-149). Evanston, IL: American Board of Medical Specialties.
18. Cronbach, L. J. (1988). Five perspectives on the validity argument. In H. Wainer & H. I. Braun (Eds.), Test Validity (pp. 3-17). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
19. Fisher, R. A. (1936). The use of multiple measurements in taxonomic problems. Annals of Eugenics, 7, 179-188.
20. Greene, E. B. (1941). Measurements of human behavior. New York: The Odyssey Press.
21. Holland, P. W., Ponte, E., Crane, E., & Malberg, R. (1998, April) Treeing in CAT: Regression trees and adaptive testing. Paper presented at the annual conference of the National Council on Measurement in Education, San Diego, CA.
22. Hubel, D. H., & Wiesel, T. V. (1962). Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. Journal of Physics, 160, 106-154.
23. Kenney, J. F. (1997). New testing methodologies for the Architect Registration Examination. CLEAR Exam Review, 8(2), 23-28.
24. Kohonen, T. (1989). Self-organization and associative memory (3rd ed.). New York: Springer-Verlag.
25. Landauer, T. K., Laham, D., & Foltz, P. W., (1998). Learning human-like knowledge by Singular Value Decomposition: A progress report. In M. I. Jordan, M. J. Kearns, & S. A. Solla (Eds.), Advances in neural information processing systems, 10 (pp. 45-51). Cambridge, MA: MIT Press.
26. Meehl, P. E. (1954). Clinical vs. statistical prediction: A theoretical analysis and a review of the evidence. Minneapolis: University of Minnesota Press.
27. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-103) New York: American Council on Education-Macmillan.
28. Michie, D., Spiegelhalter, D. J., & Taylor, C. C. (Eds.). (1994). Machine learning, neural and statistical classification. West Sussex, England: Ellis Horwood.
29. Mislevy, R. J. (1994). Evidence and inference in educational assessment. Psychometrika, 59, 439-483.
30. Mislevy, R. J. (1996). Test theory reconceived. Journal of Educational Measurement, 33, 379-416.
31. Murtagh, F., & Hernandez-Pajares, M. (1995). The Kohonen self-organizing map method: An assessment. Journal of Classification, 12, 165-190.
32. Oltman, P. K., Bejar, I. I., & Kim, S. H. (1993). An approach to automated scoring of architectural designs. In U. Flemming & S. van Wyk (Eds.), CAAD Futures 93 (pp. 215-224). Pittsburgh, PA: Elsevier.
33. Ripley, B. D. (1996). Pattern Recognition and Neural Networks. Cambridge, England: Cambridge University Press.
34. Sebrechts, M. M., Bennett, R. E., & Rock, D. A. (1991). Agreement between expert system and human raters' scores on complex constructed-response quantitative items. Journal of Applied Psychology, 76, 856-862.
35. Sheehan, K. M. (1997). A tree-based approach to proficiency scaling and diagnostic assessment. Journal of Educational Measurement, 34, 333-352.
36. Steinberg, D., & Colla, P. (1992). CART. Evanston, IL: Systat.
37. Stevens, R., Ikeda, J., Casillas, A., Palacio-Cayetano, J., & Clyman, S. (1999). Artificial neural-network based performance assessments. Computers in Human Behavior, 15, 295-313.
38. Waller, N. G., Kaiser, H. A., Illian, J. B., & Manry, M. (1998). A comparison of the classification capabilities of the 1-dimensional Kohonen neural network with two partitioning and three hierarchical cluster analysis algorithms. Psychometrika, 63, 5-22.
39. Williamson D. M., Bejar I. I., & Hone, A. S. (1999). "Mental model" comparison of automated and human scoring. Journal of Educational Measurement, 36, 158-184.
40. Winston P. H. (1992). Artificial intelligence (3rd ed.). Boston: Addison-Wesley.
41. Zhang, Y., Powers, D. E., Wright, W., & Morgan, R. (2003, April). Applying the Online Scoring Network (OSN) to Advanced Placement program (AP) tests (RR-03-12). Princeton, NJ: Educational Testing Service.