A validity-based approach to quality control and assurance of automated scoring

Assessment in Education: Principles, Policy and Practice

Volumn 18, Issue 3, 2011, Pages 319-341

  Bejar, Isaac I ^a  

^a EDUCATIONAL TESTING SERVICE   (United States)

Author keywords

Automated; Marking; Scoring

Indexed keywords

EID: 79961058822     PISSN: 0969594X     EISSN: 1465329X     Source Type: Journal    
DOI: 10.1080/0969594X.2011.555329     Document Type: Article

References (94)

1. Almond, R.G., L.S. Steinberg, and R.J. Mislevy. 2002. Enhancing the design and delivery of assessment systems: A four-process architecture. Journal of Technology, Learning, and Assessment in Education 1, no. 5. http://www.jtla.org.
2. American Institute of Architects, A. Pressman, and Smith Maran Architects. 2007. Architectural graphic standards. 11th ed. Hoboken, NJ: John Wiley & Sons.
3. Attali, Y., and D. Powers. 2010. Immediate feedback and opportunity to revise answers to open-ended questions. Educational and Psychological Measurement 70, no. 1: 22-35.
4. Baker, F.B. 1971. Automation of test scoring, reporting and analysis. In Educational measurement, 2nd ed., ed. R.L. Thorndike, 202-236. Washington, DC: American Council on Education.
5. Bejar, I.I. 1991. A methodology for scoring open-ended architectural design problems. Journal of Applied Psychology 76, no. 4: 522-532.
6. Bejar, I.I. 2002. Generative testing: From conception to implementation. In Item generation for test development, ed. S.H. Irvine and P.C. Kyllonen, 199-218. Mahwah, NJ: Erlbaum.
7. Bejar, I.I. 2010. Can speech technology improve assessment and learning? New capabilities may facilitate assessment innovations. No. RDC-15. Princeton, NJ: Educational Testing Service. http://www.ets.org/research/policy_research_reports/rdc-15.
8. Bejar, I.I., and E.A. Graf. 2010. Updating the duplex design for test-based accountability in the twenty-first century. Measurement: Interdisciplinary Research & Perspective 8, no. 2: 110-129.
9. Ben-Simon, A., and R.E. Bennett. 2007. Toward more substantively meaningful automated essay scoring. Journal of Technology, Learning, and Assessment 6, no. 1. http://www.jtla.org.
10. Bennett, R.E., and I.I. Bejar. 1998. Validity and automated scoring: It's not only the scoring. Educational Measurement: Issues and Practice 17, no. 4: 9-16.
11. Bennett, R.E., and M.M. Sebrechts. 1996. The accuracy of expert-system diagnoses of mathematical problem solutions. Applied Measurement in Education 9, no. 2: 133-150.
12. Bennett, R.E., M. Steffen, M.K. Singley, M. Morley, and D. Jacquemin. 1997. Evaluating an automatically scorable, open-ended response type for measuring mathematical reasoning in computer-adaptive tests. Journal of Educational Measurement 34, no. 2: 162-176.
13. Bernstein, J., and J. Chen. 2008. Logic and validation of a fully automatic spoken English test. In The path of speech technologies in computer assisted language learning: From research toward practice, ed. V.M. Holland and F.P. Fisher, 174-194. New York: Routledge.
14. Booz Allen Hamilton. 2006. SAT process review: Answer sheet processing. College Board. http://www.collegeboard.com/prod_downloads/satscoreprocessing/SAT_Process_Review_Final_Report.ppt.
15. Braun, H., I.I. Bejar, and D.M. Williamson. 2006. Rule-based methods for automated scoring: Applications in a licensing context. In Automated scoring of complex tasks in computer- based testing, ed. D.M. Williamson, R.J. Mislevy, and I.I. Bejar, 83-122. Mahwah, NJ: Lawrence Erlbaum.
16. Breithaupt, K., C.N. Mills, and G. Melican. 2006. Facing up to opportunities of the future. In Computer-based testing and the internet: Issues and advances, ed. D. Bartram and R. K. Hambleton, 219-251. Chichester: John Wiley & Sons.
17. Brennan, R.L. 2004. Some perspectives on inconsistencies among measurement models (CASMA Research Report Number 8). Iowa City, IA: University of Iowa. http://www.education.uiowa.edu/casma/NSF-casma-rpt.pdf.
18. Brennan, R.L., ed. 2006. Educational measurement. 4th ed. Westport, CT: Praeger.
19. Bridgeman, B., I.I. Bejar, and D. Friedman. 1999. Fairness issues in a computer-based architectural licensure examination. Computers in Human Behavior 15, nos. 3/4: 419-440.
20. Bridgeman, B., M.L. Lennon, and A. Jackental. 2001. Effect of screen size, screen resolution, and display rate on computer-based test performance. No. RR-01-23. Princeton, NJ: Educational Testing Service. http://www.ets.org/Media/Research/pdf/RR-01-23-Bridgeman.pdf.
21. Bridgeman, B., C. Trapani, and J. Bivens-Tatum. 2009. Comparability of essay question variants. Princeton, NJ: ETS.
22. Burstein, J., M. Chodorow, and C. Leacock. 2004. Automated essay evaluation. AI Magazine 25, no. 3: 27-36.
23. Butcher, P.G., and S.E. Jordan. 2010. A comparison of human and computer marking of short free-text student responses. Computers in Education 55, no. 2: 489-499.
24. Chandola, V., A. Benerjee, and V. Kumar. 2009. Anomaly detection: A survey. ACM Computing Surveys 41, no. 3: article 15.
25. Church, R.L., and J.R. Marston. 2003. Measuring accessibility for people with a disability. Geographical Analysis 35, no. 1: 83-96.
26. Cizek, G.J., and M.B. Bunch. 2007. Standard setting: A guide to establishing and evaluating performance standards on tests. Thousand Oaks, CA: Sage.
27. Clauser, B.E., M.T. Kane, and D.B. Swanson. 2002. Validity issues for performance-based tests scored with computer-automated scoring systems. Applied Measurement in Education 15, no. 4: 413-432.
28. Clauser, B.E., L.P. Rose, S.G. Clyman, K.M. Rose, M.J. Margolis, R.J. Nungester, T.E. Piemme, et al. 1997. Development of a scoring algorithm to replace expert rating for scoring a complex performance-based assessment. Applied Measurement in Education 10, no. 4: 345-358.
29. Coffman, W.E. 1971. Essay examinations. In Educational measurement, 2nd ed., ed. R.L. Thorndike, 271-302. Washington, DC: American Council on Education.
30. Cronbach, L.J., and P.E. Meehl. 1955. Construct validity in psychological tests. Psychological Bulletin 52, no. 4: 281-302.
31. Datta, R., D.J. Joshi, J. Li, and J.W. Wang. 2006. Studying aesthetics in photographic images using a computational approach. In Lecture Notes in Computer Science, vol. 3953, Proceedings of the European Conference on Computer Vision, Part III, ed. A. Leonardis, B. Horst, and A. Pinz, 288-301. Graz, Austria: Springer-Verlag.
32. Deane, P. 2006. Strategies for evidence identification through linguistic assessment of textual responses. In Automated scoring of complex tasks in computer-based testing, ed. D.M. Williamson, R.J. Mislevy, and I.I. Bejar, 313-371. Mahwah, NJ: Lawrence Erlbaum Associates.
33. Dikli, S. 2006. An overview of automated scoring of essays. Journal of Technology, Learning, and Assessment 5, no. 1. http://www.jtla.org.
34. Embretson, S.E. 1983. Construct validity: Construct representation versus nomothetic span. Psychological Bulletin 93, no. 1: 179-197.
35. Eskenazi, M. 2009. An overview of spoken language technology for education. Speech Communication 51, no. 10: 832-844.
36. Faggen, J. 1994. Setting standards for constructed-response tests: An overview. Research Memorandum No. RM-94-19. Princeton, NJ: Educational Testing Service.
37. Fife, J.H. 2011. Integrating item generation and automated scoring. Paper presented at the National Council of Measurement in Education, April, in New Orleans, LA.
38. Franco, H., L. Neumeyer, V. Digalakis, and O. Ronen. 2000. Combination of machine scores for automatic grading of pronunciation quality. Speech Communication 30, nos. 2/3: 121-130.
39. Fulcher, G. 2003. Interface design in computer-based language testing. Language Testing 20, no. 4: 384-408.
40. Haggie, D. 2008. The strategic use of marking technologies to support innovation and diversity in assessment. Paper presented at the International Association of Educational Assessment, September, in Singapore. http://www.iaea2008.cambridgeassessment.org.uk/ca/digitalAssets/180436_Haggie.pdf.
41. Hambleton, R.K., and M. Pitoniak. 2006. Setting performance standards. In Educational measurement, 4th ed., ed. R.L. Brennan, 433-470. Westport, CT: Praeger.
42. Harik, P., B. Clauser, and D. Baldwin. 2010. Comparison of alternative scoring methods for a computerized performance assessment of clinical judgment. Paper presented at the National Council of Measurement in Education, April, in Denver, CO.
43. Hjelmas, E., and B.K. Low. 2001. Face detection: A survey. Computer Vision and Image Understanding 83, no. 3: 236-274.
44. Johnson, V.E. 1996. On Bayesian analysis of multirater ordinal data: An application to automated essay grading. Journal of the American Statistical Association 91, no. 433: 42-51.
45. Jörgensen, C. 2003. Image retrieval: Theory and research. Lanham, MD: Scarecrow Press.
46. Jurafsky, D., and J.H. Martin. 2009. Speech and language processing: An introduction to natural language processing, computational linguistics, and speech recognition. 2nd ed. Upper Saddle River, NJ: Pearson Prentice Hall.
47. Kane, M.T. 2006. Validation. In Educational measurement, 4th ed., ed. R.L. Brennan, 17-64. Westport, CT: Praeger.
48. Katz, I.R., and C.M. James. 1997. Towards assessment of design skill engineering. Princeton, NJ: ETS. http://www.ets.org/Media/Research/pdf/RR-97-16-Katz.pdf.
49. Kenney, J.F. 1997. New testing methodologies for the Architect Registration Examination. CLEAR Exam Review 8, no. 2: 23-28.
50. Lane, E.S., and C.A. Stone. 2006. Performance assessment. In Educational measurement, 4th ed., ed. R.L. Brennan, 387-431. Westport, CT: Praeger.
51. Leacock, C., and M. Chodorow. 2003. C-rater: Scoring of short-answer questions. Computers and the Humanities 37, no. 4: 389-405.
52. Margolis, M.J., and B.E. Clauser. 2006. A regression-based procedure for automated scoring of a complex medical performance assessment. In Automated scoring of complex tasks in computer-based testing, ed. D. Williamson, R.J. Mislevy, and I.I. Bejar, 123-167. Mahwah, NJ: Lawrence Erlbaum Associates.
53. Martinez, M.E., J.J. Ferris, W. Kraft, and W.H. Manning. 1990. Automatic scoring of paperand- pencil figural responses. Research Report RR-90-03. Princeton, NJ: Educational Testing Service.
54. Messick, S. 1989. Validity. In Educational measurement, 3rd ed., ed. R.L. Linn, 13-103. New York: American Council on Education.
55. Mislevy, R.J. 2006. Cognitive psychology and educational assessment. In Educational measurement, 4th ed., ed. R.L. Brennan, 257-306. Westport, CT: Praeger.
56. Mislevy, R.J., and G.D. Haertel. 2006. Implications of evidence-centered design for educational testing. Educational Measurement: Issues and Practice 25, no. 4: 6-20.
57. Mislevy, R.J., L. Steinberg, R.G. Almond, and J.F. Lucas. 2006. Concepts, terminology, and basic models of evidence-centered design. In Automated scoring of complex tasks in computer-based testing, ed. D. Williamson, R.J. Mislevy, and I.I. Bejar, 49-82. Mahwah, NJ: Lawrence Erlbaum.
58. Mislevy, R.J., L.S. Steinberg, and R.G. Almond. 2002. On the roles of task model variables in assessment design. In Item generation for test development, ed. S.H. Irvine and P.C. Kyllonen, 97-128. Mahwah, NJ: Lawrence Erlbaum Associates.
59. Mislevy, R.J., L.S. Steinberg, and R.G. Almond. 2003. On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives 1, no. 1: 3-62.
60. Mislevy, R.J., F.I. Winters, I.I. Bejar, R.E. Bennett, and G.D. Haertel. 2010. Technology supports for assessment design. In International encyclopedia of education, ed. G. McGaw, E. Baker, and P. Peterson, 3rd ed., Vol. 8, 56-65. Oxford: Elsevier.
61. Mohler, M., and R. Mihalcea. 2009. Text-to-text semantic similarity for automatic short answer grading. Paper presented at the 12th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2009), March, in Athens, Greece.
62. Monaghan, W., and B. Bridgeman. 2005. E-rater as a quality control of human scores. Princeton, NJ: Educational Testing Service. http://www.ets.org/Media/Research/pdf/RD_Connections2.pdf (accessed February 13, 2009).
63. National Board of Medical Examiners. 2009. Web-based testing on wireless networks. Philadelphia, PA: NBME. http://www.nbme.org/PDF/CAS/WirelessCAS.pdf.
64. Oltman, P.K., I.I. Bejar, and S.H. Kim. 1993. An approach to automated scoring of architectural designs. In CAAD Futures 93, ed. U. Flemming and S. van Wyk, 215-224. Pittsburgh, PA: North-Holland.
65. Page, E.B. 1966. The imminence of grading essays by computer. Phi Delta Kappan 47: 238-243.
66. Pearlman, M., K. Berger, and L. Tyler. 1993. An application of multimedia software to standardized testing in music. Research Report No. 93-36. Princeton, NJ: Educational Testing Service.
67. Powers, D.E., J. Burstein, M. Chodorow, M. Fowles, and K. Kukich. 2002. Comparing the validity of automated and human scoring of essays. Journal of Educational Computing Research 26, no. 4: 407-425.
68. Pulman, S.G., and J.Z. Sukkarieh. 2005. Automatic short answer marking. Paper presented at the second workshop on Building Educational Applications Using NLP, June, in Ann Arbor, MI. http://www.clg.ox.ac.uk/pulman/pdfpapers/acl05.pdf.
69. Quinlan, T., D. Higgins, and S. Wolff. 2009. Evaluating the construct-coverage of e-rate. No. RR-09-01. Princeton, NJ: ETS.
70. Reid-Green, K.S. 1995a. An algorithm for converting contours to elevation grids. Research Report No. RR-95-7. Princeton, NJ: Educational Testing Service.
71. Reid-Green, K.S. 1995b. An algorithm for determining if an object can be viewed from another object. Research Memorandum No. RM-95-3. Princeton, NJ: Educational Testing Service.
72. Reid-Green, K.S. 1996a. A computer method for verification of wheelchair accessibility in an office. Research Memorandum No. RM-96-1. Princeton, NJ: Educational Testing Service.
73. Reid-Green, K.S. 1996b. Insolation and shadow. Research Report No. RR-96-18. Princeton, NJ: Educational Testing Service.
74. Rudner, L.M., V. Garcia, and C. Welch. 2006. An evaluation of the IntelliMetric(SM) essay scoring system. Journal of Technology, Learning, and Assessment 4, no. 4. http://www.jtla.org.
75. Rudner, L.M., and T. Liang. 2002. Automated essay scoring using Bayes' theorem. Journal of Technology, Learning, and Assessment 1, no. 2. http://www.jtla.org.
76. Sandene, B., N. Horkay, R. Bennett, N. Allen, J. Braswell, B. Kaplan, and A. Oranje. 2005. Online assessment in mathematics and writing: Reports from the NAEP Technology-Based Assessment Project. Research and Development Series (NCES 2005-457). Washington, DC: US Government Printing Office.
77. Scalise, K., and M. Wilson. 2006. Analysis and comparison of automated scoring approaches: Addressing evidence-based assessment principles. In Automated scoring of complex tasks in computer-based testing, ed. D. Williamson, R.J. Mislevy, and I.I. Bejar, 373-401. Mahwah, NJ: Lawrence Erlbaum.
78. Singley, K.S., and R.E. Bennett. 1998. Validation and extension of the mathematical expression response type: Applications of schema theory to automatic scoring and item generation in mathematics. No. ETS RR-97-19. Princeton, NJ: ETS.
79. Srihari, S., J. Collins, R. Srihari, H. Srinivasan, S. Shetty, and J. Brutt-Gifler. 2008. Automatic scoring of short handwritten essays in reading comprehension tests. Artificial Intelligence 172, nos. 2/3: 300-324.
80. Sukkarieh, J.Z., and J. Blackmore. 2009. C-rater: Automatic content scoring of short constructed responses. Paper presented at the 22nd International Conference for the Florida Artificial Intelligence Research Society, FLAIRS 2009, May, in Florida, USA.
81. Toulmin, S.E. 1958. The uses of argument. New York: Cambridge University Press.
82. Traxler, A.E. 1954. Impact of machine and devices on developments in testing and related fields. In Proceedings of the 1953 Invitational Conference on Testing Problems, 139-146. Princeton, NJ: Author.
83. Trentin, G. 2009. Using a wiki to evaluate individual contribution to a collaborative learning project. Journal of Computer Assisted Learning 25, no. 1: 43-55.
84. Vapnik, V.N. 1995. The nature of statistical learning theory. New York: Springer-Verlag.
85. Weiss, G.M., and F. Provost. 2003. Learning when training data are costly: The effect of class distribution on tree induction. Journal of Artificial Intelligence Research 19: 315-354.
86. Williamson, D.M., R.G. Almond, R.J. Mislevy, and R. Levy. 2006. An application of Bayesian networks in automated scoring of computerized simulation tasks. In Automated scoring of complex tasks in computer-based testing, ed. D. Williamson, R.J. Mislevy, and I.I. Bejar, 201-258. Mahwah, NJ: Lawrence Erlbaum.
87. Williamson, D.M., I.I. Bejar, and A. Sax. 2004. Automated tools for subject matter expert evaluation of automated scoring. Applied Measurement in Education 17, no. 4: 323-357.
88. Williamson, D.M., R.J. Mislevy, and I.I. Bejar. 2006. Automated scoring of complex tasks in computer-based testing. Mahwah, NJ: Lawrence Erlbaum Associates.
89. Wilson, M. 2005. Constructing measures: An item response modeling approach. Mahwah, NJ: Lawrence Erlbaum.
90. Witten, I.H., and E. Frank. 2005. Data mining: Practical machine learning tools and techniques. 2nd ed. Amsterdam/Boston, MA: Morgan Kaufman.
91. Xi, X., D. Higgins, K. Zechner, and D.M. Williamson. 2008. Automated scoring of spontaneous speech using SpeechRater v1.0. No. RR-07-02/TOEFL iBT 02. Princeton, NJ: Educational Testing Service. http://www.ets.org/research/researcher/RR-08-62.html.
92. Zechner, K., I.I. Bejar, and R. Hemat. 2007. Toward an understanding of the role of speech recognition in nonnative speech assessment. No. RR-07-02/TOEFL iBT 02. Princeton, NJ: Educational Testing Service. http://www.ets.org/research/researcher/RR-07-02.html.
93. Zechner, K., D. Higgins, and X. Xi. 2007. SpeechRater™: A construct-driven approach to scoring spontaneous non-native speech. Paper presented at the ISCA-SLaTE Workshop, October, in Farmington, PA.
94. Zechner, K., D. Higgins, X. Xiaoming, and D. Williamson. 2009. Automatic scoring of nonnative spontaneous speech in tests of spoken English. Speech Communication 51, no. 10: 883-895.