Rater cognition: Implications for validity

Educational Measurement: Issues and Practice

Volumn 31, Issue 3, 2012, Pages 2-9

  Bejar, Issac I ^a  

^a EDUCATIONAL TESTING SERVICE   (United States)

Author keywords

Automated scoring; Rater cognition; Scoring; Validity argument

Indexed keywords

EID: 84866452407     PISSN: 07311745     EISSN: 17453992     Source Type: Journal    
DOI: 10.1111/j.1745-3992.2012.00238.x     Document Type: Article

References (88)

1. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Organization.
2. Attali, Y., & Burstein, J. (2006). Automated essay scoring with e-rater® V.2. Journal of Technology, Learning, and Assessment, 4(3), Available from http://www.jtla.org
3. Baldwin, D., Fowles, M., & Livingston, S. (2008). Guidelines for constructed-responses and other performance assessments (Research Report No. RR-07-02/TOEFL iBT 02). Princeton, NJ: Educational Testing Service (downloadable from http://www.ets.org/Media/About_ETS/pdf/8561_ConstructedResponse_guidelines.pdf).
4. Bejar, I. I. (2011). A validity-based approach to quality control and assurance of automated scoring. Assessment in Education, 18, 319-341.
5. Bejar, I. I., Williamson, D. M., & Mislevy, R. J. (2006). Human scoring. In D. M. Williamson, R. J. Mislevy, & I. I. Bejar (Eds.), Automated scoring of complex tasks in computer-based testing (pp. 49-82). Mahwah, NJ: Lawrence Erlbaum.
6. Bennett, R. E. (2011). Automated scoring of constructed-response literacy and mathematics items. Washington, DC: Arabella Philantropic Advisors (available at http://www.ets.org/s/k12/pdf/k12_commonassess_automated_scoring_math.pdf).)
7. Bennett, R. E., & Bejar, I. I. (1998). Validity and automated scoring: It's not only the scoring. Educational Measurement: Issues and Practice, 17, 9-16.
8. Braun, H., Bejar, I. I., & Williamson, D. M. (2006). Rule-based methods for automatic scoring: Application in a licensing context. In D. M. Williamson, R. J. Mislevy, & I. I. Bejar (Eds.), Automated scoring for complex constructed response tasks in computer-based testing (pp. 83-122). Mahwah, NJ: Lawrence Erlbaum.
9. Brunswik, E. (1952). The conceptual framework of psychology. Chicago, IL: University of Chicago Press.
10. Carey, M. D., Mannell, R. H., & Dunn, P. K. (2011). Does a rater's familiarity with a candidate's pronunciation affect the rating in oral proficiency interviews? Language Testing, 28, 201-219.
11. Clauser, B. E., Kane, M. T., & Swanson, D. B. (2002). Validity issues for performance-based tests scored with computer-automated scoring systems. Applied Measurement in Education, 15, 413-432.
12. Coffman, W. E. (1971). Essay examinations. In R. L. Thorndike (Ed.), Educational Measurement (2nd ed., pp. 271-302). Washington, DC: American Council on Education.
13. Coniam, D. (2009). A comparison of on-screen and paper-based marking in the Hong Kong public examination system. Educational Research and Evaluation, 15, 243-263.
14. Coward, A. F. (1952). The comparison of two methods of grading English compositions. Journal of Educational Research, 46, 81-93.
15. Crisp, V. (2010). Towards a model of the judgement processes involved in examination marking. Oxford Review of Education, 36, 1-21.
16. Crisp, V. (2012). An investigation of rater cognition in the assessment of projects. Educational Measurement: Issues and Practice, 31, 10-20.
17. Cronbach, L. J. (1988). Five perspectives on validity argument. In H. Wainer & H. I. Braun (Eds.), Test validity (pp. 3-17). Hillsdale, NJ: Lawrence Erlbaum.
18. Cumming, A., Kantor, R., & Powers, D. E. (2002). Decision making while rating ESL/EFL writing tasks: A descriptive framework. The Modern Language Journal, 86, 67-96.
19. Darling-Hammond, L. (2010). Performance counts: Assessment systems that support high-quality learning. Washington, DC: Council of Chief State School Officers. Retrieved July 13, 2012, from http://www.ccsso.org/publications/details.cfm?PublicationID=381
20. Dawes, R. M. (1994). Psychological measurement. Psychological Review, 101, 278-281.
21. DeCarlo, L. T. (2005). A model of rater behavior in essay grading based on signal detection theory. Journal of Educational Measurement, 42, 53-76.
22. Diederich, P. B., French, J. W., & Carlton, S. T. (1961). Factors in judgments of writing ability (Research Bulletin No. RB-61-15). Princeton, NJ: Educational Testing Service.
23. Drasgow, F., Luecht, R., & Bennett, R. E. (2006). Technology and testing. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 471-515). Westport, CT: Praeger Publishers.
24. Edgeworth, F. Y. (1890). The element of chance in competitive examinations. Journal of the Royal Statistical Society, 53, 460-475, 644-663.
25. Elbow, P., & Yancey, K. B. (1994). On the nature of holistic scoring: An inquiry composed on e-mail. Assessing Writing, 1, 91-107.
26. Elliot, N. (2005). On a scale: A social history of writing assessment in America. New York, NY: Peter Lang.
27. Fechner, G. T. (1860). Elemente der Psychophysik. Leipzig, Germany: Breitkopf and Hartel.
28. Fechner, G. T. (1897). Kollektivmasslehre, ed. G. F. Lipps. Leipzig, Germany: Wilhelm Engelmann.
29. Freedman, S. W., & Calfee, R. C. (1983). Holistic assessment of writing: Experimental design and cognitive theory. In P. Mosenthal, L. Tamor, & S. A. Walmsley (Eds.), Research on writing: Principles and methods (pp. 75-98). New York, NY: Longman.
30. Fuess, C. M. (1950). The College Board, its first fifty years. New York, NY: Columbia University Press.
31. Gigerenzer, G., & Goldstein, D. G. (1996). Reasoning the fast and frugal way: Models of bounded rationality. Psychological Review, 103, 650-669.
32. Godshalk, F. I., Swineford, F., & Coffman, W. E. (1966). The measurement of writing ability. New York, NY: College Entrance Examination Board.
33. Goldberg, G. L. (2012). Judgment-based scoring by teachers as professional development: Distinguishing promises from proof. Educational Measurement: Issues and Practice, this issue, 38-47.
34. Hammond, K. R. (1996). Human judgment and social policy. New York, NY: Oxford University Press.
35. Hammond, K. R., Hursch, C. J., & Todd, F. J. (1964). Analyzing the components of clinical inference. Psychological Review, 71, 438-456.
36. Harik, P., Clauser, B. E., Grabovsky, I., Nungester, R. J., Swanson, D., & Nandakumar, R. (2009). An examination of rater drift within a generalizability theory framework. Journal of Educational Measurement, 46, 43-58.
37. Heller, J. I., Sheingold, K., & Myford, C. M. (1998). Reasoning about evidence in portfolios: Cognitive foundations for valid and reliable assessment. Educational Assessment, 5, 5-40.
38. Hill, H. C., Charalambos, Y. C., & Kraft, M. A. (2012). When rater reliability is not enough: Teacher observation systems and a case for the generalizability study. Educational Researcher, 41, 56-64.
39. Hillocks, G. (2002). The testing trap. New York, NY: Teachers College Press.
40. Hoskens, M., & Wilson, M. (2001). Real-time feedback on rater drift in constructed-response items: An example from the golden state examination. Journal of Educational Measurement, 38, 121-145.
41. Huot, B., & Neal, M. (2006). Writing assessment: A techno-history. In C. A. MacArthur, S. Graham, & J. Fitzgerald (Eds.), Handbook of writing research (pp. 417-432). New York, NY: Guilford Press.
42. Joe, J. N., Harmes, C., & Hickerson, C. A. (2011). Using verbal reports to explore rater perceptual processes in scoring: A mixed methods application to oral communication assessment Assessment in Education, 18, 239-258.
43. Johnson, M., Nádas, R., & Bell, J. F. (2010). Marking essays on screen: An investigation into the reliability of marking extended subjective texts. British Journal of Educational Technology, 41, 814-826.
44. Jörgensen, C. (2003). Image retrieval: Theory and research. Lanham, MD: Scarecrow Press.
45. Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 17-64). Westport, CT: Praeger Publishers.
46. Krosnick, J. A. (1991). Survey research. Annual Review of Psychology, 50, 537-567.
47. Lane, E. S., & Stone, C. A. (2006). Performance assessment. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 387-431). Westport, CT: Praeger.
48. Leusner, D., & Lyon, C. (2008). Providing a research basis for ETS products (Research Report No. RM-08-01). Princeton, NJ: Educational Testing Service.
49. Link, S. W. (1994). Rediscovering the past-Fechner and signal-detection-theory. Psychological Science, 5, 335-340.
50. Lumley, T. (2002). Assessment criteria in a large-scale writing test: What do they really mean to the raters? Language Testing, 19, 246-276.
51. Mabry, L. (1999). Writing to the rubric. Phi Delta Kappan, 80, 673-679.
52. McClellan, C. A. (2010). Constructed-response scoring: Doing it right. (Research Report No. RDC-13). Retrieved August 3, 2011, from http://www.ets.org/research/policy_research_reports/rdc-13
53. Messick, S. J. (1998). Alternative modes of assessment, uniform standards of validity. In M. D. Hakel (Ed.), Beyond multiple choice: Evaluating alternativesto traditional testing for selection (pp. 59-74). Mahwah, NJ: Lawrence Erlbaum.
54. Mislevy, R. J., & Haertel, G. D. (2006). Implications of evidence-centered design for educational testing. Educational Measurement: Issues and Practice, 25, 6-20.
55. Mislevy, R. J., Steinberg, L., Almond, R. G., & Lucas, J. F. (2006). Concepts, terminology, and basic models of evidence-centered design. In D. M. Williamson, R. J. Mislevy, & I. I. Bejar (Eds.), Automated scoring of complex tasks in computer-based testing (pp. 49-82). Mahwah, NJ: Lawrence Erlbaum.
56. Myers, A. E., McConville, C. B., & Coffman, W. E. (1966). Simplex structure in the grading of essay tests. Educational and Psychological Measurement, 26, 41-54.
57. Myford, C. M., Marr, D. B., & Linacre, J. M. (1996). Reader calibration and its potential role in equating for the test of written English (TOEFL Research Report No. 52). Princeton, NJ: Educational Testing Service.
58. Myford, C. M., & Mislevy, R. J. (1995). Monitoring and improving a portfolio assessment system (CSE Technical Report No. 402). Princeton, NJ: Educational Testing Service (downloadable from http://www.cse.ucla.edu/products/Reports/TECH402.pdf).
59. Myford, C. M., & Wolfe, E. W. (2009). Monitoring rater performance over time: A framework for detecting differential accuracy and differential scale category use. Journal of Educational Measurement, 46, 371-389.
60. Nijveldt, A., Beijaard, D., Brekelmans, M., Wubbels, T., & Verloop, N. (2009). Assessors' perceptions of their judgement processes: Successful strategies and threats underlying valid assessment of student teachers. Studies in Educational Evaluation, 35, 29-36.
61. Norton, L. S. (1990). Essay-writing: What really counts? Higher Education, 20, 411-442.
62. Nosofsky, R. M. (1990). Relations between exemplar-similarity and likelihood models of classification. Mathematical Psychology, 34, 393-418.
63. Partnership for Assessment of Readiness for College and Careers (2010, June 23). The Partnership for Assessment of Readiness for College and Careers (PARCC) application for the Race to the Top Comprehensive Assessment Systems competition. Retrieved July 17, 2012, from http://www.fldoe.org/parcc/pdf/apprtcasc.pdf
64. Patz, R. J., Junker, B. W., Johnson, M. S., & Mariano, L. T. (2002). The Hierarchical Rater Model for rated test items and its application to large-scale educational assessment data. Journal of Educational and Behavioral Statistics in Medicine, 27, 341-384.
65. Pollitt, A. (2004). Let's stop marking exams. Paper presented at the IAEA meeting, Philadelphia, PA. Retrieved July 17, 2012, from http://www.cambridgeassessment.org.uk/ca/digitalAssets/113942_Let_s_Stop_Marking_Exams.pdf
66. Puhan, G. (2009). Detecting and correcting scale drift in test equating: An illustration from a large scale testing program. Applied Measurement in Education, 22, 79-103.
67. Pula, J. J., & Huot, B. A. (1993). A model of background influences on holistic raters. In M. M. Williamson & B. A. Huot (Eds.), Validating holistic scoring for writing assessment: Theoretical and empirical foundations (pp. 237-265). Cresskill, NJ: Hampton Press.
68. Quinlan, T., Higgins, D., & Wolff, S. (2009). Evaluating the construct-coverage of e-rater® (Research Report No. RR-09-01). Princeton, NJ: Educational Testing Service.
69. Royal-Dawson, L., & Baird, J. (2009). Is teaching experience necessary for reliable scoring of extended English questions Educational Measurement: Issues and Practice, 28, 2-8.
70. Sakyi, A. A. (2000). Validation of holistic scoring for ESL writing assessment: How raters evaluate compositions. In J. J. Kunnan (Ed.), Fairness and validation in language assessment (pp. 129-152). Cambridge, UK: Cambridge University Press.
71. Shepard, R. N., Romney, A. K., & Nerlove, S. B. (1972). Multidimensional scaling: Theory and applications in the behavioral sciences. New York, NY: Seminar Press.
72. Shohamy, E., Gordon, C. M., & Kraemer, R. (1992). The effect of raters background and training on the reliability of direct writing tests. Modern Language Journal, 76, 27-33.
73. Simon, H. A. (1957). Models of man, social and rational: Mathematical essays on rational human behavior in a social setting. New York, NY: Wiley.
74. Simon, H. A. (1959). Theories of decision-making in economics and behavioral science. The American Economic Review, 49, 253-283.
75. SMARTER Balanced Assessment Consortium. (2010). Theory of action: An excerpt from the SMARTER Balanced Race to the Top application. Retrieved July 17, 2012, from http://www.smarterbalanced.org/wordpress/wp-content/uploads/2012/02/Smarter-Balanced-Theory-of-Action.pdf
76. Suto, I. (2012). A critical review of research methods used to explore rater cognition. Educational Measurement: Issues and Practice, 31, 21-30.
77. Suto, I., & Greatorex, J. (2008). What goes through an examiner's mind? Using verbal protocols to gain insights into the GCS marking process. British Educational Research Journal, 34, 213-233.
78. Thurstone, L. L. (1927). A law of comparative judgment. Psychological Review, 34, 273-286.
79. Tucker, L. R. (1964). A suggested alternative formulation in the developments by Hursch, Hammond, and Hursch, and by Hammond, Hursch, and Todd. Psychological Review, 71, 528-530.
80. Tversky, A. (1977). Features of similarity. Psychological Review, 84, 327-352.
81. Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185, 1124-1131.
82. Vaughn, C. (1991). Holistic assessment: What goes in the rater mind? In L. Hamp-Lyons (Ed.), Assessing second language writing in academic contexts (pp. 111-125). Norwood, NJ: Ablex.
83. Wolfe, E. W. (1997). The relationship between essay reading style and scoring proficiency in a psychometric scoring system. Assessing Writing, 4, 83-106.
84. Wolfe, E. W., & McVay, A. (2012). Application of latent trait models to identifying substantively interesting raters. Educational Measurement: Issues and Practice, this issue, 31-37.
85. Wolfe, E. W., Kao, C. W., & Ranney, M. (1998). Cognitive differences in proficient and nonproficient essay scorers. Written Communication, 15, 465-492.
86. Wolfe, E. W., Matthews, S., & Vickers, D. (2010). The effectiveness and efficiency of distributed online, regional online, and regional face-to-face training for writing assessment raters. Journal of Technology, Learning, and Assessment, 10, available at http://www.jtla.org
87. Wong, K. F. E., & Kwong, J. Y. Y. (2007). Effects of rater goals on rating patterns: Evidence from an experimental field study. Journal of Applied Psychology, 92, 577-585.
88. Xi, X., Higgins, D., Zechner, K., & Williamson, D. M. (2008). Automated scoring of spontaneous speech using Speechrater v1.0 (Research Report No. RR-07-02/TOEFL iBT 02). Princeton, NJ: Educational Testing Service. Retrieved July 17, 2012, from http://www.ets.org/research/researcher/RR-08-62.html