The stability of rater severity in large-scale assessment programs

Journal of Educational Measurement

Volumn 37, Issue 2, 2000, Pages 163-178

  Congdon, Peter J ^a   McQueen, Joy ^a  

^a Australian Cncl for Educ Research   (Australia)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0034195156     PISSN: 00220655     EISSN: None     Source Type: Journal    
DOI: 10.1111/j.1745-3984.2000.tb01081.x     Document Type: Article

References (34)

1. Barritt, L., Stock, P. L., & Clark, F. (1986). Researching practice: evaluating student essays. College Composition and Communication, 37, 315-327.
2. Buckner, D. N. (1959). The predictability of ratings as a function of interrater agreement. Journal of Applied Psychology, 43, 60-64.
3. Cantor, N. K., & Hoover, H. D. (1986, April). The reliability and validity of writing assessment: An investigation of rater, prompt within mode, and prompt between mode sources of error. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.
4. Coffman, W. E., & Kurfman, D. (1968). A comparison of two methods of reading essay examinations. American Educational Research Journal, 5, 101-120.
5. Diederich, P. B., French, J. W., & Carlton, S. T. (1961). Factors in judgments of writing ability. Research Bulletin, 61-15. Princeton, NJ: Educational Testing Service.
6. Dunbar, S. B., Koretz, D. M., & Hoover, H. D. (1991). Quality control in the development and use of performance assessments. Applied Measurement in Education, 4 (4), 289-303.
7. Engelhard, G., Jr. (1992). The measurement of writing ability with a many-faceted Rasch model. Applied Measurement in Education, 5, 171-191.
8. Engelhard, G., Jr. (1994). Examining rater errors in the assessment of written composition with a many-faceted Rasch model. Journal of Educational Measurement, 31, 93-112.
9. Engelhard, G., Jr., Gordon, B., & Gabrielson, S. (1991). The influences of mode of discourse, experiential demand, and gender on the quality of student writing. Research into the Teaching of English, 26, 315-336.
10. Gabrielson, S., Gordon, B., & Engelhard, G. Jr. (1995). The effects of task choice on the quality of writing obtained in a statewide assessment. Applied Measurement in Education, 8, 273-290.
11. Guilford, J. P. (1954). Psychometric methods (2nd ed.). New York: McGraw-Hill.
12. Hake, R. (1986). How do we judge what they write? In K. L. Greenberg, H. S. Wiener, & R. A. Donovan (Eds.), Writing assessment: Issues, strengths and strategies (pp. 153-167). New York: Longman.
13. Huddleston, E. M. (1954). Measurement of writing ability at the college-entrance level: Objective vs. subjective testing techniques. Journal of Experimental Education, 22, 165-213.
14. Linacre, J. M. (1989). Many-facet Rasch measurement. Chicago, IL: Mesa Press.
15. Linacre, J. M. (1991, April). Constructing measurement with a many-facet Rasch model. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.
16. Lumley, T., Lynch, B. K., & McNamara, T. F. (1994). A new approach to standard-setting in language assessment. Melbourne Papers in Language Testing, 3 (2), 19-40.
17. Lumley, T., & McNamara, T. F. (1995). Rater characteristics and rater bias: Implications for training. Language Testing, 12 (1), 54-71.
18. Lunz, M. E., & Stahl, J. A. (1990). Judge consistency and severity across grading periods. Evaluation and the Health Professions, 13, 425-444.
19. Lunz, M. E., Stahl, J. A., & Wright, B. D. (1994). Interjudge reliability and decision reproducibility. Educational and Psychological Measurement, 54, 913-925.
20. Lunz, M. E., Stahl, J. A., & Wright, B. D. (1996). The invariance of judge severity calibrations. In M. R. Wilson & G. Engelhard, Jr. (Eds.), Objective Measurement Theory into Practice, 3, 99-112. Norwood, NJ: Ablex.
21. Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149-174.
22. McNamara, T. F. (1996). Measuring second language performance. New York: Addison Wesley Longman Ltd.
23. Messick, S. (1994). The interplay of evidence and consequences in the validation of performance assessment. Educational Researcher, 23 (2), 13-23.
24. Myford, C. M. (1991, April). Judging acting ability: The transition from notice to expert. Paper presented at the American Educational Research Association, Chicago IL.
25. Overall, J. E., & Magee, K. N. (1992). Estimating individual rater reliabilities. Applied Psychological Measurement, 16, 77-85.
26. Ruth, L., & Murphy, S. (1988). Designing writing tasks for the assessment of writing. Norwood, NJ: Ablex Publishing.
27. Saal, F. E., Downey, R. G., & Lahey, M. A. (1980). Rating the ratings: Assessing the psychometric quality of rating data. Psychological Bulletin, 88, 413-428.
28. Thurstone, L. L. (1927). A law of comparative judgment. Psychological Review, 34, 273-286.
29. Webb, L. C., Raymond, M. R., & Houston, W. M. (1990, April). Rater stringency and consistency in performance assessment. Paper presented at the annual meeting of the American Educational Research Association and the National Council on Measurement in Education, Boston MA.
30. Weigle, S. C. (1994, February). Using FACETS to model rater training effects (Draft). Paper presented at the Language Testing Research Colloquium, Washington DC.
31. Wigglesworth, G. (1994). Patterns or oral behavior in the assessment of an oral interaction test. Australian Review of Applied Linguistics, 17 (2), 77-103.
32. Wright, B. D., & Masters, G. N. (1982). Rating scale analysis. Chicago IL: MESA Press.
33. Wu, M., Adams, R. J., & Wilson M. (1996). ConQuest: Generalised item response modelling software. Melbourne, Australia: Australian Council for Educational Research.
34. Wu, M. L. (1997). The development and application of a fit test for use with marginal maximum likelihood estimation and generalised item response models. Master's thesis, The University of Melbourne, Australia.