The critical role of anchor paper selection in writing assessment

Applied Measurement in Education

Volumn 22, Issue 3, 2009, Pages 255-271

  Popp, Sharon E Osborn ^a   Ryan, Joseph M ^a   Thompson, Marilyn S ^b  

^a ARIZONA STATE UNIVERSITY   (United States)

^b ARIZONA STATE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 74949085384     PISSN: 08957347     EISSN: None     Source Type: Journal    
DOI: 10.1080/08957340902984026     Document Type: Article

References (45)

1. Andrich, D. (1978). A rating formulation for ordered categories. Psychometrika, 43, 357-374.
2. Arizona Department of Education. (2005). Six trait rubric-Official scoring guide for AIMS. Retrieved February 9, 2007, from http://ade.state.az.us/sbtl/6traits.
3. Brennan, R. L., & Johnson, E. G. (1995). Generalizability of performance assessments. Educational Measurement: Issues and Practice, 14(4), 9-12.
4. Cherry, R. D., & Meyer, P. R. (1993). Reliability issues in holistic assessment. In M. M. Williamson &B. A. Huot (Eds.), Validating holistic scoring for writing assessment: Theoretical and empirical foundations (pp. 109-141).Cresskill, NJ: Hampton Press.
5. Coffman, W. E. (1971). Essay examinations. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 271-302). Washington, DC: American Council on Education.
6. Daiker, D. A., & Grogan, N. (1985). The selection and use of sample papers in holistic evaluation. Oxford, OH: Miami University. (ERIC Document Reproduction Service No. ED305391).
7. Driscoll, L. A. (1996). 10 steps to district performance assessment. Memphis, TN: Memphis City Schools, Department of Research, Standards, and Accountability. (ERIC Document Reproduction Service No. ED406396).
8. Engelhard, G., Jr. (1992). The measurement of writing ability with a many-faceted Rasch model. Applied Measurement in Education, 5, 171-191.
9. Engelhard, G., Jr. (1994). Examining rater errors in the assessment of written composition with a many-faceted Rasch model. Journal of Educational Measurement, 31, 93-112.
10. Engelhard, G., Jr. (1996). Evaluating rater accuracy in performance assessments. Journal of Educational Measurement, 33, 56-70.
11. Feinstein, A. R., & Cicchetti, D. V. (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43, 543-549.
12. Fenton, R., Straugh, T., Stofflet, F., & Garrison, S. (2000, April). Improving the validity and reliability of large scale writing assessment. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
13. Freedman, S. W. (1981). Influences on evaluators of expository essays: Beyond the text. Research in the Teaching of English, 15, 245-255.
14. Goertz, M. E., & Duffy, M. C. (2001). Assessment and accountability systems in the 50 states, 1999-2000 (Research Rep. No. RR-046). Philadelphia: University of Pennsylvania, Consortium for Policy Research in Education. (ERIC Document Reproduction Service No. ED450639).
15. Gordon, B., Engelhard, G., Jr., Gabrielson, S., & Bernknopf, S. (1996). Conceptual issues in equating performance assessments: Lessons from writing assessment. Journal of Research and Develoment in Education, 29, 81-88.
16. Hughes, D. C., & Keeling, B. (1984). The use of model essays to reduce context effects in essay scoring. Journal of Educational Measurement, 21, 277-281.
17. Huot, B. A. (1990). The literature of direct writing assessment: Major concerns and prevailing trends. Review of Educational Research, 60, 237-263.
18. Huot, B. A. (1993). The influence of holistic scoring procedures on reading and rating student essays. In M. M. Williamson & B. A. Huot (Eds.), Validating holistic scoring for writing assessment: Theoretical and empirical foundations (pp. 206-236). Cresskill, NJ: Hampton Press.
19. Johnson, R. L., Penny, J., & Gordon, B. (2000). The relation between score resolution methods and interrater reliability: An empirical study of an analytic scoring rubric. Applied Measurement in Education, 13, 121-138.
20. Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159-174.
21. Linacre, J. M. (1994). Many-facet Rasch measurement. Chicago: MESA Press.
22. Linacre, J. M. (2007). A user's guide to FACETS. Chicago: Winsteps.com.
23. Linacre, J. M., & Wright, B. D. (1994). Reasonable mean-square fit values. Rasch Measurement Transactions, 8, 380. Retrieved December 21, 2007 from http://www.rasch.org/rmt/rmt83b.htm.
24. Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149-174.
25. Mehrens, W. A. (1992). Using performance assessment for accountability purposes. Educational Measurement: Issues and Practice, 11(1), 3-9, 20.
26. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-103). New York: Macmillan.
27. Messick, S. (1995). Standards of validity and the validity of standards in performance assessment. Educational Measurement: Issues and Practice, 14(4), 5-8.
28. Moon, T. R., & Hughes, K. R. (2002). Training and scoring issues involved in large-scale writing assessments. Educational Measurement: Issues and Practice, 21(2), 15-19.
29. Mullis, I. (1984). Scoring direct writing assessments: What are the alternatives? Educational Measurement: Issues and Practice, 3(1), 16-18.
30. Myford, C. M., & Wolfe, E. W. (2002). When raters disagree, then what: Examining a third-rating discrepancy resolution procedure and its utility for identifying unusual patterns of ratings. Journal of Applied Measurement, 3, 300-324.
31. Nevada Department of Education. (2000). Writing proficiency examination guide 2000-2001. Carson City, NV. (ERIC Document Reproduction Service No. ED447485).
32. Novak, J. R., Herman, J. L., & Gearhart, M. (1996). Establishing validity for performance-based assessments: An illustration for collections of writing. Journal of Educational Research, 89, 220-233.
33. Odell, L., & Cooper, C. R. (1980). Procedures for evaluating writing: Assumptions and needed research. College English, 42, 35-43.
34. Quellmalz, E. (1984). Toward successful large-scale writing assessments: Where are we now? Where do we go from here? Educational Measurement: Issues and Practice, 3(1), 29-32, 35.
35. Rasch, G. (1960/1980). Probabilistic models for some intelligence and attainment tests. Copenhagen: Danish Institute for Educational Research, 1960. Expanded edition, Chicago: The University of Chicago Press, 1980.
36. Roid, G. H. (1994). Patterns of writing skills derived from cluster analysis of direct-writing assessments. Applied Measurement in Education, 7, 159-170.
37. Saal, F. E., Downey, R. G., & Lahey, M. A. (1980). Rating the ratings: Assessing the psychometric quality of rating data. Psychological Bulletin, 88, 413-428.
38. Spandel, V. (1996). Seeing with new eyes: A guidebook on teaching and assessing beginning writers (3rd ed.). Portland, OR: Northwest Regional Educational Laboratory.
39. Thorndike, E. L. (1920). A constant error in psychological ratings. Journal of Applied Psychology, 4, 25-29.
40. Washington Office of the State Superintendent of Public Instruction. (2001). Grade 10 anchor set annotations from the spring 2001 Washington Assessment of Student Learning in Writing (with) presentation guide for principals. Olympia, WA. (ERIC Document Reproduction Service No. ED466812)
41. Weigle, S. C. (1998). Using FACETS to model rater training effects. Language Testing, 15, 263-287.
42. White, E. M. (1985). Teaching and assessing writing. San Francisco, CA: Jossey-Bass.
43. Wolfe, E. W., Kao, C.-W., & Ranney, M. (1998). Cognitive differences in proficient and nonproficient essay scorers. Written Communication, 15, 465-492.
44. Wright, B. D., & Masters, G. N. (1982). Rating scale analysis: Rasch Measurement. Chicago: MESA Press.
45. Zhang, L. (2000). Delaware student testing program: Report on special writing study. Dover: Delaware State Department of Education, Assessment and Accountability Branch. (ERIC Document Reproduction Service No. ED455271).