Accuracy in the scoring of writing: Studies of reliability and validity using a New Zealand writing assessment system

Assessing Writing

Volumn 9, Issue 2, 2004, Pages 105-121

  Brown, Gavin T L ^a   Glasswell, Kath ^b   Harland, Don ^c  

^a UNIVERSITY OF AUCKLAND   (New Zealand)

^b UNIVERSITY OF ILLINOIS AT CHICAGO   (United States)

^c Howick College   (New Zealand)

Author keywords

Assessment tool; Large scale assessment; New Zealand; Reliability; Validity

Indexed keywords

EID: 4644314465     PISSN: 10752935     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.asw.2004.07.001     Document Type: Article

References (42)

1. P.W. Airasian Classroom assessment 3rd ed. 1997 McGraw-Hill New York
2. J.C. Alderson C. Clapham D. Wall Language test construction and evaluation 1995 Cambridge University Press Cambridge, UK
3. American Educational Research Association (AERA), American Psychological Association (APA) & National Council for Measurement in Education (NCME) Standards for Educational & Psychological Testing 1999 American Educational Research Association Washington, DC
4. Breland, H.M., Bridgeman, B., & Fowles, M.E. (1999). Writing assessment in admission to higher education: Review and framework (ETS Research Report No. 99-3). Princeton, NJ: Educational Testing Service
5. Brown, G.T., & Hattie, J.A. (2003). A national teacher-managed, curriculum-based assessment system: Assessment tools for teaching & learning (asTTle) (asTTle Tech. Rep. No. 41). Auckland, NZ: University of Auckland/Ministry of Education
6. G.T.L. Brown Assessment in English English in Aotearoa 36 1998 62-67
7. G.T.L. Brown The New Zealand English curriculum English in Aotearoa 35 1998 64-70
8. W. Coffman On the reliability of ratings of essay examinations in English Research in the Teaching of English 5 1971 24-36
9. J. Cohen A coefficient for agreement for nominal scales Educational and Psychological Measurement 20 1960 37-46
10. College Board. (2001). AP technical corner: Scoring the free-response section. Retrieved September 15, 2001. From http://www.collegeboard.com/ap/techman/chap3/scorefc.htm
11. Cooper, P. L. (1984). The assessment of writing ability: A review of research (ETS Research Report No. 84-12). Princeton, NJ: Educational Testing Service
12. B.G. Davis M. Scriven S. Thomas The evaluation of composition instruction 1981 Edgepress Point Reyes, CA
13. A.H. Dyson S.W. Freedman Writing in: J. Flood J.M. Jensen D. Lapp J.R. Squire eds. Handbook of research on teaching the English language arts 1991 Macmillan New York 754-774
14. Gearhart, M., Herman, J. L., Baker, E. L., & Whittaker, A. K. (1992). Writing portfolios at the elementary level: A study of methods for writing assessment (CSE Tech. Rep. No. 337). Los Angeles, CA: UCLA Graduate School of Education, National Center for Research on Evaluation, Standards and Student Testing (CRESST)
15. Gearhart, M., Herman, J.L., Novak, J.R., Wolf, S.A., & Abedi, J. (1994). Toward the instructional utility of large-scale writing assessment: Validation of a new narrative rubric (CSE Tech. Rep. No. 389). Los Angeles, CA: UCLA Graduate School of Education, National Center for Research on Evaluation, Standards and Student Testing (CRESST)
16. Glasswell, K., Parr, J., & Aikman, M. (2001). Development of the asTTle writing assessment rubrics for scoring extended writing tasks (asTTle Tech. Rep. No. 6). Auckland, NZ: University of Auckland/ Ministry of Education
17. Green, S., Johnson, M., O'Donovan, N., & Sutton, P. (2003, July). Changes in key stage two writing from 1995 to 2002. Paper presented at the United Kingdom Reading Association Conference, Cambridge, UK
18. N.E. Gronlund R.L. Linn Measurement and evaluation in teaching 6th ed. 1990 Macmillan New York
19. Harland, D. (2002). Teaching argumentative writing to Year 9 students. Unpublished Master of Arts thesis. University of Auckland, Auckland, NZ
20. J.A. Hattie rown P.J. Keegan Assessment Tools for Teaching and Learning (asTTle) manual: Version 2, 2003 2002 Learning Media Wellington, NZ
21. J.L. Herman M. Gearhart E.L. Baker Assessing writing portfolios: Issues in the validity and meaning of scores Educational Assessment 1 3 1993 201-224
22. B. Huot The literature of direct writing assessment: Major concerns and prevailing trends Review of Educational Research 60 2 1990 237-263
23. Koretz, D., Klein, S., McCaffrey, D., & Stecher, B. (1993). Interim report: The reliability of Vermont portfolio scores in the 1992-1993 School Year (CSE Tech. Rep. No. 370). Los Angeles, CA: UCLA Graduate School of Education, National Center for Research on Evaluation, Standards and Student Testing (CRESST)
24. Koretz D., McCaffrey, D., Klein, S., Bell, R., & Stecher, B. (1993). The reliability of scores from the 1992 Vermont Portfolio Assessment Program (CSE Tech. Rep. No. 355). Los Angeles, CA: UCLA Graduate School of Education, National Center for Research on Evaluation, Standards and Student Testing (CRESST)
25. Lee, Y. W. (2001). The essay scoring and scorer reliability in TOEFL CBT. Paper presented at the Annual Meeting of National Council on Measurement in Education (NCME), Seattle, WA
26. P. LeMahieu D. Gitomer J. Eresh Large-scale portfolio assessment: Difficult but not impossible Journal of Educational Measurement: Issues and Practice 14 1995 11-28
27. Linn, R. L. (1991). Cross state comparability of judgments of student writing: Results from the New Standards Project (CSE Tech. Rep. No. 335). Los Angeles, CA: UCLA Graduate School of Education, National Center for Research on Evaluation, Standards and Student Testing (CRESST)
28. R.L. Linn N.E. Gronlund Measurement and evaluation in teaching 8th ed. 2000 Macmillan New York
29. R.L. Linn V.L. Kiplinger C.W. Chapman P.G. LeMahieu Cross-state comparability of judgments of student writing: Results from the New Standards Project Applied Measurement in Education 5 2 1992 89-110
30. Longford, N. (1994). A case for adjusting subjectively rated scores in the advanced placement tests (Tech. Rep. No. 94-5). Princeton, NJ: Education Testing Service
31. J.H. McMillan Classroom assessment: Principles and practice for effective instruction 2nd ed. 2001 Allyn & Bacon Boston, MA
32. Ministry of Education Assessment: Policy to practice 1994 Learning Media Wellington, NZ
33. Ministry of Education English in the New Zealand curriculum 1994 Learning Media Wellington, NZ
34. Ministry of Education Assessment for better learning. Curriculum development: A report on national assessment developments Update 18 1997
35. Novak, J. R., Herman, J. L., & Gearhart, M. (1996). Issues in portfolio assessment: The scorability of narrative collections (CSE Tech. Rep. No. 410). Los Angeles, CA: UCLA Graduate School of Education, National Center for Research on Evaluation, Standards and Student Testing (CRESST)
36. J. Penny R.L. Johnson B. Gordon The effect of rating augmentation on inter-rater reliability: An empirical study of a holistic rubric Assessing Writing 7 2000 143-164
37. W.J. Popham Modern educational measurement: Practical guidelines for educational leaders 6th ed. 2000 Allyn & Bacon Boston
38. Resnick, L., Resnick, D., & De Stefano, L. (1993). Cross-scorer and cross-method comparability and distribution of judgments of student math, reading, and writing performance results from the New Standards Project Big Sky Scoring Conference (CSE Tech. Rep. No. 368). Los Angeles, CA: UCLA Graduate School of Education, National Center for Research on Evaluation, Standards and Student Testing (CRESST)
39. R.J. Shavelson N.M. Webb Generalizability theory: A primer 1991 Sage Newbury Park, CA
40. S.E. Stemler A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability Practical Assessment Research & Evaluation 9 4 2004
41. J. Stuhlmann C. Daniel A. Dellinger R.K. Denny T. Powers A generalisability study of the effects of training on teachers' abilities to rate children's writing using a rubric Reading Psychology 20 2 1999 107-127
42. J.A. Supovitz A. MacGowan III J. Slattery Assessing agreement: Interrater reliability of portfolio assessment in Rochester, New York Educational Assessment 4 3 1997 237-259