Score Resolution: An Investigation of the Reliability and Validity of Resolved Scores

Applied Measurement in Education

Volumn 16, Issue 4, 2003, Pages 299-322

  Johnson, Robert L ^a   Penny, Jim ^b   Fisher, Steve ^a   Kuhs, Therese ^a  

^a UNIVERSITY OF SOUTH CAROLINA   (United States)

^b CASTLE Worldwide Inc   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0142199854     PISSN: 08957347     EISSN: None     Source Type: Journal    
DOI: 10.1207/S15324818AME1604_3     Document Type: Article

References (50)

1. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
2. Baker, L., & Wigfield, A. (1999). Dimensions of children's motivation for reading and their relations to reading activity and reading achievement. Reading Research Quarterly, 34(4), 452-477.
3. Breland, H. (1983). The direct assessment of writing skill: A measurement review (Technical Report No. 83-6). Princeton, NJ: College Entrance Examination Board.
4. Brennan, R. (1996). Generalizability of performance assessments. In G. Phillips (Ed.), Technical issues in large-scale performance assessment (pp. 19-58). Washington, DC: National Center for Education Statistics.
5. Brennan, R. (2001). An essay on the history and future of reliability from the perspective of replications. Journal of Educational Measurement, 38(4), 295-317.
6. Brown, J., Hilgers, T., & Marsella, J. (1991). Essay prompts and topics: Minimizing the effect of mean differences. Written Communication, 8(4), 533-556.
7. Cherry, R., & Myer, P. (1993). Reliability issues in holistic assessment. In M. Williamson & B. Huot (Eds.) Validating holistic scoring for writing assessment: Theoretical and empirical foundations (pp. 109-141). Cresskill, NJ: Hampton.
8. Claggett, F. (1999). Integrating reading and writing in large-scale assessment. In C. Cooper & L. Odell (Eds.), Evaluating writing: The role of teacher's knowledge about text, learning, and culture (pp. 344-365). Urbana, IL: National Council of Teachers of English.
9. Clauser, B., Clyman, S., & Swanson, D. (1999). Components of rater error in a complex performance assessment. Journal of Educational Measurement, 36(1), 29-45.
10. Coffman, W. (1971). Essay examinations. In R. Thorndike (Ed.), Educational measurement (2nd ed., pp. 271-302). Washington: American Council on Education.
11. Cronbach, L., Linn, R., Brennan, R., & Haertel, E. (1997). Generalizability analysis for performance assessments of student achievement or school effectiveness. Educational and Psychological Measurement 57(3), 373-399.
12. Diederich, P. (1974). Measuring growth in English. Urbana, IL: National Council of Teachers of English.
13. Dunbar, S., Koretz, D., & Hoover, H. (1991). Quality control in the development and use of performance assessments. Applied Measurement in Education, 4(4), 289-303.
14. Godshalk, F., Swineford, F., & Coffman, W. (1966). The measurement of writing ability. Princeton: College Entrance Examination Board.
15. Hayes, J., Hatch, J., & Silk, C. (2000). Does holistic assessment predict writing performance? Estimating the consistency of student performance on holistically scored writing assignments. Written Communication, 17(1), 3-26.
16. Hieronymous, A., Hoover, H., Cantor, N., & Oberley, K. (1987). Handbook for focused holistic scoring. Chicago: Riverside Publishing.
17. Hogan, T., & Mishler, C. (1980). Relationships between essay tests and objective tests of language skills for elementary school students. Journal of Educational Measurement, 17, 219-227.
18. Huot, B. A. (1993). The influence of holistic scoring procedures on reading and rating student essays. In M. M. Williamson & B. A. Huot (Eds.), Validating holistic scoring for writing assessment (pp. 206-236). Cresskill, NJ: Hampton.
19. Janopoulos, M. (1989). Reader comprehension and holistic assessment of second language writing proficiency. Written Communication 6(2), 218-237.
20. Johnson, R., Penny, J., & Gordon, B. (2000). The relation between score resolution methods and interrater reliability: An empirical study of an analytic scoring rubric. Applied Measurement in Education, 13(2), 121-138.
21. Johnson, R., Penny, J., & Gordon, B. (2001). Score resolution and the interrater reliability of holistic scores in rating essays. Written Communication, 18(2), 229-249.
22. Johnson, R., Penny, J., & Johnson, C. (1998, June). Score resolution in the rating of performance assessments: Practices and issues. Presented at the Council of Chief State School Officers National Conference on Large Scale Assessment.
23. Johnson, R., Penny, J., & Johnson, C. (2000, April). A conceptual framework for score resolution in the rating of performance assessments: The union of validity and reliability. Presented at the annual meeting of the American Educational Research Association, New Orleans.
24. Johnson, R., Willeke, M., Bergman, T., & Steiner, D. (1997). Family literacy portfolios: Development and implementation. Window on the World of Family Literacy, 2(2), 10-17.
25. Kane, M. (1982). A sampling model for validity. Applied Psychological Measurement, 6(2), 125-160.
26. Khattri, N., Reeve, A., & Kane, M. (1998). Principles and practices of performance assessment. Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
27. Lane, S., Liu, M., Ankenmann, R., & Stone, C. (1996). Generalizability and validity of a mathematics performance assessment. Journal of Educational Measurement, 33(1), 71-92.
28. LeMahieu, P., Gitomer, D., & Eresh, J. (1995). Portfolios in large-scale assessment: Difficult but not impossible. Educational Measurement: Issues and Practice, 14(3), 11-16, 25-28.
29. Linn, R., Kiplinger, V., Chapman, C., & LeMahieu, P. (1992). Cross-state comparability of judgments of student writing results from the New Standards Project. Applied Measurement in Education, 5(2), 89-110.
30. Livingston, S. (1998, April). Results of the pilot test of the School Leaders' Licensure Assessment. Paper presented at the Annual Meeting of the American Educational Research Association, San Diego, CA.
31. Masters, J. (1992, April). A study of arbitrations in Pennsylvania's writing assessment. Paper presented at the annual meeting of the National Council on Measurement in Education, San Francisco, CA.
32. Messick, S. (1993). Validity. In R. Linn (Ed.), Educational measurement, (3rd ed., pp. 13-103). Washington: American Council on Education.
33. Moss, P., Schutz, A., & Collins, K. (1997, April). Developing coherence between assessment and reforma: An integrative approach to portfolio evaluation for teacher licensure. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.
34. Mullis, I. (1984). Scoring direct writing assessments: What are the alternatives? Educational Measurement: Issues and Practices, 3, 16-18.
35. Myers, M. (1980). A procedure for writing assessment and holistic scoring. Urbana, IL: National Council of Teachers of English.
36. National Board for Professional Teaching Standards (2001). Early and middle childhood art: Scoring guide. Washington, DC: Author.
37. Olson, J., Bond, L., & Andrews, C. (1999). Annual survey of state student assessment programs: A summary report. Washington, DC: Council of Chief State School Officers.
38. Patsula, L., Clauser, B., Clyman, S., & Fan, V. (1998, April). Resolving rater discrepancies in scoring performance assessments. Paper presented at the annual meeting of the National Council on Measurement in Education, San Diego, CA.
39. Shavelson, R., Baxter, G., & Gao, X. (1993). Sampling variability of performance assessments. Journal of Educational Measurement, 30(3), 215-232.
40. Smith, W. (1993). Assessing the reliability and adequacy of using holistic scoring of essays as a college composition placement technique. In M. Williamson & B. Huot (Eds.), Validating holistic scoring for writing assessment: Theoretical and empirical foundations (pp. 142-205). Cresskill, NJ: Hampton.
41. Stallings, W., & Gillmore, G. (1971). A note on "accuracy" and "precision." Journal of Educational Measurement, 8, 127-129.
42. Taylor, C. (1998). An investigation of scoring methods for mathematics performance-based assessments. Educational Assessment, 5(3), 195-224.
43. Tuman, D. (1999). Gender style as form and content: An examination of gender stereotypes in the subject preference of children's drawing. Studies in Art Education, 41(1), 40-60.
44. Welch, C., & Martinovich-Barhite, D. (1997, April). Reliability issues and possible solutions. Paper presented at the annual meeting of the American Educational Research Association, Chicago.
45. White, E. (1993). Holistic scoring: Past triumphs, future challenges. In M. Williamson & B. Huot (Eds.), Validating holistic scoring for writing assessment: Theoretical and empirical foundations (pp. 79-108). Cresskill, NJ: Hampton.
46. White, E. (1994). Teaching and assessing writing: Recent advances in understanding, evaluating, and improving student performance (2nd ed.). San Francisco: Jossey-Bass Publishers.
47. Wiley, D., & Haertel, E. (1996). Extended assessment tasks: Purposes, definitions, scoring, and accuracy. In M. Kane & R. Mitchell (Eds.), Implementing performance assessment: Promises, problems, and challenges (pp. 61-89). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
48. Willard-Traub, M., Decker, E., Reed, R., & Johnston, J. (2000). The development of large-scale portfolio placement assessment at the University of Michigan: 1992-1998. Assessing Writing, 5(1), 41-84.
49. Wolcott, W. (with Legg, S.) (1998). An overview of writing assessment: Theory, research, and practice. Urbana, IL: National Council of Teachers of English.
50. Wolfe, E. (1997). The relationship between essay reading style and scoring proficiency in a psychometric scoring system. Assessing Writing, 4 (1) 83-106.