Rating performance assessments of students with disabilities: A study of reliability and bias

Journal of Psychoeducational Assessment

Volumn 28, Issue 6, 2010, Pages 536-550

  Mastergeorge, Ann M ^a   Martínez, José Felipe ^b  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

bias; disabilities; generalizability theory; performance assessment; teacher judgment

Indexed keywords

EID: 78349299881     PISSN: 07342829     EISSN: None     Source Type: Journal    
DOI: 10.1177/0734282909351022     Document Type: Article

References (37)

1. Bolt, S., & Ysseldyke, J. (2008) Accommodating students with disabilities in large-scale testing: A comparison of differential item functioning (dif) identified across disability types. Journal of Psychoeducational Assessment, 26, 121-138.
2. Bradley, D.F., & Calvin, M.B. (1998). Grading modified assignments: Equity or compromise? Teaching Exceptional Children, 31, 24-29.
3. Brennan, R.L. (2000). Performance assessments from the perspective of generalizability theory. Applied Psychological Measurement, 24, 339-353.
4. Bursuck, W., Polloway, E.A., Plante, L., Epstein, M.H., Jayanthi, M., & McConeghy, J. (1996). Report card grading and adaptations: A national survey of classroom practices. Exceptional Children, 62, 301-318.
5. Carpenter, D. (1985). Grading handicapped pupils: Review and position statement. Remedial and Special Education, 6, 54-59.
6. Christenson, S., Decker, D., Triezenberg, H., Ysseldyke, J., & Reschly, A. (2007). Consequences of high-stakes assessment for students with disabilities. Educational Policy, 21, 662-690.
7. Christiansen, J., & Vogel, J.R. (1998). A decision model for grading students with disabilities. Teaching Exceptional Children, 31, 30-35.
8. Clauser B.E., Clyman S.G., & Swanson, D.B. (1999). Components of Rater Error in a Complex Performance Assessment. Journal of Educational Measurement, 36(1):29-45.
9. Cohen, J. (1982). Set correlation as a general multivariate data-analytic method. Multivariate Behavioral Research, 17, 301-341.
10. Cook, B. (2001). A Comparison of teachers' attitudes toward their included students with mild and severe disabilities. Journal of Special Education, 34, 203-213.
11. Cook, B., Tankersley, M., Cook, L., & Landrum, T.J. (2007). Teachers' attitudes toward their included students with disabilities. Exceptional Children, 67, 115-135.
12. deBettencourt, L.U. (1999). General educator's attitudes toward students with mild disabilities and their use of instructional strategies: Implications for training. Remedial and Special Education, 20(1), 27-35.
13. Goldschmidt, P., Martínez, J.F., Niemi, D.,& Baker, E. (2007) Relationships among measures as empirical evidence of validity: incorporating multiple indicators of achievement and school context. Educational Assessment, 12(3), 239-266.
14. Decker, D., & Bolt, S. (2008). Challenges and opportunities for promoting student achievement through large scale assessment results. Assessment for Effective Intervention, 34, 43-51.
15. Hale, J.B., Fiorello, C.A., Miller, J.A., Wenrich, J., Teodori, A.M., & Henzel, J. (2008). WISC-IV assessment and intervention strategies for children with specific learning disabilities. In A. Prifitera, D. H. Saklofske, & L. G. Weiss (Eds.), WISC-IV clinical assessment and intervention (2nd ed., pp. 109-171). New York: Elsevier Science.
16. Hurwitz, J., Elliott, S., & Braden, J. (2007). The influence of test familiarity and student disability status upon teachers' judgments of students' test performance. School Psychology Quarterly, 22, 115-144.
17. Hoyt, W.T. (2000). Rater bias in psychological research: when is it a problem and what can we do about it? Psychological Methods, 5, 64-86.
18. Koretz, D., McCaffrey, D., Klein, S., Bell, R., & Stecher, B. (1993). The reliability of scores from the 1992 Vermont portfolio assessment program (CSE Technical Report 355). Los Angeles: University of California, Los Angeles, Center for the Study of Evaluation.
19. Llosa, L. (2008). Building and supporting a validity argument for a standards-based classroom assessment of English proficiency based on teacher judgments. Educational Measurement: Issues and Practice, 27(3), 32-42.
20. Martinez, J.F., Goldschmidt, P., Niemi, D., Baker, E. & Sylvester, R. (2007) Language Arts Performance Assessments: Generalizability Studies of Local and Central Ratings. Educational Assessment, 12(3), 267-282.
21. Mashburn, A., & Henry, G. (2004). Assessing school readiness: Validity and bias in preschool and kindergarten teachers' ratings. Educational Measurement: Issues and Practice, 23, 16-30.
22. Mastergeorge, A., Chow, V., Miyoshi, J., Martinez, F., & Golestani, C. (2000). Teachers' familiarity and use of accommodations for students with disabilities in large-scale assessments (CSE Technical Report). Los Angeles: University of California, Los Angeles, Center for the Study of Evaluation.
23. Michael, R.J., & Trippi, J.A. (1987). Educators' views of procedures for grading mainstreamed handicapped children. Education, 107, 276-278.
24. Miller, M.D., & Linn, R.L. (2000). Validation of performance-based assessments. Applied Psychological Measurement, 24, 367-378.
25. Munk, D.D., & Bursuck, W.D. (1998). Report card grading adaptations for students with disabilities: Types and acceptability. Intervention in School and Clinic, 33, 306-308.
26. Niemi, D., Baker, E.L., & Sylvester, R.M. (2007). Scaling up, scaling down: Seven years of performance assessment development in the nation's second largest school district. Educational Assessment, 12, 195-214.
27. Robertson, K., Chamberlain, B., & Kasari, C. (2003). General education teachers' relationships with included students with autism. Journal of Autism and Developmental Disorders, 33, 123-130.
28. SAS Institute, Inc. (2006). SAS [Software]. Cary, NC.
29. Schumm, J.S., & Vaughn, S. (1991). Making adaptations for mainstreamed students: General education teachers' perspectives. Remedial and Special Education, 12, 18-27.
30. Shavelson, R.J., & Webb, N.M. (1981). Generalizability theory: 1973-1980. British Journal of Mathematical and Statistical Psychology, 34, 133-166.
31. Shavelson, R.J., & Webb, N.M. (1991). Generalizability theory: A primer. Newbury Park, CA: Sage.
32. Sofie, C., & Riccio, C. (2002). A comparison of multiple methods for the identification of children with reading disabilities. Journal of Learning Disabilities, 35, 234-244.
33. Stanovich, K.E. (1999). The sociopsychometrics of learning disabilities. Journal of Learning Disabilities, 32, 350-361.
34. Struyk, L.R., Epstein, M.H., Bursuck, W., Polloway, E.A., McConeghy, J., & Cole, K.B. (1995). Home-work, grading, and testing practices used by teachers for students with and without disabilities. The Clearing House, 69, 50-56.
35. Waltman, K., Kahn, A., & Koency, G. (1998). Alternative approaches to scoring: The effects of using different scoring methods on the validity of scores from a performance assessment (CSE Tech. Rep. 488). Retrieved September 23, 2009, from the CRESST Web site: http://www.ces.ucls.edu.
36. Warger, C.L. (1983). An analysis of curriculum and grading formats used by regular secondary teachers. In P. L. Reed (Ed.), Mainstreaming in secondary schools: Focus on research (OATE-OACTE Monograph Series No. 6, pp. 1-8). Kent: Ohio Association of Colleges for Teacher Education, Ohio Association of Teacher Educators.
37. Webb, N.M., Schlackman, J., & Sugrue, B. (2000). The dependability and interchangeability of assessment methods in science. Applied Measurement in Education, 13, 277-301.