Interrogating the generalizability of portfolio assessments of beginning teachers: A qualitative study

Education Policy Analysis Archives

Volumn 12, Issue , 2004, Pages

  Moss, Pamela A ^a   Sutherland, LeeAnn M ^a   Haniford, Laura ^a   Miller, Renee ^a   Johnson, David ^a   Geist, Pamela K ^b   Koziol Jr , Stephen M ^c   Star, Jon R ^d   Pecheone, Raymond L ^e,f,g  

^a UNIVERSITY OF MICHIGAN   (United States)

^b University of Maryland ^*

^c UNIVERSITY OF MARYLAND   (United States)

^d MICHIGAN STATE UNIVERSITY   (United States)

^e STANFORD UNIVERSITY   (United States)

^f Lincoln NE   (United States)

^g Natl Bd for Prof Teach Standards   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 4644285302     PISSN: 10682341     EISSN: None     Source Type: Journal    
DOI: 10.14507/epaa.v12n32.2004     Document Type: Article

References (83)

1. AERA, APA, & NCME (1985). Standards for educational and psychological testing. Washington, DC: American Psychological Association.
2. AERA, APA, & NCME (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
3. Ball, D. L., Gere, A. R. & Moss, P. A. (1998). Fieldwork Guide for the INTASC Beginning Teacher Case Study Project. Unpublished manuscript, University of Michigan.
4. Bleicher, J. (1980). Contemporary hermeneutics: Hermeneutics as method, philosophy, and critique. London: Routledge and Kegan Paul.
5. Bond, L., Smith, T., Baker, W.K., & Hattie, J.A. (2000). The certification system of the National Board for Professional Teaching Standards: A construct and consequential validity study. Greensboro, NC: Center for Educational Research and Evaluation, The University of North Carolina at Greensboro.
6. Breland, H. M., Camp, R., Jones, R. J., Morris, M. M., & Rock, D. A. (1987). Assessing writing skill (Research Monograph No. 11). New York: College Entrance Examination Board.
7. Brennan, R. L. (1983). The elements of generalizability theory. Iowa City, IA: American College Testing Program.
8. Brennan, R. L. (2001). An essay on the history and future of reliability from the perspective of replications. Journal of Educational Measurement, 38(4), 295-317.
9. Brennan, R., & Johnson, E. (1995). Generalizability of performance assessments. Educational Measurement: Issues and Practice, 14(4), 9-12, 27.
10. Crehan, K. D. (2001). An investigation of the validity of scores on locally developed performance measures in a school assessment program. Educational and Psychological Measurement, 61(5), 841-848.
11. Cronbach, L. J. (1988). Five perspectives on validity argument. In H. Wainer (Ed.), Test validity (pp. 3-17). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
12. Cronbach, L. J. (1989). Construct validation after thirty years. In R. L. Linn (Ed.), Intelligence: Measurement, theory and public policy (pp. 147-171). Urbana: University of Illinois Press.
13. Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: Wiley.
14. Cronbach, L. J., Linn, R. L., Brennan, R. L., & Haertel, E. H. (1997). Generalizability analysis for performance assessments of student achievement or school effectiveness." Educational and Psychological Measurement, 3(57), 373-399.
15. Delandshere, G., & Petrosky, A. R. (1994). Capturing teachers' knowledge. Educational Researcher, 23 (5), pp. 11-18.
16. Dunbar, S. B., Koretz, D. M., & Hoover, H. D. (1991). Quality control in the development and use of performance assessments. Applied Measurement in Education, 4(4), 289-303.
17. Dwyer, C. A. (1998). Psychometrics of Praxis III: Classroom performance assessments. Journal of Personnel Evaluation in Education, 12(2), 163-187.
18. Educational Testing Service (ETS) (1998). NBPTS technical analysis report,1996-97 administration. Southfield, MI: NBPTS
19. Engelhard, G. (1994). Examining rater errors in the assessment of written composition with a many faceted Rasch model. Journal of Educational Measurement, 31(2), 93-112.
20. Engelhard, G. ( 2002). Monitoring raters in performance assessments. In G. Tindal & T. M. Haladyna (Eds.), Large scale assessment programs for all students (pp. 261-288). Mahwah, NJ: Erlbaum.
21. Erickson, F. (1986). Qualitative methods in research on teaching. In M. C. Wittrock (Ed.), Handbook of research on teaching (pp. 119-161). New York: Macmillan.
22. Gallego, M. A., Cole, M., & The Laboratory of Human Cognition (2002). Classroom cultures and cultures in Classrooms. In V. Richardson (Ed.), Handbook of Research on Teaching (4th Ed.) (pp. 951-997). Washington: AERA.
23. Gao, X., & Colton, D. A. (1997). Evaluating measurement precisions of performance assessment with multiple forms, raters, and tasks. In D. A. Colton (Ed.), Reliability issues with performance assessments: A collection of papers. Iowa City, American College Testing Program. (ACT Research Report Series 97-3).
24. Gao, X., Shavelson, R. J., & Baxter, G. P. (1994). Generalizability of large-scale performance assessments in science: Promises and problems. Applied Measurement in Education, 7(4), 323-342.
25. Haertel, E. (1985). Construct validity and criterion-referenced testing. Review of Educational Research, 55, 23-46.
26. Haertel, E. H. & Lorie, W. (in press). Validating Standards-Based Test Score interpretations. Measurement: Interdisciplinary Research and Perspectives.
27. Harris, D. J. (1997). Using reliabilities to make decision. In D. A. Colton (Ed.), Reliability issues with performance assessments: A collection of papers. Iowa City, American College Testing Program. (ACT Research Report Series 97-3).
28. Interstate New Teacher Assessment and Support Consortium. (1992). Model standards for beginning teacher licensing and development: A Resource for State Dialogue. Washington, DC: Council of Chief State School Officers.
29. Jaeger, R. M. (1998). Evaluating the psychometric qualities of the National Board for Professional Teaching Standards' Assessments: A methodological accounting. Journal of Personnel Evaluation in Education, 2(2), 189-210.
30. Kadane, J.B., & Schum, D.A. (1996). A probabilistic analysis of the Sacco and Vanzetti evidence. New York: Wiley.
31. Kane, M., Crooks, Terence, & Cohen, Allan (1999). Validating measures of performance. Educational Measurement: Issues and Practice, 18(2), 5-17.
32. Knapp, M. & S. Woolverton. (1995). Social class and schooling. In James Banks & Cherry A. McGee Banks (Eds.), Handbook of Research on Multicultural Education (pp. 548-569). New York: Simon and Schuster.
33. Klein, S. P., McCaffrey, D., Stecher, B., & Koretz, D. (1995). The reliability of mathematics portfolio scores: Lessons from the Vermont experience. Applied Measurement in Education, 8 (3), 243-260.
34. Koretz, D., McCaffrey, D., Klein, S., Bell, R., & Stecher, B. (1992). The reliability of scores from the 1992 Vermont Portfolio Assessment Program: Interim report. Santa Monica, CA: Rand Institute on Education and Training, National Center for Research on Evaluation, Standards, and Student Testing.
35. Koretz, D., Klein, S., McCaffrey, D., & Stecher, B. (1993). Interim report: The reliability of Vermont portfolio scores in the 1992-93 school year (RAND, RP-260). Santa Monica, CA: RAND. (Reprinted from CSE Technical Report 370, Los Angeles, University of California, Center for Research on Evaluation, Standards, and Student Testing, December.)
36. Koretz, D., Stecher, B., Klein, S., & McCaffrey, D. (in press). The Vermont Portfolio Assessment Program: Findings and implications. Educational Measurement: Issues and Practice, 13(3), 5-16.
37. Koretz, D., Stecher, B., Klein, S., McCaffrey, D., & Deibert, E. (1993). Can portfolios assess student performance and influence instruction? The 1991-92 Vermont experience (RAND, RP-259). Santa Monica, CA: RAND. (Reprinted from CSE Technical Report 371, Los Angeles, University of California, Center for Research on Evaluation, Standards, and Student Testing, December.)
38. Koziol, S. M. Jr., Burns, L., & Brass, J (2003). Four lenses for the analysis of teaching. Supporting beginning teachers' practice. Working paper, Michigan State University.
39. Lane, S.. Liu., M., Ankenmann, R. D., & Stone, C. A. (1996). Generalizability and validity of a mathematics performance assessment. Journal of Educational Measurement, 33(1), 71-92.
40. Linn, R. L. & Burton, E. (1994). Performance-based assessment: Implications of task specificity. Educational Measurement: Issues and Practice, 13(1), 5-8, 15.
41. Lyons, N. (1998) With portfolios in hand: validation the new teacher professionalism. New York: Teachers College Press.
42. McBee, M. M., & Barnes, L. L. (1998). The generalizability of a performance assessment measuring achievement in eight-grade mathematics. Applied Measurement in Education, 11(2),179-194.
43. McLaughlin, M., Talbert, J., & Bascia, N. (Eds.). (1990). The contexts of teaching in secondary schools: teachers' realities. New York: Teachers College Press.
44. McLaughlin, M. & Little, J. W. (Eds.). (1993). Teachers' work: Individuals, colleagues, and contexts. New York: Teachers College Press.
45. McNeil, L. (1983). Contradictions of control: School structure and school knowledge. New York: Routledge and K. Paul.
46. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-103). New York: American Council on Education/Macmillan.
47. Messick, S. (1994). The interplay of evidence and consequences in the validation of performance assessments. Education Researcher, 32(2), 13-23.
48. Messick, S. (1996). Validity and washback in language testing. Language Testing, 13(3), 241-256.
49. Mislevy, R.J. (1994). Evidence and inference in educational assessment. Psychometrika, 59, 439-483.
50. Mislevy, R. J., Almond, R., & Steinberg, L. (2003). On the structure of educational assessment. Measurement: Interdisciplinary Research and Perspectives, 1 (1), pp. 3-62.
51. Mislevy, R. J., Wilson, M. R., Ercikan, K., Chdowsky, N. (2002). Psychometric principles in student assessment. Los Angeles: CRESST.
52. Moss, P. A. (1992). Shifting conceptions of validity in educational measurement: Implications for performance assessment. Review of Educational Research, 62 (3), 229-258.
53. Moss, P. A. (1994). Can there be validity without reliability? Educational Researcher, 23(2), 5-12.
54. Moss, P. A. (1996). Enlarging the dialogue in educational measurement: Voices from interpretive research traditions. Educational Researcher, 25 (1),20-28, 43.
55. Moss, P. A. (1998). The role of consequences in validity theory. Educational Measurement: Issues and Practice, 17 (2), 5-12.
56. Moss, P. A. (in press). The meaning and consequences of reliability. Journal of Educational and Behavioral Statistics.
57. Moss, P. A., Rex, L., & Geist, P. (2000a). Case Study Writing Guide for the INTASC Beginning Teacher Case Study Project. Unpublished manuscript, University of Michigan.
58. Moss, P. A., Rex, L., & Geist, P. (2000b). Fieldwork Guide for the INTASC Beginning Teacher Case Study Project. Unpublished manuscript, University of Michigan.
59. Moss, P. A. & Schutz, A. (1999). Risking frankness in educational assessment. Phi Delta Kappan, 80(9), 680-687.
60. Moss, P. A. & Schutz, A. (2001). Educational standards, assessment, and the search for "consensus". American Educational Research Journal, 38 (1), 37-70.
61. Moss, P. A., Schutz, A. M., & Collins, K. M (1998). An Integrative approach to portfolio evaluation for teacher licensure. Journal of Personnel Evaluation in Education, 12(2),139-161.
62. Moss, P. A., Schutz, A. M., Haniford, L., Miller, R., & Coggshall, J. (in preparation). High stakes assessment as ethical decision making. Unpublished manuscript, University of Michigan.
63. Myford, C. M. (1993). Formative studies of Praxis III: Classroom Performance Assessments--An overview. The Praxis Series: Professional Assessments for Beginning Teachers. Princeton, NJ: Educational Testing Service.
64. Myford, C. M., & Engelhard, G. (2001). Examining the psychometric quality of the national board for professional teaching standards early childhood/generalist assessment System. Journal of Personnel Evaluation in Education, 15(4), 253-285.
65. Myford, C. M., & Mislevy, R. J. (1995). Monitoring and improving a portfolio assessment system (Center for Performance Assessment Research Report). Princeton, NJ: Educational Testing Service.
66. National Research Council (2001a). Knowing what students know: The science and design of educational assessment. Washington, D.C.: National Academy Press.
67. National Research Council (2001b). Testing teacher candidates: The role of licensure tests in improving teacher quality. Washington, D.C.: National Academy Press.
68. Nystrand, M., Cohen, A. S., & Martinez, D. (1993). Addressing reliability problems in the portfolio assessment of college writing. Educational Assessment, 1(1), 53-70.
69. Ormiston, G. L. & Schrift, A. D. (Eds.) (1990). The hermeneutic tradition: From Ast to Ricoeur. Albany: SUNY Press.
70. Pearlman, M. (in press a). The design architecture of NBPTS certification assessments. In L. Ingvarson (Ed.), Assessing teachers for professional certification. Stamford, CT: Jai.
71. Pearlman, M. (in press b). The evolution of the scoring system for NBPTS assessments. In L. Ingvarson (Ed.), Assessing teachers for professional certification. Stamford, CT: Jai.
72. Porter, A. C., Youngs, P. & Odden, A (2003). Advances in teacher assessments and their uses. In V. Richardson (Ed.), Handbook on Research on Teaching (pp. 259-297). Washington, DC: AERA.
73. Reckase, M. D. (1995). Portfolio assessment: A theoretical estimate of score reliability. Educational Measurement: Issues and Practice, 14(1), 12-14, 31.
74. Schmidt, L. K, (1995). Introduction: Between certainty and relativism. In. L. K. Schmidt (Ed.), The specter of relativism: Truth, dialogue, and phronesis in philosophical hermeneutics (pp. 1-22). Evanston, IL: Northwestern University Press.
75. Schutz, A & Moss, P. A. (in press), Reasonable decisions in portfolio assessment. Educational Policy Analysis Archives.
76. Shavelson, R. J., Baxter, G. P., Pine, J., Yure, J., Goldman, S.R., Smith, B. (1991). Alternative assessment technologies for large scale science assessment: Instrument of education reform. School effectiveness and school improvement, 2(2), 97-114.
77. Shavelson, R. J., & Webb, N. W. (1991). Generalizability theory: A primer. Newbury Park, CA: SAGE Publications.
78. Stodolsky, S. & Grossman, P. (1995). The impact of subject matter on curricular activity: An analysis of five academic subjects. American Educational Research Journal, 32(2), 227-49.
79. Swanson, D., Norman, G. R. & Linn, R. L. (1995). Performance-based assessment: Lessons from the health professions. Educational Researcher, 24(5), 5-11,35.
80. Wilson, M. (1994). Community of judgment: A teacher-centered approach to educational accountability. In Office of Technology Assessment (Ed.), Issues in Educational Accountability. Washington, D.C.: Office of Technology Assessment, United States Congress.
81. Wilson, M., & Case, H. (1997). An examination of variation in rater severity over time: A study in rater drift. Berkeley, CA: Berkeley Evaluation and Assessment Research (BEAR) Center.
82. Wilson, M., & Sloane, K. (2000). From principles to practice: An embedded assessment system. Applied Measurement in Education, 13(2), 181-208.
83. Zeichner, K. (2000). Alverno College. In L. Darling Hammond (Ed.), Studies in excellence in teacher education: Preparation in the undergraduate years. Washington, DC: American Association of Colleges of Teacher Education