Using portfolios in program evaluation: An investigation of interrater reliability

American Journal of Evaluation

Volumn 21, Issue 1, 2000, Pages 65-80

  Johnson, Robert L ^a   McDaniel, Fred ^a   Willeke, Marjorie J ^a  

^a UNIVERSITY OF SOUTH CAROLINA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0041410033     PISSN: 10982140     EISSN: None     Source Type: Journal    
DOI: 10.1016/S1098-2140(00)00052-7     Document Type: Article

References (39)

1. Benchmarks for science literacy. 1993;Oxford University Press, New York.
2. Baker E., Freeman M., Clayton S. Cognitive assessment of history for large-scale testing. Wittrock M., Baker E. Testing and cognition. 1991;131-153 Prentice Hall, Englewood Cliffs, NJ.
3. Bond, L., & Roeber, E. (1998). What's new in state assessment? A summary of findings of the 1997 state student assessment programs survey. Paper presented at the National Conference on Large-Scale Assessment, Colorado Springs, CO.
4. Breland, H. (1983). The direct assessment of writing skill: A measurement review (College Board Report No. 83-6). New York: College Entrance Examination Board.
5. Brennan, R. (1995). Generalizability of performance assessments. Paper presented at the annual meeting of the National Council of Measurement in Education, San Francisco, CA.
6. Brennan R., Kane M. An index of dependability for mastery tests. Journal of Educational Measurement. 14:1977;277-289.
7. Brown A., Campione J., Webber L., McGilly K. Interactive learning environments A new look at assessment and instruction. Gifford B., O'Connor M. Changing assessments Alternative views of aptitude, achievement, and instruction. 1992;37-75 Kluwer, Boston.
8. Cherry R., Myer P. Reliability issues in holistic assessment. Williamson M., Huot B. Validating holistic scoring for writing assessment Theoretical and empirical foundations. 1993;109-141 Hampton Press, Cresskill, NJ.
9. Crick J., Brennan R. GENOVA [Computer software]. 1984;National Board of Medical Examiners, Philadelphia, PA.
10. Cronbach L., Linn R., Brennan R., Haertel E. Generalizability analysis for educational assessments. 1995;Center for the Study of Evaluation, Standards, and Student Testing, University of California at Los Angeles, Los Angeles.
11. Gredler M. Implications of portfolio assessment for program evaluation. Studies in Educational Evaluation. 21:1995;432-437.
12. Hayes J., Hatch J. Issues in measuring reliability Correlation versus percentage of agreement. Written Communication. 16:(3):1999;354-367.
13. Herman, J., & Golan, S. (1993). The effects of standardized testing on teaching and schools. Educational Measurement: Issues and Practice, 12(4), 20-25, 41-42.
14. Johnson, R., Penny, J., & Gordon, B. (in press). The relationship between score resolution methods and interrater reliability: An empirical study of an analytic scoring rubric. Applied Measurement in Education.
15. Johnson R., Willeke M., Bergman T., Steiner D. Family literacy portfolios Development and implementation. Window on the World of Family Literacy. 2:(2):1997;10-17.
16. Johnson R., Willeke M., Steiner D. Stakeholder collaboration in the design and implementation of a family literacy portfolio assessment. The American Journal of Evaluation. 19:(3):1998;339-353.
17. The Program Evaluation Standards How to Assess Evaluations of Educational Programs. 2nd ed. 1994;Sage, Thousand Oaks, CA.
18. Koretz D., Stecher B., Klein S., McCaffrey D. The Vermont portfolio assessment program Findings and implications. Educational Measurement: Issues and Practice. 13:(3):1994;5-16.
19. LeMahieu, P., Gitomer, D., & Eresh, J. (1995). Portfolios in large-scale assessment: Difficult but not impossible. Educational Measurement: Issues and Practice, 14(3), 11-16, 25-28.
20. Linn, R., & Burton, E. (1994). Performance-based assessment: Implications of task specificity. Educational Measurement: Issues and Practices, 13(1), 5-8, 15.
21. Mehrens W., Lehmann I. Measurement and evaluation in education and psychology. 4th ed. 1991;Harcourt Brace College Publishers, Fort Worth, TX.
22. Meisels S., Liaw F., Dorfman A., Nelson R. The Work Sampling System Reliability and validity of a performance assessment for young children. Early Childhood Research Quarterly. 10:1995;277-296.
23. Standards for the English language arts. 1996;Author, Urbana, IL.
24. Curriculum and evaluation standards for school mathematics. 1989;Author, Reston, VA.
25. Nunnally J. Psychometric theory. 2nd ed. 1978;McGraw-Hill, New York.
26. Paris S., Lawton T., Turner J., Roth J. A developmental perspective on standardized achievement testing. Educational Researcher. 20:(5):1991;12-20.
27. Popp R. Family portfolios Documenting change in parent-child relationships. 1992;National Center for Family Literacy, Louisville, KY.
28. Resnick L., Resnick D. Assessing the thinking curriculum New tools for educational reform. Gifford B., O'Connor M. Changing assessments Alternative views of aptitude, achievement, and instruction. 1992;37-75 Kluwer, Boston.
29. Shavelson R., Webb N. Generalizability theory A primer. 1991;Sage, Newbury Park, CA.
30. Shepard, L. (1991). Will national tests improve student learning? Phi Delta Kappan, 232-238.
31. Shepard L. Commentary What policy makers should know about the new psychology of intellectual ability and learning. Gifford B., O'Connor M. Changing assessments Alternative views of aptitude, achievement, and instruction. 1992;301-328 Kluwer, Boston.
32. Smith M.L. Put to the test The effects of external testing on teachers. Educational Researcher. 20:(5):1991;8-11.
33. Smith M., Rottenberg C. Unintended consequences of external testing in elementary schools. Educational Measurement: Issues and Practice. 10:(4):1991;7-11.
34. Supovitz J., MacGowan A., Slattery J. Assessing agreement An examination of the interrater reliability of portfolio assessment in Rochester, New York. Educational Assessment. 4:(3):1997;237-259.
35. Thorndike R., Cunningham G., Thorndike R., Hagen E. Measurement and evaluation in psychology and education. 5th ed. 1991;Macmillan, New York.
36. Valencia S., Au K. Portfolios across educational contexts Issues of evaluation, teacher development, and system validity. Educational Assessment. 4:(1):1997;1-35.
37. Wiggins, G. (1993). Assessment: Authenticity, context, and validity. Phi Delta Kappan, 200-214.
38. Wolf, D., Bixby, J., Glenn, J., & Gardner, H. (). To use their minds well: Investigating new forms of student assessment. In G. Grant (Ed.), Review of research in education, Vol. 17 (pp. 31-74). Washington, DC: American Educational Research Association.
39. Worthen B. A survey of Evaluation Practice readers. Evaluation Practice. 17:(1):1996;85-90.