An application of generalizability theory and many-facet Rasch measurement using a complex problem-solving skills assessment

Educational and Psychological Measurement

Volumn 64, Issue 4, 2004, Pages 617-639

  Smith Jr , Everett V ^a   Kulikowich, Jonna M ^b  

^a UNIVERSITY OF ILLINOIS AT CHICAGO   (United States)

^b PENNSYLVANIA STATE UNIVERSITY   (United States)

Author keywords

Facets; Generalizability theory; Performance assessment; Rasch

Indexed keywords

EID: 3242700805     PISSN: 00131644     EISSN: None     Source Type: Journal    
DOI: 10.1177/0013164404263876     Document Type: Article

References (41)

1. Baron, J. B. (1994, April). Using multi-dimensionality to capture verisimilitude: Criterion-referenced performance-based assessments and the ooze factor. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
2. Baxter, G. P., Shavelson, R. J., Goldman, S. R., &Pine, J. (1992). Evaluation of procedure-based scoring for hands-on science assessment. Journal of Educational Measurement, 29, 1-17.
3. Black, S. (1995). You be the judge. Executive Educator, 17, 32-34.
4. Brennan, R. L. (1992). Elements of generalizability theory. Iowa City, IA: ACT.
5. Dixon, W. J. (1992). BMDP biomedical computer programs. Berkeley: University of California Press.
6. Engelhard, G., Jr. (1992). The measurement of writing ability with a many-faceted Rasch model. Applied Measurement in Education, 5, 171-191.
7. Engelhard, G., Jr. (1994). Examining rater errors in the assessment of written composition with a many-faceted Rasch model. Journal of Educational Measurement, 31, 93-112.
8. Hambleton, R. K., Swaminathan, H., &Rogers, H. J. (1991). Fundamentals of item response theory (Vol. 2). Newbury Park, CA: Sage.
9. Jetton, T. L., &Alexander, P. A. (1995). The nature of scientific discourse: The academic and social texts that are constructed through the discourse in a science classroom. Paper presented at the annual meeting of the American Educational Research Association, San Francisco.
10. Kulikowich, J. M., &Smith, E. V. (1994, April). Performance-based assessment in the urban classroom: Using paper-pencil and technological tasks. Paper presented as the annual meeting of the American Educational Research Association, New Orleans, LA.
11. Linacre, J. M. (1989). Multi-facet Rasch measurement. Chicago: MESA.
12. Linacre, J. M. (1995). Constructing measurement with a many-facet Rasch model. In M. Wilson (Ed.), Objective measurement: Theory into practice (Vol. 2, pp. 129-144). Norwood, NJ: Ablex.
13. Linacre, J.M.(1996a). FACETS: A computer program for analysis of examinations with multiple facets. Chicago: MESA.
14. Linacre, J. M. (1996b). Generalizability theory and many-facet Rasch measurement. In G. Engelhard Jr. &M. Wilson (Eds.), Objective measurement: Theory into practice (Vol. 3, pp. 85-98). Norwood, NJ: Ablex.
15. Linacre, J. M. (1999). Investigating rating scale category utility. Journal of Outcome Measurement, 3, 103-122.
16. Linacre, J. M., Engelhard, E., Jr., Tatum, D. S., &Myford, C. M. (1994). Measurement with judges: Many-faceted conjoint measurement. International Journal of Educational Research, 21, 569-577.
17. Lunz, M. E., & Stahl, J. A. (1990). Judge consistency and severity across grading periods. Evaluation & the Health Professions, 13, 435-444.
18. Lunz, M. E., Wright, B. D., &Linacre, J. M. (1990). Measuring the impact of judge severity on examination scores. Applied Measurement in Education, 3, 331-345.
19. Lunz, M. E., Wright, B. D., Stahl, J. A., &Linacre, J. M. (1989, March). Equating practical examinations. Paper presented at the annual meeting of the National Council on Measurement in Education, San Francisco.
20. Marcoulides, G. A., &Drezner, A. (1997). A method for analyzing performance assessments. In M. Wilson, G. Engelhard Jr., &K. Draney (Eds.), Objective measurement: Theory into practice (Vol. 4, pp. 261-277). Greenwich, CT. Ablex.
21. Marcoulides, G. A., &Drezner, A. (2000). A procedure for detecting pattern clustering in measurement designs. In M. Wilson &G. Engelhard Jr. (Eds.), Objective measurement: Theory into practice (Vol. 5, pp. 287-302). Stamford, CT: Ablex.
22. Mislevy, R. J. (1996). Test theory reconceived. Journal of Educational Measurement, 33, 379-416.
23. Muraki, E., Hombo, C., &Lee, Y. (2000, April). Equating and linking of performance assessments. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
24. Rodgers, Y. v. d. M. (1996). A role-playing exercise for development and international economics courses. Journal of Economic Education, 27, 217-223.
25. Roth, W.-M., &Lucas, K. B. (1997). From "truth" to "invented reality": A discourse analysis of high school physics students' talk about scientific knowledge. Journal of Research in Science Teaching, 34, 145-179.
26. Roth, W.-M., &Roychoudhury, A. (1994). Physics students' epistemologies and views about knowing and learning. Journal of Research in Science Teaching, 31, 5-30.
27. Sanders, W. L., &Horn, S. P. ( 1995, March). Educational assessment reassessed: The usefulness of standardized and alternative measures of student achievement as indicators for the assessment of educational outcomes. Educational Policy Analysis Archives, 3. Retrieved from http://olam.ed.asu.edu/ epaa/v3n6.html
28. Schumacker, R. E. (1999). Many-facet Rasch analysis with crossed, nested, and mixed designs. Journal of Outcome Measurement, 3, 323-338.
29. Schumacker, R. E., &Lunz, M. E. (1997). Interpreting the chi-square statistics reported in the many-faceted Rasch model. Journal of Outcome Measurement, 1, 239-257.
30. Shavelson, R. J., &Baxter, G. P. (1992). What we've learned about assessing hands-on science. Educational Leadership, 49, 20-25.
31. Shavelson, R. J., &Webb, N. M. (1991). Generalizability theory. Newbury Park, CA: Sage.
32. Shavelson, R. J., Webb, N. M., &Rowley, G. L. (1989). Generalizability theory. American Psychologist, 44, 922-932.
33. Smith, E.V., Jr. (2001). Evidence for the reliability of measures and the validity of measure interpretation: A Rasch measurement perspective. Journal of Applied Measurement, 2, 281-311.
34. Smith, E. V., Jr., Wakely, M., de Kruif, R., &Swartz, C. (2003). Optimizing ratings scales for self-efficacy (and other) research. Educational and Psychological Measurement, 63, 369-391.
35. Stahl, J. A., Lunz, M. E., &Wright, B. D. (1991, April). Equating examinations that include judges (multiple facets). Paper presented at the annual meeting of the National Council of Measurement in Education, Chicago.
36. Twing, J. S., &Williams, K. T. (1992). An investigation of writing assessment using a many-faceted Rasch model. Paper presented at the annual meeting of the American Educational Research Association, San Francisco.
37. van der Linden, W. J., &Hambleton, R. K. (Eds.). (1997). Handbook of modern item response theory. New York: Springer-Verlag.
38. Wright, B. D., &Masters, G. N. (1982). Rating scale analysis: Rasch measurement. Chicago: MESA.
39. Wright, B. D., &Stone, M. H. (1979). Best test design. Chicago: MESA.
40. Young, M. F. (1993). Instructional design for situated learning. Educational Technology Research and Development, 41, 43-58.
41. Zhu, W., Updyke, W. F., &Lewandowski, C. (1997). Post-hoc Rasch analysis of optimal categorization of an ordered-response scale. Journal of Outcome Measurement, 1, 286-304.