Detecting and Measuring Rater Effects Using Many-Facet Rasch Measurement: Part II

Journal of Applied Measurement

Volumn 5, Issue 2, 2004, Pages 189-227

  Myford, Carol M ^a,c   Wolfe, Edward W ^b  

^a UNIVERSITY OF ILLINOIS AT CHICAGO   (United States)

^b MICHIGAN STATE UNIVERSITY   (United States)

^c Department of Kinesiology   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CENTRAL TENDENCY; MANY-FACET RASCH MEASUREMENT (MFRM); RATER EFFECTS;

COMPUTATIONAL LINGUISTICS; COMPUTER PROGRAMMING; MATHEMATICAL MODELS; PROBLEM SOLVING; STATISTICAL METHODS;

MEASUREMENT THEORY;

ARTICLE; HUMAN; METHODOLOGY; OBSERVER VARIATION; PSYCHOLOGICAL MODEL; PSYCHOMETRY; STATISTICS;

HUMANS; MODELS, PSYCHOLOGICAL; OBSERVER VARIATION; PSYCHOMETRICS;

EID: 1842843697     PISSN: 15297713     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (85)

1. Adams, R. J., Wilson, M. R., and Wang, W. (1997). The multidimensional random coefficients multinomial logit model. Applied Psychological Measurement, 21, 1-23.
2. Adams, R. J., Wilson, M., and Wu, M. (1997). Multilevel item response modeling: An approach to errors in variables regression. Journal of Educational and Behavioral Statistics, 22, 47-76.
3. Andrich, D. (1998). Thresholds, steps and rating scale conceptualization. Rasch Measurement: Transactions of the Rasch Measurement SIG, 12, 648.
4. Baker, E. L., Abedi, J., Linn, R. L., and Niemi, D. (1996). Dimensionality and generalizability of domain-independent performance assessments. Journal of Educational Research, 89, 197-205.
5. Beale, E. M. L., and Little, R. J. A. (1975). Missing data in multivariate analysis. Journal of the Royal Statistical Society (B), 129-145.
6. Bernardin, H. J., and Pence, E. C. (1980). Effects of rater training: Creating new response sets and decreasing accuracy. Journal of Applied Psychology, 65, 60-66.
7. Borman, W. C. (1977). Consistency of rating accuracy and rating errors in the judgment of human performance. Organizational Behavior and Human Performance, 20, 238-252.
8. Braun, H. I. (1988). Understanding scoring reliability: Experiments in calibrating essay readers. Journal of Educational Statistics, 13, 1-18.
9. Brennan, R. L. (1983). Elements of generalizability theory. Iowa City, IA: The American College Testing Program.
10. Brennan, R. L. (2000). (Mis) conceptions about generalizability theory. Educational Measurement: Issues and Practice, 19, 5-10.
11. Brennan, R. L. (2001). Generalizability theory, New York: Springer-Verlag.
12. Cason, G. J., and Cason, C. L. (1984). A deterministic theory of clinical performance rating. Evaluation and the Health Professions, 7, 221-247.
13. Clauser, B. E., Clyman, S. G., and Swanson, D. B. (1999). Components of rater error in a complex performance assessment. Journal of Educational Measurement, 36, 29-45.
14. Cracker, L., and Algina, J. (1986). Introduction to classical and modern test theory. New York: Holt, Rinehart, and Winston.
15. Cronbach, L. J., Gleser, G. C., Nanda, H., and Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability scores and profiles. New York: Wiley.
16. DeGruijter, D. N. M. (1984). Two simple models for rater effects. Applied Psychological Measurement, 8, 213-218.
17. Engelhard, G., Jr. (1994). Examining rater errors in the assessment of written composition with a many-faceted Rasch model. Journal of Educational Measurement, 31, 93-112.
18. nd ed.). New York: McGraw Hill.
19. Hedge, J. W., and Kavanagh, M. J. (1988). Improving the accuracy of performance evaluations: Comparison of three methods of performance appraiser training. Journal of Applied Psychology, 73, 68-73.
20. Hill, C. E., O'Grady, K. E., and Price, P. (1988). A method for investigating sources of rater bias. Journal of Counseling Psychology, 35, 346-350.
21. Hoskens, M., and Wilson, M. (2001). Real-time feedback on rater drift in constructed response items: An example from the Golden State Examination. Journal of Educational Measurement, 38, 121-146.
22. Houston, W. M., Raymond, M. R., and Svec, J. C. (1991). Adjustments for rater effects in performance assessment. Applied Psychological Measurement, 15, 409-421.
23. Hoyt, W. T. (2000). Rater bias in psychological research: When is it a problem and what can we do about it? Psychological Methods, 5, 64-86.
24. Hoyt, W. T., and Kerns, M. D. (1999). Magnitude and moderators of bias in observer ratings: A meta-analysis. Psychological Methods, 4, 403-424.
25. Johnson, R. L., Penny, J., and Johnson, C. (2000). The relationship between score resolution methods and interrater reliability: An empirical study of an analytic scoring rubric. Applied Measurement in Education, 13, 121-138.
26. Keaveny, T. J., and McGann, A. F. (1975). A comparison of behavioral expectation scales and graphic rating scales. Journal of Applied Psychology, 60, 695-703.
27. Kenny, D. A., and Kashy, D. A. (1992). Analysis of the multitrait-multimethod matrix by confirmatory factor analysis. Psychological Bulletin, 112, 165-172.
28. Lance, C. E., LaPointe, J. A., and Stewart, A. M. (1994). A test of the context dependency of three causal models of halo rater error. Journal of Applied Psychology, 79, 332-340.
29. Lane, S., Liu, M., Ankenmann, R. D., and Stone, C. A. (1996). Generalizability and validity of a mathematics performance assessment. Journal of Educational Measurement, 33, 71-92.
30. Latham, G. P., Wexley, K. N., and Pursell, E. D. (1975). Training managers to minimize rating errors in the observation of behavior. Journal of Applied Psychology, 60, 550-555.
31. Linacre, J. M. (1989). Many-faceted Rasch measurement. Chicago: MESA Press.
32. Linacre, J. M. (1996). Generalizability and many-facet Rasch measurement. In G. Engelhard, Jr., and M. Wilson (Eds.), Objective measurement: Theory into practice: Vol. 3 (pp. 85-98). Norwood, NJ: Ablex.
33. Linacre, J. M. (1999). Investigating rating scale category utility. Journal of Outcome Measurement, 3, 103-122.
34. Linacre, J. M. (2001a). FACETS [Computer program, version 3.36.2]. Chicago: MESA Press.
35. Linacre, J. M. (2001b). A user's guide to Facets: Rasch measurement computer program [Computer program manual]. Chicago: MESA Press.
36. Linacre, J. M. (2002). Number of person or item strata. Rasch Measurement: Transactions of the Rasch Measurement SIG, 16, 888.
37. Linn, R. L., Burton, E., DeStefano, L., and Hanson, M. (1996). Generalizability of New Standards Project 1993 pilot study tasks in mathematics. Applied Measurement in Education, 9, 201-214.
38. Little, R. J. A., and Rubin, D. B. (1987). Statistical analysis with missing data. New York: John Wiley.
39. Longford, N. T. (1994a). Reliability of essay rating and score adjustment. Journal of Educational and Behavioral Statistics, 19, 171-201.
40. Longford, N. T. (1994b). A case for adjusting subjectively rated scores in the Advanced Placement tests (ETS Technical Report 94-5). Princeton, NJ: Educational Testing Service.
41. Longford, N. T. (1995). Measurement of uncertainty in educational testing. New York: Springer-Verlag.
42. Longford, N. T. (1996). Adjustment for reader rating behavior in the Test of Written English (TOEFL Research Report No. 55). Princeton, NJ: Educational Testing Service.
43. Lumley, T., and McNamara, T. F. (1995). Rater characteristics and rater bias: Implications for training. Language Testing, 12, 54-71.
44. Lunz, M. E., Stahl, J. A., and Wright, B. D. (1996). The invariance of rater severity calibrations. In G. Engelhard, Jr., and M. Wilson (Eds.), Objective measurement: Theory into practice: Vol. 3 (pp. 99-112). Norwood, NJ: Ablex.
45. Lynch, B. K., and McNamara, T. F. (1998). Using G-theory and many-facet Rasch measurement in the development of performance assessments of the ESL speaking skills of immigrants. Language Testing, 15, 158-180.
46. MacMillan, P.D. (2000). Classical, generalizability, and multifaceted Rasch detection of interrater variability in large, sparse data sets. Journal of Experimental Education, 68, 167-190.
47. Marcoulides, G. A. (1999). Generalizability theory: Picking up where the Rasch IRT model leaves off? In S. E. Embretson, and S. L. Hershberger (Eds.), The new rules of measurement: What every psychologist and educator should know (pp. 129-152). Mahwah, NJ: Lawrence Erlbaum.
48. Marcoulides, G. A., and Drezner, Z. (1997). A method for analyzing performance assessments. In M. Wilson, G. Engelhard, Jr., and K. Draney (Eds.), Objective measurement: Theory into practice: Vol. 4 (pp. 261-277). Greenwich, CT: Ablex.
49. Marsh, H. W. (1989). Confirmatory factor analysis of multitrait- multimethod data: Many problems and a few solutions. Applied Psychological Measurement, 13, 335-361.
50. Marsh, H. W., and Bailey, M. (1991). Confirmatory factor analyses of multitrait-multimethod data: A comparison of alternative models. Applied Psychological Measurement, 15, 47-70.
51. McNamara, T. F., and Adams, R. J. (2000). The implications of halo effects and item dependencies for objective measurement. In M. Wilson, and G. Engelhard, Jr. (Eds.), Objective measurement: Theory into practice: Vol. 5 (pp. 243-257). Stamford, CT: Ablex.
52. Muraki, E. (1999, April). The introduction of essay questions to the GRE: Toward a synthesis of item response theory and generalizability theory. Paper presented at the annual meeting of the American Educational Research Association, Montréal, Canada.
53. Muraki, E., and Bock, R. D. (2003). PARSCALE [Computer program, version 4]. St. Paul MN: Assessment Systems Corporation.
54. Murphy, K. R., and Anhalt, R. L. (1992). Is halo error a property of the rater, ratees, or the specific behavior observed? Journal of Applied Psychology, 77, 494-500.
55. Murphy, K. R., and DeShon, R. (2000a). Interrater correlations do not estimate the reliability of job performance ratings. Personnel Psychology, 53, 873-900.
56. Murphy, K. R., and DeShon, R. (2000b). Progress in psychometrics: Can industrial and organizational psychology catch up? Personnel Psychology, 53, 913-924.
57. Myford, C. M., Marr, D. B., and Linacre, J. M. (1996). Reader calibration and its potential role in equating for the TWE (TOEFL Research Report No. 95-40). Princeton, NJ: Educational Testing Service.
58. Myford, C. M., and Mislevy, R. J. (1995). Monitoring and improving a portfolio assessment system (MS #94-05). Princeton, NJ: Educational Testing Service, Center for Performance Assessment.
59. Myford, C. M., and Wolfe, E. W. (2002). When raters disagree, then what: Examining a third-rating discrepancy resolution procedure and its utility for identifying unusual patterns of ratings. Journal of Applied Measurement, 3, 300-324.
60. O'Grady, K. E., and Medoff, D. R. (1991). Rater reliability - a maximum-likelihood confirmatory factor-analytic approach. Multivariate Behavioral Research, 26, 363-387.
61. O'Neill, T. R., and Lunz, M. E. (2000). A method to study rater severity across several administrations. In M. Wilson, and G. Engelhard, Jr. (Eds.), Objective measurement: Theory into practice: Vol. 5 (pp. 135-146). Stamford, CT: Ablex.
62. Patz, R. J., Junker, B. W., Johnson, M. S., and Mariano, L. T. (2002). The hierarchical rater model for rated test items and its application to large-scale educational assessment data. Journal of Educational and Behavioral Statistics, 27, 341-384.
63. Patz, R. J., Wilson, M. J., and Hoskens, M. (1997). Optimal rating procedures and methodology for NAEP open-ended items (Working Paper No. 97-37). Washington, DC: U. S. Department of Education, Office of Educational Research and Improvement, National Center for Education Statistics. Retrieved Sept. 5, 2001 from the World Wide Web: http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid= 9737
64. Pulakos, E. D., Schmitt, N., and Ostroff, C. (1986). A warning about the use of a standard deviation across dimensions within ratees to measure halo. Journal of Applied Psychology, 71, 29-32.
65. Raymond, M. R. (1986). Missing data in evaluation research. Evaluation and the Health Professions, 9, 395-420.
66. Raymond, M. R., and Houston, W. H. (1990). Detecting and correcting for rater effects in performance assessment (ACT Research Report Series 90-14). Iowa City, IA: The American College Testing Program.
67. Raymond, M. R., and Viswesvaran, C. (1993). Least-squares models to correct for rater effects in performance assessment. Journal of Educational Measurement, 30, 253-268.
68. Raymond, M. R., Webb, L. C., and Houston, W. M. (1991). Correcting performance-rating errors in oral examinations. Evaluation and the Health Professions, 14, 100-122.
69. Rost, J. (1988). Rating scale analysis with latent class models. Psychometrika, 53, 327-348.
70. Saal, F. E., Downey, R. G., and Lahey, M. A. (1980). Rating the ratings: Assessing the psychometric quality of rating data. Psychological Bulletin, 88, 413-428.
71. Scullen, S. E. (1999). Using confirmatory factor analysis of correlated uniquenesses to estimate method variance in multitrait-multimethod matrices. Organizational Research Methods, 2, 275-292.
72. Scullen, S. E., Mount, M. K., and Goff, M. (2000). Understanding the latent structure of job performance ratings. Journal of Applied Psychology, 85, 956-970.
73. Shavelson, R. J., Webb, N. M., and Rowley, G. L. (1989). Generalizability theory. American Psychologist, 44, 922-932.
74. Wang, W. (1997). Estimating rater severity with multilevel and multidimensional item response modeling. Taipei, Taiwan: Taiwan National Science Council. (ERIC Document Reproduction Service No. ED 408 340).
75. Wang, W., Wilson, M. R., and Adams, R. J. (1997). Rasch models for multidimensionality between and within items. In M. Wilson, G. Engelhard, Jr., and K. Draney (Eds.), Objective measurement: Theory into practice: Vol. 4 (pp. 139-156). Greenwich, CT: Ablex.
76. Widaman, K. F. (1985). Hierarchically nested covariance structure models for multitrait-multimethod data. Applied Psychological Measurement, 9, 1-26.
77. Wilson, H. G. (1988). Parameter estimation for peer grading under incomplete design. Educational and Psychological Measurement, 48, 69-81.
78. Wilson, M. R., and Adams, R. J. (1995). Rasch models for item bundles. Psychometrika, 60, 181-198.
79. Wilson, M. R., and Case, H. (2000). An examination of variation in rater severity over time: A study of rater drift. In M. Wilson, and G. Engelhard, Jr. (Eds.), Objective measurement: Theory into practice: Vol. 5 (pp. 113-134). Stamford, CT: Ablex.
80. Wilson, M. R., and Hoskens, M. (1999, April). The rater bundle model for constructed-response items: An example in the context of real-time feedback on rater effects. Paper presented at the annual meeting of the American Educational Research Association, Montréal, Canada.
81. Wolfe, E. W. (1998, April). A two-parameter logistic rater model (2PLRM): Detecting rater harshness and centrality. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA.
82. Wolfe, E. W. (in press). Identifying rater effects using latent trait models. Psychology Science.
83. Wolfe, E. W., Chiu, C. W. T., and Myford, C. M. (1999). The manifestation of common rater effects in multi-faceted Rasch analyses (MS #97-02). Princeton, NJ: Educational Testing Service, Center for Performance Assessment.
84. Wolfe, E. W., Moulder, B. M., and Myford, C. M. (2001). Detecting differential rater functioning over time (DRIFT) using a Rasch multi-faceted rating scale model. Journal of Applied Measurement, 2, 256-280.
85. Wu, M., Adams, R., and Wilson, M. (1997). ConQuest [Computer program]. Melbourne, Australia: Australian Council for Educational Research.