Rater bias in psychological research: When is it a problem and what can we do about it?

Psychological Methods

Volumn 5, Issue 1, 2000, Pages 64-86

  Hoyt, William T ^a  

^a UNIVERSITY OF WISCONSIN MADISON   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; EPIDEMIOLOGY; HUMAN; OBSERVER VARIATION; PSYCHOMETRY; STATISTICAL MODEL;

BIAS (EPIDEMIOLOGY); HUMANS; MODELS, STATISTICAL; OBSERVER VARIATION; PSYCHOMETRICS;

EID: 0034152760     PISSN: 1082989X     EISSN: None     Source Type: Journal    
DOI: 10.1037/1082-989X.5.1.64     Document Type: Article

References (43)

1. American Educational Research Association/American Psychological Association/National Council on Measurement in Education. (1985). Standards for educational and psychological testing. Washington, DC: American Psychological Association.
2. Bollen, K. A. (1989). Structural equations with latent variables. New York: Wiley.
3. Brennan, R. L. (1992). Elements of generalizability theory (2nd ed.). Iowa City, IA: American College Testing.
4. Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.
5. Cleary, T. A., & Linn, R. L. (1969). Error of measurement and the power of a statistical test. British Journal of Mathematical and Statistical Psychology, 22, 49-55.
6. Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum.
7. Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155-159.
8. Cooper, W. H. (1981). Ubiquitous halo. Psychological Bulletin, 90, 218-244.
9. Cronbach, L. J. (1947). Test "reliability": Its meaning and determination. Psychometrika, 12, 1-16.
10. Cronbach, L. J. (1976). On the design of educational measures. In D. N. M. de Gruijter & L. J. T. van der Kamp (Eds.), Advances in psychological and educational measurement (pp. 199-208). New York: Wiley.
11. Cronbach, L. J. (1995). Giving method variance its due. In P. E. Shrout and S. T. Fiske (Eds.), Personality research, methods, and theory: A Festschrift in honor of Donald Fiske (pp. 145-157). Hillsdale, NJ: Erlbaum.
12. Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: Wiley.
13. DeShon, R. P. (1998). A cautionary note on measurement error corrections in structural equation models. Psychological Methods, 3, 412-423.
14. Fleiss, J. L. (1981). Balanced incomplete block designs for inter-rater reliability studies. Applied Psychological Measurement, 5, 105-112.
15. Ford, A. (1931). Neutralizing inequalities in rating. Personnel Journal, 9, 466-469.
16. Guilford, J. P. (1954). Psychometric methods (2nd ed.). New York: McGraw-Hill.
17. Hill, C. E., O'Grady, K. E., & Price, P. (1988). A method for investigating sources of rater bias. Journal of Counseling Psychology, 35, 346-350.
18. Hoyt, W. T., & Kerns, M.-D. (1999). Magnitude and moderators of bias in observer ratings: A meta-analysis. Psychological Methods, 4, 403-424.
19. Hoyt, W. T., & Melby, J. N. (1999). Dependability of measurement in counseling psychology: An introduction to generalizability theory. The Counseling Psychologist, 27, 325-352.
20. Hunter, I.E., & Schmidt, F. L. (1990). Methods of metaanalysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.
21. Hunter, I.E., & Schmidt, F. L. (1994). Correcting for sources of artificial variation across studies. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 323-336). New York: Russell Sage Foundation.
22. Kane, J. S., Bernardin, H. J., Villanova, P., & Peyrefitte, J. (1995). Stability of rater leniency: Three studies. Academy of Management Journal, 38, 1036-1051.
23. Kane, M. T., Gillmore, G. M., & Crooks, T. J. (1976). Student evaluations of teaching: The generalizability of class means. Journal of Educational Measurement, 13, 171-183.
24. Kenny, D. A. (1979). Correlation and causality. New York: Wiley.
25. Kenny, D. A. (1991). A general model of consensus and accuracy in interpersonal perception. Psychological Review, 98, 155-163.
26. Kenny, D. A. (1994). Interpersonal perception: A social relations analysis. New York: Guilford.
27. Kenny, D. A., & LaVoie, L. (1984). The social relations model. In L. Berkowitz (Ed.), Advances in experimental social psychology (Vol. 18, pp. 141-182). New York: Academic Press.
28. Kline, R. B. (1998). Principles and practice of structural equation modeling. New York: Guilford Press.
29. Kneeland, N. (1929). That lenient tendency in rating. Personnel Journal, 7, 356-366.
30. Lance, C. E., LaPointe, J. A., & Fisicaro, S. A. (1994). Tests of three causal models of halo rater error. Organizational Behavior and Human Decision Processes, 57, 83-96.
31. Murphy, K. R., Jako, R. A., & Anhalt, R. L. (1993). Nature and consequences of halo error: A critical analysis. Journal of Applied Psychology, 78, 218-225.
32. Raymond, M. R., & Viswesvaran, C. (1993). Least squares models to correct for rater effects in performance assessment. Journal of Educational Measurement, 30, 253-268.
33. Rothstein, H. R. (1990). Interrater reliability of job performance ratings: Growth to asymptote level with increasing opportunity to observe. Journal of Applied Psychology, 75, 322-327.
34. Saal, F. E., Downey, R. G., & Lahey, M. A. (1980). Rating the ratings: Assessing the psychometric quality of rating data. Psychological Bulletin, 88, 4130-428.
35. Schmidt, F. L. (1992). What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist, 47, 1173-1181.
36. Schmidt, F. L., & Hunter, J. E. (1996). Measurement error in psychological research: Lessons from 26 research scenarios. Psychological Methods, 1, 199-223.
37. Shavelson, R. J., & Webb, N.M. (1991). Generalizability theory: A primer. Newbury Park, CA: Sage.
38. Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420-428.
39. Spearman, C. (1910). Correlation calculated from faulty data. British Journal of Psychology, 3, 271-295.
40. Sutcliffe, J. P. (1980). On the relationship of reliability to statistical power. Psychological Bulletin, 88, 509-515.
41. Thorndike, E. L. (1920). A constant error in psychological ratings. Journal of Applied Psychology, 4, 25-29.
42. Webb, N. M., & Shavelson, R. J. (1981). Multivariate generalizability of General Education Development ratings. Journal of Educational Measurement, 18, 13-22.
43. Wilson, H. G. (1988). Parameter estimation for peer grading under incomplete design. Educational and Psychological Measurement, 48, 69-81.