Magnitude and moderators of bias in observer ratings: A meta-analysis

Psychological Methods

Volumn 4, Issue 4, 1999, Pages 403-424

  Hoyt, William T ^a   Kerns, Michael David ^b  

^a UNIVERSITY OF WISCONSIN MADISON   (United States)

^b NONE

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0041029763     PISSN: 1082989X     EISSN: None     Source Type: Journal    
DOI: 10.1037/1082-989X.4.4.403     Document Type: Article

References (91)

1. Achenbach, T. M., McConaughty, S. H., & Howell, C. T. (1987). Child/adolescent behavioral and emotional problems: Implications of cross-informant correlations for situational specificity. Psychological Bulletin, 101, 213-232.
2. Adams, D. C., Gurevitch, J., & Rosenberg, M. S. (1997). Resampling test for meta-analysis of ecological data. Ecology, 78, 1277-1283.
3. * Axelrod, B. N., Goldman, R. S., & Woodard, J. L. (1992). Interrater reliability in scoring the Wisconsin Card Sorting Test. Clinical Neuropsychologist, 6, 143-155.
4. * Baker, E. L., Abedi, J., Linn, R. L., & Niemi, D. (1995). Dimensionality and generalizability of domain-independent performance assessments. Journal of Educational Research, 89, 197-205.
5. Baucom, D. H., & Sayers, S. (1989). The behavioral observation of couples: Where have we lagged and what is the next step in the sequence? Behavioral Assessment, 11, 149-159.
6. * Bennett, R. E., & Rock, D. L. (1995). Generalizability, validity, and examinee perceptions of a computer-delivered Formulating Hypotheses test. Journal of Educational Measurement, 32, 19-36.
7. * Berk, R. A. (1979). Generalizability of behavioral observations: A clarification of interobserver agreement and interobserver reliability. American Journal of Mental Deficiency, 83, 460-472.
8. Bernardin, H. J., & Pence, E. C. (1980). Effects of rater training: Creating new response sets and decreasing accuracy. Journal of Applied Psychology, 65, 60-66.
9. * Boodoo, G. M., & Garlinghouse, P. (1983). Use of the essay examination to investigate the writing skills of undergraduate education majors. Educational and Psychological Measurement, 43, 1005-1014.
10. * Booth, C. L., Mitchell, S. K., & Solin, F. K. (1979). The generalizability study as a method of assessing intra-and interobserver reliability in observational research. Behavior Research Methods & Instrumentation, 11, 491-494.
11. Brennan, R. L. (1992). Elements of generalizability theory (Rev. ed.). Iowa City: American College Testing Program.
12. * Brennan, R. L., Gao, X., & Colton, D. A. (1995). Generalizability analyses of Work Keys Listening and Writing Tests. Educational and Psychological Measurement, 55, 157-176.
13. * Brennan, R. L., & Lockwood, R. E. (1980). A comparison of the Nedelsky and Angoff cutting score procedures using generalizability theory. Applied Psychological Measurement, 4, 219-240.
14. Buckley, M. R., Cote, J. A., & Comstock, S. M. (1990). Measurement errors in the behavioral sciences: The case of personality/attitude research. Educational and Psychological Measurement, 50, 447-474.
15. * Cain, P. S., & Green, B. F. (1983). Reliabilities of selected ratings available from the Dictionary of Occupational Titles. Journal of Applied Psychology, 1, 155-165.
16. Campbell, D. T., & Fiske, D.W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.
17. * Carbno, W. C. (1981). Student evaluation of teacher effectiveness: A case study. Educational and Psychological Measurement, 41, 937-951.
18. * Chalmers, D. J., & Knight, R. G. (1985). The reliability of ratings of the familiarity of the environmental stimuli. Environment and Behavior, 17, 223-238.
19. * Coates, T. J., Rosekind, M. R., Strossen, R. J., Thoresen, C. E., & Kirmil-Gray, K. (1979). Sleep recordings in the laboratory and home: A comparative analysis. Neurophysiology, 16, 339-346.
20. * Coates, T. J., & Thoresen, C. E. (1978). Using generalizability theory in behavioral observation. Behavior Therapy, 9, 605-613.
21. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46.
22. * Conger, A. J., Conger, J. C., Wallander, J., Ward, D., & Dygdon, J. (1983). A generalizability study of the Conners' Teacher Rating Scale - Revised. Educational and Psychological Measurement, 43, 1019-1031.
23. * Conger, J. C., Moisan-Thomas, P. C., & Conger, A. J. (1989). Cross-situational generalizability of social competence: A multilevel analysis. Behavioral Assessment, 11, 411-431.
24. * Conger, R. D., Brainerd, D. W., Birch, L. L., Friedberg, P. J., & Navarro, L. A. (1986). Assessing the quality of family observations: A comparative analysis. Journal of Marriage and the Family, 48, 361-373.
25. Cooper, H., & Hedges, L. V. (1994). Research synthesis as a scientific enterprise. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 3-14). New York: Sage.
26. Cooper, W. H. (1981). Ubiquitous halo. Psychological Bulletin, 90, 218-244.
27. * Crocker, L., Llabre, M., & Miller, M. D. (1988). The generalizability of content validity ratings. Journal of Educational Measurement, 25, 287-299.
28. Cronbach, L. J. (1995). Giving method variance its due. In P. E. Shrout & S. T. Fiske (Eds.), Personality research, methods, and theory: A festschrift honoring Donald W. Fiske (pp. 145-157). Hillsdale, NJ: Erlbaum.
29. Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: Wiley.
30. Cronbach, L. J., Lynn, R. L., Brennan, R. L., & Haertel, E.H. (1997). Generalizability analysis for performance assessments of student achievement or school effectiveness. Educational and Psychological Measurement, 57, 373-399.
31. * Crooks, T. J., & Kane, M. T. (1981). The generalizability of student ratings of instructors: Item specificity and section effects. Research in Higher Education, 15, 305-313.
32. * Curran, J. P., Monti, P. M., Corriveau, D. P., Hay, L. R., Hagerman, S., Zwick, W. R., & Farrell, A. D. (1980). The generalizability of a procedure for assessing social skills and social anxiety in a psychiatric population. Behavioral Assessment, 2, 389-401.
33. * Doverspike, D., Carlisi, A. M., Barrett, G. V., & Alexander, R. A. (1983). Generalizability analysis of a point-method job evaluation instrument. Journal of Applied Psychology, 68, 476-483.
34. Epstein, S. (1983). Aggregation and beyond: Some issues on the aggregation of behavior. Journal of Personality, 51, 360-392.
35. * Erlich, O., & Borich, G. (1978). Factorial analysis of generalizability. Educational and Psychological Measurement, 38, 125-133.
36. * Farrell, A. D., Curran, J. P., Zwick, W. R., & Monti, P. M. (1983). Generalizability and discriminant validity of anxiety and social skills ratings in two populations. Behavioral Assessment, 6, 1-14.
37. * Farrell, A.D., Mariotto, M. J., & Conger, A. J. (1979). Self-ratings and judges' ratings of heterosexual social anxiety and skill: A generalizability study. Journal of Consulting and Clinical Psychology, 47, 164-175.
38. Fleiss, J. L. (1981). Balanced incomplete block designs for inter-rater reliability studies. Applied Psychological Measurement, 5, 105-112.
39. Ford, A. (1931). Neutralizing inequalities in ratings. Personnel Journal, 9, 466-469.
40. * Fraser, S. L., Cronshaw, S. F., & Alexander, R. A. (1984). Generalizability analysis of a point-method job evaluation instrument: A field study. Journal of Applied Psychology, 69, 643-647.
41. Funder, D. C. (1987). Errors and mistakes: Evaluating the accuracy of social judgment. Psychological Bulletin, 101, 75-90.
42. * Gillmore, G. M., Kane, M. T., & Naccarato, R. W. (1978). The generalizability of student ratings of instruction: Estimation of the teacher and course components. Journal of Educational Measurement, 15, 1-13.
43. * Goodwin, L. D., Sands, D. J., & Kozleski, E. B. (1991). Estimating interviewer reliability for interview schedules used in special education research. The Journal of Special Education, 25, 73-89.
44. * Green, R. S., & Jerrell, J. M. (1994). The generalizability of brief ratings of psychosocial functioning. Evaluation and Program Planning, 17, 141-151.
45. Guilford, J. P. (1954). Psychometric methods (2nd ed.) New York: McGraw-Hill.
46. Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.
47. Hill, C. E., Nutt, E. A., & Jackson, S. (1994). Trends in psychotherapy process research: Samples, measures, researchers, and classic publications. Journal of Counseling Psychology, 41, 364-377.
48. * Hill, C. E., O'Grady, K. E., & Price, P. (1988). A method for investigating sources of rater bias. Journal of Counseling Psychology, 35, 346-350.
49. Hoyt, W. T. (1994). Development and awareness of reputations in newly formed groups: A social relations analysis. Personality and Social Psychology Bulletin, 20, 464-472.
50. Hoyt, W. T. (in press). Rater bias in psychological research: When is it a problem and what can we do about it? Psychological Methods.
51. Hoyt, W. T., & Melby, J. N. (1999). Dependability of measurement in counseling: An introduction to generalizability theory. The Counseling Psychologist, 27, 325-352.
52. * Huysamen, G. K. (1990). The application of generalizability theory to the reliability of ratings. South African Journal of Psychology, 20, 200-205.
53. * Kane, M. T., Gillmore, G. M., & Crooks, T. J. (1976). Student evaluations of teaching: The generalizability of class means. Journal of Educational Measurement, 13, 171-183.
54. Kavanagh, M. J., MacKinney, A. C., & Wolins, L. (1971). Issues in managerial performance: Multitrait-multi-method analyses of ratings. Psychological Bulletin, 75, 34-49.
55. Kenny, D. A. (1991). A general model of consensus and accuracy in interpersonal perception. Psychological Review, 98, 155-163.
56. Kenny, D. A. (1994). Interpersonal perception: A social relations analysis. New York: Guilford.
57. Kenny, D. A., & LaVoie, L. (1984). The Social Relations Model. In L. Berkowitz (Ed.), Advances in experimental social psychology (Vol. 18). Orlando, FL: Academic Press.
58. * Lakey, B., McCabe, K. M., Fisicaro, S. A., & Drew, J. B. (1996). Environmental and personal determinants of support perceptions: Three generalizability studies. Journal of Personality and Social Psychology, 70, 1270-1280.
59. Landy, F. J., & Farr, J. L. (1980). Performance rating. Psychological Bulletin, 87, 72-107.
60. * Lane, S., Liu, M., Ankenmann, R. D., & Stone, C. A. (1996). Generalizability and validity of a mathematics performance assessment. Journal of Educational Measurement, 33, 71-92.
61. * Lavigueur, S., Tremblay, R. E., & Saucier, J. F. (1993). Can spouse support be accurately and reliably rated? A generalizability study of families with disruptive boys. Journal of Child Psychology and Psychiatry and Allied Disciplines, 34, 689-714.
62. * Levy, P. (1974). Generalizability studies in clinical settings. British Journal of Social and Clinical Psychology, 13, 161-172.
63. * Lomax, R. G. (1982). An application of generalizability theory to observational research. Journal of Experimental Education, 51, 22-30.
64. Lorenz, P.O., & Melby, J. N. (1994). Family stress and adaptation: Methods of study. In R. D. Conger & G. H. Elder, Jr. (Eds.), Families in troubled times: Adapting to change in rural America (pp. 21-54). New York: Aldine.
65. * Marcoulides, G. A. (1989). Performance appraisal: Issues of validity. Performance Improvement Quarterly, 2(2), 3-12.
66. * McWilliam, R. A., & Ware, W. B. (1994). The reliability of observations of young children's engagement: An application of generalizability theory. Journal of Early Intervention, 18, 34-47.
67. * Novak, J. R., Herman, J. L., & Gearhart, M. (1996). Establishing validity for performance-based assessments: An illustration for collections of student writing. Journal of Educational Research, 89, 220-233.
68. * Nussbaum, A. (1984). Multivariate generalizability theory in educational measurement: An empirical study. Applied Psychological Measurement, 8, 219-230.
69. Ozer, D. J. (1989). Construct validity in personality assessment. In D. M. Buss & N. Cantor (Eds.), Personality psychology: Recent trends and emerging directions (pp. 224-234). New York: Springer Verlag.
70. * Raymond, M. R., & Viswesvaran, C. (1993). Least squares models to correct for rater effects in performance assessment. Journal of Educational Measurement, 30, 253-268.
71. Raymond, M. R., Webb, L. C., & Houston, W. M. (1991). Correcting performance-rating errors in oral examinations. Evaluation and the Health Professions, 14, 100-122.
72. Rosenberg, M. S., Adams, D. C., & Gurevitch, J. (1997). MetaWin: Statistical software for meta-analysis with resampling tests. Sunderland, MA: Sinauer.
73. Rosenthal, R. (1979). The "file drawer problem" and tolerance for null results. Psychological Bulletin, 86, 638-641.
74. Saal, F. E., Downey, R. G., & Lahey, M. A. (1980). Rating the ratings: Assessing the psychometric quality of rating data. Psychological Bulletin, 88, 413-428.
75. Schmidt, F. L. (1996). What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist, 47, 1173-1181.
76. * Shavelson, R. J., Baxter, G. P., & Gao, X. (1993). Sampling variability of performance assessments. Journal of Educational Measurement, 30, 215-232.
77. * Shavelson, R. J., Mayberry, P. W., Li, W., & Webb, N. M. (1990). Generalizability of job performance measurements: Marine Corps rifleman. Military Psychology, 2, 129-144.
78. Shavelson, R. J., & Webb, N.M. (1991). Generalizability theory: A primer. Newbury Park, CA: Sage.
79. Shrout, P. E. (1995). Measuring the degree of consensus in personality judgments. In P. E. Shrout & S. T. Fiske (Eds.), Personality research, methods, and theory: A. festschrift honoring Donald W. Fiske (pp. 79-92). Hillsdale, NJ: Erlbaum.
80. Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420-428.
81. Smith, P. L. (1978). Sampling errors of variance components in small sample multifacet generalizability studies. Journal of Educational Statistics, 3, 319-346.
82. * Smith, P. L. (1979). The generalizability of student ratings of courses: Asking the right questions. Journal of Educational Measurement, 16, 77-87.
83. Thorndike, E. L. (1920). A constant error in psychological ratings. Journal of Applied Psychology, 4, 25-29.
84. * Tracey, T. J., &Guinee, J. P. (1990). Generalizability of Interpersonal Communication Rating Scale ratings across presentation modes. Journal of Counseling Psychology, 37, 330-336.
85. * Ulrich, D. A., Riggen, K. J., Ozmun, J. C., Screws, D. P., & Cleland, F. E. (1989). Assessing movement control in children with mental retardation: A generalizability analysis of observers. American Journal on Mental Retardation, 94, 170-176.
86. * Webb, N. M., Shavelson, R. J., Kim, K. S., & Chen, Z. (1989). Reliability (generalizability) of job performance measurements: Navy machinist mates. Military Psychology, 1, 91-110.
87. * Webb, N. M., Shavelson, R. J., Shea, J., & Morello, E. (1981). Generalizability of general education development ratings of jobs in the United States. Journal of Applied Psychology, 66, 186-192.
88. West, S. G., Newsom, J., & Fenaughty, A. (1991). Publication trends in JPSP: Stability and change in topics, methods, and theories across two decades. Personality and Social Psychology Bulletin, 18, 473-484.
89. Wherry, R. J. (1952). The control of bias in rating: A theory of rating (PRB Report No. 922, Contract No. DA-49-083 OSA 69). Washington, DC: U.S. Department of the Army.
90. Wherry, R. J., & Bartlett, C. J. (1982). The control of bias in ratings: A theory of rating. Personnel Psychology, 35, 521-551.
91. * Wieder, G. B., & Weiss, R. L. (1980). Generalizability theory and the coding of marital interactions. Journal of Consulting and Clinical Psychology, 48, 469-477.