Building a better mouse trap: Item characteristics associated with rating discrepancies in 360-degree feedback

Consulting Psychology Journal

Volumn 57, Issue 4, 2005, Pages 235-245

  Kaiser, Robert B ^a   Craig, S Bartholomew ^a,b  

^a Kaplan DeVries Inc   (United States)

^b NORTH CAROLINA STATE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 33745164579     PISSN: 10659293     EISSN: None     Source Type: Journal    
DOI: 10.1037/1065-9293.57.4.235     Document Type: Review

References (31)

1. Bartko, J. J. (1976). On various intraclass correlation reliability coefficients. Psychological Bulletin, 83, 762-765.
2. Borman, W. C. (1997). 360-degree ratings: An analysis of assumptions and a research agenda for evaluating their validity. Human Resource Management Review, 7, 299-315.
3. Brutus, S., & Facteau, J. D. (2003). Short, simple, and specific: The influence of item design characteristics in multi-source assessment contexts. International Journal of Selection and Assessment, 11, 313-325.
4. Chomsky, N. (1965). Aspects of the theory of syntax. Cambridge, MA: MIT Press.
5. Chomsky, N. (1986). Knowledge of language: Its nature, origin, and use. New York: Praeger.
6. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). New Jersey: Erlbaum.
7. Cohen, J. (1994). The earth is round (p <.05). American Psychologist, 49, 997-1003.
8. Cohen, J., & Cohen, P. (1983). Applied multiple regression/ correlation analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
9. Conway, J., & Huffcutt, A. (1997). Psychometric properties of multi-source performance ratings: A meta-analysis of subordinate, supervisor, peer, and self-ratings. Human Performance, 10, 331-360.
10. Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Orlando, FL: Harcourt Brace Jovanovich.
11. DeNisi, A. S. (1996). A cognitive approach to performance appraisal. London: Routledge.
12. Fleenor, J. W., Fleenor, J. B., & Grossnickle, W. F. (1996). Inter-rater reliability and agreement of performance ratings: A methodological comparison. Journal of Business and Psychology, 10, 367-380.
13. Funder, D. C., & Sneecl, C. D. (1993). Behavioral manifestations of personality: An ecological approach to judgmental accuracy. Journal of Personality and Social Psychology, 64, 479-490.
14. Greguras, G. J., & Robie, C. (1998). A new look at within-source inter-rater reliability of 360-degree feedback ratings. Journal of Applied Psychology, 83, 960-968.
15. James, L. R., Demaree, R. G., & Wolf, G. (1984). Estimating within-group inler-rater reliability with and without response bias. Journal of Applied Psychology, 69, 85-98.
16. WG: An assessment of wilhin-group inter-rater agreement. Journal of Applied Psychology, 78, 306-309.
17. th annual Conference of the Society for Industrial and Organizational Psychology, Toronto, Ontario, Canada.
18. Kozlowski, S. W. J., & Hattrup, K. (1992). A disagreement about, within-group agreement: Disentangling issues of consistency versus consensus. Journal of Applied Psychology, 77, 161-167.
19. LeBreton, J. M., Burgess, J. R. D., Kaiser, R. B., Atchley, K. K., & James, L. R. (2003). The restriction of variance hypothesis and inter-rater reliability and agreement: Are ratings from multiple sources really dissimilar? Organizational Research Methods, 6, 78-126.
20. th annual Conference of the Society for Industrial and Organizational Psychology, Chicago, IL.
21. Lombardo, M., & McCauley, C. (1994). Benchmarks: A manual and trainer's guide. Greensboro, NC: Center for Creative Leadership.
22. McCauley, C. D., & Lombardo, M. (1990). Benchmarks: An instrument for diagnosing managerial strengths and weaknesses. In K. K. Clark & M. B. Clark (Hds.), Measures of leadership (pp. 535-546). West Orange, NJ: Leadership Library of America.
23. Murphy, K. R., & Cleveland, J. N. (1995). Understanding performance appraisal: Social, organizational, and goal based perspectives. Thousand Oaks. CA: Sage.
24. Schwartz, N. (1999). Self reports: How the questions shape the answers. American Psychologist, 54, 93-105.
25. Scullen, S. E., Mount, M. K., & Goff, M. (2000). Understanding the latent, structure of job performance ratings. Journal of Applied Psychology, 85, 956-970.
26. Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420-428.
27. Sloan, E. B. (1994). Assessing and developing versatility: Executive survival skill for the brave new world. Consulting Psychology Journal: Practice and Research, 46, 24-31.
28. Sudman, S., & Bradburn, N. N. (1982). Asking questions: A practical guide to questionnaire design. San Francisco: Jossey-Bass.
29. Sudman, S., Bradburn, N. N., & Schwartz, N. (1996). Thinking about answers: The application of cognitive processes to survey methodology. San Francisco: Jossey-Bass.
30. Tsui, A. S., & Ohlott, P. (1988). Multiple assessment of managerial effectiveness: Interrater agreement and consensus in effectiveness models. Personnel Psychology, 41, 779-803.
31. Zedeck, S. (1995). Review of Benchmarks. In J. Conoley & J. Impara (Eds.), The twelfth mental measurements yearbook (Vol. 1, pp. 128-129). Lincoln, NE: Buros Institute of Mental Measurements.