Rater source effects are alive and well after all

Personnel Psychology

Volumn 63, Issue 1, 2010, Pages 119-151

  Hoffman, Brian ^a   Lance, Charles E ^b   Bynum, Bethany ^b   Gentry, William A ^c  

^a The University of Georgia ^*  (United States)

^b UNIVERSITY OF GEORGIA   (United States)

^c Center for Creative Leadership   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 77649326574     PISSN: 00315826     EISSN: 17446570     Source Type: Journal    
DOI: 10.1111/j.1744-6570.2009.01164.x     Document Type: Article

References (85)

1. Arthur W, Villado AJ. The importance of distinguishing between constructs and methods when comparing predictors in personnel selection research and practice. Journal of Applied Psychology 2008, 93:435-442.
2. Austin JT, Villanova P. The criterion problem: 1917-1992. Journal of Applied Psychology 1992, 77:836-874.
3. Bentler PM. Comparative fit indexes in structural models. Psychological Bulletin 1990, 107:238-246.
4. Bentler PM, Bonett DG. Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin 1980, 88:588-606.
5. Bentler PM, Woodward JA. The greatest lower bound to reliability. Principals of modern psychological measurement: A Festschrift to Frederic M. Lord 1983, 237-253. Wainer H, Messick S. In, Eds., Hillsdale, NJ, Erlbaum
6. Borman WC. The rating of individuals in organizations: An alternative approach. Organizational Behavior and Human Performance 1974, 12:105-124.
7. Borman WC. 360 ratings: An analysis of assumptions and research agenda for evaluating their validity. Human Resource Management Review 1997, 7:299-315.
8. Borman WC, Brush DH. More progress toward a taxonomy of managerial performance requirements. Human Performance 1993, 6:1-21.
9. Bowler MC, Woehr DJ. A meta-analytic evaluation of the impact of dimension and exercise factors on assessment center ratings. Journal of Applied Psychology 2006, 91:1114-1124.
10. Browne MW, Cudeck R. Alternative ways of assessing model fit. Testing structural models 1993, 136-162. Bollen K, Long JS. In, Eds., Newbury Park, CA, Sage
11. Campbell DT, Fiske DW. Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin 1959, 56:81-105.
12. Campbell JP, McHenry JJ, Wise LL. Modeling job performance in a population of jobs. Personnel Psychology 1990, 43:313-333.
13. Carty HM. Review of BENCHMARKS®[revised]. The fifteenth mental measurements yearbook 2003, 123-124. Plake BSImpara JSpies RA. In, Eds., Lincoln, NE, Buros Institute of Mental Measurements
14. Cheung GW, Rensvold RB. Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling 2002, 9:233-255..
15. Church AH, Allen DW. Advancing the state of the art of 360-degree feedback. Group and Organization Management 1997, 22:149-161.
16. Conway JM. Analysis and design of multitrait-multirater performance appraisal studies. Journal of Management 1996, 22:139-162.
17. Conway JM. Distinguishing contextual performance from task performance for managerial jobs. Journal of Applied Psychology 1999, 84:3-13.
18. Conway JM, Huffcutt AI. Psychometric properties of multisource performance ratings: A meta-analysis of subordinate, supervisor, peer, and self-ratings. Human Performance 1997, 10:331-360.
19. Conway JM, Lievens F, Scullen SE, Lance CE. Bias in the correlated uniqueness model for MTMM data. Structural Equation Modeling 2004, 11:535-559.
20. Cooper WH. Ubiquitous halo. Psychological Bulletin 1981, 90:218-244.
21. Coovert MD, Craiger JP, Teachout MS. Effectiveness of the direct product versus confirmatory factor model for reflecting the structure of multimethod-multirater job performance data. Journal of Applied Psychology 1997, 82:271-280.
22. Dansereau F, Graen G, Haga W. A vertical dyad linkage approach to leadership within formal organizations: A longitudinal investigation of the role making process. Organizational Behavior and Human Performance 1975, 13:46-78.
23. DeVries DL, Morrison AM, Shullman SL, Gerlach ML. Performance appraisal on the line 1986, Greensboro, NC, Center for Creative Leadership
24. Feldman JM. Beyond attribution theory: Cognitive processes in performance appraisal. Journal of Applied Psychology 1981, 66:127-148.
25. Fleenor JW, McCauley CD, Brutus S. Self-other rating agreement and leader effectiveness. Leadership Quarterly 1996, 7:487-506.
26. Guilford JP. Psychometric methods 1954, 2nd ed., New York, McGraw-Hill
27. Harris MM, Schaubroeck J. A meta-analysis of self-supervisor, self-peer, and peer-supervisor ratings. Personnel Psychology 1988, 41:43-62.
28. Hoffman BJ, Blair C, Meriac J, Woehr DJ. Expanding the criterion domain? A meta-analysis of the OCB literature. Journal of Applied Psychology 2007, 92:555-566.
29. Hoffman BJ, Woehr DJ. Disentangling the meaning of multisource feedback source and dimension factors. Personnel Psychology 2009, 62:735-765.
30. Holzbach RL. Rater bias in performance ratings: Superior, self, and peer ratings. Journal of Applied Psychology 1978, 63:579-588.
31. Hu LT, Bentler PM. Fit indices in covariance structure modeling: Sensitivity to underparameterization model misspecification. Psychological Methods 1998, 3:424-453.
32. Hu LT, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis. Structural Equation Modeling 1999, 6:1-55.
33. Jöreskog K, Sörbom D. LISREL 8.70 2004, Chicago, Scientific Software International Inc
34. Kenny DA, Berman JS. Statistical approaches to the correction of correlational bias. Psychological Bulletin 1980, 88:288-295.
35. King LM, Hunter JE, Schmidt FL. Halo in a multidimensional forced-choice performance evaluation scale. Journal of Applied Psychology 1980, 65:507-516.
36. Kingstrom PO, Bass AR. A critical analysis of studies comparing behaviorally anchored rating scale (BARS) and other rating formats. Personnel Psychology 1981, 34:263-289.
37. Klimoski RJ, London M. Role of the rater in performance appraisal. Journal of Applied Psychology 1974, 59:445-451.
38. Lakey CE, Goodie AS, Lance CE, Stinchfield S, Winters KC. Examining DSM-IV criteria for pathological gambling: Psychometric properties and evidence from cognitive biases. Journal of Gambling Studies 2007, 23:479-498.
39. Lance CE, Baxter D, Mahan RP. Multi-source performance measurement: A reconceptualization. Performance measurement: Current perspectives and future challenges 2006, 49-76. Bennett W, Lance CE, Woehr DJ. In, Eds., Mahwah, NJ, Erlbaum
40. Lance CE, Hoffman BJ, Gentry WA, Baranik LE. Rater source factors represent important subcomponents of the criterion construct space, not rater bias. Human Resource Management Review, 18, 223-232 2008
41. Lance CE, Noble CL, Scullen SE. A critique of the correlated trait-correlated method (CTCM) and correlated uniqueness (CU) models for multitrait-multimethod (MTMM) data. Psychological Methods 2002, 7:228-244.
42. Lance CE, Teachout MS, Donnelly TM. Specification of the criterion construct space: An application of hierarchical confirmatory factor analysis. Journal of Applied Psychology 1992, 77:437-452.
43. Lance CE, Woehr DJ. Statistical control of halo: Clarification from two cognitive models of the performance appraisal process. Journal of Applied Psychology 1986, 71:679-687.
44. Lance CE, Woehr DJ, Fisicaro SA. Cognitive categorization processes in performance evaluation: Confirmatory tests of two models. Journal of Organizational Behavior 1991, 12:1-20.
45. Lance CE, Woehr DJ, Meade AW. Case study: A Monte Carlo investigation of assessment center construct validity models. Organizational Research Methods 2007, 10:430-448.
46. Landy FJ, Farr JL. Performance rating. Psychological Bulletin 1980, 87:72-107.
47. Lawler EE. The multitrait-multirater approach to measuring managerial job performance. Journal of Applied Psychology 1967, 51:369-381.
48. LeBreton JM, Burgess JD, Kaiser RB, Atchley EK, James LR. The restriction of variance hypothesis and interrater reliability and agreement: Are ratings from multiple sources really dissimilar?. Organizational Research Methods 2003, 6:80-128.
49. Li H, Rosenthal R, Rubin DB. Reliability of measurement in psychology: From Spearman-Brown to maximal reliability. Psychological Methods 1996, 1:98-107.
50. Lombardo MM, McCauley CD. BENCHMARKS®: A manual and trainer's guide 1994, Greensboro, NC, Center for Creative Leadership
51. Lombardo MM, McCauley CD, McDonald-Mann D, Leslie JB. BENCHMARKS® developmental reference points 1999, Greensboro, NC, Center for Creative Leadership
52. Mabe PA, West SG. Validity of self-evaluation of ability: A review and meta-analysis. Journal of Applied Psychology 1982, 67:280-296.
53. Mann FC. Toward an understanding of the leadership role in formal organizations. Leadership and productivity 1965, 68-77. Dubin RHomans GCMann FCMiller DC. In, Eds., San Francisco, Chandler
54. McCauley C, Lombardo M. BENCHMARKS®: An instrument for diagnosing managerial strengths and weaknesses. Measures of leadership 1990, 535-545. Clark KE, Clark MB. In, Eds., West Orange, NJ, Leadership Library of America
55. McCauley C, Lombardo M, Usher C. Diagnosing management development needs: An instrument based on how managers develop. Journal of Management 1989, 15:389-403.
56. Mintzberg H. The manager's job: Folklore and fact. Harvard Business Review 1975, 53:49-61.
57. Mount MK, Judge TA, Scullen SE, Sytsma MR, Hezlett SA. Trait, rater, and level effects in 360-degree performance ratings. Personnel Psychology 1998, 51:557-576.
58. Murphy KR. Three models of the performance appraisal process. Industrial and Organizational Psychology Perspectives on Research and Practice 2008
59. Murphy KR, DeShon R. Interrater correlations do not estimate the reliability of job performance ratings. Personnel Psychology 2000, 53:873-900.
60. Newcomb T. An experiment designed to test the validity of a rating technique. Journal of Educational Psychology 1931, 22:279-289..
61. Saal FE, Downey RG, Lahey MA. Rating the ratings: Assessing the psychometric quality of rating data. Psychological Bulletin 1980, 88:413-428.
62. Schmidt FL, Viswesvaran C, Ones DS. Reliability is not validity and validity is not reliability. Personnel Psychology 2000, 53:901-912.
63. Scullen SE, Mount MK, Goff M. Understanding the latent structure of job performance ratings. Journal of Applied Psychology 2000, 85:956-970.
64. Sevy BA, Olson RD, McGuire DP, Frazier ME, Paajanen G. Managerial skills profile technical manual 1985, Minneapolis, MN, Personnel Decisions, Inc
65. Smith CA, Organ DW, Near JP. Organizational citizenship behavior: Its nature and antecedents. Journal of Applied Psychology 1983, 68:653-663.
66. Spangler M. Review of BENCHMARKS®[revised]. The fifteenth mental measurements yearbook 2003, 124-126. Plake BSImpara JSpies RA. In, Eds., Lincoln, NE, Buros Institute of Mental Measurements
67. Sparrowe RT, Liden RC. Process and structure in leader-member exchange. Academy of Management Review 1997, 22:522-552.
68. Steiger JH. Structural model evaluation and modification: An interval estimation approach. Multivariate Behavioral Research 1990, 25:173-180.
69. Thorndike EL. A constant error in psychological ratings. Journal of Applied Psychology 1920, 4:25-29.
70. Thornton G. The relationship between supervisory and self appraisals of executive performance. Personnel Psychology 1980, 21:441-455.
71. Timmreck CW, Bracken DW. Multisource feedback: A study of its use in decision-making. Employment Relations Today 1997, 24:21-27.
72. Tucker LR, Lewis C. A reliability coefficient for maximum likelihood factor analysis. Psychometrika 1973, 38:1-10.
73. Vance RJ, MacCallum RC, Coovert MD, Hedge JW. Construct validity of multiple job performance measures using confirmatory factor analysis. Journal of Applied Psychology 1988, 73:74-80.
74. Viswesvaran C, Ones DS, Schmidt FL. Comparative analysis of the reliability of job performance ratings. Journal of Applied Psychology 1996, 81:557-574.
75. Viswesvaran C, Schmidt FL, Ones DS. The moderating influence of job performance dimension on convergence of supervisory and peer ratings of job performance: Unconfounding construct-level convergence and rating difficulty. Journal of Applied Psychology 2002, 87:245-354.
76. Viswesvaran C, Schmidt FL, Ones DS. Is there a general factor in ratings of job performance? A meta-analytic framework for disentangling substantive and error influences. Journal of Applied Psychology 2005, 90:108-131..
77. Wells FL. A statistical study of literary merit. (Columbia Univ. Cont. to Phil. & Psych., 16, 3.). Archives of Psychology 1907, 7:5-25.
78. Wherry RJ. The control of bias in ratings: VIII. A theory of rating 1952, PRB report No. 922, Contract No. DA-49-083 OSA69, Department of the Army, Columbus, OH, Ohio State University Research Foundation
79. Wherry RJ, Bartlett CJ. The control of bias in ratings: A theory of rating. Personnel Psychology 1982, 35:521-551.
80. Wildaman KF. Hierarchically nested covariance structure models for multitrait-multimethod data. Applied Psychological Measurement 1985, 9:1-26.
81. Woehr DJ, Huffcutt AI. Rater training for performance appraisal: A quantitative review. Journal of Occupational and Organizational Psychology 1994, 67:189-206.
82. Woehr DJ, Sheehan MK, Bennett W. Assessing measurement equivalence across ratings sources: A multitrait-multirater approach. Journal of Applied Psychology 2005, 90:592-600.
83. Yukl GA. Leadership in organizations 2006, 6th ed., Englewood Cliffs, NJ, Prentice-Hall
84. Zedeck S. Review of BENCHMARKS®. The twelfth mental measurements yearbook 1995, 1:128-129. Conoley JImpara J. In, Eds., Vol, Lincoln, NE, Buros Institute of Mental Measurements
85. Zedeck S, Baker HT. Nursing performance as measured by behavioral expectation sales: A multitrait-multirater analysis. Organizational Behavior and Human Performance 1972, 7:457-466.