A Meta-Analysis of Incremental Validity and Nomological Networks for Subordinate and Peer Ratings

Human Performance

Volumn 14, Issue 4, 2001, Pages 267-303

  Conway, James M ^a   Lombardo, Kristie ^b   Sanders, Kelley C ^b  

^a CENTRAL CONNECTICUT STATE UNIVERSITY   (United States)

^b NONE

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0035605317     PISSN: 08959285     EISSN: None     Source Type: Journal    
DOI: 10.1207/S15327043HUP1404_1     Document Type: Article

References (127)

1. Agle, B. R. (1993). Charismatic chief executive officers: Are they more effective? An empirical test of charismatic leadership theory. (Doctoral dissertation, University of Washington, 1993). Dissertation Abstracts International, 55, 317A.
2. Alker, H. A., Straub, W. F., & Leary, J. (1973). Achieving consistency: A study of basketball officiating. Journal of Vocational Behavior, 3, 335-343.
3. Atwater, L. E., & Yammarino, F. J. (1992). Does self-other agreement on leadership perceptions moderate the validity of leadership and performance predictions? Personnel Psychology, 45, 141-164.
4. Atwater, L. E., & Yammarino, F. J. (1993). Personal attributes as predictors of superiors' and subordinates' perceptions of military academy leadership. Human Relations, 46, 645-668.
5. Avolio, B. J., & Howell, J. M. (1992). The impact of leadership behavior and leader-follower personality match on satisfaction and unit performance. In K. E. Clark, M. B. Clark, & D. P. Campbell (Eds.), Impact of leadership (pp. 225-235). Greensboro, NC: Center for Creative Leadership.
6. Barrick, M. R., & Mount, M. K. (1991). The Big Five personality dimensions and job performance: A meta-analysis. Personnel Psychology. 44, 1-26.
7. Bass, B. M. (1985). Leadership and performance beyond expectations. New York: Free Press.
8. Bass, B. M., & Avolio, B. J. (1997). Full range leadership development: Manual for the Multifactor Leadership Questionnaire. Redwood City, CA: Mind Garden.
9. Bobko, P., Roth, P. L., & Potosky, D. (1999). Derivation and implications of a meta-analytic matrix incorporating cognitive ability, alternative predictors, and job performance. Personnel Psychology, 52, 561-589.
10. Bommer, W. H., Johnson, J. L., Rich, G. A., Podsakoff, P. M., & MacKenzie, S. B. (1995). On the interchangeability of objective and subjective measures of employee performance: A meta-analysis. Personnel Psychology, 48, 587-605.
11. Borg, W. R. (1967). Teacher effectiveness in team teaching. The Journal of Experimental Education, 35, 65-70.
12. Borman, W. C. (1974). The rating of individuals in organizations: An alternate approach. Organizational Behavior and Human Performance, 12, 105-124.
13. Borman, W. C. (1997). 360° ratings: An analysis of assumptions and a research agenda for evaluating their validity. Human Resource Management Review, 7, 299-315.
14. Borman, W. C., & Motowidlo, S. J. ( 1993). Expanding the criterion domain to include elements of contextual performance. In N. Schmitt & W. C. Borman (Eds.), Personnel selection in organizations (pp. 71-98). San Francisco: Jossey-Bass.
15. Borman, W. C., White, L. A., & Dorsey, D. W. (1995). Effects of ratee task performance and interpersonal factors on supervisor and peer performance ratings. Journal of Applied Psychology, 80, 168-177.
16. Bozeman, D. P. (1997). Interrater agreement in multi-source performance appraisal: A commentary. Journal of Organizational Behavior, 18, 313-316.
17. Brooks, J. H., II, & Stuhlmacher, A. (1999, May). Personal characteristics and rater congruence in multi-rater feedback. Paper presented at the 14th annual conference of the Society for Industrial and Organizational Psychology, Atlanta, GA.
18. Carter, G. C. (1952). Measurement of supervisory ability. Journal of Applied Psychology, 36, 393-395.
19. Center for Creative Leadership. (1996). [360-degree feedback ratings]. Unpublished raw data.
20. Center for Creative Leadership, (undated). Correlations between CPP instruments and benchmarks: Section I: MBT1 with benchmarks. Greensboro, NC: Author.
21. Church, A. H., & Bracken, D. W. (1997). Advancing the state of the art of 360-degree feedback: Guest editors' comments on the research and practice of multirater assessment methods. Group & Organization Management, 22, 149-161.
22. Church, A. H., & Waclawski, J. (1998). The relationship between individual personality orientation and executive leadership behavior. Journal of Occupational and Organizational Psychology, 71, 99-125.
23. Conway, J. M. (1999). Distinguishing contextual performance from task performance for managerial jobs. Journal of Applied Psychology, 84, 3-13.
24. Conway, J. M. (2000). Managerial performance development constructs and personality correlates. Human Performance, 13, 23-46.
25. Conway, J. M., & Huffcutt, A. I. (1997). Psychometric properties of multi-source performance ratings: A meta-analysis of subordinate, supervisor, peer, and self-ratings. Human Performance, 10, 331-360.
26. Cortina, J. M., Doherty, M. L., Schmitt, N., Kaufman, G., & Smith, R. G. (1992). The "big five" personality factors in the IPI and MMPI: Predictors of police performance. Personnel Psychology, 45, 119-140.
27. Cowen, S. S. (1990). A study of the relationships between perceived leadership behaviors of presidents at public four-year institutions of higher education in the U.S. and the changes in FTE enrollment, perceptions of effectiveness, subordinate satisfaction, and other factors of the presidency (Doctoral dissertation, Gonzaga University, 1990). Dissertation Abstracts International, 53, 78A.
28. Cronbach, L. J., & Meehl, P. E. (1995). Construct validity in psychological tests. Psychologist Bulletin, 52, 281-302.
29. Curphy, G. J. (1991). An empirical evaluation of Bass' (1985) theory of transformational and transactional leadership (Doctoral dissertation, University of Minnesota, 1991). Dissertation Abstracts International, 52, 2804B.
30. Dicken, C. (1969). Predicting the success of Peace Corps community development workers. Journal of Consulting and Clinical Psychology, 33, 597-606.
31. Edgerton, H. A., Feinberg, M. R., & Thomson, K. F. (1957). Prediction of the "human relations" effectiveness of industrial supervisors. Personnel Psychology, 10, 421-431.
32. Feist, G. J. (1993). A structural model of scientific eminence. Psychological Science, 4, 366-371.
33. Ferris, G. R., & Judge, T. A. (1991). Personnel/human resources management: Apolitical influence perspective. Journal of Management, 17, 447-488.
34. Fiedler, F. E., & Leister, A. F. (1977). Leader intelligence and task performance: A test of a multiple screen model. Organizational Behavior and Human Performance, 20, 1-14.
35. Fleenor, J. W. (1991, June). Personality measures and managerial effectiveness: A failure to find expected relationships. Paper presented at the 3rd annual conference of the American Psychological Society, Washington, DC.
36. Fleenor, J. W., & Brutus, S. (2001). Multisource feedback forpersonnel decisions. In D. W. Bracken, C. W. Timmreck, & A. H. Church (Eds.), The handbook of multisource feedback (pp. 335-351). San Francisco: Jossey-Bass.
37. Fleishman, E. A. (1953). The description of supervisory behavior. Journal of Applied Psychology, 37, 1-6.
38. Fleishman, E. A. (1998). Consideration and structure: Another look at their role in leadership research (pp. 51-60). In F. Dansereau & F. J. Yammarino (Eds.), Leadership: The multiple-level approaches: Classical and new wave. Stamford, CT: JAI.
39. Flom, P. K. (1970). Performance in the medical internship (Doctoral dissertation. University of California, Berkeley, 1970). Dissertation Abstracts International, 32, 1188B.
40. Forsythe, G. B., McGaghie, W. C., & Friedman, C. P. (1986). Construct validity of medical clinical competence measures: A multitrait-multimethod matrix study using confirmatory factor analysis. American Educational Research Journal, 23, 315-336.
41. Fox, S., Haboucha, S., & Dinur, Y. (1981). The predictive validity of the Sixteen Personality Factors questionnaire relative to three independent criterion measures of military performance. Educational and Psychological Measurement, 41, 515-521.
42. Funderburg, S. A., & Levy, P. E. (1997). The influence of individual and contextual variables on 360-degree feedback system attitudes. Group & Organization Management, 22, 210-235.
43. Geyer, A. L. J., & Steyrer, J. M. (1998). Transformational leadership and objective performance in banks. Applied Psychology: An International Review, 47, 397-420.
44. Godfrey, B., & Timmerman, T. A. (1998, April). Employee and supervisor personality effects on employee performance ratings. Paper presented at the 13th annual conference of the Society for Industrial and Organizational Psychology, Dallas, TX.
45. Gordon, T. R., & Brannick, M. T. (1999, April). Sailing crews and measures of team process. Paper presented at the 14th annual conference of the Society for Industrial and Organizational Psychology, Atlanta, GA.
46. Gough, H. G. (1990). Testing for leadership with the California Psychological Inventory. In K. E. Clark & M. B. Clark (Eds.), Measures of leadership (pp. 355-379). West Orange, NJ: Leadership Library of America, Inc.
47. Harris, M. M., & Schaubroeck, J. (1988). A meta-analysis of self-supervisor, self-peer, and peer-supervisor ratings. Personnel Psychology, 41, 43-62.
48. Heneman, R. L. (1986). The relationship between supervisory ratings and results-oriented measures of performance: A meta-analysis. Personnel Psychology, 39, 811-826.
49. Hicks, J. A., & Stone, J. B. (1962). The identification of traits related to managerial success. Journal of Applied Psychology. 46, 428-432.
50. Hogan, J., Brinkmeyer, K., & Hogan, R. (1994). Technical manual for the Hogan Personality Inventory Management Development Report. Tulsa, OK: Hogan Assessment Systems, Inc.
51. Hogan, R., & Shelton, D. (1998). A socioanalytic perspective on job performance. Human Performance, 11, 129-144.
52. Hollander, E. P. (1954). Peer nominations on leadership as a predictor of the pass-fail criterion in naval air training. Journal of Applied Psychology, 38, 150-153.
53. Hough, L. M. (1992). The "Big Five" personality variables - construct confusion: Description versus prediction. Human Performance, 5, 139-155.
54. Hough, L. M. (1998, April). Job performance models & personality taxonomies. In J. C. Hogan (Chair), Meaning and models of the personality-job performance relation. Symposium conducted at the 13th annual conference of the Society for Industrial and Organizational Psychology, Dallas, TX.
55. Hough, L. M., Ones, D. S., & Viswesvaran, C. (1998, April). Personality correlates of managerial performance constructs. In R. C. Page (Chair), Personality determinants of managerial potential performance, progression and ascendancy. Symposium conducted at the 13th annual conference of the Society for Industrial and Organizational Psychology, Dallas, TX.
56. Hunter, J. E. (1996). REGRESS.BAS: A multiple regression program in BASICA Manual. Unpublished manuscript, Michigan State University.
57. Hunter, J. E., & Hunter, R. F. (1984). Validity and utility of alternative predictors of job performance. Psychological Bulletin, 96, 72-98.
58. Hunter, J. E., & Schmidt, F. L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.
59. Hunter, J. E., Schmidt, F. L., & Judiesch, M. K. (1990). Individual differences in output variability as a function of job complexity. Journal of Applied Psychology, 75, 28-42.
60. Jansen, D. G., Bonk, E. C., & Garvey, F. J. (1973). Relationships between Personal Orientation Inventory and Shipley-Hartford Scale scores and supervisor and peer ratings of counseling competency for clergymen in clinical training. Journal of Community Psychology, 1, 182-184.
61. Jauch, L. R., & Glueck, W. F. (1975). Evaluation of university professors' research performance. Management Science, 22, 66-75.
62. Johnson, J. W., Schneider, R. J., & Oswald, F. L. (1997). Toward a taxonomy of managerial performance profiles. Human Performance, 10, 227-250.
63. Kamp, J. D., & Hough. L. M. (1988). Utility of temperament for predicting job performance. In L. M. Hough (Ed.), Utility of temperament, biodata, and interest assessment for predicting job performance (pp. 1-90; Army Research Institute Research Note 88-02). Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.
64. Kane, J. S., & Lawler, E. E., III (1978). Methods of peer assessment. Psychological Bulletin, 85, 555-586.
65. Keller, R. T., & Holland, W. E. (1978). Individual characteristics of innovativeness and communication in research and development organizations. Journal of Applied Psychology, 63, 759-762.
66. Koene, B. A. S., Pennings, J. M., & Schreuder, H. (1992). Leadership, organizational culture, and organizational outcomes. In K. E. Clark, M. B. Clark, & D. P. Campbell (Eds.), Impact of leadership (pp. 215-223). Greensboro, NC: Center for Creative Leadership.
67. Koh, W. L. K. (1990). An empirical validation of the theory of transformational leadership in secondary schools in Singapore (Doctoral dissertation, University of Oregon, 1990). Dissertation Abstracts International, 52, 602A.
68. Kramer, W. (1999, April). Center for Leadership Studies [On-line]. Available: http://CLS.Binghamton.Edu
69. Kremer, J. F. (1990). Construct validity of multiple measures in teaching, research, and service and reliability of peer ratings. Journal of Educational Psychology, 82, 213-218.
70. Lawler, E. E., III, & Suttle, J. L. (1973). Expectancy theory and job behavior. Organizational Behavior and Human Performance, 9, 482-503.
71. Leslie, J. B., & Fleenor, J. W. (1998). Feedback to managers: A review and comparison of multi-rater instruments for management development (3rd ed.). Greensboro, NC: Center for Creative Leadership.
72. Levine, E. L., Spector, P. E., Menon, S., Narayanan, L., & Cannon-Bowers, J. ( 1996). Validity generalization for cognitive, psychomotor, and perceptual tests for craft jobs in the utility industry. Human Performance, 9, 1-22.
73. London, M. (2001). The great debate: Should multisource feedback be used for administration or development only? In D. W. Bracken, C. W. Timmreck, & A. H. Church (Eds.), The handbook of multisource feedback (pp. 368-385). San Francisco: Jossey-Bass.
74. London, M., & Smither, J. W. (1995). Can multi-source feedback change perceptions of goal accomplishment, self-evaluations, and performance-related outcomes? Theory-based applications and directions for research. Personnel Psychology, 48, 803-839.
75. Lowe, K. B, Kroeck, K. G., & Sivasubramaniam, N. (1996). Effectiveness correlates of transformational and transactional leadership: A meta-analytic review of the MLQ literature. Leadership Quarterly, 7, 385-425.
76. Mayfield, E. C. (1972). Value of peer nominations in predicting life insurance sales performance. Journal of Applied Psychology, 56, 319-323.
77. Mayo, G. D. (1956). Peer ratings and halo. Educational and Psychological Measurement, 16, 317-323.
78. McCloy, R. A., Campbell, J. P., & Cudeck, R. (1994). A confirmatory test of a model of performance determinants. Journal of Applied Psychology, 79, 493-505.
79. McDermid, C. D. (1965). Some correlates of creativity in engineering personnel. Journal of Applied Psychology, 49, 14-19.
80. McDougall, W. P., & Reitan, H. M. (1961). The use of a peer rating technique in appraising selected attributes of counselor trainees. Counselor Education and Supervision, 1, 72-76.
81. Mischel, W. (1965). Predicting the success of Peace Corps volunteers in Nigeria. Journal of Personality and Social Psychology, 1, 510-517.
82. Morrison, E. W., & Phelps, C. C. (1999). Taking charge at work: Extrarole efforts to initiate workplace change. Academy of Management Journal, 42, 403-419.
83. Mosier, C. I. (1943). On the reliability of a weighted composite. Psychometrika, 8, 161-168.
84. Mount, M. K., Barrick, M. R., & Strauss, J. P. (1994). Validity of observer ratings of the big five personality factors. Journal of Applied Psychology, 79, 272-280.
85. Mount, M. K., Judge, T. A., Scullen, S. E., Sytsma, M. R., & Hezlett, S. A. (1998). Trait, rater and level effects in 360-degree performance ratings. Personnel Psychology, 51, 557-576.
86. Murphy, K. R., & Cleveland, J. N. (1995). Understanding performance appraisal: Social, organizational, and goal-based perspectives. Thousand Oaks, CA: Sage.
87. Murphy, K. R., Cleveland, J. N., & Mohler, C. J. (2001). Reliability, validity, and meaningfulness of multisource ratings. In D. W. Bracken, C. W. Timmreck, & A. H. Church (Eds.), The handbook of multisource feedback (pp. 130-148). San Francisco: Jossey-Bass.
88. Neuman, G. A., & Wright, J. (1999). Team effectiveness: Beyond skills and cognitive ability. Journal of Applied Psychology, 84, 376-389.
89. Nunnally, J. C. (1978). Psychometric theory (2nd ed.). New York: McGraw-Hill.
90. Olson, D. A., Shultz, K. S., & Scott, G. (2000, April). The association between personality preferences and behavioral ratings for physician leaders. Paper presented at the 15th annual conference of the Society for Industrial and Organizational Psychology, New Orleans, LA.
91. Organ, D. W. (1997). Organizational citizenship behavior: It's construct clean-up time. Human Performance, 10, 85-97.
92. Oz, S., & Eden, D. (1994). Restraining the golem: Boosting performance by changing the interpretation of low scores. Journal of Applied Psychology, 79, 744-754.
93. Parker, T. C. (1963). Relationships among measures of supervisory behavior, group behavior, and situational characteristics. Personnel Psychology, 16, 319-334.
94. Romano, C. (1994). Conquering the fear of feedback. HRfocus, 71, 9-10.
95. Ross, S. M., & Offermann, L. R. (1997). Transformational leaders: Measurement of personality attributes and work group performance. Personality and Social Psychology Bulletin, 23, 1078-1086.
96. Rowland, K. M., & Scott, W. E., Jr. (1968). Psychological attributes of effective leadership in a formal organization. Personnel Psychology, 21, 365-377.
97. Russell, C. J., Mattson, J., Devlin, S. E., & Atwater, D. (1990). Predictive validity of biodata items generated from retrospective life experience essays. Journal of Applied Psychology, 75, 569-580.
98. Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124, 262-274.
99. Schmitt, N., Rogers, W., Chan, D., Sheppard, L., & Jennings, D. (1997). Adverse impact and predictive efficiency of various predictor combinations. Journal of Applied Psychology, 82, 719-730.
100. Scullen, S. E., Mount, M. K., & Goff, M. (2000). Understanding the latent structure of job performance ratings. Journal of Applied Psychology, 85, 956-970.
101. Seltzer, J., & Bass, B. M. (1990). Transformational leadership: Beyond initiation and consideration. Journal of Management, 16, 693-703.
102. Shipper, F., & Wilson, C. L. (1992). The impact of managerial behaviors on group performance, stress, and commitment. In K. E. Clark, M. B. Clark, & D. P. Campbell (Eds.), Impact of leadership (pp. 119-129). Greensboro, NC: Center for Creative Leadership.
103. Siegel, L. (1982). Paired comparison evaluations of managerial effectiveness by peers and supervisors. Personnel Psychology, 35, 843-852.
104. Skarlicki, D. P., Folger, R., &Tesluk, P. (1999). Personality as a moderator in the relationship between fairness and retaliation. Academy of Management Journal, 42, 100-108.
105. Spitzer, M. E., & McNamara, W. J. (1964). A managerial selection study. Personnel Psychology, 17, 19-40.
106. Stogdill, R. M., & Shartle, C. L. (1955). Methods in the study of administrative leadership (Monograph No. 80). Columbus, OH: Ohio State University, The Bureau of Business Research.
107. Stumpf, S. A. (1981). Career roles, psychological success, and job attitudes. Journal of Vocational Behavior, 19, 98-112.
108. Tesluk, P., & Gerstner, C. (1998, April). What makes for responsive andproactive teams? Keys to team effectiveness. Paper presented at the 13th annual conference of the Society for Industrial and Organizational Psychology, Dallas, TX.
109. Tett, R. P., Jackson, D. N., & Rothstein, M. (1991). Personality measures as predictors of job performance: A meta-analytic review. Personnel Psychology, 44, 703-742.
110. Timmreck, C. W., & Bracken, D. W. (1997). Multisource feedback: A study of its use in decision making. Employment Relations Today, 24, 21-27.
111. Titus, H. E. (1969). Prediction of supervisory success by use of standard psychological tests. The Journal of Psychology, 72, 35-40.
112. Tomow, W. W. (1993). Perceptions or reality: Is multi-perspective measurement a means or an end? Human Resource Management. 32, 221-229.
113. Trites, D. K. (1960). Adaptability measures as predictors of performance ratings. Journal of Applied Psychology, 44, 349-353.
114. Validity Information Exchange. (1954a). No. 7-024. Personnel Psychology, 7, 159-160.
115. Validity Information Exchange. (1954b). No. 7-045. Personnel Psychology, 7, 279.
116. Validity Information Exchange. (1954c). No. 7-082. Personnel Psychology, 7, 553-554.
117. Van Scotter, J. R., & Motowidlo, S. J. (1996). Interpersonal facilitation and job dedication as separate facets of contextual performance. Journal of Applied Psychology, 81, 525-531.
118. Viswesvaran, C., & Ones, D. S. (1995). Theory testing: Combining psychometric meta-analysis and structural equations modeling. Personal Psychology, 48, 865-885.
119. Viswesvaran, C., Ones, D. S., & Schmidt, F. L. (1996). Comparative analysis of the reliability of job performance ratings. Journal of Applied Psychology, 81, 557-574.
120. Waters, L. K., & Waters, C. W. (1970). Peer nominations as predictors of short-term sales performance. Journal of Applied Psychology, 54, 42-44.
121. Westman, M. (1990). The relationship between stress and performance: The moderating effect of hardiness. Human Performance, 3, 141-155.
122. Wherry, R. J., & Fryer, D. H. (1949). Buddy ratings: Popularity contest or leadership criteria? Personnel Psychology, 2, 147-159.
123. Williams, S. B., & Leavitt, H. J. (1947). Group opinion as a predictor of military leadership. Journal of Consulting Psychology, 11, 283-291.
124. Yammarino, F. J., & Atwater, L. E. (1997). Do managers see themselves as others see them? Implications of self-other rating agreement for human resources management. Organizational Dynamics, 25(4), 35-44.
125. Yammarino, F. J., & Atwater, L. E. (2001). Understanding agreement in multisource feedback. In D. W. Bracken, C. W. Timmreck, & A. H. Church (Eds.), The handbook of multisource feedback (pp. 204-220). San Francisco: Jossey-Bass.
126. Yammarino, F. J., & Bass, B. M. (1990). Long-term forecasting of transformational leadership and its effects among naval officers: Some preliminary findings. In K. E. Clark & M. B. Clark (Eds.), Measures of leadership (pp. 151-169). West Orange, NJ: Leadership Library of America, Inc.
127. Yukl, G. A., & Kanuk, L. (1979). Leadership behavior and effectiveness of beauty salon managers. Personnel Psychology, 32, 663-675.