Alternate Approaches to Understanding the Psychometric Properties of Assessment Centers: An analysis of the structure and equivalence of exercise ratings

International Journal of Selection and Assessment

Volumn 20, Issue 1, 2012, Pages 82-97

  Hoffman, Brian J ^a   Meade, Adam ^b  

^a University of Georgia   (United States)

^b NORTH CAROLINA STATE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84857254960     PISSN: 0965075X     EISSN: 14682389     Source Type: Journal    
DOI: 10.1111/j.1468-2389.2012.00581.x     Document Type: Article

References (86)

1. Anderson, N., Lievens, F., Van Dam, K., & Born, M. P. (2006). A construct-driven investigation of gender differences in a leadership-role assessment center. Journal of Applied Psychology, 91, 555-566.
2. Arthur, W., Day, E. A., McNelly, T. L., & Edens, P. S. (2003). A meta-analysis of the criterion-related validity of assessment center dimensions. Personnel Psychology, 56, 125-154.
3. Arthur, W., Day, E. A., & Woehr, D. J. (2008). Mend it, don't end it: An alternate view of assessment center construct-related validity evidence. Industrial and Organizational Psychology: Perspectives on Research and Practice, 1, 105-111.
4. Arthur, W., & Villado, A. J. (2008). The importance of distinguishing between constructs and methods when comparing predictors in personnel selection research and practice. Journal of Applied Psychology, 93, 435-442.
5. Arthur, W., Woehr, D. J., & Maldegen, R. (2000). Convergent and discriminant validity of assessment center dimensions: A conceptual and empirical reexamination of the assessment center construct-related validity paradox. Journal of Management, 26, 813-835.
6. Atkins, P., & Wood, R. (2002). Self-versus others' ratings as predictors of assessment center ratings: Validation evidence for 360-degree feedback programs. Personnel Psychology, 55, 871-904.
7. Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107, 238-246.
8. Bollen, K. A. (1989). Structural equations with latent variables. New York: John Wiley & Sons.
9. Borman, W. C., & Brush, D. H. (1993). More progress toward a taxonomy of managerial performance requirements. Human Performance, 6, 1-21.
10. Bowler, M. C., & Woehr, D. J. (2006). A meta-analytic evaluation of the impact of dimension and exercise factors on assessment center ratings. Journal of Applied Psychology, 91, 1114-1124.
11. Brannick, M. T. (2008). Back to basics of test construction and scoring. Industrial and Organizational Psychology: Perspectives on Research and Practice, 1, 131-133.
12. Brannick, M. T., & Spector, P. E. (1990). Estimation problems in the block-diagonal model of the multitrait-multimethod matrix. Applied Psychological Measurement, 14, 325-339. doi: 10.1177/014662169001400401.
13. Bray, D. W. (1982). The assessment center and the study of lives. American Psychologist, 37, 180-189.
14. Campbell, J. P., McHenry, J. J., & Wise, L. L. (1990). Modeling job performance in a population of jobs. Personnel Psychology, 43, 313-333.
15. Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9, 233-255.
16. Collins, J. M., Schmidt, F. L., Sanchez-Ku, M., Thomas, L., McDaniel, M. A., & Le, H. (2003). Can basic individual differences shed light on the construct meaning of assessment center evaluations? International Journal of Selection and Assessment, 11, 17-29.
17. Connelly, B. S., Ones, D. S., Ramesh, A., & Goff, M. (2008). A pragmatic view of dimensions and exercises in assessment center ratings. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 121-124.
18. Conway, J. M. (1996). Analysis and design of multitrait-multirater performance appraisal studies. Journal of Management, 22, 139-162.
19. Conway, J. M. (1999). Distinguishing contextual performance from task performance for managerial jobs. Journal of Applied Psychology, 84, 3-13.
20. Diefendorff, J. M., Silverman, S. B., & Greguras, G. J. (2005). Measurement equivalence and multisource ratings for non-managerial positions: Recommendations for research and practice. Journal of Business and Psychology, 19, 399-425.
21. Dilchert, S., & Ones, D. S. (2009). Assessment center dimensions: Individual differences correlates and meta-analytic incremental validity. International Journal of Selection and Assessment, 17, 254-270.
22. Feldman, J. M. (1981). Beyond attribution theory: Cognitive processes in performance appraisal. Journal of Applied Psychology, 66, 127-148.
23. Fleishman, E. A. (1957). The leader opinion questionnaire. In R. M. Stogdill & A. E. Coons (Eds.), Leader behavior: Its description and measurement (pp. 120-133). Columbus, OH: Bureau of Business Research.
24. Gaugler, B. B., Rosenthal, D. B., Thornton, G. C., & Bentson, B. (1987). Meta-analysis of assessment center validity. Journal of Applied Psychology, 72, 493-511.
25. Guilford, J. P. (1954). Psychometric methods (2nd ed.) New York: McGraw-Hill.
26. Hoffman, B. J., Blair, C., Meriac, J., & Woehr, D. J. (2007). Expanding the criterion domain? A meta-analysis of the OCB literature. Journal of Applied Psychology, 92, 555-566.
27. Hoffman, B. J., Lance, C., Bynum, B., & Gentry, B. (2010). Rater source effects are alive and well after all. Personnel Psychology, 63, 119-151.
28. Hoffman, B. J., Melchers, K., Blair, C. A., Kleinmann, M., & Ladd, R. T. (2011). Exercises and dimensions are the currency of assessment centers. Personnel Psychology, 64, 351-395.
29. Hoffman, B. J., & Woehr, D. J. (2009). Disentangling the meaning of multisource feedback: An examination of the nomological network surrounding source and dimension factors. Personnel Psychology, 62, 735-765.
30. Howard, A. (2008). Making assessment centers work the way they are supposed to. Industrial and Organizational Psychology: Perspectives on Research and Practice, 1, 98-104.
31. Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis. Structural Equation Modeling, 6, 1-55.
32. Jackson, D. J. R., Stillman, J. A., & Atkins, S. G. (2005). Rating tasks versus dimensions in assessment centers: A psychometric comparison. Human Performance, 18, 213-241.
33. Jackson, D. J. R., Stillman, J. A., & Englert, P. (2010). Task-based assessment centers: Empirical support for a systems model. International Journal of Selection and Assessment, 18, 141-154.
34. Jones, R. G., & Born, M. (2008). Assessor constructs in use as the missing component in validation of assessment center dimensions: A critique and directions for research. International Journal of Selection and Assessment, 16, 229-238.
35. Jöreskog, K., & Sörbom, D. (1996). LISREL 8: Users reference guide. Chicago: Scientific Software International.
36. Judge, T. A., & Piccolo, R. F. (2004). Transformational and transactional leadership: A meta-analytic test of their relative validity. Journal of Applied Psychology, 89, 755-768.
37. Judge, T. A., Piccolo, R. F., & Ilies, R. (2004). The forgotten ones? The validity of consideration and initiating structure in leadership research. Journal of Applied Psychology, 89, 36-51.
38. Katz, D., & Kahn, R. L. (1951). Human organization and worker motivation. In L. R. Tripp (Ed.), Industrial productivity (pp. 146-171). Madison, WI: Industrial Relations Research Association.
39. Kingsbury, F. A. (1922). Analyzing ratings and training raters. Journal of Personnel Research, 1, 377-383.
40. Kolk, N. J., Born, M. P., & van der Flier, H. (2004). A triadic approach to the construct validity of the assessment center: The effect of categorizing dimensions into a feeling, thinking, and power taxonomy. European Journal of Psychological Assessment, 20, 149-156.
41. Kram, K. E. (1985). Mentoring at work. Glenview, IL: Scott, Foresman and Company.
42. Lance, C. E. (2008a). Why assessment centers do not work the way they're supposed to. Industrial and Organizational Psychology Perspectives on Research and Practice, 1, 84-97.
43. Lance, C. E. (2008b). Where have we been, how did we get there, and where shall we go? Industrial and Organizational Psychology: Perspectives on Research and Practice, 1, 140-146.
44. Lance, C. E., Dawson, B., Birkelbach, D., & Hoffman, B. J. (2010). Method effects, measurement error, and substantive conclusions. Organizational Research Methods, 13, 435-455.
45. Lance, C. E., Foster, M. R., Gentry, W. A., & Thoresen, J. D. (2004). Assessor cognitive processes in an operational assessment center. Journal of Applied Psychology, 89, 22-35.
46. Lance, C. E., Lambert, T. A., Gewin, A. G., Lievens, F., & Conway, J. M. (2004). Revised estimates of dimension and exercise variance components in assessment center postexercise dimension ratings. Journal of Applied Psychology, 89, 377-385.
47. Lance, C. E., Newbolt, W. H., Gatewood, R. D., Foster, M. R., French, N. R., & Smith, D. E. (2000). Assessment center exercise factors represent cross-situational specificity, not method bias. Human Performance, 13, 323-353.
48. Lance, C. E., Woehr, D. J., & Meade, A. W. (2007). Case study: A Monte Carlo investigation of assessment center construct validity models. Organizational Research Methods, 10, 430-448.
49. Lievens, F. (2002). Trying to understand the different pieces of the construct validity puzzle of assessment centers: An examination of assessor and assessee effects. Journal of Applied Psychology, 87, 675-686.
50. Lievens, F. (2008). What does exercise-based assessment really mean? Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 112-115.
51. Lievens, F., Chasteen, C. S., Day, E. A., & Christiansen, N. D. (2006). Large-scale investigation of the role of trait activation theory for understanding assessment center convergent and discriminant validity. Journal of Applied Psychology, 91, 247-258.
52. Lievens, F., De Corte, W., & Schollaert, E. (2008). A closer look at the frame-of-reference effect in personality scale scores and validity. Journal of Applied Psychology, 93, 268-279.
53. Marsh, H. W. (1989). Confirmatory factor analyses of multitrait-multimethod data: Many problems and a few solutions. Applied Psychological Measurement, 13, 335-361. doi: 10.1177/014662168901300402.
54. Maurer, T., Raju, N. S., & Collins, W. C. (1998). Peer and subordinate performance measurement equivalence. Journal of Applied Psychology, 83, 693-702.
55. McCauley, C., & Lombardo, M. (1990). BENCHMARKS®: An instrument for diagnosing managerial strengths and weaknesses. In K. E. Clark & M. B. Clark (Eds.), Measures of leadership (pp. 535-545). West Orange, NJ: Leadership Library of America.
56. Meade, A. W., Johnson, E. C., & Braddy, P. W. (2008). Power and sensitivity of alternative fit indices in tests of measurement invariance. Journal of Applied Psychology, 93, 568-592.
57. Meriac, J. P., Hoffman, B. J., Woehr, D. J., & Fleisher, D. J. (2008). Further evidence for the validity of assessment center dimensions: A meta-analysis of the incremental criterion-related validity of assessment center dimension ratings. Journal of Applied Psychology, 93, 1042-1052.
58. Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performance as scientific inquiry into score meaning. American Psychologist, 50, 741-749.
59. Moses, J. (2008). Assessment centers work, but for different reasons. Industrial and Organizational Psychology: Perspectives on Research and Practice, 1, 134-136.
60. Mount, M. K., Judge, T. A., Scullen, S. E., Sytsma, M. R., & Hezlett, S. A. (1998). Trait, rater, and level effects in 360-degree performance ratings. Personnel Psychology, 51, 557-576.
61. Murphy, K. R. (1989). Dimensions of job performance. In R. F. Dillon & J. W. Pellegrino (Eds.), Testing: Theoretical and applied perspectives (pp. 218-247). New York: Praeger.
62. Norton, S. D. (1977). The empirical and content validity of assessment centers vs. traditional methods for predicting managerial success. Academy of Management Review, 2, 442-453.
63. OSS Assessment Staff (1948). Assessment of men. New York: Rinehart.
64. Pedhazur, E. H., & Schmelkin, L. P. (1991). Measurement, design, and analysis: An integrated approach. Hillsdale, NJ: Erlbaum.
65. Putka, D. (2011). A closer look at reliable and unreliable sources of variance in dimension-exercise units. Unpublished manuscript.
66. Rupp, D. E., Thornton, G. C., III, & Gibbons, A. M. (2008). The construct validity of the assessment center method and usefulness of dimensions as focal constructs. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 116-120.
67. Russell, C. J., & Domm, D. R. (1995). Two field tests of an explanation of assessment centre validity. Journal of Occupational and Organizational Psychology, 68, 25-47.
68. Sackett, P. R. (1987). Assessment centers and content validity: Some neglected issues. Personnel Psychology, 40, 13-25.
69. Sackett, P. R., & Dreher, G. F. (1982). Constructs and assessment center dimensions: Some troubling empirical findings. Journal of Applied Psychology, 67, 401-410.
70. Sackett, P. R., & Wilson, M. A. (1982). Factors affecting the consensus judgment process in managerial assessment centers. Journal of Applied Psychology, 67, 10-17.
71. Salgado, J. F., Moscoso, S., & Lado, M. (2003). Test-retest reliability of ratings of job performance dimensions in managers. International Journal of Selection and Assessment, 11, 98-101.
72. Schmitt, N. (1977). Interrater agreement and dimensionality and combination of assessment center judgments. Journal of Applied Psychology, 62, 171-176.
73. Scullen, S. E., Mount, M. K., & Judge, T. A. (2003). Evidence of the construct validity of developmental ratings of managerial performance. Journal of Applied Psychology, 88, 50-66.
74. Shore, T. H., Thornton, G. C., & Shore, L. M. (1990). Construct validity of two categories of assessment center dimension ratings. Personnel Psychology, 43, 101-116.
75. Steiger, J. H. (1990). Structural model evaluation and modification: An interval estimation approach. Multivariate Behavioral Research, 25, 173-180.
76. Stogdill, R. M. (1963). Manual for the leader behavior description questionnaire, form XII. Columbus, OH: Bureau of Business Research, Ohio State University.
77. Thorndike, E. L. (1920). A constant error in psychological ratings. Journal of Applied Psychology, 4, 25-29.
78. Vandenberg, R. J., & Lance, C. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4-69.
79. Viswesvaran, C., Ones, D. S., & Schmidt, F. L. (1996). Comparative analysis of the reliability of job performance ratings. Journal of Applied Psychology, 81, 557-574.
80. Viswesvaran, C., Schmidt, F. L., & Ones, D. S. (2002). The moderating influence of job performance dimension on convergence of supervisory and peer ratings of job performance: Unconfounding construct-level convergence and rating difficulty. Journal of Applied Psychology, 87, 245-354.
81. Viswesvaran, C., Schmidt, F. L., & Ones, D. S. (2005). Is there a general factor in ratings of performance? A meta-analytic framework for disentangling substantive and error influences. Journal of Applied Psychology, 90, 108-131.
82. Wells, F. L. (1907). A statistical study of literary merit. Columbia University Contributions to Philosophy and Psychology, 16, 5-30.
83. Wherry, R. J. (1952). The control of bias in ratings: VIII. A theory of rating (PRB report No. 922, Contract No. DA-49-083 OSA69, Department of the Army). Columbus, OH: Ohio State University Research Foundation.
84. Widaman, K. F. (1992). Multitrait-multimethod models in aging research. Experimental Aging Research, 18, 185-201.
85. Woehr, D. J., & Arthur, W., Jr. (2003). The construct-related validity of assessment center ratings: A review and meta-analysis of the role of methodological factors. Journal of Management, 29, 231-258.
86. Yukl, G. (2006). Leadership in organizations (6th ed.) Upper Saddle River, NJ: Pearson, Prentice-Hall.