Revised Estimates of Dimension and Exercise Variance Components in Assessment Center Postexercise Dimension Ratings

Journal of Applied Psychology

Volumn 89, Issue 2, 2004, Pages 377-385

  Lance, Charles E ^a,d   Lambert, Tracy A ^a   Gewin, Amanda G ^a   Lievens, Filip ^b   Conway, James M ^c  

^a UNIVERSITY OF GEORGIA   (United States)

^b GHENT UNIVERSITY   (Belgium)

^c CENTRAL CONNECTICUT STATE UNIVERSITY   (United States)

^d University of Georgia ^*  (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ANALYSIS OF VARIANCE; ARTICLE; CLINICAL TRIAL; FACTORIAL ANALYSIS; HUMAN; META ANALYSIS; PERSONNEL MANAGEMENT; PSYCHOMETRY; REPRODUCIBILITY; THEORETICAL MODEL;

ANALYSIS OF VARIANCE; FACTOR ANALYSIS, STATISTICAL; HUMANS; MODELS, THEORETICAL; PERSONNEL SELECTION; PSYCHOMETRICS; REPRODUCIBILITY OF RESULTS;

EID: 1842810086     PISSN: 00219010     EISSN: None     Source Type: Journal    
DOI: 10.1037/0021-9010.89.2.377     Document Type: Article

References (60)

1. Arthur, W. A., Jr., Woehr, D. J., & Maldegen, R. (2000). Convergent and discriminant validity of assessment center dimensions: A conceptual and empirical re-examination of the assessment center construct-related validity paradox. Journal of Management 26, 813-835.
2. Becker, A. S. (1990). The effects of a reduction in assessor roles on the convergent and discriminant validity of assessment center ratings. Unpublished doctoral dissertation, University of Missouri, St. Louis.
3. Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107, 238-246.
4. Bobrow, W., & Leonards, J. S. (1997). Development and validation of an assessment center during organizational change. Journal of Social Behavior and Personality, 12, 217-236.
5. Brannick, M. T., & Spector, P. E. (1990). Estimation problems in the block-diagonal model of the multitrait-multimethod matrix. Applied Psychological Measurement, 14, 325-339.
6. Browne, M. W. (1984). The decomposition of multitrait-multimethod matrices by generalized least squares. British Journal of Mathematical and Statistical Psychology, 37, 1-21.
7. Bycio, P., Alvares, K. M., & Hahn, J. (1987). Situational specificity in assessment center ratings: A confirmatory factor analysis. Journal of Applied Psychology, 72, 463-474.
8. Cheung, G. W., & Rensvold, R. B. (2001). The effects of model parsimony and sampling error on the fit of structural equation models. Organizational Research Methods, 4, 236-264.
9. Chorvat, V. P. (1994). Toward the construct validity of assessment center leadership dimensions: A multitrait-multimethod investigation using confirmatory factor analysis. Unpublished doctoral dissertation, University of South Florida, Tampa.
10. Conway, J. M., Lievens, F., Scullen, S. E., & Lance, C. E. (2002, April). Bias in the correlated uniqueness model for MTMM data: A simulation study. Paper presented at the meeting of the Society for Industrial and Organizational Psychology, Toronto, Ontario, Canada.
11. Donahue, L. M., Truxillo, D. M., Cornwell, J. M., & Gerrity, M. J. (1997). Assessment center construct validity and behavioral checklists: Some additional findings. Journal of Social Behavior and Personality, 12, 85-108.
12. Fleenor, J. W. (1996). Constructs and developmental assessment centers: Further troubling empirical findings. Journal of Business and Psychology, 10, 319-333.
13. Fredricks, A. J. (1989). Assessment center ratings: Models and processes. Unpublished doctoral dissertation, University of Nebraska, Lincoln.
14. Gaugler, B. B., Rosenthal, D. B., Thornton, G. C., III, & Bentson, C. (1987). Meta-analysis of assessment center validity. Journal of Applied Psychology, 72, 493-511.
15. Harris, M. M., Becker, A. S., & Smith, D. E. (1993). Does the assessment center scoring method affect the cross-situational consistency of ratings? Journal of Applied Psychology, 78, 675-678.
16. Howard, A. (1997). A reassessment of assessment centers: Challenges for the 21st century. Journal of Social Behavior and Personality, 12(5), 13-52.
17. Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis. Structural Equation Modeling, 6, 1-55.
18. International Task Force on Assessment Center Guidelines (2000). Guidelines and ethical considerations for assessment center operations. Public Personnel Management, 28, 315-331.
19. James, L. R. (1980). The unmeasured variables problem in path analysis. Journal of Applied Psychology, 65, 415-421.
20. James, L. R., Demaree, R. G., & Mulaik, S. A. (1986). A note on validity generalization procedures. Journal of Applied Psychology, 71, 440-450.
21. Joyce, L. W., Thayer, P. W., & Pond, S. B. (1994). Managerial functions: An alternative to traditional assessment center dimensions? Personnel Psychology, 47, 109-121.
22. Kenny, D. A., & Kashy, D. A. (1992). Analysis of the multitrait-multimethod matrix by confirmatory factor analysis. Psychological Bulletin, 112, 165-172.
23. Kleinmann, M. (1997). Transparenz der Anforderungsdimensionen: Ein Moderator der Konstrukt- und Kriteriumsvalidität des Assessment-Centers [Transparency of the required dimensions: A moderator of assessment centers' construct and criterion validity]. Zeitschrift für Arbeits- und Organisationspsychologie, 41, 171-181.
24. Kleinmann, M., Andres, J., Fedtke, C., Godbersen, F., & Köller, O. (1994). Der Einfluss unterscheidlicher auswertungsmethoden auf die Konstruktvalidität von Assessment-Centern [The influence of different rating procedures on the construct validity of assessment center methods]. Zeitschrift für Experimentelle und Angewandte Psychologie, 41, 184-210.
25. Kleinmann, M., Kuptsch, C., & Köller, O. (1996). Transparency: A necessary requirement for the construct validity of assessment centers. Applied Psychology: An International Review, 45, 67-84.
26. Kolk, N. J., Born, M. P., & Van der Flier, H. (2000, April). The transparent assessment center: The effect of revealing dimensions to applicants. Paper presented at the meeting of the Society for Industrial and Organizational Psychology, New Orleans, LA.
27. Kolk, N. J., Born, M. P., & van der Flier, H. (2001, April). Common rater variance as explanation for the lack of construct validity of assessment center dimensions. Paper presented at the meeting of the Society for Industrial and Organizational Psychology, San Diego, CA.
28. Kudisch, J. D., Ladd, R. T., & Dobbins, G. H. (1997). New evidence on the construct validity of diagnostic assessment centers: The findings may not be so troubling after all. Journal of Social Behavior and Personality, 12, 129-144.
29. Lance, C. E., Foster, M. R., Gentry, W. A. & Thorensen, J. D. (2004). Assessor cognitive processes in an operational assessment center. Journal of Applied Psychology, 89, 22-35.
30. Lance, C. E., Johnson, C. D., Douthitt, S. S., Bennett, W., Jr., & Harville, D. L. (2000). Good news: Work sample administrators' global performance judgments are (about) as valid as we've suspected. Human Performance, 13, 253-277.
31. Lance, C. E., Newbolt, W. H., Gatewood, R, D., Foster, M. R., French, N., & Smith, D. E. (2000). Assessment center exercise factors represent cross-situational specificity, not method bias. Human Performance, 13, 323-353.
32. Lance, C. E., Noble, C. L., & Scullen, S. E. (2002). A critique of the correlated trait-correlated method and correlated uniqueness models for multitrait-multimethod data. Psychological Methods, 7, 228-244.
33. Lievens, F. (1998). Factors which improve the construct validity of assessment centers: A review. International Journal of Selection and Assessment, 6, 141-152.
34. Lievens, F. (2001). Assessors and use of assessment center dimensions: A fresh look at a troubling issue. Journal of Organizational Behavior, 22, 203-221.
35. Lievens, F. (2002). Trying to understand the different pieces of the construct validity puzzle of assessment centers: An examination of assessor and assessee effects. Journal of Applied Psychology, 87, 675-686.
36. Lievens, F., & Conway, J. M. (2001). Dimension and exercise variance in assessment center scores: A large-scale evaluation of multitrait-multimethod studies. Journal of Applied Psychology, 86, 1202-1222.
37. Lievens, F., & Klimoski, R. J. (2001). Understanding the assessment centre process: Where are we now? International Review of Industrial and Organizational Psychology, 16, 246-286.
38. Lievens, F., & Van Keer, E. (1999, May). Modeling method effects in assessment centers: An application of the correlated uniqueness approach. Paper presented at the meeting of the Society for Industrial and Organizational Psychology, Atlanta, GA.
39. Lowry, P. E. (1995). The assessment center process: Assessing leadership in the public sector. Public Personnel Management, 22, 443-450.
40. Lowry, P. E. (1997). The assessment center process: New directions. Journal of Social Behavior and Personality, 12, 53-62.
41. Marsh, H. W. (1989). Confirmatory factor analysis of multitrait- multimethod data: Many problems and a few solutions. Applied Psychological Measurement, 13, 335-261.
42. Marsh, H. W. (1994). Confirmatory factor analyses of factorial invariance: A multifaceted approach. Structural Equation Modeling, 1, 5-34.
43. Marsh, H. W., & Bailey, M. (1991). Confirmatory factor analysis of multitrait-multimethod data: A comparison of alternative models. Applied Psychological Measurement, 15, 47-70.
44. Parker, M. W. (1992). A construct validation of the Florida Principal Competencies Assessment Center using confirmatory factor analysis. Unpublished doctoral dissertation, University of South Florida, Tampa.
45. Robertson, I., Gratton, L., & Sharpley, D. (1987). The psychometric properties and design of managerial assessment centres: Dimensions into exercises won't go. Journal of Occupational Psychology, 60, 187-195.
46. Robie, C., Adams, K. A., Osburn, H. G., Morris, M. A., & Etchegaray, J. M. (2000). Effects of the rating process on the construct validity of assessment center dimension evaluations. Human Performance, 13, 355-370.
47. Sackett, P. R., & Dreher, G. F. (1982). Constructs and assessment center dimensions: Some troubling empirical findings. Journal of Applied Psychology, 67, 401-410.
48. Sackett, P. R., & Tuzinski, K. (2001). The role of dimensions and exercises in assessment center judgments. In M. London (Ed.), How people evaluate others in organizations (pp. 111-129). Mahwah, NJ: Erlbaum.
49. Sagie, A., & Magnezy, R. (1997). Assessor type, number of distinguishable dimension categories, and assessment centre construct validity. Journal of Occupational and Organizational Psychology, 70, 103-108. (Sagie - 1 & Sagie - 2).
50. Schleicher, D. J., Day, D. V., Mayes, B. T., & Riggio, R. E. (1999, May). A new frame for frame-of-reference training: Enhancing the construct validity of assessment centers. Paper presented at the meeting of the Society for Industrial and Organizational Psychology, Atlanta, GA.
51. Schleicher, D. J., Day, C. V., Mayes, B. T., & Riggio, R. E. (2002). A new frame for frame-of-reference training: Enhancing the construct validity of assessment centers. Journal of Applied Psychology, 87, 735-746.
52. Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124, 262-274.
53. Schneider, J. R., & Schmitt, N. (1992). An exercise design approach to understanding assessment center dimension and exercise constructs. Journal of Applied Psychology, 77, 32-41.
54. Scullen, S. E. (1999). Using confirmatory factor analysis of correlated uniquenesses to estimate method variance in multitrait-multimethod matrices. Organizational Research Methods, 2, 275-292.
55. Smith, F. D. (1991). Work samples as measures of performance. In A. K. Wigdor & B. F. Green, Jr. (Eds.), Performance assessment for the workplace (Vol. 2, pp. 27-52). Washington, DC: National Academy Press.
56. Sweeney, D. C. (1976). The development and analysis of rating scales for the Chicago Police Recruit Assessment Center. Unpublished manuscript, Bowling Green State University, OH.
57. Teachout, M. S., & Pellum, M. W. (1991). Air force research to link standards for enlistment to on-the-job performance (AFHRL-TR-90-90). Brooks AFB, TX: Air Force Human Resources Laboratory, Training Systems Division.
58. Van der Velde, E. G., Born, M. P., & Hofkes, K. (1994). Begripsvalidering van een assessment center met behulp van confirmatorische factoranalyse [Construct validity of an assessment center using confirmatory factor analysis]. Gedrag en Organisatie, 7, 18-26.
59. Veldman, W. M. (1994). Assessment centers and candidates' personal qualities: A study on the correlations between assessment center ratings. Unpublished doctoral dissertation, Vrije Universiteit Amsterdam, Amsterdam, the Netherlands.
60. Woehr, D. J., & Arthur, W., Jr. (1999, April). The construct-related validity of assessment center ratings: A review and meta-analysis of the role of methodological factors. Paper presented at the meeting of the Society for Industrial and Organizational Psychology, Atlanta, GA.