Assessment center construct-related validity: Stepping beyond the MTMM matrix

Journal of Vocational Behavior

Volumn 75, Issue 2, 2009, Pages 173-182

  Bowler, Mark C ^a   Woehr, David J ^b  

^a EAST CAROLINA UNIVERSITY   (United States)

^b UNIVERSITY OF TENNESSEE   (United States)

Author keywords

Assessment center; Construct related validity; Generalizability theory; Variance partitioning

Indexed keywords

EID: 68949130110     PISSN: 00018791     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jvb.2009.03.008     Document Type: Article

References (36)

1. Arthur W., Day E.D., McNelly T.L., and Edens P.S. A meta-analysis of the criterion-related validity of assessment center dimensions. Personnel Psychology 56 (2003) 125-154
2. Arthur W., Woehr D.J., and Maldegen R. Convergent and discriminant validity of assessment center dimensions: A conceptual and empirical re-examination of the assessment center construct-related validity paradox. Journal of Management 26 (2000) 813-835
3. Bernardin H.J., and Buckley M.R. Strategies in rater training. Academy of Management Review 6 (1981) 205-212
4. Bowler M.C., and Woehr D.J. A meta-analytic evaluation of the impact of dimension and exercise factors on assessment center ratings. Journal of Applied Psychology 91 (2006) 1114-1124
5. Bowler, M. C., & Woehr, D. J. (2008). Evaluating assessment center construct-related validity via variance partitioning. In B. J. Hoffman (Chair), Reexamining assessment centers: Alternate approaches. Paper presented at the 23rd annual meeting of the society for industrial and organizational psychology, San Francisco, CA.
6. Brannick M.T., Michaels C.E., and Baker D.P. Construct validity of in-basket scores. Journal of Applied Psychology 74 (1989) 957-963
7. Bray D.W., and Grant D.L. The assessment center in the measurement of potential for business management. Psychological Monographs: General and Applied 80 (1966) 1-27
8. Brennan R.L. Variance components in generalizability theory. In: Reynolds C.R. (Ed). Cognitive assessment: A multidisciplinary perspective (1994), Plenum Press, New York, NY 175-207
9. Bycio P., Alvares K.M., and Hahn J. Situational specificity in assessment center ratings: A confirmatory factor analysis. Journal of Applied Psychology 72 (1987) 463-474
10. Chen H.C. Assessment center: A critical mechanism for assessing HRD effectiveness and accountability. Advances in Developing Human Resources 8 (2006) 247-264
11. Cochran W.G., and Cox G.M. Experimental designs. 2nd ed. (1957), John Wiley & Sons, Inc., New York, NY
12. Conner, W. S., & Young, S. (1961). Fractional factorial designs for experiments with factors at two and three levels (Applied Mathematics Series, Vol. 58). Washington, DC: National Bureau of Standards.
13. Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York, NY: Wiley, Inc.
14. Federer W.T. Experimental design (1955), Macmillan, New York
15. Fleenor J.W. Constructs and developmental assessment centers: Further troubling empirical findings. Journal of Business and Psychology 3 (1996) 319-335
16. Howard A. A reassessment of assessment centers: Challenges for the 21st century. Journal of Social Behavior and Personality 12 (1998) 13-52
17. Highhouse S., and Harris M.M. The measurement of assessment center situations: Bem's template matching technique for examining exercise similarity. Journal of Applied Social Psychology 23 (1993) 140-155
18. Jackson D.J., Stillman J.A., and Atkins S.G. Rating task versus dimensions in assessment centers: A psychometric comparison. Human Performance 18 (2005) 213-241
19. Kenny D.A., and Kashy D.A. Analysis of the multitrait-multimethod matrix by confirmatory factor analysis. Psychological Bulletin 112 (1992) 165-172
20. Kraiger K., and Teachout M.S. Generalizability theory as construct-related evidence of the validity of job performance ratings. Human Performance 3 (1990) 19-35
21. Kudish J.D., Ladd R.T., and Dobbins G.H. New evidence on the construct validity of diagnostic assessment centers: The findings may not be so troubling after all. Journal of Social Behavior and Personality 12 (1997) 129-144
22. Lance C.E., Foster M.R., Gentry W.A., and Thoresen J.D. Assessor cognitive processes in an operational assessment center. Journal of Applied Psychology 89 (2004) 22-35
23. Lance C.E., Lambert T.A., Gewin A.G., Lievens F., and Conway J.M. Revised estimates of dimension and exercise variance components in assessment center postexercise dimension ratings. Journal of Applied Psychology 89 (2004) 377-385
24. Lance C.E., Newbolt W.H., Gatewood R.D., Foster M.R., French N.R., and Smith D.E. Assessment center exercises factors represent cross-situational specificity, not method bias. Human Performance 13 (2000) 323-353
25. Lance C.E., Woehr D.J., and Meade A.W. Case study: A Monte Carlo investigation of assessment center construct validity models. Organizational Research Methods 10 (2007) 430-448
26. Lievens F., and Conway J.M. Dimension and exercise variance in assessment center scores: A large-scale evaluation of multitrait-multimethod studies. Journal of Applied Psychology 86 (2001) 1202-1222
27. Lievens F., Chasteen C.S., Day E.A., and Christiansen N.D. Large-scale investigation of the role of trait activation theory for understanding assessment center convergent and discriminant validity. Journal of Applied Psychology 91 (2006) 247-258
28. Marsh H.W. Confirmatory factor analysis of multitrait-multimethod data: Many problems and a few solutions. Applied Psychological Measurement 13 (1989) 335-361
29. Marsh, H. W., & Grayson, D. (1995). Latent variable models of multitrait-multimethod data. In R. H. Hoyle (Ed.), Structural equation modeling: Concepts, issues, and applications (pp. 177-198). Thousand Oaks, CA: Sage Publications, Inc.
30. McHenry J.J., and Schmitt N. Multimedia testing. In: Rumsey M.G., and Walker C.B. (Eds). Personnel selection and classification (1994), Erlbaum, Hillsdale, NJ 193-232
31. Saal F.E., Downey R.G., and Lahey M.A. Rating the ratings: Assessing the psychometric quality of rating data. Psychological Bulletin 88 (1980) 413-428
32. Schneider J.R., and Schmitt N. An exercise design approach to understanding assessment center dimension and exercise constructs. Journal of Applied Psychology 77 (1992) 32-41
33. Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: A primer. Thousand Oaks, CA, US: Sage Publications, Inc.
34. Spychalski A., Quinones M.A., Gaugler B.A., and Pohley K. A survey of assessment center practices in organization in the United States. Personnel Psychology 50 (1997) 71-90
35. Turnage J.J., and Muchinsky P.M. Transsituational variability in human performance within assessment centers. Organizational Behavior and Human Performance 30 (1982) 174-200
36. Woehr D.J., and Arthur W. The construct-related validity of assessment center ratings: A review and meta-analysis of the role of methodological factors. Journal of Management 29 (2003) 231-258