If it ain't trait it must be method: (Mis)application of the multitrait-multimethod design in organizational research

Statistical and Methodological Myths and Urban Legends: Doctrine, Verity and Fable in the Organizational and Social Sciences

Volumn , Issue , 2008, Pages 337-360

  Lance, Charles E ^a   Baranik, Lisa E ^b   Lau, Abby R ^b   Scharlau, Elizabeth A ^b  

^a UNIVERSITY OF GEORGIA   (United States)

^b NONE

Author keywords

[No Author keywords available]

Indexed keywords

EID: 70449462419     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.4324/9780203867266     Document Type: Chapter

References (95)

1. Auken, S. V., Barry, T. E., & Bagozzi, R. P. (2006). A cross-country construct validation of cognitive age. Journal of the Academy of Marketing Science, 34, 439-455.
2. Avison, W. R. (1978). Auxiliary theory and multitrait-multimethod validation:A review of two approaches. Applied Psychological Measurement, 2, 431-447.
3. Baltes, B. B., Bauer, C. C., Bajdo, L. M., & Parker, C. P. (1999). The use of multitrait-multimethod data for detecting nonlinear relationships:The case of psychological climate and job satisfaction. Journal of Business and Psychology, 17, 3-17.
4. Becker, T. E., & Cote, J. A. (1994). Additive and multiplicative method effects in applied psychological research: An empirical assessment of three models. Journal of Management, 20, 625-641.
5. Benson, J. (1998). Developing a strong program of construct validation: A test anxiety example. Educational Measurement: Issues and Practice, 17, 10-22.
6. Bernard, R. S., Cohen, L. L., McClellan, C. B., & MacLaren, J. E. (2004). Pediatric procedural approach-avoidance coping and distress: A multitrait-multimethod study. Journal of Pediatric Psychology, 29, 131-141.
7. Blickle, G. (2003). Convergence of agents' and targets' reports on intraorganizational influence attempts. European Journal of Psychological Assessment, 19, 40-53.
8. Bollen, K. A., & Paxton, P. (1998). Detection and determinants of bias in subjective measures. American Sociological Review, 63, 465-478.
9. Bong, M., & Hocevar, D. (2002). Measuring self-efficacy: Multitrait-multimethod comparison of scaling procedures. Applied Measurement in Education, 15, 143-171.
10. Borman, W. C. (1974). The rating of individuals in organizations: An alternative approach. Organizational Behavior and Human Performance, 12, 105-124.
11. Boruch, R. F., Larkin, J. D., Wolins, L., & McKinney, A. C. (1970). Alternative method of analysis: Multitrait-multimethod data. Educational and Psychological Measurement, 30, 833-854.
12. Bourke, S. C., McColl, E., Shaw, P. J., & Gibson, G. J. (2004). Validation of quality of life instruments in ALS. Amyotrophic Lateral Sclerosis and Other Motor Neuron Disorders, 5, 55-60.
13. Bozeman, D. (1997). Interrater agreement in multi-source performance appraisal: A commentary. Journal of Organizational Behavior, 18, 313-316.
14. Bresnahan, M. J., Levine, T. R., Shearman, S. M., & Lee, S. Y. (2005). A multimethod multitrait validity assessment of self-construal in Japan, Korea, and the United States. Human Communication Research, 31, 33-59.
15. Browne, M. W. (1984). The decomposition of multitrait-multimethod matrices. British Journal of Mathematical and Statistical Psychology, 37, 1-21.
16. Campbell, D. T., & Fiske, D. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.
17. Campbell, D. T., & O'Connell, E. J. (1967). Method factors in multitraitmultimethod matrices: Multiplicative rather than additive? Multivariate Behavioral Research, 2, 409-426.
18. Carmines, E. G., & Zeller, R. A. (1979). Reliability and validity assessment. Beverly Hills, CA: Sage.
19. Cattell, R. B. (1946). Description and measurement of personality. Yonkerson-Hudson, NY: World Book.
20. Cattell, R. B. (1966). The data box: Its ordering of total resources in terms of possible relational systems. In R. B. Cattell (Ed.), Handbook of multivariate experimental psychology (pp. 67-128). Chicago: Ran McNally.
21. Chisolm, M. A., Lance, C. E., Williamson, G. M., & Mulloy, L. L. (2004). Development and validation of the immunosuppressant adherence instrument (ITAS). Patent Education and Counseling, 59, 13-20.
22. Cole, D. A. (2006). Coping with longitudinal data in research on developmental psychopathology. International Journal of Behavioral Development, 30, 20-25.
23. Cole, D. A., Martin, N. C., & Steiger, J. H. (2005). Empirical and conceptual problems with longitudinal trait-state models: Introducing a traitstate-occasion model. Psychological Methods, 10, 3-20.
24. Conway, J. M. (1996). Analysis and design of multitrait-multirater performance appraisal studies. Journal of Management, 22, 139-162.
25. Conway, J. M. (1998). Understanding method variance in multitrait-multirater performance appraisal matrices: Examples using general impressions and interpersonal affect as method factors. Human Performance, 11, 29-55.
26. Conway, J. M., & Huffcutt, A. I. (1997). Psychometric properties of multisource performance ratings: A meta-analysis of subordinate, supervisor, peer, and self-ratings. Human Performance, 10, 331-360.
27. Cook, W. L., & Kenny, D. A. (2006). Examining the validity of self-report assessments of family functioning: A question of the level of analysis. Journal of Family Psychology, 20, 209-216.
28. Cresswell, S. L., & Eklund, R. C. (2006). The convergent and discriminant validity of burnout measures in sport: A multitrait-multimethod analysis. Journal of Sports Sciences, 2006, 209-220.
29. Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281-302.
30. DiStefano, C., & Motl, R. W. (2006). Further investigating method effects associated with negatively worded items on self-report surveys. Structural Equation Modeling, 13, 440-464.
31. Doty, D. H., & Glick, W. H. (1998). Common method bias: Does common methods variance really bias results? Organizational Research Methods, 1, 374-406.
32. Eid, M. (2000). A multitrait-multimethod model with minimal assumptions. Psychometrika, 65, 241-261.
33. Eid, M., & Langeheine, R. (1999). The measurement of consistency and occasion specificity with latent class models: A new model and its application to the measurement of affect. Psychological Methods, 4, 100-116.
34. Fiske, D. W., & Campbell, D. T. (1992). Citations do not solve problems. Psychological Bulletin, 112, 393-395.
35. French, D. P., Marteau, T. M., Senior, V., & Weinman, J. (2005). How valid are measures of beliefs about the causes of illness? The example of myocardial infarction. Psychology and Health, 20, 615-635.
36. Funke, F. (2005). The dimensionality of right-wing authoritarianism: Lessons from the dilemma between theory and measurement. Political Psychology, 26, 195-218.
37. Glancy, M., & Little, S. L. (1995). Studying the social aspects of leisure-Development of the multiple-method field investigation model (MMFI). Journal of Leisure Research, 27, 305-325.
38. Guilford, J. P. (1954). Psychometric methods. New York: McGraw-Hill.
39. Harris, R. D. (2004). Organizational task environments: An evaluation of convergent and discriminant validity. Journal of Management Studies, 41, 857-882.
40. Hernández, A., & González-Romá, V. (2002). Analysis of multitrait-multioccasion data: Additive versus multiplicative models. Multivariate Behavioral Research, 37, 59-87.
41. Hoffman, B. J., & Woehr, D. J. (2007). Disentangling the meaning of multisource feedback: An examination of the nomological network surrounding source and dimension factors. Manuscript submitted for publication.
42. Horan, P. M., DiStefano, C., & Motl, R. W. (2003). Wording effects in selfesteem scales: Methodological artifact or response style? Structural Equation Modeling, 10, 444-455.
43. Ittenbach, F. R., Buison, A. M., Stallings, V. A., & Zemel, B. S. (2006). Statistical validation of air-displacement plethysmography for body composition assessment in children. Annals of Human Biology, 33, 187-201.
44. Jackson, D. J. R., Stillman, J. A., & Atkins, S. G. (2005). Rating tasks versus dimensions in assessment centers: A psychometric comparison. Human Performance, 18, 213-241.
45. Janssens, J. M. A. M., DeBruyn, E. E. J., Manders, W. A., & Scholte, R. H. J. (2005). The multitrait-multimethod approach in family assessment. European Journal of Psychological Assessment, 21, 232-239.
46. Kavanagh, M. J., MacKinney, A. C., & Wolins, L. (1971). Issues in managerial performance: Multitrait-multimethod analyses of ratings. Psychological Bulletin, 75, 34-49.
47. Kenny, D. A., & Berman, J. S. (1980). Statistical approaches to the correction of correlational bias. Psychological Bulletin, 88, 288-295.
48. Kenny, D. A., & Zautra, A. (2001). Trait-state models for longitudinal data. In L. M. Collins & A. G. Sayer (Eds.), New methods for the analysis of change (pp. 243-263). Washington, DC: American Psychological Association.
49. Ketokivi, M. A., & Schroeder, R. G. (2004). Perceptual measures of performance:Fact or fiction? Journal of Operations Management, 22, 247-264.
50. Kim, C., & Lee, H. (1997). Development of family triadic measures for children's purchase influence. Journal of Marketing Research, 34, 307-321.
51. King, L. M., Hunter, J. E., & Schmidt, F. L. (1980). Halo in a multidimensional forced-choice performance evaluation scale. Journal of Applied Psychology, 65, 507-516.
52. Krehan, K. D. (2001). An investigation of the validity of scores on locally developed performance measures in a school assessment program. Educational and Psychological Measurement, 61, 841-848.
53. Kuncel, N. R., Hezlett, S. A., & Ones, D. S. (2001). A comprehensive metaanalysis of the predictive validity of the graduate record examinations:Implications for graduate selection and performance. Psychological Bulletin, 127, 162-181.
54. Lance, C. E. (2008). Why assessment centers do not work the way they are supposed to. Industrial and Organizational Psychology, 1, 84-97.
55. Lance, C. E., Baxter, D., & Mahan, R. P. (2006). Multi-source performance measurement: A reconceptualization. In W. Bennett, C. E. Lance, & D. J. Woehr (Eds.), Performance measurement: Current perspectives and future challenges (pp. 49-76). Mahwah, NJ: Erlbam.
56. Lance, C. E., Foster, M. R., Gentry, W. A., & Thoresen, J. D. (2004). Assessor cognitive processes in an operational assessment center. Journal of Applied Psychology, 89, 22-35.
57. Lance, C. E., Lambert, T. A., Gewin, A. G., Lievens, F., & Conway, J. M. (2004). Revised estimates of dimension and exercise variance components in assessment center post-exercise dimension ratings. Journal of Applied Psychology, 89, 377-385.
58. Lance, C. E., Newbolt, W. H., Gatewood, R. D., Foster, M. R., French, N., & Smith, D. E. (2000). Assessment center exercise factors represent cross-situational specificity, not method bias. Human Performance, 13, 323-353.
59. Lance, C. E., Noble, C. L., & Scullen, S. E. (2002). A critique of the correlated trait-correlated method (CTCM) and correlated uniqueness (CU) models for multitrait-multimethod (MTMM) data. Psychological Methods, 7, 228-244.
60. Lance, C. E., & Sloan, C. E. (1993). Relationships between overall and life facet satisfaction: A multitrait-multimethod (MTMM) study. Social Indicators Research, 30, 1-15.
61. Lance, C. E., Teachout, M. S., & Donnelly, T. M. (1992). Specification of the criterion construct space: An application of hierarchical confirmatory factor analysis. Journal of Applied Psychology, 77, 437-452.
62. Lance, C. E., Woehr, D. J., & Meade, A. W. (2007). Case study: A Monte Carlo investigation of assessment center construct validity models. Organizational Research Methods, 10, 449-462.
63. Landy, F. L. (1986). Stamp collecting versus science: Validation as hypothesis testing. American Psychologist, 41, 1183-1192.
64. Lehmann, D. R. (1988). An alternative procedure for assessing convergent and discriminant validity. Applied Psychological Measurement, 12, 411-423.
65. Lievens, F. (2001). Assessors and use of assessment centre dimensions: A fresh look at a troubling issue. Journal of Organizational Behavior, 22, 203-221.
66. Lievens, F., & Conway, J. M. (2001). Dimension and exercise variance in assessment center scores: A large-scale evaluation of multitrait-multimethod studies. Journal of Applied Psychology, 86, 1202-1222.
67. Lim, B. C., & Ployhart, R. E. (2006). Assessing the convergent and discriminant validity of Goldberg's international personality item pool-A multitrait-multimethod examination. Organizational Research Methods, 9, 29-54.
68. London, M., & Smither, J. W. (1995). Can multi-source feedback change perceptions of goal accomplishment, self-evaluations, and performance-related outcomes? Theory-based applications and directions for research. Personnel Psychology, 48, 803-839.
69. Marsh, H. W. (1989). Confirmatory factor analysis of multitrait-multimethod data: Many problems and a few solutions. Applied Psychological Measurement, 13, 335-361.
70. Marsh, H. W., Ellis, L. A., Parada, R. H., Richards, G., & Heubeck, B. G. (2005). A short version of the Self Description Questionnaire II:Operationalizing criteria for short-form evaluation with new applications of confirmatory factor analysis. Psychological Assessment, 17, 81-102.
71. Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741-749.
72. Millsap, R. E. (1995). The statistical analysis of method effects in multitraitmultimethod data: A review. In P. E. Shrout & S. T. Fiske (Eds.), Personality research, methods, and theory: A festschrift honoring Donald W. Fiske (pp. 93-109). Hillsdale, NJ: Erlbaum.
73. Motl, R. W., & DiStefano, C. (2002). Longitudinal invariance of self-esteem and method effects associated with negatively worded items. Structural Equation Modeling, 9, 562-578.
74. Mount, M. K., Judge, T. A., Scullen, S. E., Sytsma, M. R., & Hezlett, S. A. (1998). Trait, rater, and level effects in 360-degree performance ratings. Personnel Psychology, 51, 557-576.
75. Pitoniak, M. J., Sireci, S. G., & Luecht, R. M. (2002). A multitrait-multimethod validity investigation of scores from a professional licensure examination. Educational and Psychological Measurement, 62, 498-516.
76. Podsakoff, P. M., MacKenzie, S. B., Podsakoff, N. P., & Lee, J. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88, 879-903.
77. Quilty, L. C., Oakman, J. M., & Risko, E. (2006). Correlates of the Rosenberg Self-Esteem Scale method effects. Structural Equation Modeling, 13, 99-117.
78. Rogers, R., Sewell, K. W., Ustad, K., Reinhardt, V., & Edwards, W. (1995). The referral decision scale with mentally disordered inmates-A preliminary study of convergent and discriminant validity. Law and Human Behavior, 19, 481-492.
79. Rosenberg, M. (1965). Society and the adolescent child. Princeton, NJ:Princeton University Press.
80. Sackett, P. R. (1987). Assessment centers and content validity: Some neglected issues. Personnel Psychology, 40, 13-25.
81. Sackett, P. R., & Dreher, G. F. (1982). Constructs and assessment center dimensions: Some troubling findings. Journal of Applied Psychology, 67, 401-410.
82. Schermelleh-Engel, K., Keith, N., Moosbrugger, H., & Hodapp, V. (2004). Decomposing person and occasion-specific effects: An extension of latent state-trait theory to hierarchical models. Psychological Methods, 9, 198-219.
83. Schleicher, D. J., Day, D. V., Mayes, B. T., & Riggio, R. E. (2002). A new frame for frame-of-reference training: Enhancing the construct validity of assessment centers. Journal of Applied Psychology, 87, 735-746.
84. Schmitt, N. (1978). Path analysis of multitrait-multimethod matrices. Applied Psychological Measurement, 2, 157-173.
85. Scullen, S. E., Mount, M. K., & Judge, T. A. (2003). Evidence of the construct validity of developmental ratings of managerial performance. Journal of Applied Psychology, 88, 50-66.
86. Spector, P. E. (2006). Method variance in organizational research: Truth or urban legend? Organizational Research Methods, 9, 221-232.
87. Stevens, S. S. (1946). On the theory of scales for measurement. Science, 103, 677-680.
88. Tomás, J. M., & Oliver, A. (1999). Rosenberg's self-esteem scale: Two factors or method effects. Structural Equation Modeling, 6, 84-98.
89. Tornow, W. W. (1993). Perceptions or reality: Is multi-perspective measurement a means or an end? Human Resource Management, 32, 221-229.
90. Viswesvaran, C., Ones, D. S., & Schmidt, F. L. (1996). Comparative analysis of the reliability of job performance ratings. Journal of Applied Psychology, 81, 557-574.
91. Wherry, R. J., & Bartlett, C. J. (1982). The control of bias in ratings: A theory of rating. Personnel Psychology, 35, 521-551.
92. Widaman, K. F. (1985). Hierarchically nested covariance structure models for multitrait-multimethod data. Applied Psychological Measurement, 9, 1-26.
93. Wothke, W. (1996). Models for multitrait-multimethod matrix analysis. In G. A. Marcoulides & R. E. Schumacker (Eds.), Advanced structural equation modeling: Issues and techniques (pp. 7-56). Mahwah, NJ: Erlbaum.
94. Yang, W., Lance, C. E., & Hui, H. C. (2006). Psychometric properties of the Chinese Self-Directed Search (1994 edition). Journal of Vocational Behavior, 68, 560-576.
95. Zedeck, S., Imparto, N., Krausz, M., & Oleno, T. (1974). Development of behaviorally anchored rating scales as a function of organizational level. Journal of Applied Psychology, 59, 249-252.