Method effects, measurement error, and substantive conclusions

Organizational Research Methods

Volumn 13, Issue 3, 2010, Pages 435-455

  Lance, Charles E ^a   Dawson, Bryan ^a   Birkelbach, David ^a   Hoffman, Brian J ^a  

^a The University of Georgia ^*  (United States)

Author keywords

construct validation procedures; factor analysis; measurement design; measurement models; method variance

Indexed keywords

EID: 78049486567     PISSN: 10944281     EISSN: 15527425     Source Type: Journal    
DOI: 10.1177/1094428109352528     Document Type: Article

References (87)

1. Allen, J.G., & Hamsher, J.H. (1974). The development and validation of a test of emotional styles. Journal of Consulting and Clinical Psychology, 42, 663-668.
2. Arora, R. (1982). Validation of an S-O-R model for situation, enduring and response components of involvement. Journal of Marketing Research, 19, 505-516.
3. Arthur, W. Jr., Woehr, D.J., & Maldegan, R. (2000). Convergent and discriminant validity of assessment center dimensions: A conceptual and empirical reexamination of the assessment center construct related validity paradox. Journal of Management, 26, 813-835.
4. Avolio, B.J., Yammarino, F.J., & Bass, B.M. (1991). Identifying common methods variance with data collected from a single source: An unresolved sticky issue. Journal of Management, 17, 571-587.
5. Bagozzi, R.P., & Yi, Y. (1990). Assessing method variance in multitrait-multimethod matrices: The case of self-reported affect and perceptions at work. Journal of Applied Psychology, 75, 547-560.
6. Bentler, P.M., Jackson, D.N., & Messick, S. (1971). Identification of content and style: A two-dimensional interpretation of acquiescence. Psychological Bulletin, 76, 186-204.
7. Birnbaum, P.H., Farh, J., & Wong, G.Y.Y. (1986). The job characteristics model in Hong Kong. Journal of Applied Psychology, 71, 598-605.
8. Borman, W.C. (1974). The rating of individuals in organizations; An alternative approach. Organizational Behavior and Human Performance, 12, 105-124.
9. Bowler, M.C., & Woehr, D.J. (2006). A meta-analytic evaluation of the impact of dimension and exercise factors on assessment center ratings. Journal of Applied Psychology, 91, 1114-1124.
10. Brannick, M.T., & Spector, P.E. (1990). Estimation problems in the block-diagonal model of the multitrait-multimethod matrix. Applied Psychological Measurement, 14, 325-339.
11. Buckley, M.R., Cote, J.A., & Comstock, S.M. (1990). Measurement errors in the behavioral sciences: The case of personality/attitude research. Educational and Psychological Measurement, 50, 447-474.
12. Campbell, D.T., & Fiske, D.W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.
13. Campbell, J.P. (1982). Editorial: Some remarks from the outgoing editor. Journal of Applied Psychology, 67, 691-700.
14. Cattell, R.B. (1966). The data box: Its ordering of total resources in terms of possible relational systems. In R. B. Cattell (ed.), Handbook of multivariate experimental psychology (pp. 67-128). Chicago: Rand McNally.
15. Choi, N. (2004). A psychometric examination of the personal attributes questionnaire. Journal of Social Psychology, 144, 348-352.
16. Cole, D.A. (2006). Coping with longitudinal data in research on developmental psychopathology. International Journal of Behavioral Development, 30, 20-25.
17. Conway, J.M., & Huffcutt, A.I. (1997). Psychometric properties of multisource performance ratings: A meta-analysis of subordinate, supervisor, peer, and self-ratings. Human Performance, 10, 331-360.
18. Conway, J.M., Lievens, F., Scullen, S.E., & Lance, C.E. (2004). Bias in the correlated uniqueness model for MTMM data. Structural Equation Modeling, 11, 535-559.
19. Cote, J.A., & Buckley, M.R. (1987). Estimating trait, method and error variance: Generalizing across 70 construct validation studies. Journal of Marketing Research, 26, 315-318.
20. Crampton, S.M., & Wagner, J.A. III. (1994). Percept-percept inflation in microorganizational research: An investigation of prevalence and effect. Journal of Applied Psychology, 79, 67-76.
21. Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Fort Worth, TX: Harcourt, Brace, Jovanovich.
22. Cronbach, L.J., & Meehl, P.E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281-302.
23. D'Mello, S.D., & Graesser, A. (2009). Automatic language detection of learner's affect from gross body language. Applied Artificial Intelligence, 23, 123-150.
24. DiStefano, C., & Motl, R.W. (2006). Further investigating method effects associated with negatively worded items on self-report surveys. Structural Equation Modeling, 13, 440-464.
25. Doty, D.H., & Glick, W.H. (1998). Common methods bias: Does common methods variance really bias results? Organizational Research Methods, 1, 374-406.
26. Flamer, S. (1983). Assessment of the MTMM matrix validity of Likert scales via confirmatory factor analysis. Multivariate Behavioral Research, 18, 275-308.
27. Franic, D.M., & Bothe, A.K. (2008). Psychometric evaluation of condition-specific instruments used to assess health-related quality of life, attitudes, and related constructs in stuttering. American Journal of Speech-Language Pathology, 17, 60-80.
28. Frese, M., & Zapf, D. (1988). Methodological issues in the study of work stress: Objective vs. subjective measurement of work stress and the question of longitudinal studies. In C. L. Cooper & R. Payne (Eds.), Causes, coping and consequences at work (pp. 375-410). Chichester: Wiley.
29. Greenleaf, E.A. (1992a). Improving rating scale measures by detecting and correcting bias components in some response styles. Journal of Marketing Research, 29, 176-188.
30. Greenleaf, E.A. (1992b). Measuring extreme response style. Public Opinion Quarterly, 56, 328-351.
31. Hackman, J.R., & Oldham, G.R. (1976). Motivation through the design of work: Test of a theory. Organizational Behavior and Human Performance, 16, 250-279.
32. Heisenberg, W. (1927). Ãœber den anschaulichen Inhalt der quantentheoretischen Kinematik und Mechanik. [About the graphic content of quantum theoretic kinematics and mechanics]. Zeitschrift fÃ1/4r Physik, 43, 172-198.
33. Hewitt, A.K., Foxcroft, D.R., & MacDonald, J. (2004). Multitrait-multimethod confirmatory factory analysis of the attributional style questionnaire. Personality and Individual Differences, 37, 1483-1491.
34. Hoffman, B.J., Lance, C.E., Bynum, B.H., & Gentry, W.A. (in press). Rater source effects are alive and well after all. Personnel Psychology.
35. Hoffman, B.J., & Woehr, D.J. (2009). Disentangling the meaning of multisource feedback: Expanding the nomological network surrounding source and dimension factors. Personnel Psychology, 62, 735-765.
36. Hunter, J.E., & Schmidt, F.L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks, CA: Sage.
37. Johnson, S.M., Smith, P.C., & Tucker, S.M. (1982). Response format of the Job Descriptive Index: Assessment of reliability and validity by the multitrait-multimethod matrix. Journal of Applied Psychology, 67, 500-505.
38. Jöreskog, K.G., & Sörbom, D. (1993). LISREL 8 user's reference guide. Chicago: Scientific Software.
39. Keaveny, T.J., & McGann, A.F. (1975). A comparison of behavioral expectation scales and graph rating scales. Journal of Applied Psychology, 60, 695-703.
40. Kenny, D.A., & Kashy, D.A. (1992). Analysis of the multitrait-multimethod matrix by confirmatory factor analysis. Psychological Bulletin, 112, 165-172.
41. Kenny, D.A., & Zautra, A. (2001). Trait-state models for longitudinal data. In L. M. Collins & A. G. Sayer (Eds.), New methods for the analysis of change (pp. 243-263). Washington, DC: American Psychological Association.
42. King, L.M., Hunter, J.E., & Schmidt, F.L. (1980). Halo in a multidimensional forced-choice performance evaluation scale. Journal of Applied Psychology, 65, 507-516.
43. Kothandapani, V. (1971). Validation of feeling, belief, and intention to act as three components of attitude and their contribution to prediction of contraceptive behavior. Journal of Personality and Social Psychology, 19, 321-333.
44. Lance, C.E. (2008a). Why assessment centers do not work the way they are supposed to. Industrial and Organizational Psychology, 1, 84-97.
45. Lance, C.E. (2008b). Where have we been, how did we get there, and where shall we go? Industrial and Organizational Psychology, 1, 140-146.
46. Lance, C.E., Baranik, L.E., Lau, A.R., & Scharlau, E.A. (2009). If it ain't trait it must be method: (Mis)application of the multitrait-multimethod methodology in organizational research. In C. E. Lance & R. J. Vandenberg (Eds.), Statistical and methodological myths and urban legends: Received doctrine, verity, and fable in organizational and social research (pp. 339-362). New York: Routledge.
47. Lance, C.E., Baxter, D., & Mahan, R.P. (2006). Multi-source performance measurement: A reconceptualization. In W. Bennett, C. E. Lance, & D. J. Woehr (Eds.), Performance measurement: Current perspectives and future challenges (pp. 49-76). Mahwah, NJ: Erlbaum.
48. Lance, C.E., Hoffman, B.J., Gentry, W.A., & Baranik, L.E. (2008). Rater source factors represent important subcomponents of the criterion construct space, not rater bias. Human Resource Management Review, 18, 223-232.
49. Lance, C.E., Lambert, T.A., Gewin, A.G., Lievens, F., & Conway, J.M. (2004). Revised estimates of dimension and exercise variance components in assessment center post-exercise dimension ratings. Journal of Applied Psychology, 89, 377-385.
50. Lance, C.E., Newbolt, W.H., Gatewood, R.D., Foster, M.R., French, N., & Smith, D.E. (2000). Assessment center exercise factors represent cross-situational specificity, not method bias. Human Performance, 13, 323-353.
51. Lance, C.E., Noble, C.L., & Scullen, S.E. (2002). A critique of the correlated trait - correlated method (CTCM) and correlated uniqueness (CU) models for multitrait-multimethod (MTMM) data. Psychological Methods, 7, 228-244.
52. Lance, C.E., & Sloan, C.E. (1993). Relationships between overall and life facet satisfaction: A multitrait-multimethod (MTMM) study. Social Indicators Research, 30, 1-15.
53. Lance, C.E., Teachout, M.S., & Donnelly, T.M. (1992). Specification of the criterion construct space: An application of hierarchical confirmatory factor analysis. Journal of Applied Psychology, 77, 437-452.
54. Lance, C.E., & Vandenberg, R.J. (2009). Statistical and methodological myths and urban legends: Doctrine, verity, and fable in organizational and social research. New York: Routledge.
55. Lievens, F., Chasteen, C.S., Day, E.A., & Christiansen, N.D. (2006). Large-scale investigation of the role of trait activation theory for understandning assessment center convergent and discriminant validity. Journal of Applied Psychology, 91, 247-258.
56. Lievens, F., & Conway, J.M. (2001). Dimension and exercise variance in assessment center scores: A large-scale evaluation of multitrait-multimethod studies. Journal of Applied Psychology, 86, 1202-1222.
57. Lindell, M.K., & Whitney, D.J. (2001). Accounting for common method variance in cross-sectional research designs. Journal of Applied Psychology, 86, 114-121.
58. London, M., & Tornow, W.W. (1998). Maximizing the value of 360-degree feedback. Greensboro, NC: Center for Creative Leadership.
59. Meehl, P.E. (1990a). Why summaries of research on psychological theories are often uninterpretable. Psychological Reports, 66, 195-244.
60. Meehl, P.E. (1990b). Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant it. Psychological Inquiry, 1, 108-141.
61. Messiah, A. (1961-1962). Quantum mechanics. New York: Interscience.
62. Mosher, D.L. (1966). The development and multitrait-multimethod matrix analysis of three measures of three aspects of guilt. Journal of Consulting Psychology, 30, 25-29.
63. Mount, M.K., Judge, T.A., Scullen, S.E., Sytsma, M.R., & Hezlett, S.A. (1998). Trait, rater, and level effects in 360-degree performance ratings. Personnel Psychology, 51, 557-576.
64. Ostroff, C., Kinicki, A.J., & Clark, M.A. (2002). Substantive and operational issues of response bias across levels of analysis: An example of climate-satisfaction relationships. Journal of Applied Psychology, 87, 355-368.
65. Palmer, J.K., & Loveland, J.M. (2004). Further investigation of the psychometric properties of saucier's big five "mini-marker:" Evidence for criterion and construct validity. Individual Differences Research, 2, 231-238.
66. Pitoniak, M.J., Sireci, S.G., & Leucht, R.M. (2002). A multitrait-multimethod validity investigation of scores from a professional licensure examination. Educational and Psychological Measurement, 62, 498-516.
67. Podsakoff, P.M., MacKenzie, S.B., Lee, J.-Y., & Podsakoff, N.P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88, 879-903.
68. Quilty, L.C., Oakman, J.M., & Risko, E. (2006). Correlates of the Rosenberg Self-Esteem Scale method effects. Structural Equation Modeling, 13, 99-117.
69. Rindskopf, D. (1983). Parameterizing inequality constraints on unique variances in linear structural models. Psychometrika, 48, 73-83.
70. Rose, G.L., & Andiappan, P. (1978). Sex effects on managerial hiring decisions. Academy of Management Journal, 21, 104-112.
71. Sackett, P.R., & Dreher, G.F. (1982). Constructs and assessment center dimensions: Some troubling findings. Journal of Applied Psychology, 67, 401-410.
72. Schermelleh-Engel, K., Keith, N., Moosbrugger, H., & Hodapp, V. (2004). Decomposing person and occasion-specific effects: An extension of latent state-trait theory to hierarchical models. Psychological Methods, 9, 198-219.
73. Scullen, S.E., Mount, M.K., & Goff, M. (2000). Understanding the latent structure of job performance ratings. Journal of Applied Psychology, 85, 956-970.
74. Seligson, J.L., Huebner, E.S., & Valois, R.F. (2003). Preliminary validation of the brief multidimensional student life satisfaction scale (bmslss). Social Indication Research, 61, 121-145.
75. Spector, P.E. (1987). Method variance as an artifact in self-reported affect and perceptions at work: Myth or significant problem? Journal of Applied Psychology, 72, 438-443.
76. Spector, P.E. (2006). Method variance in organizational research: Truth or urban legend? Organizational Research Methods, 9, 221-232.
77. Stevens, S.S. (1946). On the theory of scales of measurement. Science, 103, 677-680.
78. Thompson, V.C. & Berven, N.L. (2002). Development of an instrument to measure cultural attitudes and behaviors affecting vocational rehabilitation. Rehabilitation Counseling Bulletin, 45, 76-104.
79. Vandenberg, R.J. (2006). Statistical and methodological myths and urban legends: Where, pray tell, did they get this idea? Organizational Research Methods, 9, 194-201.
80. Viswesvaran, C., Ones, D.S., & Schmidt, F.L. (1996). Comparative analysis of the reliability of job performance ratings. Journal of Applied Psychology, 81, 557-574.
81. Weems, G.H., & Onwuegbuzie, A.J. (2001). The impact of midpoint responses and reverse coding on survey data. Measurement and Evaluation in Counseling and Development, 34, 166-176.
82. Weems, G.H., Onwuegbuzie, A.J., Schreiber, J.B., & Eggers, S.J. (2003). Characteristics of respondents who respond differently to positively and negatively worded items on rating scales. Assessment & Evaluation in Higher Education, 28, 587-607.
83. Wherry, R.J., & Bartlett, C.J. (1982). The control of bias in ratings: A theory of rating. Personnel Psychology, 35, 521-551.
84. Williams, L.J., & Anderson, S.E. (1994). An alternative approach to method effects by using latent-variable models: Applications in organizational behavior research. Journal of Applied Psychology, 79, 323-331.
85. Williams, L.J., Cote, J.A., & Buckley, M.R. (1989). Lack of method variance in self-reported affect and perceptions at work: Reality or artifact? Journal of Applied Psychology, 74, 462-468.
86. Woehr, D.J., & Arthur, W. Jr. (2003). The construct related validity of assessment center ratings: A review and meta-analysis of the role of methodological factors. Journal of Management, 29, 231-258.
87. Yang, W., Lance, C.E., & Hui, H.C. (2006). Psychometric properties of the Chinese self-directed search (1994 ed.). Journal of Vocational Behavior, 68, 560-576.