The other side of method bias: The perils of distinct source research designs

Multivariate Behavioral Research

Volumn 45, Issue 2, 2010, Pages 294-321

  Kammeyer Mueller, John ^a   Steel, Piers D G ^b   Rubenstein, Alex ^a  

^a UNIVERSITY OF FLORIDA   (United States)

^b UNIVERSITY OF CALGARY   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 77953660789     PISSN: 00273171     EISSN: None     Source Type: Journal    
DOI: 10.1080/00273171003680278     Document Type: Article

References (84)

1. Allen, D., & Fraser, B. J. (2007). Parent and student perceptions of classroom learning environment and its association with student outcomes. Learning Environments Research, 10, 67-82.
2. American Academy of Pediatrics: Committee on Quality Improvement, Subcommittee on Attention- Deficit/Hyperactivity Disorder. (2000). Clinical practice guideline: Diagnosis and evaluation of the child with attention-deficit/hyperactivity disorder. Pediatrics, 105, 1158-1170.
3. Barger, P., & Grandey, A. (2006). "Service with a smile" and encounter satisfaction: Emotional contagion and appraisal mechanisms. Academy of Management Journal, 49, 1229-1238.
4. Baugh, F. (2002). Correcting effect sizes for score reliability: A reminder that measurement and sub- stantive issues are linked inextricably. Educational and Psychological Measurement, 62, 254-263.
5. Berry, C. M., Ones, D. S., & Sackett, P. R. (2007). Interpersonal deviance, organizational deviance, and their common correlates: A review and meta-analysis. Journal of Applied Psychology, 92, 410-424.
6. Bing, M. N., Stewart, S. M., Davison, H., Green, P. D., McIntyre, M. D., & James L. R. (2007). An integrative typology of personality assessment for aggression: Implications for predicting counterproductive workplace behavior. Journal of Applied Psychology, 92, 722-744.
7. Bird, H. R., Gould, M. S., & Staghezza, B. (1992). Aggregating data from multiple informants in child psychiatry epidemiological research. Journal of the American Academy of Child & Adolescent Psychiatry, 31, 78-85.
8. Boring, E. G. (1961). The beginning and growth of measurement in psychology. Isis, 42, 238-257.
9. Brown, K. G., Le, H., & Schmidt, F. L. (2006). Specific aptitude theory revisited: Is there incremental validity for training performance? International Journal of Selection and Assessment, 14, 87-100.
10. Byrne, B. M., & Bazana, P. G. (1996). Investigating the measurement of social and academic competencies for early/late preadolescents and adolescents: A multitrait-multimethod analysis. Applied Measurement in Education, 9, 113-132.
11. Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait- multimethod matrix. Psychological Bulletin, 56, 81-105.
12. Campbell, J. (1982). Editorial: Some remarks from the outgoing editor. Journal of Applied Psychology, 67, 691-700.
13. Clement D. L., De Buyzere, M. L., De Bacquer, D. A., de Leeuw, P. W., Duprez, D. A., Fagard, & O'Brien, E. (2003). Office versus ambulatory pressure study investigators. Prognostic value of ambulatory blood-pressure recordings in patients with treated hypertension. New England Journal of Medicine, 348, 2407-2415.
14. Cortina, J., & DeShon, R. P. (1998). Determining relative importance of predictors with the obser- vational design. Journal of Applied Psychology, 83, 798-804.
15. Cote, J. A., & Buckley, R. (1987). Estimating trait, method, and error variance: Generalizing across 70 construct validation studies. Journal of Marketing Research, 24, 315-318.
16. Courvoisier, D. S., Nussbeck, F. W., Eid, M., Geiser, C., & Cole, D. A. (2008). Analyzing the convergent and discriminant validity of states and traits: Development and application of multimethod latent state-trait models. Psychological Assessment, 20, 270-280.
17. Cronbach, L.J., Gleser, G.C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York, NY: Wiley.
18. Cureton, E. E. (1971). The stability coefficient. Educational and Psychological Measurement, 31, 45-55.
19. Dalal, R. S. (2005). A meta-analysis of the relationship between organizational citizenship behavior and counterproductive work behavior. Journal of Applied Psychology, 90, 1241-1255.
20. Deinzer, R., Steyer, R., Eid, M., Notz, P., Schwenkmezger, P., Ostendorf, F., & Neubauer, A. (1995). Situational effects in trait assessment: The FPI, NEOFFI, and EPI questionnaires. European Journal of Personality, 9, 1-23.
21. DeShon, R. P. (2002). Generalizability theory. In F. Drasgow & N. Schmitt (Eds.), Measuring and analyzing behavior in organizations: Advances in measurement and data analysis (pp. 189-220). San Francisco, CA: Jossey-Bass.
22. Doty, D. H., & Glick, W. H. (1998). Common methods bias: Does common methods variance really bias results? Organizational Research Methods, 1, 374-406.
23. Doucet, L. (2004). Service provider hostility and service quality. Academy of Management Journal, 47,761-771.
24. Dunning, D., Heath, C., & Suls, J. M. (2004). Flawed self-assessment: Implications for health, education, and the workplace. Psychological Science in the Public Interest, 5, 69-106.
25. Emslie, G. J., & Mayes, T. L. (1999). Depression in children and adolescents: A guide to diagnosis and treatment. CNS Drugs, 11, 181-189.
26. Funder, D. C., & Colvin, C. R. (1997). Congruence of others' and self-judgments of personality. In R. Hogan, J. Johnson, & S. Briggs (Eds.), Handbook of personality psychology (pp. 617-647). San Diego, CA: Academic Press.
27. Hadorn, D. C., & Hays, R. D. (1991). Multitrait-multimethod analysis of health-related quality-of-life measures. Medical Care, 29, 829-840.
28. Hull, A. L., Hodder, S., Berger, B., Ginsberg, D., Lindheim, N., Quan, J., & Kleinhenz, M. E. (1995). Validity of three clinical performance assessments of internal medicine clerks. Academic Medicine, 70, 517-522.
29. Ilies, R., Scott, B. A., & Judge, T. A. (2006). The interactive effects of personal traits and experienced states on intraindividual patterns of citizenship behavior. Academy of Management Journal, 49, 561-575.
30. Janssen, O., & Van Yperen, N. W. (2004). Employees'goal orientations, the quality of leader-member exchange, and the outcomes of job performance and job satisfaction. Academy of Management Journal, 47, 368-384.
31. Johnson, J. W. (2004). Factors affecting relative weights: The influence of sampling and measurement error. Organizational Research Methods, 7, 283-299.
32. Jöreskog, K., & Sörbom, D. (2001). LISREL (Version 8.51) [Computer software]. Chicago, IL: Scientific Software International.
33. Judd, C. M., & McClelland, G. H. (1998). Measurement. In D. T. Gilbert, S. T. Fiske, & G. Lindzey (Eds.), Handbook of social psychology (4th ed., Vol. 1, pp. 180-232). New York, NY: McGraw- Hill.
34. Kammeyer-Mueller, J. D., Wanberg C. R., Glomb, T. M., & Ahlburg, D. (2005). The role of temporal shifts in turnover processes: It's about time. Journal of Applied Psychology, 90, 644-658.
35. Kenny, D. A. (1995). The multitrait-multimethod matrix: Design, analysis, and conceptual issues. In P. E. Shrout & S. T. Fiske (Eds.), Personality research, methods, and theory: A Festschrift honoring Donald W. Fiske (pp. 111-124). Hillsdale, NJ: Erlbaum.
36. Kenny, D. A., & Zautra, A. (2001). Trait-state models for longitudinal data. In L. M. Collins & A. G. Sayer (Eds.), New methods for the analysis of change (pp. 243-263). Washington, DC: American Psychological Association.
37. Kolar, D. W., Funder, D. C., & Colvin, C. R. (1996). Comparing the accuracy of personality judgments by the self and knowledgeable others. Journal of Personality, 64, 311-337.
38. Lance, C. E., Hoffman, B. J., Gentry, W. A., & Baranik, L. E. (2008). Rater source factors represent important subcomponents of the criterion construct space, not rater bias. Human Resource Management Review, 18, 223-232.
39. Lance, C. E., Noble, C. L, & Scullen, S. E. (2002). A critique of the correlated trait-correlated method (CTCM) and correlated uniqueness (CU) models for multitrait-multimethod (MTMM) data. Psychological Methods, 7, 228-244.
40. Lau, E., Dolovich, L., & Austin, Z. (2007). Comparison of self, physician, and simulated patient ratings of pharmacist performance in a family practice simulator. Journal of Interprofessional Care, 21, 129-140.
41. Levinson, M. H. (2004). Time-binding time: A history of time-measurement and time-management in America. A Review of General Semantics, 61, 9-19.
42. Lindell, M. K., & Whitney, D. J. (2001). Accounting for common method variance in cross-sectional research designs. Journal of Applied Psychology, 86, 114-121.
43. Littauer, S. B. (1960). On some aspects of technological measurement. Management Technology, 1, 36-45.
44. Lockyer, J. M., Violato, C., & Fidler, H. (2006). A multi source feedback program for anesthesiol- ogists. Canadian Journal of Anesthesia, 53, 33-39.
45. Marsh, H. W. (1989). Confirmatory factor analyses of multitrait-multimethod data: Many problems and a few solutions. Applied Psychological Measurement, 13, 335-361.
46. Mayer, R. C., & Gavin, M. B. (2005). Trust in management and performance: Who minds the shop while the employees watch the boss? Academy of Management Journal, 48, 874-888.
47. Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741-749.
48. Mielenz, T., Jackson, E., Currey, S., DeVellis, R,. & Callahan, L. F. (2006). Psychometric properties of the Centers for Disease Control and Prevention health-related quality of life (CDC HRQOL) items in adults with arthritis. Health and Quality of Life Outcomes, 4, 66.
49. Millsap, R. E. (1995). The statistical analysis of method effect in multitrait-multimethod data: A review. In P. E. Shrout & S. T. Fiske (Eds.), Personality research, methods, and theory: A Festschrift honoring Donald W. Fiske (pp. 93-109). Hillsdale, NJ: Erlbaum.
50. Muis, K. R., Winne, P. H., & Jamieson-Noel, D. (2007). Using a multitrait-multimethod analysis to examine conceptual similarities of three self-regulated learning inventories. British Journal of Educational Psychology, 77, 177-195.
51. Murphy, K. R., & DeShon, R. (2000). Interrater correlations do not estimate the reliability of job performance ratings. Personnel Psychology, 53, 873-900.
52. Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory. New York: McGraw-Hill.
53. Ostroff, C., Kinicki, A. J., & Clark, M. A. (2002). Substantive and operational issues of response bias across level of analysis: An example of climate-satisfaction relationships. Journal of Applied Psychology, 87, 355-368.
54. Papageorgiou, V., Kalyva, E., Dafoulis, V., & Vostanis, P. (2008). Differences in parents' and teachers' ratings of ADHD symptoms and other mental health problems. European Journal of Psychiatry, 22, 200-210.
55. Piccolo, R. F., & Colquitt, J. A. (2006). Transformational leadership and job behaviors: The mediating role of core job characteristics. Academy of Management Journal, 49, 327-340.
56. Piedmont, R. L., McCrae, R. R., Riemann, R., & Angleitner, A. (2000). On the invalidity of validity scales: Evidence from self-reports and observer ratings in volunteer samples. Journal of Personality and Social Psychology, 78, 582-593.
57. Podsakoff, P. M., MacKenzie, S. B., Lee, J.-Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: A critical review of literature and recommended remedies. Journal of Applied Psychology, 88, 879-903.
58. Raja, U., Johns, G., & Ntalianis, F. (2004). The impact of personality on psychological contracts. Academy of Management Journal, 47, 350-367.
59. Ree, M. J., & Carretta, T. R. (2006). The role of measurement error in familiar statistics. Organizational Research Methods, 9, 99-112.
60. Rogosa, D. (1980). A critique of cross-lagged correlation. Psychological Bulletin, 88, 245-258.
61. Rogosa, D., Floden, R., & Willett, J. B. (1984). Assessing the stability of teacher behavior. Journal of Educational Psychology, 76, 1000-1027.
62. Rubin, R. S., Munz, D. C., & Bommer, W. H. (2005). Leading from within: The effects of emotion recognition and personality on transformational leadership behavior. Academy of Management Journal, 48, 845-858.
63. Sackett, P. R., & Larson, J. R., Jr. (1990). Research strategies and tactics in industrial and or- ganizational psychology. In M. D. Dunette & L. M. Hough (Eds.), Handbook of industrial and organizationalpsychology(2nd ed., Vol. 1, pp. 419-489).Palo Alto, CA: Consulting Psychologists Press.
64. Shechtman, Z. (1992). A group assessment procedure as a predictor of on-the-job performance of teachers. Journal of Applied Psychology, 77, 383-387.
65. Schmidt, F. L., Hunter, J. E., & Caplan, J. R. (1981). Validity generalization results for two job groups in the petroleum industry. Journal of Applied Psychology, 66, 261-273.
66. Schuerger, J. M., Tait, E., & Tavernelli, M. (1982). Temporal stability of personality by questionnaire. Journal of Personality & Social Psychology, 43, 176-182.
67. Schuerger, J. M., Zarrella, K. L., & Hotz, A. S. (1989). Factors that influence the temporal stability of personality by questionnaire. Journal of Personality & Social Psychology, 56, 777-783.
68. Scullen, S. E., Mount, M. K., & Goff, M. (2000). Understanding the latent structure of job performance ratings. Journal of Applied Psychology, 85, 956-970.
69. Sobel, D. (1996). Longitude: The true story of a lone genius who solved the greatest scientific problem of his time. New York, NY: Penguin Books Ltd.
70. Spector, P. E. (2006). Method variance in organizational research. Truth or urban legend? Organizational Research Methods, 9, 221-232.
71. Steyer, R., Ferring, D., & Schmitt, M. J. (1992). States and traits in psychological assessment. European Journal of Psychological Assessment, 8, 79-98.
72. Steyer, R., Schmitt, M., & Eid, M. (1999). Latent state-trait theory and research in personality and individual differences. European Journal of Personality, 13, 389-408.
73. Tan, H. H., Foo, M. D., & Kwek, M. H. (2004). The effects of customer personality traits on the display of positive emotions. Academy of Management Journal, 47, 287-296.
74. Tannock, R. (2003). Neuropsychology of attention disorders. In S. J. Segalowitz & I. Rapin (Eds.), Handbook of neuropsychology (2nd ed., Vol. 8, Part II, p. 758). Amsterdam, The Netherlands: Elsevier.
75. Taylor, F. S. (1942). The origin of the thermometer. Annals of Science, 5, 129-156.
76. Tripp, G., Schaughency, E. A., & Clarke, B. (2006). Parent and teacher rating scales in the evaluation of attention-deficit/hyperactivity disorder: Contribution to diagnosis and differential diagnosis in clinically referred children. Developmental and Behavioral Pediatrics, 27, 209-218.
77. Vaidya, J. G., Gray, E. K., Haig, J., & Watson, D. (2002). On the temporal stability of personality: Evidence for differential stability and the role of life experiences. Journal of Personality & Social Psychology, 83, 1469-1484.
78. Verhulst, F. C. (1995). Recent developments in the assessment and diagnosis of child psychopathology. European Journal of Psychological Assessment, 11, 203-212.
79. Violato, C., Lockyer, J. M., & Fidler, H. (2008). Changes in performance: A 5-year longitudinal study of participants in a multi-source feedback programme. Medical Education, 42, 1007-1013.
80. Viswesvaran, C., Schmidt, F. L., & Ones, D. S. (2005). Is there a general factor in ratings of job performance? A meta-analytic framework for disentangling substantive and error influences. Journal of Applied Psychology, 90, 108-131.
81. Wall, T. D., & Wood, S. J. (2005). The romance of human resource management and business performance, and the case for big science. Human Relations, 58, 429-462.
82. Widaman, K. F. (1985). Hierarchically nested covariance structure models for multitrait-multimethod data. Applied Psychological Measurement, 9, 1-26.
83. Wilkinson, J. R., Crossley, J. G. M., Wragg, A., Mills, P., Cowan, G., & Wade, W. (2008). Implementing workplace-based assessment across the medical specialties in the United Kingdom. Medical Education, 42, 364-373.
84. Ziegert, J. C., & Hanges, P. J. (2005) Employment discrimination: The role of implicit attitudes, motivation, and a climate for racial bias. Journal of Applied Psychology, 90, 553-562.