The measurement equivalence of Web-based and paper-and-pencil measures of transformational leadership: A multinational test

Organizational Research Methods

Volumn 9, Issue 3, 2006, Pages 339-368

  Cole, Michael S ^a   Bedeian, Arthur G ^b   Feild, Hubert S ^c,d  

^a UNIVERSITY OF ST GALLEN   (Switzerland)

^b LOUISIANA STATE UNIVERSITY   (United States)

^c AUBURN UNIVERSITY   (United States)

^d AUBURN UNIVERSITY   (United States)

Author keywords

Measurement invariance; Transformational leadership

Indexed keywords

EID: 33745026012     PISSN: 10944281     EISSN: 15527425     Source Type: Journal    
DOI: 10.1177/1094428106287434     Document Type: Article

References (74)

1. American Psychological Association. (2002). Ethical principles of psychologists and code of conduct. American Psychologist, 57, 1060-1073.
2. Antonakis, J., Avolio, B. J., & Sivasubramaniam, N. (2003). Context and leadership: An examination of the nine-factor full-range leadership theory using the multifactor leadership questionnaire. Leadership Quarterly, 14, 261-295.
3. Arbuckle, J. L. (1996). Full information estimation in the presence of missing data. In G. A. Marcoulides & R. E. Schumaker (Eds.), Advanced structural equation modeling: Issues and techniques (pp. 243-277). Mahwah, NJ: Erlbaum.
4. Arbuckle, J. L., & Wothke, W. (1999). Amos 4.0 user's guide. Chicago: Small Waters.
5. Asci, F. H., Eklund, R. C., Whitehead, J. R., Kirazci, S., & Koca, C. (2005). Use of the CY-PSPP in other cultures: A preliminary investigation of its factorial validity for Turkish children and youth. Psychology of Sport and Exercise, 6, 33-50.
6. Avolio, B. J., Bass, B. M., & Jung, D. I. (1999). Re-examining the components of transformational and transactional leadership using the Multifactor Leadership Questionnaire. Journal of Occupational and Organizational Psychology, 72, 441-462.
7. Bass, B. M. (1985). Leadership and performance beyond expectations. New York: Free Press.
8. Bass, B. M. (1997). Does the transactional-transformational leadership paradigm transcend organizational and national boundaries? American Psychologist, 52, 130-139.
9. Bass, B. M., & Avolio, B. J. (2000). Multifactor Leadership Questionnaire: Technical report, leader form, rater form, and scoring key for MLQ form 5x-short (2nd ed.). Redwood City, CA: Mindgarden.
10. Booth-Kewley, S., Edwards, J. E., & Rosenfeld, P. (1992). Impression management, social desirability, and computer administration of attitude questionnaires: Does the computer make the difference? Journal of Applied Psychology, 77, 562-566.
11. Brannick, M. T. (1995). Critical comments on applying covariance structure modeling. Journal of Organizational Behavior, 16, 201-213.
12. Buchanan, T. (2002). Online assessment: Desirable or dangerous? Professional Psychology: Research and Practice, 33, 148-154.
13. Buchanan, T., Johnson, J. A., & Goldberg, L. R. (2005). Implementing a five-factor personality inventory for use on the Internet. European Journal of Psychological Assessment, 21, 115-127.
14. Buchanan, T., & Smith, J. L. (1999). Using the Internet for psychological research: Personality testing on the World Wide Web. British Journal of Psychology, 90, 125-144.
15. Bycio, P., Hackett, R. D., & Allen, J. S. (1995). Further assessments of Bass's (1985) conceptualization of transactional and transformational leadership. Journal of Applied Psychology, 80, 468-478.
16. Byrne, B. M. (2001). Structural equation modeling with AMOS: Basic concepts, applications, and programming. Mahwah, NJ: Erlbaum.
17. Byrne, B. M., Shavelson, R. J., & Muthén, B. (1989). Testing for equivalence of factor covariance and mean structures: The issue of partial measurement invariance. Psychological Bulletin, 105, 456-466.
18. Carless, S. A. (1998). Assessing the discriminant validity of transformational leadership behavior as measured by the MLQ. Journal of Occupational and Organizational Psychology, 71, 353-358.
19. Chan, D. (1998). Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models. Journal of Applied Psychology, 83, 234-246.
20. Chen, F., Bollen, K. A., Paxton, P., Curran, P. J., & Kirby, J. B. (2001). Improper solutions in structural equation models: Causes, consequences, and strategies. Sociological Methods and Research, 29, 468-508.
21. Cheung, G. W., & Rensvold, R. B. (1999). Testing factorial invariance across groups: A reconceptualization and proposed new method. Journal of Management, 25, 1-27.
22. Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9, 233-255.
23. Cohen, R. J., Swerdlik, M. E., & Phillips, S. M. (1996). Psychological testing and assessment (3rd ed.). Mountain View, CA: Mayfield.
24. Den Hartog, D. N., House, R. J., Hanges, P. J., Ruiz-Quintanilla, S. A., & Dorfman, P. W. (1999). Culture specific and cross-culturally generalizable implicit leadership theories: Are attributes of charisma/transformational leadership universally endorsed? Leadership Quarterly, 10, 219-256.
25. Den Hartog, D. N., Van Muijen, J. J., & Koopman, P. L. (1997). Transactional versus transformational leadership: An analysis of the MLQ. Journal of Occupational and Organizational Psychology, 70, 19-34.
26. Dillon, W. R., Kumar, A., & Mulani, N. (1987). Offending estimates in covariance structure analysis: Comments on the causes of and solutions to Heywood cases. Psychological Bulletin, 101, 126-135.
27. Donovan, M. A., Drasgow, F., & Probst, T. M. (2000). Does computerizing paper-and-pencil job attitude scales make a difference? New IRT analyses offer insight. Journal of Applied Psychology, 85, 305-313.
28. Drasgow, F. (1984). Scrutinizing psychological tests: Measurement equivalence and equivalent relations with external variables are the central issue. Psychological Bulletin, 95, 134-135.
29. Eaton, J., & Struthers, C. W. (2002). Using the internet for organizational research: A study of cynicism in the workplace. CyberPsychology and Behavior, 5, 305-313.
30. Epstein, J., Klinkenberg, W. D., Wiley, D., & McKinley, L. (2001). Insuring sample equivalence across Internet and paper-and-pencil assessments. Computers in Human Behavior, 17, 339-346.
31. Evan, W. M., & Miller, J. R. (1969). Differential effects on response bias of computer vs. conventional administration of a social science questionnaire: An exploratory methodological experiment. Behavioral Science, 14, 216-227.
32. Fouladi, R. T., McCarthy, C. J., & Moller, N. P. (2002). Paper-and-pencil or online? Evaluating mode effects on measures of emotional functioning and attachment. Assessment, 9, 204-215.
33. French, D. J., & Tait, R. J. (2004). Measurement invariance in the General Health Questionnaire-12 in young Australian adolescents. European Child and Adolescent Psychiatry, 13, 1-7.
34. Gosling, S. D., Vazire, S., Srivastava, S., & John, O. P. (2004). Should we trust Web-based studies? A comparative analysis of six preconceptions about Internet questionnaires. American Psychologist, 59, 93-104.
35. Herrero, J., & Meneses, J. (in press). Short Web-based versions of the Perceived Stress (PSS) and Center for Epidemiological Studies-Depression (CESD) scales: A comparison to pencil-and-paper responses among Internet users. Computers in Human Behavior.
36. Hogg, A. (2002). Conducting online research. Burk Incorporated White Paper Series, 3 (2). Retrieved July 8, 2004, from http://www.burke.com/ whitepapers/PDF/B.WhitePaperVol3-2002-Iss2.pdf
37. Hollenbeck, J. R., Klein, H. J., O' Leary, A. M., & Wright, P. M. (1989). Investigation of the construct validity of a self-report measure of goal commitment. Journal of Applied Psychology, 74, 951-956.
38. Hu, L., & Bentler, P. M. (1998). Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychological Methods, 3, 424-453.
39. King, W. C., Jr., & Miles, E. W. (1995). A quasi-experimental assessment of the effect of computerizing noncognitive paper-and-pencil measurements: A test of measurement equivalence. Journal of Applied Psychology, 80, 643-651.
40. Knapp, H., & Kirk, S. A. (2003). Using pencil and paper, Internet, and touch-tone phones for self-administered surveys: Does methodology matter? Computers in Human Behavior, 19, 117-134.
41. Kraut, R., Olson, J., Banaji, M., Bruckman, A., Cohen, J., & Couper, M. (2004). Psychological research online: Report of board of scientific affairs' advisory group on the conduct of research on the Internet. American Psychologist, 59, 105-117.
42. Lautenschlager, G. J., & Flaherty, V. L. (1990). Computer administration of questions: More desirable or more social desirability? Journal of Applied Psychology, 75, 310-314.
43. Lawley, D. N., & Maxwell, A. E. (1971). Factor analysis as a statistical method (2nd ed.). London: Butterworth.
44. Lievens. F., & Anseel, F. (2004). Confirmatory factor analysis and invariance of an organizational citizenship behavior measure across samples in a Dutch-speaking context. Journal of Occupational and Organizational Psychology, 77, 299-306.
45. Little, T. D. (1997). Mean and covariance structures (MACS) analyses of cross-cultural data: Practical and theoretical issues. Multivariate Behavioral Research, 32, 53-76.
46. Lowe, K. B., Kroeck, K. G., & Sivasubramaniam, N. (1996). Effectiveness correlates of transformation and transactional leadership: A meta-analytic review of the MLQ literature. Leadership Quarterly, 7, 385-425.
47. Mantzicopoulos, P., French, B. F., & Maller, S. J. (2004). Factor structure of the pictorial scale of perceived competence and social acceptance with two pre-elementary samples. Child Development, 75, 1214-1228.
48. Marsh, H. W., Balla, J. R., & McDonald, R. P. (1988). Goodness-of-fit indexes in confirmatory factor analysis: The effect of sample size. Psychological Bulletin, 103, 391-410.
49. Meade, A. W., & Lautenschlager, G. J. (2004a). A comparison of item response theory and confirmatory factor analytic methodologies for establishing measurement equivalence/invariance. Organizational Research Methods, 7, 361-388.
50. Meade, A. W., & Lautenschlager, G. J. (2004b). A Monte-Carlo study of confirmatory factor analytic tests of measurement equivalence/invariance. Structural Equation Modeling, 11, 60-72.
51. Millsap, R. E., & & Kwok, O. (2004). Evaluating the impact of partial factorial invariance on selection in two populations. Psychological Methods, 9, 93-115.
52. Mullen, M. R. (1995). Diagnosing measurement equivalence in cross-national research. Journal of International Business Studies, 26, 573-596.
53. Naglieri, J. A., Drasgow, F., Schmit, M., et al. (2004). Psychological testing on the Internet: New problems, old issues. American Psychologist, 59, 150-162.
54. Ployhart, R. E., & Oswald, F. L. (2004). Applications of mean and covariance structure analysis: Integrating correlational and experimental approaches. Organizational Research Methods, 7, 27-65.
55. Ployhart, R. E., Wiechmann, D., Schmitt, N., Sacco, J. M., & Rogg, K. (2003). The cross-cultural equivalence of job performance ratings. Human Performance, 16, 49-79.
56. Potosky, D., & Bobko, P. (1997). Computer versus paper-and-pencil administration mode and response distortion in noncognitive selection tests. Journal of Applied Psychology, 82, 293-299.
57. Richman, W. L., Kiesler, S., Weisband, S., & Drasgow, F. (1999). A meta-analytic study of social desirability distortion in computer-administered questionnaires, traditional questionnaires, and interviews. Journal of Applied Psychology, 84, 754-775.
58. Riggs, M. L., & Knight, P. A. (1994). The impact of perceived group success-failure on motivational beliefs and attitudes: A causal study. Journal of Applied Psychology, 79, 755-766.
59. Riordan, C. M., & Vandenberg, R. J. (1994). A central question in cross-cultural research: Do employees of different cultures interpret work-related measures in an equivalent manner? Journal of Management, 20, 643-671.
60. Riordan, C. M., & Weatherly, E. W. (1999). Defining and measuring employees' identification with their work groups. Educational and Psychological Measurement, 59, 310-324.
61. Rosenfeld, P., Booth-Kewley, S., Edwards, J. E., & Thomas, M. D. (1996). Responses on computer surveys: Impression management, social desirability, and the Big Brother syndrome. Computers in Human Behavior, 12, 263-274.
62. Ryan, A. M., Chan, D., Ployhart, R. E., & Slade, L. A. (1999). Employee attitude surveys in a multinational organization: Considering language and culture in assessing measurement equivalence. Personnel Psychology, 52, 37-58.
63. Scandura, T., & Dorfman, P. (2004). Leadership research in an international and cross-cultural context. Leadership Quarterly, 15, 277-307.
64. Simsek, Z., & Veiga, J. F. (2001). A primer on Internet organizational surveys. Organizational Research Methods, 4, 218-235.
65. Stanton, J. M. (1998). An empirical assessment of data collection using the Internet. Personnel Psychology, 51, 709-725.
66. Stanton, J. M., & Rogelberg, S. G. (2001). Using Internet/Intranet Web pages to collect organizational research data. Organizational Research Methods, 4, 200-217.
67. Steenkamp, J. B. E. M., & Baumgartner, H. (1998). Assessing measurement invariance in cross-national consumer research. Journal of Consumer Research, 25, 78-90.
68. Synodinos, N. E., & Brennan, J. M. (1990). Evaluating microcomputer interactive survey software. Journal of Business and Psychology, 4, 483-493.
69. Thompson, L. F., Surface, E. A., Martin, D. L., & Sanders, M. G. (2003). From paper to pixels: Moving personnel surveys to the Web. Personnel Psychology, 56, 197-227.
70. Vandenberg, R. J. (2002). Toward a further understanding of and improvement in measurement invariance methods and procedures. Organizational Research Methods, 5, 139-158.
71. Vandenberg, R. J., & Lance, C. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4-70.
72. Whitener, E. M., & Klein, H. J. (1995). Equivalence of computerized and traditional research methods: The roles of scanning, social environment, and social desirability. Computers in Human Behavior, 11, 65-75.
73. Wothke, W. (2000). Longitudinal and multigroup modeling with missing data. In T. D. Little, K. U. Schnable, & J. Baumert (Eds.), Modeling longitudinal and multilevel data (pp. 219-240). Mahwah, NJ: Erlbaum.
74. Zaccaro, S. J., Rittman, A. L., & Marks, M. A. (2001). Team leadership. Leadership Quarterly, 12, 451-483.