Measurement equivalence of paper-and-pencil and internet organisational surveys: A large scale examination in 16 countries

Applied Psychology

Volumn 58, Issue 2, 2009, Pages 336-361

  De Beuckelaer, Alain ^a   Lievens, Filip ^b  

^a RADBOUD UNIVERSITY NIJMEGEN   (Netherlands)

^b GHENT UNIVERSITY   (Belgium)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 61849144207     PISSN: 0269994X     EISSN: 14640597     Source Type: Journal    
DOI: 10.1111/j.1464-0597.2008.00350.x     Document Type: Article

References (61)

1. Bentler, P.M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107, 238 246.
2. Bentler, P.M. (1995). EQS structural equations program manual. Encino, CA : Multivariate Software.
3. Bentler, P.M. Bonett, D.G. (1980). Significance tests and goodness of fit in the analysis of covariance structure. Psychological Bulletin, 88, 588 606.
4. Birnbaum, M.H. (2001). Introduction to behavioral research on the internet. Upper Saddle River, NJ : Prentice Hall.
5. Booth-Kewley, S., Edwards, J.E. Rosenfeld, P. (1992). Impression management, social desirability, and computer administration of attitude questionnaires: Does this make a difference? Journal of Applied Psychology, 77, 562 566.
6. Bowling, A. (2005). Mode of questionnaire administration can have serious effects on data quality. Journal of Public Health, 27, 281 291.
7. Brannick, M.T. (1995). Critical comments on applying covariance structure modeling. Journal of Organizational Behavior, 16, 201 213.
8. Buchanan, T. Smith, J.L. (1999). Using the internet for psychological research: Personality testing on the World Wide Web. British Journal of Psychology, 90, 125 144.
9. Chan, D. (2000). Detection of differential item functioning on the Kirton adaption-innovation inventory using multi-group mean and covariance structure analyses. Multivariate Behavioral Research, 35, 169 199.
10. Cheung, G.W. Rensvold, R.B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling: An Interdisciplinary Journal, 9, 233 255.
11. Church, A.H. Waclawski, J. (2001). Designing and using organizational surveys: A seven-step process. San Francisco, CA : Jossey-Bass.
12. Cole, M.S., Bedeian, A.G. Field, H.S. (2006). The measurement equivalence of web-based and paper-and-pencil measures of transformational leadership. Organizational Research Methods, 9, 339 368.
13. Cook, C., Heath, F., Thompson, R.L. Thompson, B. (2001). Score reliability in Web-or-Internet-based surveys: Unnumbered graphic rating scales versus Likert-type scales. Educational and Psychological Measurement, 61, 697 706.
14. Dillman, D. (2000). Mail and internet surveys: The tailored design method (2nd edn.). New York : J. Wiley.
15. Drasgow, F. (1984). Scrutinizing psychological tests: Measurement equivalence and equivalent relations with external variables are central issues. Psychological Bulletin, 95, 134 135.
16. Drasgow, F. (1987). Study of measurement bias of two standardized psychological tests. Journal of Applied Psychology, 72, 19 29.
17. Epstein, J., Klinkenberg, W.D., Wiley, D. McKinley, L. (2001). Ensuring sampling equivalence across Internet and paper-and-pencil assessments. Computers in Human Behavior, 17, 339 346.
18. Fenlason, K.J. Suckow-Zimberg, K. (2006). Online surveys: Critical issues in using the web to conduct surveys. In A.I. Kraut (Ed.), Getting action from organizational surveys (pp. 183 212). San Francisco, CA : Jossey-Bass.
19. Ferrando, P.J. Lorenzo-Seva, U. (2005). IRT-based factor analytic procedures for testing the equivalence of paper-and-pencil and internet-administered questionnaires. Psychological Methods, 10, 193 205.
20. Gerbing, D.W. Hamilton, J.G. (1996). Viability of exploratory factor analysis as a precursor to confirmatory factor analysis. Structural Equation Modeling: An Interdisciplinary Journal, 3, 62 72.
21. Hair J.F., Jr., Black, W.C., Babin, B.J., Anderson, R.E. Tatham, R.L. (2006). Multivariate data analysis (6th edn.). Upper Saddle River, NJ : Pearson Prentice Hall.
22. Hu, L. Bentler, P.M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: An Interdisciplinary Journal, 6, 1 55.
23. Hurley, A.E., Scandura, T.A., Schriesheim, C.A., Brannick, M.T., Seers, A., Vandenberg, R.J. Williams, L.J. (1997). Exploratory and confirmatory factor analysis: Guidelines, issues, and alternatives. Journal of Organizational Behavior, 18, 667 683.
24. James, L.R. Jones, A.P. (1974). Organizational climate: A review of theory and research. Psychological Bulletin, 81, 1096 1112.
25. James, L.R. McIntyre, M.D. (1996). Perceptions of organizational climate. In K. Murphy (Ed.), Individual differences and behavior in organizations (pp. 416 450). San Francisco, CA : Jossey-Bass.
26. James, L.R. Sells, S.B. (1981). Psychological climate: Theoretical perspectives and empirical research. In D. Magnusson (Ed.), Toward a psychology of situations: An interactional perspective (pp. 275 295). Hillsdale, NJ : Erlbaum.
27. Johnson, S.R. (1996). The multinational opinion survey. In A.I. Kraut (Ed.), Organizational surveys: Tools for assessment and change (pp. 310 329). San Francisco, CA : Jossey-Bass.
28. Johnson, T., Kulesa, P., Ik Cho, P. Shavitt, S. (2005). The relation between culture and response styles: Evidence from 19 countries. Journal of Cross-Cultural Psychology, 36, 264 277.
29. Kelloway, E.K. (1995). Structural equation modeling in perspective. Journal of Organizational Behavior, 16, 215 224.
30. Kopelman, R.E., Brief, A.P. Guzzo, R.A. (1990). The role of climate and culture in productivity. In B. Schneider (Ed.), Organisational climate and culture (pp. 282 318). San Francisco, CA : Jossey-Bass.
31. Kraut, A.I. (1996). Organizational surveys: Tools for assessment and change. San Francisco, CA : Jossey-Bass, Little.
32. Kraut, A.I. Saari, L.M. (1999). Organizational surveys: Coming of age for a new era. In A.I. Kraut A.K. Korman (Eds.), Evolving practices in human resource management (pp. 302 327). San Francisco, CA : Jossey-Bass.
33. Lautenschlager, G.J. Flaherty, V.L. (1990). Computer administration of questions: More desirable or more social desirability? Journal of Applied Psychology, 75, 310 314.
34. Little, T.D. (1997). Mean and covariance (MACS) analysis of cross-cultural data: Practical and theoretical issues. Multivariate Behavioural Research, 32, 53 76.
35. Mann, C. Stewart, F. (2000). Internet communication and qualitative research: A handbook for researching online. London : Sage.
36. Meredith, W. (1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58, 525 543.
37. Muthén, L.K. Muthén, B.O. (1999). Mplus: The comprehensive modeling program for applied researchers (User's guide - 2nd printing). Los Angeles, CA : Muthén and Muthén.
38. Muthén, L.K. Muthén, B.O. (2003). Mplus version 2.13: Addendum to the Mplus user's guide. Los Angeles, CA : Muthén and Muthén.
39. Ostroff, C. (1993). The effects of climate and personal influences on individual behavior and attitudes in organizations. Organizational Behavior and Human Decision Processes, 56, 56 90.
40. Parker, C.P., Baltes, B.B., Young, S.A., Huff, J.W., Altmann, R.A., Lacost, H.A. Roberts, J.E. (2003). Relationships between psychological climate perceptions and work outcomes: A meta-analytic review. Journal of Organizational Behavior, 24, 389 416.
41. Ployhart, R.E., Weekley, J.A., Holtz, B.C. Kemp, C. (2003). Web-based and paper-and-pencil testing of applicants in a proctored setting: Are personality, biodata, and situational judgment tests comparable? Personnel Psychology, 56, 733 752.
42. Reips, U.D. (2000). The web experiment method: Advantages, disadvantages and solutions. In M.H. Birnbaum (Ed.), Psychological experiments on the internet (pp. 89 117). San Diego, CA : Academic Press.
43. Riordan, C.R. Vandenberg, R.J. (1994). A central question in cross-cultural research: Do employees of different cultures interpret work-related measures in an equivalent manner? Journal of Management, 20, 643 671.
44. Roberts, L.L., Konczak, L.J. Macan, T.H. (2004). Effects of data collection method on organizational climate survey results. Applied HRM Research, 9, 13 26.
45. Ryan, A.M., Chan, D., Ployhart, R.E. Slade, A.L. (1999). Employee attitude surveys in a multinational organization: Considering language and culture in assessing measurement equivalence. Personnel Psychology, 52, 37 58.
46. Schaeffer, D. Dillman, D. (1998). Development of a standard e-mail methodology: Results of an experiment. Public Opinion Quarterly, 62, 378 397.
47. Simsek, Z. Veigha, J.F. (2001). A primer on internet organizational surveys. Organizational Research Methods, 4, 218 235.
48. Sörbom, D. (1974). A general method for studying differences in factor means and factor structure across groups. British Journal of Mathematical and Statistical Psychology, 27, 229 239.
49. Sörbom, D. (1978). An alternative to the methodology for analysis of covariance. Psychometrika, 43, 381 396.
50. Sproull, L.S. (1986). Using electronic mail for data collection in organizational research. Academy of Management Journal, 29, 159 169.
51. Stanton, J.M. (1998). An empirical assessment of data collection using the internet. Personnel Psychology, 51, 709 725.
52. Stanton, J.M. Rogelberg, S.G. (2001). Using internet/intranet web pages to collect organizational research data. Organizational Research Methods, 4, 200 217.
53. Steiger, J.H. (1990). Structural model evaluation and modification: An interval estimation approach. Multivariate Behavioral Research, 25, 173 180.
54. Thompson, L.F., Surface, E.A., Martin, D.L. Sanders, M.G. (2003). From paper to pixels: Moving personnel surveys to the web. Personnel Psychology, 56, 197 227.
55. Vandenberg, R.J. (2002). Toward a further understanding of and improvement in measurement invariance methods and procedures. Organizational Research Methods, 5, 139 158.
56. Vandenberg, R.J. Lance, C.E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4 70.
57. Van Herk, H., Poortinga, Y.H. Verhallen, T.M.M. (2004). Response styles in rating scales: Evidence of method bias in data from six EU countries. Journal of Cross-Cultural Psychology, 35, 346 360.
58. Van Selm, M. Jankowski, N.W. (2006). Conducting online surveys. Quality & Quantity, 40, 435 456.
59. Vispoel, W.P., Boo, J. Bleiler, T. (2001). Computerized and paper-and-pencil versions of the Rosenberg self-esteem scale: A comparison of psychometric features and respondent preferences. Educational and Psychological Measurement, 61, 461 474.
60. Weijters, B. (2006). Response styles in consumer research. Unpublished doctoral dissertation, University of Ghent, Belgium.
61. Yun, G.W. Trumbo, C.W. (2000). Comparative response to a survey executed by post, email and web form. Journal of Computer-Mediated Communication, 6, 1 26.