Are internet and paper-and-pencil personality tests truly comparable?: An experimental design measurement invariance study

Organizational Research Methods

Volumn 10, Issue 2, 2007, Pages 322-345

  Meade, Adam W ^a   Michels, Lawrence C ^b   Lautenschlager, Gary J ^b  

^a NORTH CAROLINA STATE UNIVERSITY   (United States)

^b UNIVERSITY OF GEORGIA   (United States)

Author keywords

Factorial invariance; Internet testing; Measurement equivalence; Measurement invariance; Online assessment; Online testing

Indexed keywords

EID: 34147108681     PISSN: 10944281     EISSN: 15527425     Source Type: Journal    
DOI: 10.1177/1094428106289393     Document Type: Article

References (80)

1. American Educational Research Association, American Psychological Association, National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Psychological Association.
2. Anderson, J.C., Gerbing, D.W. (1984). The effect of sampling error on convergence, improper solutions, and goodness-of-fit indices for maximum likelihood confirmatory factor analysis. Psychometrika, 49, 155 173.
3. Anderson, J.C., Gerbing, D.W. (1988). Structural equation modeling in practice: A review and recommended two stage approach. Psychological Bulletin, 103, 411 423.
4. Bagozzi, R.P., Edwards, J.R. (1998). A general approach for representing constructs in organizational research. Organizational Research Methods, 1, 45 87.
5. Bandalos, D.L. (2002). The effects of item parceling on goodness-of-fit and parameter estimate bias in structural equation modeling. Structural Equation Modeling, 9, 78 102.
6. Bandalos, D.L., Finney, S.J. (2001). Item parceling issues in structural equation modeling. In G. A. Marcoulides (Ed.), New developments and techniques in structural equation modeling (pp. 269 296). Mahwah, NJ: Lawrence Erlbaum.
7. Barrett, P., Kline, P., Paltiel, L., Eysenck, H.J. (1996). An evaluation of the psychometric properties of the concept 5.2 Occupational Personality Questionnaire. Journal of Occupational and Organizational Psychology, 69, 1 19.
8. Barrick, M.R., Mount, M.K. (1991). The Big Five personality dimensions and job performance: A meta-analysis. Personnel Psychology, 44, 1 26.
9. Bartram, D., Brown, A. (2004). Information exchange article: Online testing: Mode of administration and the stability of OPQ 32i scores. International Journal of Selection and Assessment, 12, 278 284.
10. Beaujouan, Y.M. (2000). The convergence of the factorial structure of the OPQ (Occupational Personality Questionnaire) towards the Five Factor model, in seven countries. European Review of Applied Psychology/Revue Europeenne de Psychologie Appliquee, 50, 349 357.
11. Booth-Kewley, S., Edwards, J.E., Rosenfeld, P. (1992). Impression management, social desirability, and computer administration of attitude questionnaires: Does the computer make a difference? Journal of Applied Psychology, 77, 562 566.
12. Borman, W.C., Penner, L.A., Allen, T.D., Motowidlo, S.J. (2001). Personality predictors of citizenship performance. International Journal of Selection and Assessment, 9, 52 69.
13. Brehm, S.S., Brehm, J.W. (1981). Psychological reactance: A theory of freedom and control. New York: Academic Press.
14. Byrne, B.M., Shavelson, R.J., Muthen, B. (1989). Testing for the equivalence of factor covariance and mean structures: The issue of partial measurement invariance. Psychological Bulletin, 105, 456 466.
15. Chan, D. (2000). Detection of differential item functioning on the Kirton Adaptation-Innovation Inventory using multiple-group mean and covariance structure analyses. Multivariate Behavioral Research, 35, 169 199.
16. Chapman, D.S., Webster, J. (2003). The use of technologies in the recruiting, screening, and selection processes for job candidates. International Journal of Selection and Assessment, 11, 113 120.
17. Cooper, J., Weaver, K.D. (2003). Gender and computers: Understanding the digital divide. Mahwah, NJ: Lawrence Erlbaum.
18. Cortina, J.M. (1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78, 98 104.
19. Ding, L., Velicer, W.F., Harlow, L.L. (1995). Effects of estimation methods, number of indicators per factor, and improper solution on structural equation modeling fit indices. Structural Equation Modeling, 2, 119 144.
20. Donovan, M.A., Drasgow, F., Probst, T.M. (2000). Does computerizing paper-and-pencil job attitude scales make a difference? New IRT analyses offer insight. Journal of Applied Psychology, 85, 305 313.
21. Drasgow, F., Kanfer, R. (1985). Equivalence of measurement in heterogeneous populations. Journal of Applied Psychology, 70, 662 680.
22. Dulewicz, V. (1995). A validation of Belbin's team roles from 16PF and OPQ using bosses' ratings of competence. Journal of Occupational and Organizational Psychology, 68, 81 99.
23. Dwight, S.A., Fiegelson, M.E. (2000). A quantitative review of the effect of computerized testing on the measurement of social desirability. Educational and Psychological Measurement, 60, 340 360.
24. Ellingson, J.E., Smith, D.B., Sackett, P.R. (2001). Investigating the influence of social desirability on personality factor structure. Journal of Applied Psychology, 86, 122 133.
25. Facteau, J.D., Craig, S.B. (2001). Are performance appraisal ratings from different rating sources comparable? Journal of Applied Psychology, 86, 215 227.
26. Ferguson, E., Payne, T., Anderson, N. (1994). Occupational personality assessment: Theory, structure and psychometrics of the OPQ FMX5-Student. Personality and Individual Differences, 17, 217 225.
27. Golembiewski, R.T., Billingsley, K., Yeager, S. (1975). Measuring change and persistence in human affairs: Types of change generated by OD designs. Journal of Applied Behavioral Science, 12, 133 157.
28. Hacker, K.L., Steiner, R. (2002). The digital divide for Hispanic Americans. Howard Journal of Communications, 13, 267 283.
29. Harris, M.M. (2001). Trends and issues in I-O Psychology: A glimpse into the crystal ball. The Industrial-Organizational Psychologist, 38 (4). 70 73.
30. Harris, M.M. (2003). Speeding down the information highway: Internet recruitment and testing. The Industrial-Organizational Psychologist, 41 (2). 103 106.
31. Hogan, R., Hogan, J., Roberts, B.W. (1996). Personality measurement and employment decisions. American Psychologist, 51, 469 477.
32. Horn, J.L., McArdle, J.J. (1992). A practical and theoretical guide to measurement invariance in aging research. Experimental Aging Research, 18(3-4). 117 144.
33. Hu, L.T., Bentler, P.M. (1998). Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychological Methods, 3, 424 453.
34. Jackson, L.A., Eye, A.V., Barbatsis, G., Biocca, F., Zhao, Y., Fitzgerald, H.E. (2003). Internet attitudes and Internet use: Some surprising findings from the HomeNetToo project. International Journal of Human-Computer Studies, 59, 355 382.
35. Jöreskog, K.G. (1993). Testing structural equation models. In K. A. Bollen and J. S. Long (Eds.), Testing structural equation models. Newbury Park, CA: Sage.
36. J̈reskog, K., and Sörbom, D. (1996). LISREL 8: Users reference guide. Chicago: Scientific Software International.
37. Lautenschlager, G.J. (1989). ALPHATST: Testing for differences in coefficient alpha. Applied Psychological Measurement, 13, 284.
38. Lautenschlager, G.J., Flaherty, V.L. (1990). Computer administration of questions: More desirable or more social desirability? Journal of Applied Psychology, 75, 310 314.
39. Lievens, F., Harris, M.M. (2003). Research on Internet recruitment and testing: Current status and future directions. In C. L. Cooper I. T. Robertson (Eds.), International review of industrial and organizational psychology (Vol. 18). New York: Wiley.
40. Lievens, F., Van Dam, K., Anderson, N. (2002). Recent trends and challenges in personnel selection. Personnel Review, 31, 580 601.
41. Loges, W.E., Jung, J.Y. (2001). Exploring the digital divide: Internet connectedness and age. Communication Research, 28, 536 562.
42. Martin, C.L., Nagao, D.H. (1989). Some effects of computerized interviewing on job applicant responses. Journal of Applied Psychology, 74, 72 80.
43. Matthews, G., Stanton, N. (1994). Item and scale factor analyses of the Occupational Personality Questionnaire. Personality and Individual Differences, 16, 733 743.
44. McBride, J.R. (1998). Innovations in computer-based ability testing: Promise, problems, and perils. In M. D. Hakel (Ed.), Beyond multiple choice: Evaluating alternatives to traditional testing for selection (pp. 23-39). Mahwah, NJ: Lawrence Erlbaum.
45. McFarland, L.A., Ryan, A.M., Paul, K.B. (1998, April). Equivalence of an organizational attitude survey across administration modes. Paper presented at the 13th annual Society for Industrial and Organizational Psychology conference, Dallas, TX.
46. Meade, A. W., Kroustalis, C. M. (in press). Problems with item parceling for confirmatory factor analytic tests of measurement invariance. Organizational Research Methods.
47. Meade, A.W., Lautenschlager, G.J. (2004). A Monte-Carlo study of confirmatory factor analytic tests of measurement equivalence/invariance. Structural Equation Modeling, 11, 60 72.
48. Meade, A.W., Lautenschlager, G.J., Michels, L.C., Gentry, W.A. (2003, April). Equivalence of online and paper and pencil assessments. In A.W. Meade (Chair). Applications of IRT to measurement in organizations. Symposium presented at the annual Society for Industrial and Organizational Psychology conference, Orlando, FL.
49. Meredith, W. (1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58, 525 543.
50. Michels, L.C., Meade, A.W., Lautenschlager, G.J., and Gentry, W.A. (2003, May). The effect of test format and respondent choice on socially desirable responding. Paper presented at the 15th annual meeting of the American Psychological Society, Atlanta, GA.
51. Mount, M.K. Barrick, M.R. (1998). Five reasons why the Big Five article has been frequently cited. Personnel Psychology, 98, 849 858.
52. Park, D., Smith, A., Lautenschlager, G.J., Earles, J., Frieske, D., Zwahr, M., Gaines, C. (1996). Mediators of long-term memory performance across the life span. Psychology and Aging, 11, 621 637.
53. Pew Internet and American Life Project. (2000 a). Who's not online: 57% of those without Internet access say they do not plan to log on. Washington, DC: Pew Foundation. Retrieved from http://www.pewInternet.org/.
54. Pew Internet and American Life Project. (2000 b). African Americans and the Internet. Washington, DC: Pew Foundation. Retrieved from http://www.pewInternet.org/.
55. Ployhart, R.W., Oswald, F.L. (2004). Applications of mean and covariance structure analysis: Integrating correlational and experimental approaches. Organizational Research Methods, 7, 27 65.
56. Ployhart, R.E., Weekley, J.A., Holtz, B.C., Kemp, C. (2003). Web-based and paper-and-pencil testing of applicants in a proctored setting: Are personality, biodata, and situational judgment tests comparable? Personnel Psychology, 56, 733 752.
57. Potosky, D., Bobko, P. (1997). Computer versus paper-and-pencil mode and response distortion in noncognitive selection tests. Journal of Applied Psychology, 82, 293 299.
58. Potosky, D., Bobko, P. (2004). Selection testing via the Internet: Practical considerations and exploratory empirical findings. Personnel Psychology, 57, 1003 1034.
59. Rensvold, R.B., Cheung, G.W. (2000, August). Beyond two-group comparisons: Identifying sets of invariant groups. Paper presented at the annual Academy of Management meeting, Toronto, Ontario, Canada.
60. Richman, W.L., Kiesler, S., Weisband, S., Drasgow, F. (1999). A meta-analytic study of social desirability distortion in computer-administered questionnaires, traditional questionnaires, and interviews. Journal of Applied Psychology, 84, 754 775.
61. Rigdon, E.E. (1996). CFI vs. RMSEA: A comparison of two fit indexes for structural equation modeling. Structural Equation Modeling, 3, 369 379.
62. Riordan, C.M., Vandenberg, R.J. (1994). A central question in cross-cultural research: Do employees of different cultures interpret work-related measures in an equivalent manner? Journal of Management, 20, 643 671.
63. Robert, C., Probst, T.M., Martocchio, J.H., Drasgow, F., Lawler, J.J. (2000). Empowerment and continuous improvement in the United States, Mexico, Poland, and India: Predicting fit on the basis of the dimensions of power distance and individualism. Journal of Applied Psychology, 85, 643 658.
64. Rosenfeld, P., Doherty, L.M., Vicino, S.M., Kantor, J., Greaves, J. (1989). Attitude assessment in organizations: Testing three microcomputer-based survey systems. Journal of General Psychology, 116, 145 154.
65. Salgado, J.F. (1996). Personality and job competences: A comment on the Robertson Kinder (1993) study. Journal of Occupational and Organizational Psychology, 69, 373 375.
66. Salgado, J.F., Moscoso, S. (2003). Internet-based personality testing: Equivalence of measures and assessees' perceptions and reactions. International Journal of Selection and Assessment, 11, 194 205.
67. Saville, P., Sik, G., Nyfield, G., Hackston, J., Maciver, R. (1996). A demonstration of the validity of the Occupational Personality Questionnaire (OPQ) in the measurement of job competencies across time and in separate organizations. Applied Psychology: An International Review, 45, 243 262.
68. Schmit, M.J., Ryan, A.M. (1993). The Big Five in personnel selection: Factor structure in applicant and nonapplicant populations. Journal of Applied Psychology, 78, 966 974.
69. Sharf, J. (2000). As if g-loaded adverse impact isn't bad enough, Internet recruiters can expect to be accused of e-loaded impact. The Industrial-Organizational Psychologist, 38 (2). 156.
70. Shl. (2000). Notes accompanying the OPQ 32 survey. Boulder, CO: SHL.
71. Smith, D.B., Hanges, P.J., Dickson, M.W. (2001). Personnel selection and the five-factor model: Reexamining the effects of applicant's frame of reference. Journal of Applied Psychology, 86, 304 315.
72. Spera, S.D., Moye, N.A. (2001, April). Paper and Web survey methods in a multinational company. Paper presented at the annual meeting of the Society for Industrial and Organizational Psychology, San Diego, CA.
73. Stanton, J.M. (1998). An empirical assessment of data collection using the Internet. Personnel Psychology, 51, 709 726.
74. Steiger, J.H. (2000). Point estimation, hypothesis testing, and interval estimation using the RMSEA: Some comments and a reply to Hayduck and Glaser. Structural Equation Modeling, 7, 149 162.
75. Taris, T.W., Bok, I.A., Meijer, Z.Y. (1998). Assessing stability and change of psychometric properties of multi-item concepts across different situations: A general approach. Journal of Psychology, 132, 301 316.
76. Vandenberg, R.J., Lance, C.E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4 69.
77. West, S.G., Finch, J.F., and Curran, P.J. (1995). Structural equation models with nonnormal variables. In R. H. Hoyle (Ed.), Structural equation modeling: Concepts, issues, and applications. Thousand Oaks, CA: Sage.
78. Williams, L.J., Holahan, P.J. (1994). Parsimony-based fit indices for multiple-indicator models: Do they work? Structural Equation Modeling, 1, 161 189.
79. Yost, P.R., Homer, L.E. (1998, April). Electronic versus paper surveys: Does the medium affect the response? Paper presented at the annual meeting of the Society for Industrial and Organizational Psychology, Dallas, TX.
80. Young, S.A., Daum, D.L., Robie, C., Macey, W.H. (2000). Paper versus Web survey administration: Do different methods yield different results ? Paper presented at the annual meeting of the Society for Industrial and Organizational Psychology, New Orleans, LA.