Computerizing organizational attitude surveys: An investigation of the measurement equivalence of a multifaceted job satisfaction measure

Educational and Psychological Measurement

Volumn 67, Issue 4, 2007, Pages 658-678

  Mueller, Karsten ^a   Liebig, Christian ^a   Hattrup, Keith ^b  

^a UNIVERSITY OF MANNHEIM   (Germany)

^b SAN DIEGO STATE UNIVERSITY   (United States)

Author keywords

Administration mode; Attitude measure; Measurement equivalence; Survey

Indexed keywords

EID: 34547155248     PISSN: 00131644     EISSN: 15523888     Source Type: Journal    
DOI: 10.1177/0013164406292084     Document Type: Article

References (66)

1. Bartram, D., Bayliss, R. (1984). Automated testings: Past, present, and future. Journal of Occupational Psychology, 57, 221-237.
2. Bentler, P.M. (1988). Causal modeling via structural equation systems. In J. R. Nesselroade R. B. Cattell (Eds.), Handbook of multivariate experimental psychology. Perspectives on individual differences (2 nd ed., pp. 317-335). New York: Plenum.
3. Bentler, P.M., Bonett, D.G. (1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88, 588-606.
4. Bollen, K.A. (1989). Structural equations with latent variables. Oxford, UK: John Wiley.
5. Bollen, K.A., Long, J.S. (1993). Testing structural equation models. Newbury Park, CA: Sage.
6. Booth-Kewley, S., Edwards, J.E., Rosenfeld, P. (1992). Impression management, social desirability, and computer administration of attitude questionnaires: Does the computer make a difference ? Journal of Applied Psychology, 77, 562-566.
7. Burke, M.J. (1992). Computerized psychological testing: Impacts on measuring predictor constructs and future job behavior. In N. Schmitt W. C. Borman (Eds.), Personnel selection in organizations (pp. 203-238). San Francisco: Jossey-Bass.
8. Byrne, B.M., Shavelson, R.J., Muthen, B. (1989). Testing for the equivalence of factor covariance and mean structures. The issue of partial measurement invariance. Psychological Bulletin, 105, 456-466.
9. Cheung, G.W., Rensvold, R.B. (1999). Testing factorial invariance across groups: A reconceptualization and proposed new method. Journal of Management, 25, 1-27.
10. Cheung, G.W., Rensvold, R.B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9, 233-255.
11. Cook, T.D., Campbell, T.L. (1979). Quasi-experimentation: Design analysis issues for field settings. Boston: Houghton-Mifflin.
12. Davis, C., Cowles, M. (1989). Automated psychological testing: Method of administration, need for approval, and measures of anxiety. Educational and Psychological Measurement, 49, 311-320.
13. Davis, R.N. (1999). Web-based administration of a personality questionnaire: Comparison with traditional methods. Behavior Research Methods, Instruments Computers, 31, 572-577.
14. Dieffendorff, J.M., Silverman, S.B. Greguras, G.J. (2005). Measurement equivalence and multisource ratings for non-managerial positions: Recommendations for research and practice. Journal of Business and Psychology, 19, 399-425.
15. Donovan, M.A., Drasgow, F., Probst, T.M. (2000). Does computerizing paper-and-pencil job attitude scales make a difference? New IRT analyses offer insight. Journal of Applied Psychology, 85, 305-313.
16. Drasgow, F. (1984). Scrutinizing psychological tests: Measurement equivalence and equivalent relations with external variables are the central issues. Psychological Bulletin, 95, 134-135.
17. Drasgow, F., Kanfer, R. (1985). Equivalence of psychological measurement in heterogeneous populations. Journal of Applied Psychology, 70, 662-680.
18. Dwight, S.A., Feigelson, M.E. (2000). A quantitative review of the effect of computerized testing on the measurement of social desirability. Educational and Psychological Measurement, 60, 340-360.
19. Erdman, H., Klein, M.H., Greist, J.H. (1983). The reliability of a computer interview for drug-use abuse information. Behavior Research Methods Instrumentation, 15, 66-68.
20. Fox, S., Schwartz, D. (2002). Social desirability and controllability in computerized and paper-and-pencil personality questionnaires. Computers in Human Behavior, 18, 389-410.
21. Henson, R.K. (2001). Understanding internal consistency reliability estimates: A conceptual primer on coefficient alpha. Measurement Evaluation in Counseling Development, 34, 177-189.
22. Horn, J.L., McArdle, J.J. (1992). A practical and theoretical guide to measurement invariance in aging research. Experimental Aging Research, 18, 117-144.
23. Hu, L., Bentler, P.M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55.
24. Joinson, A. (1999). Social desirability, anonymity, and Internet-based questionnaires. Behavior Research Methods, Instruments Computers, 31, 433-438.
25. Kantor, J. (1991). The effects of computer administration and identification on the job descriptive index (JDI). Journal of Business and Psychology, 5, 309-323.
26. Katz, L., Dalby, J.T. (1981). Computer and manual administration of the Eysenck Personality Inventory. Journal of Clinical Psychology, 37, 586-588.
27. Kiesler, S., Sproull, L.S. (1986). Response effects in the electronic survey. Public Opinion Quarterly, 50, 402-413.
28. King, W.C., Miles, E.W. (1995). A quasi-experimental assessment of the effect of computerizing noncognitive paper-and-pencil measurements: A test of measurement equivalence. Journal of Applied Psychology, 80, 643-651.
29. Kline, R.B. (1998). Principles and practice of structural equation modeling. New York: Guilford.
30. Knapp, H., Kirk, S.A. (2003). Using pencil and paper, Internet and touch-tone phones for self-administered surveys: Does methodology matter? Computers in Human Behavior, 19, 117-134.
31. Lautenschlager, G.J., Flaherty, V.L. (1990). Computer administration of questions: More desirable or more social desirability? Journal of Applied Psychology, 75, 310-314.
32. Little, T.D. (2000). On the comparability of constructs in cross-cultural research. A critique of Cheung and Rensvold. Journal of Cross-Cultural Psychology, 31, 213-219.
33. Locke, S.D., Gilbert, B.O. (1995). Method of psychological-assessment, self-disclosure, and experiential differences. A study of computer, questionnaire, and interview assessment formats. Journal of Social Behavior and Personality, 10, 255-263.
34. Lushene, R.E., O'Neil, H.F., Dunn, T. (1974). Equivalent validity of a completely computerized MMPI. Journal of Personality Assessment, 38, 353-361.
35. Martin, C.L., Nagao, D.H. (1989). Some effects of computerized interviewing on job applicant responses. Journal of Applied Psychology, 74, 72-80.
36. Mead, A.D., Drasgow, F. (1993). Equivalence of computerized and paper-and-pencil cognitive ability tests: A meta-analysis. Psychological Bulletin, 114, 449-458.
37. Meredith, W. (1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58, 525-543.
38. Millstein, S.G. (1987). Acceptability and reliability of sensitive information collected via computer interview. Educational and Psychological Measurement, 47, 523-533.
39. Mowday, R.T., Steers, R.M., Porter, L.W. (1979). Measurement of organizational commitment. Journal of Vocational Behavior, 14, 224-247.
40. Niehoff, B.P., Moorman, R.H. (1993). Justice as a mediator of the relationship between methods of monitoring and organizational citizenship behavior. Academy of Management Journal, 36, 527-556.
41. Ployhart, R.E., Weekley, J.A., Holtz, B.C., Kemp, C. (2003). Web-based and paper-and-pencil testing of applicants in a proctored setting: Are personality, biodata, and situational judgment tests comparable ? Personnel Psychology, 56, 733-752.
42. Potosky, D., Bobko, P. (1997). Computer versus paper-and-pencil administration mode and response distortion in noncognitive selection tests. Journal of Applied Psychology, 82, 293-299.
43. Reise, S.P., Widaman, K.F., Pugh, R.H. (1993). Confirmatory factor analysis and item response theory: Two approaches for exploring measurement invariance. Psychological Bulletin, 114, 552-566.
44. Richman, W.L., Kiesler, S., Weisband, S., Drasgow, F. (1999). A meta-analytic study of social desirability distortion in computer-administered questionnaires, traditional questionnaires, and interviews. Journal of Applied Psychology, 84, 754-775.
45. Riordan, C.M., Vandenberg, R.J. (1994). A central question in cross-cultural research: Do employees of different cultures interpret work-related measures in an equivalent manner? Journal of Management, 20, 643-671.
46. Rosenfeld, P., Doherty, L.M., Carroll, L. (1987). Microcomputer-based organizational survey assessment: Applications to training. Journal of Business and Psychology, 2, 182-193.
47. Rosenfeld, P., Doherty, L.M., Vicino, S.M., Kantor, J., Greaves, J. (1989). Attitude assessment in organizationsTesting three microcomputer-based survey-systems. Journal of General Psychology, 116, 145-154.
48. Rosenfeld, P., Giacalone, R.A., Knouse, S.B., Doherty, L.M., Vicino, S.M., Kantor, J., et al. (1991). Impression management, candor, and microcomputer-based organizational surveysan individual-differences approach. Computers in Human Behavior, 7, 23-32.
49. Ryan, A.M., Chan, D., Ployhart, R.E., Slade, L.A. (1999). Employee attitude surveys in a multinational organization: Considering language and culture in assessing measurement equivalence. Personnel Psychology, 52, 37-58.
50. Schuldberg, D. (1988). The MMPI is less sensitive to the automated testing format than it is to repeated testing: Item and scale effects. Computers in Human Behavior, 4, 285-298.
51. Shadish, W.R., Cook, T.D., Campbell, T.L. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton-Mifflin.
52. Smith, P.C., Kendall, L.M., Hulin, C.L. (1969). The measurement of satisfaction in work and retirement: A strategy for the study of attitudes. Oxford, UK: Rand McNally.
53. Stanton, J.M. (1998). An empirical assessment of data collection using the Internet. Personnel Psychology, 51, 709-725.
54. Steenkamp, J.-B.E.M., Baumgartner, H. (1998). Assessing measurement invariance in cross-national consumer research. Journal of Consumer Research, 25, 78-90.
55. Steiger, J.H. (1990). Structural model evaluation and modification: An interval estimation approach. Multivariate Behavioral Research, 25, 173-180.
56. Synodinos, N.E., Brennan, J.M. (1990). Evaluating microcomputer interactive survey software. Journal of Business and Psychology, 4, 483-493.
57. Synodinos, N.E., Papacostas, C.S., Okimoto, G.M. (1994). Computer-administered versus paper-and-pencil surveys and the effect of sample selection. Behavior Research Methods, Instruments, Computers, 26, 395-401.
58. Turner, C.F., Ku, L., Rogers, S.M., Lindberg, L.D., Pleck, J.H., Sonenstein, F.L. (1998). Adolescent sexual behavior, drug use, and violence: Increased reporting with computer survey technology. Science, 280, 867-873.
59. Vandenberg, R.J., Lance, C.E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4-69.
60. van de Vijver, F.J.R., Poortinga, Y.H. (1997). Towards an integrated analysis of bias in cross-cultural assessment. European Journal of Psychological Assessment, 13, 29-37.
61. Vansickle, T.R., Kimmel, C., Kapes, J.T. (1989). Test retest equivalency of the computer-based and paper pencil versions of the Strong-Campbell Interest Inventory. Measurement and Evaluation in Counseling and Development, 22, 88-93.
62. Weiss, D.J., Dawis, R.V., England, G.W., Lofquist, L.H. (1967). Manual for the Minnesota Satisfaction Questionnaire. Minneapolis: University of Minnesota Press.
63. White, D.M., Clements, C.B., Fowler, R.D. (1985). A comparison of computer administration with standard administration of the MMPI. Computers in Human Behavior, 1, 153-162.
64. Widaman, K.F. (1985). Hierarchically nested covariance structure models for multitrait-multimethod data. Applied Psychological Measurement, 9, 1-26.
65. Wilkerson, J.M., Nagao, D.H., Martin, C.L. (2002). Socially desirable responding in computerized questionnaires: When questionnaire purpose matters more than the mode. Journal of Applied Social Psychology, 32, 544-559.
66. Wilson, F.R., Genco, K.T., Yager, G.G. (1985). Assessing the equivalence of paper-and-pencil vs. computerized tests: Demonstration of a promising methodology. Computers in Human Behavior, 1, 265-275.