Effects of Personalized Versus Generic Implementation of an Intra-Organizational Online Survey on Psychological Anonymity and Response Behavior: A Field Experiment

Journal of Business and Psychology

Volumn 29, Issue 2, 2014, Pages 169-181

  Mueller, Karsten ^a   Straatmann, Tammo ^a   Hattrup, Kate ^b   Jochum, Marco ^c  

^a UNIVERSITY OF MANNHEIM   (Germany)

^b SAN DIEGO STATE UNIVERSITY   (United States)

^c Group Strategy   (Germany)

Author keywords

Anonymity; Field experiment; Nonresponse; Online survey implementation; Response behavior; Survey

Indexed keywords

EID: 84900991524     PISSN: 08893268     EISSN: None     Source Type: Journal    
DOI: 10.1007/s10869-012-9262-9     Document Type: Article

References (71)

1. Anseel, F., Lievens, F., Schollaert, E., & Choragwicka, B. (2010). Response rates in organizational science, 1995-2008: A meta-analytic review and guidelines for survey researchers. Journal of Business and Psychology, 25, 335-349.
2. Armenakis, A. A., & Bedian, A. G. (1999). Organizational change: A review of theory and research in the 1990s. Journal of Management, 25(3), 293-315.
3. Baumgartner, H., & Steenkamp, J.-B. E. M. (2001). Response styles in marketing research: A cross-national investigation. Journal of Marketing Research, 38(2), 143-156.
4. Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107(2), 238-246.
5. Bentler, P. M., & Bonett, D. G. (1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88, 588-606.
6. Bergkvist, L., & Rossiter, J. R. (2007). The predictive validity of multiple-item versus single-item measures of the same constructs. Journal of Marketing Research, 44(May), 175-184.
7. Chapman, D. S., & Webster, J. (2003). The use of technologies in the recruiting, screening, and selection processes for job candidates. International Journal of Selection and Assessment, 11, 113-120.
8. Church, A., & Waclawski, J. (1998). Designing and using organizational surveys. Aldershot: Gower.
9. Church, A. H., & Waclawski, J. (2001). Designing and using organizational surveys: A seven step process. San Francisco: Jossey-Bass.
10. Churchill, G. A. (1979). A paradigm for developing better measures of marketing constructs. Journal of Marketing Research, 16, 64-73.
11. Cook, C., Heath, F., & Thompson, R. L. (2000). A meta-analysis of response rates in web- or internet-based surveys. Educational and Psychological Measurement, 60, 821-836.
12. Couper, M. P., Tourangeau, R., Conrad, F. G., & Crawford, S. D. (2004). What they see is what we get: Response options for web surveys. Social Science Computer Review, 22, 111-127.
13. Cycyota, C. S., & Harrison, D. A. (2006). What (not) to expect when surveying executives: A meta-analysis of top manager response rates and techniques over time. Organizational Research Methods, 9, 133-160.
14. Cycyota, C. S., Harrison, D. A., & Stahl, A. S. (2002). Enhancing response rates at the executive level: Are employee- or consumer-level techniques effective? Journal of Management, 28, 163-189.
15. de Jong, M. G., Steenkamp, J.-B. E. M., Fox, J.-P., & Baumgartner, H. (2008). Using item response theory to measure extreme response style in marketing research: A global investigation. Journal of Marketing Research, 45(1), 104-115.
16. Diamantopoulos, A., Reynolds, N. L., & Simintiras, A. C. (2006). The impact of response styles on the stability of cross-national comparisons. Journal of Business Research, 59(8), 925-935.
17. Dieffendorff, J. M., Silverman, S. B., & Greguras, G. J. (2005). Measurement equivalence and multisource ratings for non-managerial positions: Recommendations for research and practice. Journal of Business and Psychology, 19, 399-425.
18. Dillman, D. A. (2000). Mail and internet surveys: The tailored design method. New York: Wiley.
19. Dommeyer, C. J., & Moriarty, E. (2000). Comparing two forms of an e-mail survey: Embedded vs. attached. International Journal of Market Research, 42(1), 39-50.
20. Dunnette, M. D., & Heneman, H. G. (1956). Influence of scale administrator on employee attitude responses. Journal of Applied Psychology, 40, 73-77.
21. Falletta, S. V., & Combs, W. (2002). Surveys as a tool for organization development and change. In J. Waclawski & A. H. Church (Eds.), Organization development: A data-driven approach to organizational change (pp. 78-102). San Francisco: Jossey-Bass.
22. Fenlason, K. J., & Suckow-Zimberg, K. (2006). Online surveys. Critical issues in using the web to conduct surveys. In A. I. Kraut (Ed.), Getting action from organizational surveys (pp. 183-212). San Francisco: Jossey-Bass.
23. Fox, S., & Schwartz, D. (2002). Social desirability and controllability in computerized and paper-and-pencil personality questionnaires. Computers in Human Behaviour, 18(4), 389-410.
24. Gardner, D. G., Cummings, L. L., Dunham, R. B., & Pierce, J. L. (1998). Single-item versus multiple-item measurement scales: An empirical comparison. Educational and Psychological Measurement, 58(6), 898-915.
25. Göritz, A. S., Reinhold, N., & Batinic, B. (2002). Online panels. In B. Batinic, U. Reips, & M. Bosnjak (Eds.), Online social sciences (pp. 27-47). Seattle, WA: Hogrefe & Huber.
26. Graeff, T. R., & Harmon, S. (2002). Collecting and using personal data: Consumers' awareness and concerns. Journal of Consumer Marketing, 19(4), 302-318.
27. Harman, R. P., Thompson, L. F., & Surface, E. A. (2009, April). Understanding survey comment nonresponse and the characteristics of nonresponders. Paper presented at the 24th annual meeting of the Society for Industrial and Organizational Psychology, New Orleans, LA.
28. Heerwegh, D., Vanhove, T., Matthijs, K., & Loosveldt, G. (2005). The effect of personalization on response rates and data quality in web surveys. International Journal of Social Research Methodology, 8(2), 85-99.
29. Holt, D. T., Armenakis, A. A., Harris, S. G., & Feild, H. S. (2007). Readiness for organizational change: The systematic development of a scale. Journal of Applied Behavioral Science, 43(2), 232-255.
30. Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1-55.
31. Jensen, C., Potts, C., & Jensen, C. (2005). Privacy practices of internet users: Self-reports versus observed behavior. International Journal of Human-Computer Studies, 63(1-2), 203-227.
32. Kantor, J. (1991). The effects of computer administration and identification on the job descriptive index. Journal of Business and Psychology, 75, 309-323.
33. Kerin, R. A., & Petersen, R. A. (1977). Personalization, respondent anonymity, and response distortion in mail surveys. Journal of Applied Psychology, 62, 86-89.
34. Kraut, A. I. (1996). An overview of organizational surveys. In A. I. Kraut (Ed.), Organizational surveys: Tools for assessment and change (pp. 1-14). San Francisco: Jossey-Bass.
35. Kraut, A. I. (2006). Getting action from organizational surveys. New concepts, technologies and applications. San Francisco: Jossey Bass.
36. Kraut, A. I., & Saari, L. M. (1999). Organization surveys: Coming of age for a new era. In A. I. Kraut & A. K. Korman (Eds.), Evolving practices in human resource management (pp. 302-327). San Francisco: Jossey-Bass.
37. McFarland, L. A., Ryan, A. M., & Paul, K. B. (1998, April). Equivalence of an organizational attitude survey across administration modes. Paper presented at the 13th Annual Conference of the Society for Industrial and Organizational Psychology, Dallas, TX.
38. Mowday, R. T., Steers, R. M., & Porter, L. W. (1979). The measurement of organizational commitment. Journal of Vocational Behavior, 14, 224-247.
39. Mueller, K., Liebig, C., & Hattrup, K. (2007). Computerizing organizational attitudes surveys: An investigation of the measurement equivalence of a multifaceted job satisfaction measure. Educational and Psychological Measurement, 67(4), 658-678.
40. Nunnally, J. (1978). Psychometric theory. New York: McGraw-Hill.
41. Penny, J. A. (2003). Exploring differential item functioning in a 360-degree assessment: Rater source and method of delivery. Organizational Research Methods, 6(1), 61-79.
42. Postmes, T., Spears, R., & Lea, M. (1998). Breaching or building social boundaries? SIDE-effects of computer-mediated communication. Communication Research, 25, 689-715.
43. Price, J. L., & Mueller, C. W. (1986). Handbook of organizational measurement. Marshfield, MA: Pitman.
44. Richman, W. L., Kiesler, S., Weisband, S., & Drasgow, F. (1999). A meta-analytic study of social desirability distortion in computer administered questionnaires, traditional questionnaires, and interviews. Journal of Applied Psychology, 84, 754-775.
45. Rogelberg, S. G., Fisher, G. G., Maynard, D. C., Hakel, M. D., & Horvath, M. (2001). Attitudes toward surveys: Development of a measure and its relationship to respondent behavior. Organizational Research Methods, 4, 3-25.
46. Rogelberg, S. G., Spitzmüller, C., Little, I. S., & Reeve, C. L. (2006). Understanding response behavior to an online special topics organizational satisfaction survey. Personnel Psychology, 59, 903-923.
47. Rose, D. S., Sidle, S. D., & Griffith, K. H. (2007). A penny for your thoughts: Monetary incentives improve response rates for company-sponsored employee surveys. Organizational Research Methods, 10(2), 225-240.
48. Rosenfeld, P., Booth-Kewley, S., Edwards, J. E., & Thomas, M. D. (1996). Responses on computer surveys: Impression management, social desirability, and the Big Brother syndrome. Computers in Human Behaviour, 12, 263-274.
49. Rosenfeld, P., Doherty, L. M., & Carroll, L. (1987). Microcomputer-based organizational survey assessment: Applications to training. Journal of Business and Psychology, 2, 182-193.
50. Saari, L. M., & Judge, T. A. (2004). Employee attitudes and job satisfaction. Human Resource Management, 43(4), 395-407.
51. Saari, L. M., & Scherbaum, C. A. (2011). Identified employee surveys: Potential promise, perils, and professional practice guidelines. Industrial and Organizational Psychology, 4, 435-448.
52. Sheehan, K. B. (2001). E-mail survey response rates: A review. Journal of Computer-Mediated Communication, 6(2). Retrieved March 2, 2008, from http://jcmc. indiana. edu/vol6/issue2/sheehan. html.
53. Singer, E., Mathiowetz, N. A., & Couper, M. P. (1993). The impact of privacy and confidentiality concerns on survey participation: The case of the 1990 U. S. census. Public Opinion Quarterly, 57, 465-482.
54. Smith, P. B. (2003). Meeting the challenge of cultural difference. In D. Tjosvold & K. Leung (Eds.), Cross-cultural management: Foundation and future (pp. 59-73). Aldershot: Ashgate.
55. Spears, R., & Lea, M. (1994). Panacea or panopticon: The hidden power in computer mediated communication. Communication Research, 21, 427-459.
56. Stanton, J. M. (1998). An empirical assessment of data collection using the internet. Personnel Psychology, 51, 709-725.
57. Steiger, J. H. (1990). Structural model evaluation and modification: An internal estimation approach. Multivariate Behavior Research, 25, 173-180.
58. Thompson, L. F., & Surface, E. A. (2007). Employee surveys administered online: Attitudes toward the medium, nonresponse, and data representativeness. Organizational Research Methods, 10, 241-261.
59. Thompson, L. F., Surface, E. A., Martin, D. L., & Sanders, M. G. (2003). From paper to pixels: Moving personnel surveys to the web. Personnel Psychology and Marketing, 56, 197-227.
60. Tourangeau, R., Groves, R. M., Kennedy, C., & Yan, T. (2009). The presentation of a web survey, nonresponse, and measurement error among members of web panel. Journal of Official Statistics, 25(3), 299-321.
61. Tourangeau, R., & Yan, T. (2007). Sensitive questions in surveys. Psychological Bulletin, 133(5), 859-883.
62. Uriell, Z. A., & Dudley, C. M. (2009). Sensitive topics: Are there modal differences? Computers in Human Behavior, 25, 76-87.
63. Vandenberg, R. J., & Lance, C. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4-69.
64. Vehovar, V., Batagelj, Z., Manfreda, K. L., & Zaletel, M. (2002). Nonresponse in web surveys. In R. M. Groves, D. A. Dillman, J. L. Eltinge, & R. J. A. Little (Eds.), Survey nonresponse (pp. 229-242). New York: Wiley.
65. Venkatesh, V., Morris, M. G., Davis, F. D., & Davis, G. B. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27(3), 425-478.
66. West, S. G., Finch, J. F., & Curran, P. J. (1995). Structural equation models with nonnormal variables: Problems and remedies. In R. H. Hoyle (Ed.), Structural equation modeling: Concepts, issues, and applications (pp. 56-75). Thousand Oaks, CA: Sage.
67. Whelan, T. J. (2008, April). Antecedents of anonymity perceptions in Web-based surveys. Paper presented at the 23rd annual meeting of the Society for Industrial and Organizational Psychology, San Francisco, CA.
68. Whelan, T. J., & Meade, A. W. (2009, April). Examining DIF in perceived anonymity when sensitive items are endorsed. Paper presented at the 24th annual conference of the Society for Industrial and Organizational Psychology, New Orleans, LA.
69. Whelan, T. J., & Thompson, L. F. (2009, April). Development/validation of the PANON scale assessing survey anonymity perceptions. Paper presented at the 24th annual conference of the Society for Industrial and Organizational Psychology, New Orleans, LA.
70. Yost, P. R., & Homer, L. E. (1998, April). Electronic versus paper surveys: Does the medium affect the response? Paper presented at the 13th annual conference of the Society for Industrial and Organizational Psychology, Dallas, TX.
71. Zimbardo, P. G. (1969). The human choice: Individuation, reason, and order vs. deindividuation, impulse and chaos. In W. J. Arnold & D. Levine (Eds.), Nebraska symposium on motivation (pp. 237-307). Lincoln, NE: University of Nebraska Press.