The survey effect: Does administering surveys affect trainees' behavior?

Learning and Individual Differences

Volumn 37, Issue , 2015, Pages 1-12

  Sitzmann, Traci ^a   Wang, Mo ^b  

^a UNIVERSITY OF COLORADO   (United States)

^b UNIVERSITY OF FLORIDA   (United States)

Author keywords

Attrition; Data quality; Interruptions; Self regulation; Survey effect; Training

Indexed keywords

EID: 84910015578     PISSN: 10416080     EISSN: 18733425     Source Type: Journal    
DOI: 10.1016/j.lindif.2014.11.003     Document Type: Article

References (83)

1. Arthur W., Day D.V. Development of a short form for the Raven Advanced Progressive Matrices Test. Educational and Psychological Measurement 1994, 54:394-403.
2. Beal D.J., Weiss H.M., Barros E., MacDermid S.M. An episodic process model of affective influences on performance. Journal of Applied Psychology 2005, 90:1054-1068. 10.1037/0021-9010.90.6.1054.
3. Beck J.W., Schmidt A.M. State-level goal orientations as mediators of the relationship between time pressure and performance: A longitudinal study. Journal of Applied Psychology 2013, 98(2):354.
4. Beier M.E., Kanfer R. Motivation in training and development: A phase perspective. Learning, Training, and Development in Organizations 2010, 65-97.
5. Berthold K., Nuckles M., Renkl A. Do learning protocols support learning strategies and outcomes? The role of cognitive and metacognitive prompts. Learning and Instruction 2007, 17:564-577. 10.1016/j.learninstruc.2007.09.007.
6. Bliese P.D., Ployhart R.E. Growth modeling using random coefficient models: Model building, testing, and illustrations. Organizational Research Methods 2002, 5:362-387. 10.1177/109442802237116.
7. Bradburn N. Respondent burden. Health survey research methods: Second Biennial Conference (Williamsburg, VA) 1977, U.S. Government Printing Office, Washington, DC.
8. Brown, C. A. (2005). An investigation of the influence of metacognition, reading comprehension skill, and background knowledge on studying. Unpublished doctoral dissertation, Columbia University, New York.
9. Butler D.L., Winne P.H. Feedback and self-regulated learning: A theoretical synthesis. Review of Educational Research 1995, 65:245-281. 10.2307/1170684.
10. Cape P. Questionnaire length, fatigue effects and response quality revisited. Survey Sampling International. Originally presented at ARF re: Think 2010. New York 2010.
11. Chapin G. Field work and social research 1920, The Century Co. Commission on Federal Paperwork, New York, (Retrieved from http://archive.org/stream/fieldworkandsoc01chapgoog#page/n66/mode/1up).
12. Colquitt J.A., LePine J.A., Noe R.A. Toward an integrative theory of training motivation: A meta-analytic path analysis of 20years of research. Journal of Applied Psychology 2000, 85:678-707. 10.1037/0021-9010.85.5.678.
13. Colquitt J.A., Simmering M.S. Conscientiousness, goal orientation, and motivation to learn during the learning process: A longitudinal study. Journal of Applied Psychology 1998, 83:654-665. 10.1037/0021-9010.83.4.654.
14. de la Fuente J., Justicia F. The DEDEPRO model of regulated teaching and learning: Recent advances. Electronic Journal of Research in Educational Psychology 2007, 5(3):535-564.
15. Diener E., Wallbom M. Effects of self-awareness on antinormative behavior. Journal of Research in Personality 1976, 10(1):107-111.
16. Dillman D.A., Smyth J.S., Christian L.M. Internet, mail, and mixed-mode surveys: A tailored design method 2009, John Wiley & Sons, Hoboken, NJ.
17. Duval T.S., Wicklund R.A. A theory of objective self-awareness 1972, Academic, New York.
18. Ely K., Sitzmann T., Falkiewicz C. The influence of goal orientation dimensions on time to train in a self-paced training environment. Learning and Individual Differences 2009, 19:146-150. 10.1016/j.lindif.2008.08.003.
19. Fischer B., Glanzer M. Short-term storage and the processing of cohesion during reading. The Quarterly Journal of Experimental Psychology A: Human, Experimental Psychology 1986, 38(3-A):431-460.
20. Frese M., Zapf D. Action as the core of work psychology: A German approach. Handbook of industrial and organizational psychology 1994, Vol. 4:271-340. Consulting Psychologists Press, Palo Alto, CA. 2nd ed. M. Dunnette, L. Hough, H.C. Triandis (Eds.).
21. Galesic M., Bosnjak M. Effects of questionnaire length on participation and indicators of response quality in a web survey. Public Opinion Quarterly 2009, 73:349-360. 10.1093/poq/nfp031.
22. Golinelli D., Gery R., Green H.D., Kennedy D.P., Tucker J.S., Wenzel S.L. Sampling to reduce respondent burden in personal network studies and its effect on estimates of structural measures. Field Methods 2010, 22(3):217-230. 10.1177/1525822X10370796.
23. Green M., McGill E. State of the industry report 2011 2011, American Society for Training & Development, Alexandria, VA.
24. Heberlein T.A., Baumgartner R. Factors affecting response rates to mailed questionnaires: A quantitative analysis of the published literature. American Sociological Review 1978, 43(4):447-462. 10.2307/2094771.
25. Hodgetts H.M., Jones D.M. Contextual cues aid recovery from interruption: The role of associative activation. Journal of Experimental Psychology: Learning, Memory, and Cognition 2006, 32(5):1120-1132. 10.1037/0278-7393.32.5.1120.
26. Hübner S., Nückles M., Renkl A. Prompting cognitive and metacognitive processing in writing-to-learn enhances learning outcomes. Proceedings of the 28th Annual Conference of the Cognitive Science Society 2006, 357-362. Erlbaum, Mahwah. R. Sun, N. Miyake, C. Schunn (Eds.).
27. Jett Q.R., George J.M. Work interrupted: A closer look at the role of interruptions in organizational life. Academy of Management Review 2003, 28:494-509.
28. Kanfer R., Ackerman P.L. Motivation and cognitive abilities: An integrative/aptitude-treatment interaction approach to skill acquisition. Journal of Applied Psychology 1989, 74:657-690. 10.1037/0021-9010.74.4.657.
29. Karoly P. Mechanisms of self-regulation: A systems view. Annual Review of Psychology 1993, 44:23-52. 10.1002/job.4030140409.
30. Kirkpatrick D.L. Evaluation. The ASTD training and development handbook 1996, 294-312. McGraw-Hill, New York. 4th ed. R.L. Craig (Ed.).
31. Korn, M., & Levitz, J. (2013, Jan. 2). Online Courses Look for a Business Model. The Wall Street Journal.
32. Lee F.K., Sheldon K.M., Turban D.B. Personality and the goal-striving process: The influence of achievement goal patterns, goal level, and mental focus on performance and enjoyment. Journal of Applied Psychology 2003, 88:256-265. 10.1037/0021-9010.88.2.256.
33. Littell R.C., Milliken G.A., Stroup W.W., Wolfinger R.D., Schabenberger O. SAS for mixed models 2006, SAS Institute Inc., Cary, NC. 2nd ed.
34. Loi R., Yang J., Diefendorff J.M. Four-factor justice and daily job satisfaction: A multilevel investigation. Journal of Applied Psychology 2009, 94:770-781.
35. MacKinnon D.P., Lockwood C.M., Hoffman J.M., West S.G., Sheets V. A comparison of methods to test mediation and other intervening variable effects. Psychological Methods 2002, 7:83-104. 10.1037//1082-989X.7.1.83.
36. Miller R.B., Behrens J.T., Greene B.A., Newman D. Goals and perceived ability: Impact on student valuing, self-regulation, and persistence. Contemporary Educational Psychology 1993, 18:2-14. 10.1006/ceps.1993.1002.
37. Mobley W.H., Hand H.H., Baker R.L., Meglino B.M. Conceptual and empirical analysis of military recruit training attrition. Journal of Applied Psychology 1979, 64:10-18.
38. Muraven M., Baumeister R.F. Self-regulation and depletion of limited resources: Does self-control resemble a muscle?. Psychological Bulletin 2000, 126:247-259. 10.1037/0033-2909.126.2.247.
39. Noe R.A. Trainees' attributes and attitudes: Neglected influences on training effectiveness. Academy of Management Review 1986, 11:736-749. 10.2307/258393.
40. Noe R.A., Schmitt N. The influence of trainee attitudes on training effectiveness: Test of a model. Personnel Psychology 1986, 39:497-523. 10.1111/j.1744-6570.1986.tb00950.x.
41. Petros T.V., Bentz B., Hammes K., Zehr H.D. The components of text that influence reading times and recall in skilled and less skilled college readers. Discourse Processes 1990, 13(4):387-400. 10.1080/01638539009544767.
42. Phillips J.M., Gully S.M. Role of goal orientation, ability, need for achievement, and locus of control in the self-efficacy and goal-setting process. Journal of Applied Psychology 1997, 82:792-802. 10.1037/0021-9010.82.5.792.
43. Pintrich P.R., Cross D.R., Kozma R.B., McKeachie W.J. Instructional psychology. Annual Review of Psychology 1986, 37:611-651.
44. Porter S.R., Whitcomb M.E., Weitzer W.H. Multiple surveys of students and survey fatigue. New Directions for Institutional Research 2004, 121:63-73.
45. Raudenbush S.W., Bryk A.S. Hierarchical linear models: Applications and data analysis methods 2002, Sage Publications, Thousand Oaks, California. 2nd ed.
46. Ruch F. Effects of repeated interviewing on the respondents' answers. Journal of Consulting Psychology 1941, 5(4):179-182.
47. Schmidt A.M., DeShon R.P. The moderating effects of performance ambiguity on the relationship between self-efficacy and performance. Journal of Applied Psychology 2010, 95:572-581. 10.1037/a0018289.
48. Schmidt A.M., Ford J.K. Learning within a learner control training environment: The interactive effects of goal orientation and metacognitive instruction on learning outcomes. Personnel Psychology 2003, 56:405-429. 10.1111/j.1744-6570.2003.tb00156.x.
49. Scott B.A., Barnes C.M. A multilevel field investigation of emotional labor, affect, work withdrawal and gender. Academy of Management Journal 2011, 54:116-136.
50. Sharp L.M., Frankel J. Respondent burden: A test of some common assumptions. Public Opinion Quarterly 1983, 47:36-53.
51. Silvia P.J., Duval T.S. Objective self-awareness theory: Recent progress and enduring problems. Personality and Social Psychology Review 2001, 5(3):230-241.
52. Sitzmann T. A theoretical model and analysis of the role of self-regulation in the attrition process. Learning and Individual Differences 2012, 22(1):46-54. 10.1016/j.lindif.2011.11.004.
53. Sitzmann T., Bell B.S., Kraiger K., Kanar A.M. A multilevel analysis of the effect of prompting self-regulation in technology-delivered instruction. Personnel Psychology 2009, 62:697-734. 10.1111/j.1744-6570.2009.01155.x.
54. Sitzmann T., Brown K.G., Casper W.J., Ely K., Zimmerman R. A review and meta-analysis of the nomological network of trainee reactions. Journal of Applied Psychology 2008, 93:280-295. 10.1037/0021-9010.93.2.280.
55. Sitzmann T., Brown K.G., Ely K., Kraiger K., Wisher R.A. A cyclical model of motivational constructs in Web-based courses. Military Psychology 2009, 21:534-551. 10.1080/08995600903206479.
56. Sitzmann T., Ely K. Sometimes you need a reminder: The effects of prompting self-regulation on regulatory processes, learning, and attrition. Journal of Applied Psychology 2010, 95:132-144. 10.1037/a0018080.
57. Sitzmann T., Ely K. A meta-analysis of self-regulated learning in work-related training and educational attainment: What we know and where we need to go. Psychological Bulletin 2011, 137:421-442.
58. Sitzmann T., Ely K., Bell B.S., Bauer K.N. The effects of technical difficulties on learning and attrition during online training. Journal of Experimental Psychology: Applied 2010, 16:281-292. 10.1037/a0019968.
59. Sitzmann T., Johnson S.K. The best laid plans: Examining the conditions under which a planning intervention improves learning and reduces attrition. Journal of Applied Psychology 2012, 97:967-981. 10.1037/a0027977.
60. Sitzmann T., Johnson S.K. When is ignorance bliss? The effects of inaccurate self-assessments of knowledge on learning and attrition. Organizational Behavior and Human Decision Processes 2012, 117:192-207. 10.1016/j.obhdp.2011.11.004.
61. Sitzmann T., Kraiger K., Stewart D., Wisher R. The comparative effectiveness of Web-based and classroom instruction: A meta-analysis. Personnel Psychology 2006, 59:623-664. 10.1111/j.1744-6570.2006.00049.x.
62. Sonnentag S., Mojza E.J., Demerouti E., Bakker A.B. Reciprocal relations between recovery and work engagement: The moderating role of job stressors. Journal of Applied Psychology 2012, 10.1037/a0028292.
63. Sosdian C.P., Sharp L.M. Nonresponse in mail surveys: Access failure or respondent resistance. The Public Opinion Quarterly 1980, 44(3):396-402.
64. Speier C., Valacich J.S., Vessey I. The influence of task interruption on individual decision making: An information overload perspective. Decision Sciences 1999, 30:337-360.
65. Speier C., Vessey I., Valacich J.S. The effects of interruptions, task complexity, and information presentation on computer-supported decision-making performance. Decision Sciences 2003, 34:771-797.
66. Straus S.G., Ward D. Evaluating delivery venues for teaching critical thinking skills. Tenth Annual Army dL Workshop, Newport News, VA, June, 2011 2011.
67. To M.L., Fisher C.D., Ashkanasy N.M., Rowe P.A. Within-person relationships between mood and creativity. Journal of Applied Psychology 2012, 97(3):599-612. 10.1037/a0026097.
68. Tourangeau R., Rips L.J., Rasinski K. The psychology of survey response 2000, Cambridge University Press, Cambridge, UK.
69. Vallacher R.R., Solodky M. Objective self-awareness, standards of evaluation, and moral behavior. Journal of Experimental Social Psychology 1979, 15(3):254-262.
70. Vancouver J.B., Kendall L.N. When self-efficacy negatively relates to motivation and performance in a learning context. Journal of Applied Psychology 2006, 91:1146-1153. 10.1037/0021-9010.91.5.1146.
71. Vancouver J.B., Thompson C.M., Tischner E.C., Putka D.J. Two studies examining the negative effect of self-efficacy on performance. Journal of Applied Psychology 2002, 87:506-516. 10.1037/0021-9010.87.3.506.
72. Warr P., Downing J. Learning strategies, learning anxiety and knowledge acquisition. British Journal of Psychology 2000, 91:311-333. 10.1348/000712600161853.
73. Weinstein C.E., Schulte A.C., Palmer D.R. LASSI: Learning and Study Strategies Inventory 1987, H & H Publishing, Clearwater, FL.
74. Winne P.H. A metacognitive view of individual differences in self-regulated learning. Learning and Individual Differences 1996, 8:327-351.
75. Wu C.-H., Griffin M.A. Longitudinal relationships between core self-evaluations and job satisfaction. Journal of Applied Psychology 2011, September 26, (Advance online publication). 10.1037/a0025673.
76. Yeo G.B., Frederiks E.R. Cognitive and affective regulation: Scale validation and nomological network analysis. Applied Psychology 2011, 60(4):546-575.
77. Yeo G., Loft S., Xiao T., Kiewitz C. Goal orientations and performance: Differential relationships across levels of analysis and as a function of task demands. Journal of Applied Psychology 2009, 94:710-726. 10.1037/a0015044.
78. Yeo G.B., Neal A. A multilevel analysis of effort, practice, and performance: Effects; of ability, conscientiousness, and goal orientation. Journal of Applied Psychology 2004, 89(2):231-247. 10.1037/0021-9010.89.2.231.
79. Yeo G.B., Neal A. An examination of the dynamic relationship between self-efficacy and performance across levels of analysis and levels of specificity. Journal of Applied Psychology 2006, 91:617-631. 10.1037/0021-9010.91.5.1088.
80. Yeo G., Neal A. Subjective cognitive effort: A model of states, traits, and time. Journal of Applied Psychology 2008, 93(3):617-631. 10.1037/0021-9010.93.3.617.
81. Young P. Scientific social surveys and research 1939, Prentice Hall, New York.
82. Zijlstra F.R.H., Roe R.A., Leonora A.B., Krediet I. Temporal factors in mental work: Effects of interrupted activities. Journal of Occupational and Organizational Psychology 1999, 72:163-185.
83. Zimmerman B.J., Labuhn A.S. Self-regulation of learning: Process approaches to personal development. APA educational psychology handbook 2012, Vol. 1:339-425. American Psychological Association, Washington.