The development and evaluation of a survey to measure user engagement

Journal of the American Society for Information Science and Technology

Volumn 61, Issue 1, 2010, Pages 50-69

  O'Brien, Heather L ^a,b   Toms, Elaine G ^a,b  

^a UNIVERSITY OF BRITISH COLUMBIA   (Canada)

^b DALHOUSIE UNIVERSITY   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

CORE SET; EXPLORATORY FACTOR ANALYSIS; INTERACTIVE SYSTEM; ONLINE SHOPPING; PERCEIVED USABILITY; RELIABILITY AND VALIDITY; SOFTWARE APPLICATIONS; STRUCTURAL EQUATION MODELING; USER ENGAGEMENT; USER EXPERIENCE; USER-SYSTEM INTERACTION;

ELECTRONIC COMMERCE; FELT;

RELIABILITY ANALYSIS;

EID: 72849147106     PISSN: 15322882     EISSN: 15322890     Source Type: Journal    
DOI: 10.1002/asi.21229     Document Type: Review

References (92)

1. Aboulafia, A., & Bannon, L.J. (2004). Understanding affect in design: An outline conceptual framework. Theoretical Issues in Ergonomic Science, 5(1), 4-15.
2. Adams, D.A., Nelson, R.R., &Todd, P.A. (1992). Perceived usefulness, ease of use, and usage of information technology: A replication. MIS Quarterly, 16(2), 227-247.
3. Arnold, M.J., & Reynolds, K.E. (2003). Hedonic shopping motivations. Journal of Retailing, 79, 77-95.
4. Babin, B.J., Darden, W.R., & Griffin, M. (1994). Work and/or fun: Measuring hedonic and utilitarian shopping value. The Journal of Consumer Research, 20(4), 644-656.
5. Bannon, L. (2005). A human-centred perspective on interaction design. In A. Pirhonen, H. Isomäki, C. Roast, & P. Saariluoma (Eds.), Future Interaction Design (pp. 9-30). Berlin, Germany: Springer.
6. Beardsely, M. (1982). The Aesthetic Point of View: Selected Essays. Ithaca, NY: Cornell University Press.
7. Bojko, A. (2006). Using eye tracking to compate web page designs: A case study. Journal of Usability Studies, 3(1), 112-120.
8. Boyle, G.J. (1983). Critical review of state-trait curiosity test development. Motivation and Emotion, 7(4), 377-397.
9. Brooke, J. (1996). SUS: A 'quick and dirty' usability scale. In P.W. Jordan, B. Thomas, I.L. McClelland, & B. Weerdmeester (Eds.), Usability Evaluation in Industry. Philadelphia: Taylor and Francis.
10. Champion, E. (2003). Applying game design theory to virtual heritage environments. In Proceedings of the First International Conference on Computer Graphics and Interactive Techniques in Australasia and South East Asia (pp. 273-274). Melbourne, Australia. New York: ACM Press.
11. Chapman, P. (1997). Models of Engagement: Intrinsically Motivated Interaction with Multimedia Learning Software. Unpublished master's thesis, University of Waterloo, Waterloo, Ontario, Canada.
12. Chapman, P., Selvarajah, S., & Webster, J. (1999). Engagement in multimedia training systems. In Proceedings of the 32nd Hawaii International Conference on System. Sciences (HICSS-32) (Vol/Track 1, pp. 1-9). Los Alamitos, CA: IEEE Press. Retrieved September 17, 2009, from http://www.ieeexplore.ieee. org/stamp/stamp.jsp?tp=&arnumber = 772808 &isnumber= 16781
13. Childers, T.L., Carr, C.L., Peck, J., & Carson, S. (2001). Hedonic and utilitarian motivations for online retail shopping behavior. Journal of Retailing, 77,511-535.
14. Choi, D., & Kim, J. (2004). Why people continue to play online games: In search of critical design, factors to increase customer loyalty to online contests. Cyber Psychology and Behavior, 7(1), 11-24.
15. Clip Culture: Internet video. (2006). The Economist, 379(8475), 73.
16. Cook, C., Heath, E, & Thompson, R. (2000). A meta-analysis of response rates in Web- or Internet-based surveys. Educational & Psychological Measurement, 60(6), 821-836.
17. Cortina, J.M. (1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 28(1), 98-104.
18. Davis, F.D., Bagozzi, R.P., & Warshaw, P.R. (1989). User acceptance of computer technology: A comparison of two theoretical models. Management Science, 35(8), 982-1003.
19. DeLone, WH., & McLean, E.R. (1992). Information systems success: The quest for the dependent variable. Information Systems Research, 3(1), 60-95.
20. DeVellis, R.F. (2003). Scale development: Theory and applications (2nd ed. Vol. 26). Thousand Oaks, CA: Sage.
21. Douglas, Y., & Hargadon, A. (2000). The pleasure principle: Immersion., engagement, flow. In F. Shipman (Ed.), Proceedings of the 11th ACM Conference on Hypertext and Hypermedia (Hypertext 2000) (pp. 153-160). New York: ACM Press.
22. Feedback. (1995). The Penguin dictionary of psychology. Retrieved 29 April 2005, from xreferplus. http://www.xreferplus.com/entry/150004FinneranZhang
23. Forlizzi, J., & Battarbee, K. (2004, August 1-4). Understanding experience in interactive systems. In D. Benyon, P. Moody, D. Gruen, & I. McAra-McWilliam (Eds.), Proceedings of the Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques (DIS 2004) (pp. 261-268). New York: ACM Press.
24. Gefen, D., & Straub, D. (2000). The relative importance of perceived ease of use in IS adoption: A study of e-commerce adoption. Journal of the Association of Information Systems, 1(8), 1-20.
25. Ghani, J. A., Supnick, R., & Rooney, P. (1991, December 16-18). The experience of flow in computer mediated and in face to face groups. Paper presented at the International Conference on Information Systems, New York, NY.
26. Guay, F., Vallerand, R.J., & Blanchard, C. (2000). On the assessment of situational intrinsic and extrinsic motivation: The Situational Motivation Scale (SIMS). Motivation and Emotion, 24(3), 175-213.
27. Hargens, L.L. (2001). Sampling procedures. In Encyclopedia of Sociology, (2nd ed., Vol. 4, pp. 2444-2449). New York: Macmillan Reference.
28. Haywood, N., & Cairns, P. (2005). Engagement with an interactive museum exhibit. In T. McEwan, J. Gulliksen, & D. Benyon (Eds.), People and Computers XIX-The Bigger Picture, Proceedings of HCI2005. London: Springer.
29. Hebb, D.O. (1972). Textbook of psychology, 3rd edition. Philadelphia: WB Saunders.
30. Huang, M. (2003). Designing website attributes to induce experiential encounters. Computers in Human Behavior, 19, 425-442.
31. Hull, R., & Reid, J. (2003). Designing engaging experiences with children and artistis. In M.A. Blythe, A.F. Monk, K. Overbeeke & P.C. Wright (Eds.), Funology:From usability to enjoyment (pp. 179-187). Amsterdam: Kluwer Academic Publishers.
32. Hutcheson, G.D., Sofroniou, N. (1999). The multivariate social Scientist: Introductory statistics using generalized linear models. London: Sage.
33. Hutchins E.L., Holland J.D., & Norman D.A. (1986). Direct manipulation interfaces. In D.A. Norman, S.W. Draper (Eds.), User centered system, design: New perspectives on human-computer interaction. Mahwah, NJ: Lawrence Erlbaum Associates.
34. Jacques, R.D. (1996). The nature of engagement and its role in hypermedia evaluation and design. Unpublished doctoral dissertation, South Bank University, London.
35. Jacques, R., Preece, J., & Carey, T. (1995). Engagement as a design concept for multimedia. Canadian Journal of Educational Communication, 24(1), 49-59.
36. Jennings, M. (2000). Theory and models for creating engaging and immersive ecommerce websites. In Proceedings of the 2000 ACM SIGCPR Conference on Computer Personnel Research (pp. 77-85). New York: ACM Press.
37. Jones, M. (n.d.). Creating engagement in computer-based learning environments. ITFORUM Paper 30. Retrieved 5 March, 2005, from http://itech1. coe.uga.edu/itforum/paper30/paper30.ht.ml
38. Kappelman, L.A. (1995). Measuring user involvement: A. diffusion of innovation perspective. Database Advances, 26(2/3), 65-86.
39. Kline, T.J.B., Sulsky, L.M., & Rever-Moriyama, S.D. (2000). Common method variance and specification errors: A practical approach to detection. The Journal of Psychology, 134(4), 401-421.
40. Konradt, U., & Sulz, K. (2001). The experience of flow in interacting with a hypermedia learning environment. Journal of Educational Multimedia and Hypermedia, 10(1), 69-84.
41. Kelloway, E.K. (1998). Using LISREL for structural equation modeling: A researcher's guide. Thousand Oaks, CA: Sage Publications.
42. Kelley, E. (2005). Context and leadership in the remote environment. Doctoral dissertation. Saint Mary's Univeristy, Halifax, Nova Scotia, Canada. [Publication No. AATNR03071], Retrieved October 6,2009, from http://www.proquest.com/en-US/catalogs/databases/detail/dai.shtml
43. Laarni N.S., Ravaja J., Kallinen, K., & Saari, T. (2004). Transcendent experience in the use of computer-based media. In Proceedings Third Nordic conference on. Human-computer interaction (NordiCHI '04) (pp. 409-412). New York: ACM Press.
44. Laurel, B. (1993). Computers as Theatre. Reading, MA: Addison-Wesley.
45. pervievedLavie, T, & Tractinsky, N. (2004). Assessing dimensions of pervieved visual aesthetics of web sites. International Journal of Human-Computer Studies, 60, 269-298.
46. Lindgaard, G. (2004). Adventurers versus nit-pickers on affective computing. Interacting with Computers, 16, 723-728.
47. Lindgaard, G., Fernandes, G., Dudek, C., & Brown, J. (2006). Attention web designers: You have 50 milliseconds to make a good first impression ! Behaviour & Information Technology, 25, 115-126.
48. Litman, J. A., & Spielberger, C. D. (2003). Measuring epistemic curiosity and its diverse and specific components. Journal of Personality Assessment, 90(1), 75-86.
49. Liu, Y. (2003). Developing a scale to measure the interactivity of Web sites. Journal of Advertising Research, 43(2), 207-216.
50. Luszczynska, A., Diehl, M., Gutierrez-Dona, B., Kuusinen, P., & Schwarzer, R. (2004). Measuring one component of dispositional selfregulation: Attention control in goal pursuit. Personality and Individual Differences, 37, 555-566. (Pubitemid 39007460)
51. Madden, M. (2007). Online Video. Washington, DC: Pew Internet & American Life Project.
52. Makkonen, P. (1997). Does collaborative hypertext support better engagement in learning the basics of informatics? SIGCSE Bulletin, 29(3), 130-132.
53. Mandryk, R.L. (2004, April 24-29). Objectively Evaluating Entertainment Technology. In Proceedings of ACM CHI 2004 Conference on Human Factors in Computing Systems, Extended Abstracts in Computing Systems (CHI '04) (pp. 1057-1058). New York: ACM Press.
54. Mathwick, C., Malhotra, N., & Rigdon, E. (2001). Experiential value: Conceptualization, measurement and application in the catalog and internet shopping environment. Journal of Retailing, 77, 39-56. (Pubitemid 33668950)
55. Mathwick, C., & Rigdon, E. (2004). Play, flow, and the online search, experience. Journal of Consumer Research, 3.1(2), 324-332. (Pubitemid 39518977)
56. Matlin, M.W. (1994). Cognition. (3rd ed.). Orlando, FL: Harcourt Brace.
57. Nahl, D. (2007). The centrality of affect in information behavior. In D. Nahl & D. Bilal (Eds.), Information and emotion: The emergent affective paradigm, in information behavior research and theory (pp. 3-38). Medford, NJ: Information Today.
58. Novak, T.P., Hoffman, D.L., & Yung, Y.F. (2000). Measuring the customer experience in online environments: A structural modeling approach. Marketing Science, 19(1), 22-42.
59. Nunnally, J.C. (1978). Pyschometric theory (2nd ed). New York: McGraw Hill.
60. O'Brien, H.L., & Toms, E.G. (2008). What is user engagement? A conceptual framework for defining user engagement with technology. Journal of the American Society for Information Science and Technology, 59(6), 938-955.
61. Overbeeke, K., Djajadiningrat, T., Hummels, C., Wensveen, S., & Frens, J. (2003). Let's make things engaging. In M.A. Blythe, A.F. Monk, K. Overbeeke, & P.C. Wright (Eds.), Funology: From usability to enjoyment (pp. 7-17). Amsterdam: Kluwer Academic Press.
62. Pace, S. (2004). A grounded theory of the flow experiences of Web users. International Journal of Human-Computer Studies, 60, 327-363.
63. Park, S.-E., Choi, D., & Kim, J. (2004). Critical factors for the aesthetic fidelity of web pages: Empirical studies with professional web designers and users. Interacting with Computers, 16, 351-376.
64. Peterson, R.A. (2000). Constructing effective questionnaires. Thousand Oaks, CA: Sage.
65. Pine B.J. II, & Gilmore, J.H. (1999). The experience economy. Boston, MA: Harvard Business School Press.
66. Quesenbery, W. (2003). Dimensions of usability. In M. Albers & B. Mazur (Eds.), Content and complexity: Information design in technical communications (pp. 81-102). Mahwah, NJ: Lawrence Erlbaum.
67. Reio, T.G.J. (1997). Effects of curiosity on socialization-related learning and job performance in adults. Unpublished doctoral dissertation, Virginia Polytechnic Institute and State University, Falls Church, VA.
68. Rieber, L.P. (1996). Seriously considering play: Designing interactive learning environments based on the blending of microworlds, simulations, and games. Educational Technology Research & Development, 44(2), 45-58.
69. Said, N.S. (2004). An engaging multimedia design model. In Proceedings of the 2004 Conference in Interaction Design and Children: Building a Community (pp. 169-172). New York: ACM Press.
70. Schmidt, K.E., Bauerly, M., Liu, Y., & Sridharan, S. (2003). Web page aesthetics and performance: A survey and an experimental study. Paper presented at the International Conference on Industrial Engineering, Las Vegas, NV.
71. Schraw, G., Flowerday, T., & Reisetter, M.F. (1998). The role of choice in reader engagement. Journal of Educational Psychology, 90(4), 705-714.
72. Seah, M.-L., & Cairns, P. (2008). From immersion to addiction in video games. In England, D., & Beale, R. (Eds.), Proceedings of Human Computer Interaction Conference, Vol.1 (HCI 2008) (pp. 55-63). Bedfordshire, UK: British Computer Society. Retrieved October 8, 2009, from http://www.users.cs. york.ac.uk/∼pcairns/papers/Seah-Cairns-HCI2008.pdf
73. Shang, R.-A., Chen, Y.-C. & Shen, L. (2005). Extrinsic versus intrinsic motivations for consumers to shop on-line. Information Management, 42(3), 401-413.
74. Shenkman, B.N., & Jonsson, F. (2000). Aesthetics and preferences of web pages. Behaviour and Information Technology, 19(5), 367-377.
75. Shernoff, D.., Csikszentmihalyi, M., Schneider, B., & Shernoff, E.S. (2003). Student engagement in high school classrooms from the perspective of flow theory. School Psychology Quarterly, 18(2), 158-176. (Pubitemid 37001697)
76. Singleton, R.A., Straits, B.C., & Straits, M.M. (1993). Approaches to social science research. New York: Oxford University Press.
77. Skelly, T.C., Fries, K., Linnett, B., Nass, C., & Reeves, B. (1994, April 24-28). Seductive interfaces: Satisfying a mass audience. In C Plaisant (Ed.), Conference Companion on Human Factors in Computing Systems (pp. 359-360). New York: ACM Press.
78. Smith, C.A., & Ellsworth, P.C. (1985). Patterns of cognitive appraisal in emotion. Journal of Personality and Social Psychology, 48(4), 813-838.
79. Stone, D., Jarrett, C., Woodroffe, M., & Minocha, S. (2005). User interface design and evaluation. London: Morgan Kaufman Publishers.
80. Tabachnick, B.G., & Fidell, L.S. (2001). Using multivariate statistics (fourth ed.). Needham Heights, MA: Allyn & Bacon.
81. Tractinsky, N. (1997). Aesthetics and apparent usability: Empirically assessing cultural and methodological issues. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems (pp. 115-122). New York: ACM Press.
82. Tractinsky, N., Katz, A.S., & Ikar, D. (2000). What is beautiful is usable. Interacting with Computers, 13, 127-145.
83. Unger, L.S., & Kernan, J.B. (1983). On the meaning of leisure: An investigation of some determinants of the subjective experience. The Journal of Consumer Research, 9(4), 381-392.
84. Vick, R.M., & Ikehara, C.S. (2002). Methodological issues of real time data acquisition, from multiple sources of physiological data. In Proceedings of the 36th. Hawaii International Conference on System Sciences (HICSS36). Los Alamitos, CA: IEEE. Retrieved September 17, 2009, from http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&earnumber=1174295 &isnumber=26341
85. Walker, G.H., Stanton, N.A., & Young, M.S. (2008). Feedback and driver situation awareness (SA): A comparison of SA measures and contexts. Transportation Research Part F: Traffic Psychology and Behaviour, 11 (4), 282-299.
86. Watson, D., Clark, L.A., & Tellegen, A. (1988). Development and validation of brief measures of positive and negative affect: The PANAS Scales. Journal of Personality and Social Psychology, 54, 1063-1070.
87. Webster, J., & Ahuja, J.S. (2006). Enhancing the design of web navigation systems: the influence of user disorientation on engagement and performance. MIS Quarterly, 30(3), 661-678. (Pubitemid 46051916)
88. Webster, J., & Ho, H. (1997). Audience engagement in multimedia presentations. The DATA BASE for Advances in Information Systems, 28(2), 63-77.
89. Webster, J., Trevino, L.K., & Ryan, L. (1993). The dimensionality and correlates of flow in human-computer interactions. Computers in Human Behavior, 9, 411-426.
90. Witmer, B.G., & Singer, M.J. (1998). Measuring presence in virtual environments: A presence questionnaire. Presence, 7(3), 225-240. (Pubitemid 128084456)
91. Wright, P.C., & McCarthy, J. (2005). The value of the novel in designing for experience. In A. Pirhonen, H. Isomäki, C. Roast, & P. Saariluoma (Eds.), Future interaction design (pp. 9-30). Berlin: Springer.
92. Zhang, P., & von Dran, G.M. (2000). Satisfiers and dissatisfiers: A two-factor model for website design and evaluation. Journal of the American Society for Information Science, 51(14), 1253-1268. (Pubitemid 32074825)