Multimodal behavior and interaction as indicators of cognitive load

ACM Transactions on Interactive Intelligent Systems

Volumn 2, Issue 4, 2012, Pages

  Chen, Fang ^a   Ruiz, Natalie ^a   Choi, Eric ^a   Epps, Julien ^a   Khawaja, M Asif ^a   Taib, Ronnie ^a   Yin, Bo ^a   Wang, Yang ^a  

^a CSIRO   (Australia)

Author keywords

Assessment; Cognitive load; Multimodal; Pen input

Indexed keywords

BEHAVIORAL RESEARCH; ELECTROPHYSIOLOGY; FLIGHT PATHS; NUCLEAR REACTORS;

ASSESSMENT; COGNITIVE LOADS; GALVANIC SKIN RESPONSE; INTELLIGENT INTERACTIVE SYSTEMS; MULTI-MODAL; MULTI-MODAL INTERACTIONS; PEN INPUT; REAL WORLD ENVIRONMENTS;

COGNITIVE SYSTEMS;

EID: 84983587904     PISSN: 21606455     EISSN: 21606463     Source Type: Journal    
DOI: 10.1145/2395123.2395127     Document Type: Article

References (109)

1. ALIBALI, M. W., KITA, S., AND YOUNG, A. J. 2000. Gesture and the process of speech production: We think, therefore we gesture. Lang. Cogn. Process. 15, 6, 593-613.
2. ARK, W. S., DRYER, D. C., AND LU, D. J. 1999. The emotion mouse. In Proceedings of the 8th International Conference on Human-Computer Interaction (HCI'99): Ergonomics and User Interfaces. Vol. 1, Lawrence Erlbaum Association, London.
3. BACKS, R. W. AND WALRATH, L. C. 1992. Eye movement and pupillary response indices of mental workload during visual search of symbolic displays. Appl. Ergonom. 23, 243-254.
4. BADDELEY, A. 2003. Working memory and language: An overview. J. Comm. Disord. 36, 189-208.
5. BADDELEY, A. D. 1992. Working memory. Science 255, 556-559.
6. BERTHOLD, A. AND JAMESON, A. 1999. Interpreting symptoms of cognitive load in speech input. In Proceedings of the 7th International Conference on User Modeling (UM'99).
7. BRENNER,M.,SHIPP, T., DOHERTY, E., ANDMORRISSEY, P. 1985. Voice measures of psychological stress: Laboratory and field data. In Vocal Fold Physiology, Biomechanics, Acoustics, and Phonatory Control, I. Titze and R. Scherer, Eds., 239-248.
8. BRUNKEN, R., PLASS, J. L., AND LEUTNER, D. 2003. Direct measurement of cognitive load in multimedia learning. Educ. Psychol. 38, 1, 53-61.
9. BYRNE, A. J., SELLEN, A. J., AND JONES, J. G. 1998. Errors on anaesthetic record charts as a measure of anaesthetic performance during simulated critical incidents. Brit. J. Anaesth. 80, 58-62.
10. CHALKER, S. AND WEINER, E. 1998. The Oxford Dictionary of English Grammar. Oxford University Press, New York.
11. CHANDLER, P. AND SWELLER, J. 1991. Cognitive load theory and the format of instruction. Cogn. Instruct. 8, 4, 293-332.
12. COWAN, N. 2001. The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behav. Brain Sci. 24, 1, 87-114.
13. DALE, R., ROCHE, J., SNYDER, K., AND MCCALL, R. 2008. Exploring action dynamics as an index of pairedassociate learning. PLoS ONE 3, 3.
14. DECHERT, H. W. AND RAUPACH, M. 1980. Towards a Cross-Linguistic Assessment of Speech Production. Lang, Frankfurt.
15. DELIS, D. C., KRAMER, J. H., AND KAPLAN, E. 2001. The Delis-Kaplan Executive Function System. The Psychological Corporation.
16. DONNER, W. AND HANCOCK, J. T. 2011. Upset now? Emotion contagion distributed groups. In Proceedings of the International Conference on Computer-Human Interaction (CHI'11).
17. ESPOSITO, A., STEJSKAL, V., SMEKAL, Z., AND BOURBAKIS, N. 2007. The significance of empty speech pauses: Cognitive and algorithmic issues. In Proceedings of the International Symposium on Brain, Vision and Artificial Intelligence (BVAI'07). Lecture Notes in Computer Science, vol. 4729. Springer.
18. FERNANDEZ, R. AND PICARD, R. W. 2003. Modeling drivers' speech under stress. Speech Comm. 40, 1-2.
19. FLESCH, R. 1948. A new readability yardstick. J. Appl. Psychol. 32, 3, 221-233.
20. FRANKISH, G., HULL, R., AND MORGAN, P. 1995. Recognition accuracy and user acceptance of pen interfaces. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computer Systems (CHI'95)
21. FREUND, Y. 1995. Boosting a weak learning algorithm by majority. Inf. Comput. 121, 2, 256-285.
22. FREUND, Y. AND SCHAPIRE, R. E. 1997. A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. Syst. Sci. 55, 1, 119-139.
23. GALEN, G. P. AND VAN HUYGEVOORT, M. 2000. Error, stress and the role of neuron-motor noise in space oriented behaviour. Biol. Psychol. 51, 151-171.
24. GAWRON, V. J. 2000. Human Performance Measures Handbook. Lawrence Erlbaum Associates.
25. GOLDIN-MEADOW, S., NUSBAUM, H., KELLY, S., AND WAGNER, S. 2001. Explaining math: Gesturing lightens the load. Psychol. Sci. 12, 516-522.
26. GOPHER, D. AND BRAUNE, R. 1984. On the psychophysics of workload: Why bother with subjective measures? Hum. Factors 26, 519-532.
27. GUNNING, R. 1952. The Technique of Clear Writing. McGraw-Hill.
28. GUTL, C.,PIVEC, M.,TRUMMER, C.,GARCABARRIOS, V. M.,MDRISTSCHER, F.,PRIPFL, J., AND UMGEHER, M. 2005. Adele (adaptive e-learning with eye-tracking): Theoretical background, system architecture and application scenarios. Euro. J. Open Dist. E-Learn. 2.
29. HANCOCK, J. T., GEE, K., CIACCIO, K., AND LIN, J. M. 2008. I'm sad, you're sad: Emotion contagion in cmc. In Proceedings of the ACM Conference on Computer Supported Cooperative Work (CSCW'08)
30. HANSEN, J. H. L. 1996. Analysis and compensation of speech under stress and noise for environmental robustness in speech recognition. Speech Comm. 20, 1-2, 151-173.
31. HOCKEY, G. R. J. 2003. Operator functional state as a framework for the assessment of performance degradation. In NATO Advances Research Workshop on Operator Functional State and Impaired Performance in Complex Work Environments II
32. IKEHARA, C. S. AND CROSBY, M. E. 2005. Assessing cognitive load with physiological sensors. In Proceedings of the 38th Annual Hawaii International Conference on System Sciences (HICSS'05)
33. IQBAL, S. T.,ZHENG, X. S., ANDBAILEY,B. P. 2004. Task-Evoked pupillary response to mental workload in humancomputer interaction. In Proceedings of the International Conference on Computer-Human Interaction (CHI'04)
34. JACOBS, S. C., FRIEDMAN, R., PARKER, J. O., TOFLER, G. H., JIMENEZ, A. H., MULLER, J. E., AND STONE, P. H. 1994. Use of skin conductance changes during mental stress testing as an index of autonomic arousal in cardiovascular research. Amer. Heart J. 128, 1, 1170-1177.
35. JAMESON, A., KEIFER, J.,MULLER, C., GROSSMANN-HUTTER, B.,WITTIG, F., AND RUMMER, R. 2009. Assessment of a user's time pressure and cognitive load on the basis of features of speech. In Resource-Adaptive Cognitive Processes, M. W. Crocker and J. Siekmann, Eds., Springer, 171.
36. KALYUGA, S. 2007. Enhancing instructional efficiency of interactive e-learning environments: A cognitive load perspective. Educ. Psychol. Rev. 19, 387-399.
37. KATZ, C., FRASER, E. B., AND WAGNER, T. L. 1998. Rotary-Wing crew communication patterns across workload levels. In Proceedings of the RTO HFM Symposium on Current Aeromedical Issues in Rotary Wing Operations
38. KENNEDY, D. AND SCHOLEY, A. 2000. Glucose administration, heart rate and cognitive performance: Effects of increasing mental effort. Psychopharm. 149, 1, 63-71.
39. KERANEN, H.,VAYRYNEN, E., PAAKKONEN, R., LEINO, T., KURONEN, P., TOIVANEN, J., AND SEPPANEN, T. 2004. Prosodic features of speech produced by military pilots during demanding tasks. In Proceedings of the Fonetiikan Paivat Conference
40. KERR, B. 1973. Processing demands during mental operations: Memory and cognition. J. Memor. Cogn. 1, 401-412.
41. KETTEBEKOV, S. 2004. Exploiting prosodic structuring of coverbal gesticulation. In Proceedings of the 6th International Conference on Multimodal Interfaces (ICMI'04)
42. KHAWAJA, M. A., CHEN, F., AND MARCUS, N. 2010. Using language complexity to measure cognitive load for adaptive interaction design. In Proceedings of the International Conference on Intelligent User Interfaces (IUI'10).
43. KHAWAJA, M. A., CHEN, F., OWEN, C., AND HICKEY, G. 2009. Cognitive load measurement from user's linguistic speech features for adaptive interaction design. In Proceedings of the International Conference on Human-Computer Interaction (INTERACT'09), Part I. Lecture Notes in Computer Science, vol. 5726, Springer.
44. KIRSCHNER, F., PAAS, F., AND KIRSCHNER, P. A. 2009. Cognitive load approach to collaborative learning: United brains for complex tasks. Educ. Psychol. Rev. 21, 1.
45. KRAMER, A. F. 1991. Physiological metrics of mental workload: A review of recent progress. In Multiple Task Performance, D. L. Damos, Ed., Taylor and Francis, London, 279-328.
46. LAJOIE, S. P. 2000. Computers as Cognitive Tools: No More Walls. Lawrence Erlbaum, Hillsdale, NJ.
47. LE, P., AMBIKAIRAJAH, E., EPPS, J., VIDHYASAHARAN, S., AND CHOI, E. 2011. Investigation of spectral centroid features for cognitive load classification. Speech Comm. 53, 4, 540-551.
48. LE, P., EPPS, J., AMIKAIRAJAH, E., AND SETHU, V. 2010a. Robust speech-based cognitive load classification using a multi-band approach. In Proceedings of the APSIPA Annual Summit and Conference (APSIPA'10)
49. LE, P., EPPS, J., CHOI, E., AND AMBIKAIRAJAH, E. 2010b. A study of voice source and vocal tract filter based features in cognitive load classification. In Proceedings of the International Conference on Pattern Recognition (ICPR'10)
50. LENNON, C. AND BURDICK, H. 2004. The lexile framework as an approach for reading measurement and success. http://www.Lexile.com.
51. LIPP, O. V. AND NEUMANN, D. L. 2004. Attentional blink reflex modulation in a continuous performance task is a modality specific. Psychophysiol. 41, 3, 417-425.
52. LIU, J., WONG, C. K., AND HUI, K. K. 2003. An adaptive user interface based on personalised learning. IEEE Intell. Syst. 18, 2, 52-57.
53. LIVELY, E., PISONI, D. B., SUMMERS, W. V., AND BERNACKI, R. 1993. Effects of cognitive workload on speech production: Acoustic analyses and perceptual consequences. J. Acout. Soc. Amer. 93, 2962-2973.
54. MACKINTOSH, C. 2010. AIS React Software v.6.6. Australian Institute of Sport.
55. MARCUS, N., COOPER, M., AND SWELLER, J. 1996. Understand instructions. Educ. Psychol. 88, 1, 49-63.
56. MARSHALL, S. P., PLEYDELL-PEARCE, C. W., AND DICKSON, B. T. 2003. Integrating psychological measures of cognitive workload and eye movements to detect strategy shifts. In Proceedings of the 36th Hawaii International Conference on System Sciences (HICSS'03). Vol. 5, IEEE Computer Society, Los Alamitos, CA.
57. MCKINSTRY, C., DALE, R., AND SPIVEY, M. J. 2008. Action dynamics reveal parallel competition in decision making. Psychol. Sci. 19, 1, 22-24.
58. MCLAUGHLIN, H. G. 1969. SMOG grading: A new readability formula. J. Read. 12, 8, 639-646.
59. MOUSAVI, S. Y.,LOW, R., AND SWELLER, J. 1995. Measurement and analysis methods of heart rate and respiration for use in applied environments. J. Educ. Psychol. 87, 2, 319-334.
60. MULLER, C., GROSSMANN-HUTTER, B., JAMESON, A., RUMMER, R., AND WITTIG, F. 2001. Recognising time pressure and cognitive load on the basis of speech: An experimental study. In Proceedings of the 18th International Conference on User Modeling (UM'01)
61. NICKEL, P. AND NACHREINER, F. 2000. Psychometric properties of the 0.1Hz component of hrv as an indicator of mental strain. In Proceedings of the 14th Triennial Congress of the International Ergonomics Association and 44th Annual Meeting of the Human Factors and Ergonomics Society (IEA'00/HFES'00)
62. O'DONNELL, R. D. AND EGGEMEIER, F. T. 1986. Workload assessment methodology. In Handbook of Perception and Human Performance, vol. 2, K. R. Boff, L. Kaufman, and J. P. Thomas, Eds., Wiley, New York, 1-49.
63. OVIATT, S. 1997. Multimodal interactive maps: Designing for human performance. Hum.-Comput. Interact. 12, 93-129.
64. OVIATT, S. 2006. Human-Centered design meets cognitive load theory: Designing interfaces that help people think. In Proceedings of the ACM Conference on Multimedia
65. OVIATT, S. 2009. Designing interfaces that stimulate ideational superfluency. In Proceedings of the Conference on Research Foundations for Understanding Book and Reading in the Digital Age: Implementing New Knowledge Environments
66. OVIATT, S., COULSTON, R., AND LUNSFORD, R. 2004. When do we interact multimodally?: Cognitive load and multimodal communication patterns. In Proceedings of the 6th International Conference on Multimodal Interfaces (ICMI'04)
67. PAAS, F., AYERS, P., AND PACHMAN, M. 2008. Assessment of cognitive load in multimedia learning: Theory, methods and applications. In Recent Innovations in Educational Technology that Facilitate Student Learning, D. H. Robinson and G. Schraw, Eds., 11-36.
68. PAAS, F.,TUOVINEN, J. E., TABBERS, H., AND GERVEN, P. 2003. Cognitive load measurement as a means to advance cognitive load theory. Educ. Psychol. 38, 1, 63-71.
69. PALIWAL, K. K. 1998. Spectral subband centroid features for speech recognition. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP'98).
70. PARASURAMAN, R., SHERIDAN, T. B., AND WICKENS, C. D. 2008. Situation awareness, mental workload, and trust in automation: Viable, empirically supported cognitive engineering constructs. J. Cogn. Engin. Decis. Making 2, 2, 140-160.
71. PICARD, R. 1997. Affective Computing. MIT Press.
72. RECK, R. P. AND RECK, R. A. 2007. Generating and rendering readability scores for project Gutenberg tests. In Proceedings of the Corpus Linguistics Conference
73. REYNOLDS, D. A. AND ROSE, R. C. 1992. An integrated speech-background model for robust speaker identification. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP'92)
74. ROSENBAUM, D. A. 2005. The Cinderella of psychology: The neglect of motor control in the science of mental life and behavior. Amer. Psychol. 60, 308-317.
75. RUBINE, D. 1991. Specifying gestures by example. In Proceedings of the 18th Annual ACM Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'91). ACM Press, New York.
76. RUFFELL SMITH, H. P. 1979. A simulator study of the interaction of pilot workload with errors, vigilance, and decisions. NASA tech. memo, Moffett Field, CA: NASA Ames Research Center.
77. RUIZ, N. 2011. Cognitive load measurement in multimodal interfaces. Ph.D. dissertation, University of New South Wales, Sydney, Australia.
78. RUIZ, N., LIU, G., YIN, B., FARROW, D., AND CHEN, F. 2010. Teaching athletes cognitive skills: Detecting load in speech input. In Proceedings of the 24th BCS Conference on Human-Computer Interaction (HCI'10)
79. RUIZ, N., TAIB, R., AND CHEN, F. 2006. Examining the redundancy of multimodal input. In Proceedings of the Annual Conference of the Australian Computer-Human Interaction Special Interest Group (OzCHI'06)
80. RUIZ, N., TAIB, R., AND CHEN, F. 2011. Freeform pen input as evidence of cognitive load and expertise. In Proceedings of the International Conference on Multimodal Interfaces
81. RUIZ, N., TAIB, R., SHI, Y., CHOI, E., AND CHEN, F. 2007. Using pen input features as indices of cognitive load. In Proceedings of the 9th International Conference on Multimodal Interfaces (ICMI'07)
82. SCHAPIRE, R. E. 1990. The strength of weak learnability. Mach. Learn. 5, 2, 197-227.
83. SCHAPIRE, R. E., FREUND, Y., BARTLETT, P., AND LEE,W. S. 1998. Boosting the margin: A new explanation for the effectiveness of voting methods. Ann. Statist. 26, 5, 1651-1686.
84. SCHILPEROORD, J. 2001. On the cognitive status of pauses in discourse production. In Contemporary Tools and Techniques for Studying Writing, T. Olive and C. M. Levy, Eds., Kluwer Academic Publishers, London.
85. SCHWARTZ, D. L. AND HEISER, J. 2006. Spatial representations and imagery in learning. In Handbook of the Learning Sciences, K. Sawywe, Ed., Lawrence Erlbaum.
86. SEXTON, J. B. AND HELMREICH, R. L. 2000. Analyzing cockpit communication: The links between language, performance, error and workload. J. Hum. Perform. Extreme Environ. 5, 1, 63-68.
87. SHI, Y., RUIZ, N., TAIB, R., CHOI, E., AND CHEN, F. 2007. Galvanic skin response (gsr) as an index of cognitive load. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI'07)
88. SPIVEY, M. J., GROSJEAN, M., AND KNOBLICH, G. 2005. Continuous attraction toward phonological competitors. Proc. Nat. Acad. Sci. Unit. Stat. Amer. 102, 29, 10393-10398.
89. STROOP, J. R. 1935. Studies of interference in serial verbal reactions. J. Experiment. Psychol.
90. SWELLER, J., MERRIENBOER, J., AND PAAS, F. 1998. Cognitive architecture and instructional design. Educ. Psychol. Rev. 10, 3, 251-296.
91. TOLKMITT, E. J. AND SCHERER, K. R. 1986. Effect of experimentally induced stress on vocal parameters. J. Experiment. Psychol. 12, 302-312.
92. URE, J. 1971. Lexical density and register differentiation. In Applications of Linguistics, G. Perren and T. Trim, Eds., Cambridge University Press, 443-452.
93. VANDERBERG, R. AND SWANSON, H. L. 2007. Which components of working memory are important in the writing process? Read. Writ. Interdiscpl. J. 20, 7.
94. WICKENS, C. D. AND HOLLANDS, J. G. 2000. Engineering Psychology and Human Performance 3rd Ed. Pearson/Prentice-Hall, Upper Saddle River, NJ.
95. WILSON, G. F. AND RUSSELL, C. A. 2003. Real-Time assessment of mental workload using psychological measures and artificial neural network. Hum. Factors 45, 4, 635-643.
96. WOOD, C., TORKKOLA, K., AND KUNDALKAR, S. 2004. Using driver's speech to detect cognitive workload. In Proceedings of the 9th Conference on Speech and Computer (SPECOM'04). International Speech Communication Association Press.
97. YAP, T. 2011. Speech production under cognitive load: Effects and classification. Ph.D. thesis, University of New South Wales, Sydney, Australia.
98. YAP, T. F., AMBIKAIRAJAH, E., EPPS, J., AND CHOI, E. 2010a. Cognitive load classification using formant features. In Proceedings of the IEEE International Conference on Information Sciences, Signal Processing and Their Applications (ISSPA'10)
99. YAP, T. F., EPPS, J., AMBIKAIRAJAH, E., AND CHOI, E. 2010b. An investigation of formant frequencies for cognitive load classification. In Proceedings of the Annual Conference of the International Speech Communication Association (InterSpeech'10)
100. YAP, T. F., EPPS, J., AMBIKAIRAJAH, E., AND CHOI, E. 2011. Formant frequencies under cognitive load: Effects and classification. EURASIP J. Adv. Signal Process
101. YAP, T. F., EPPS, J., CHOI, E., AND AMBIKAIRAJAH, E. 2010c. Glottal features for speech-based cognitive load classification. In Proceedings of the IEEE International Conference on Acoustic, Speech and Signal Processing (ICASSP'10)
102. YIN, B. AND CHEN, F. 2007. Towards automatic cognitive load measurement from speech analysis. In Human- Computer Interaction, Interaction Design and Usability, J. Jacko, Ed. Springer, 1011-1020.
103. YIN, B.,CHEN, F., RUIZ, N., AND AMBIKAIRAJAH, E. 2008. Speech-Based cognitive load monitoring system. In Proceedings of the IEEE International Conference on Acoustic, Speech and Signal Processing (ICASSP'08)
104. YIN, B., RUIZ, N., CHEN, F., AND KHAWAJA, M. A. 2007. Automatic cognitive load detection from speech features. In Proceedings of the Australian Computer-Human Interaction Conference (OzCHI'07)
105. YEH, Y. Y. AND WICKENS, C. D. 1988. Dissociation of performance and subjective measures of workload. Hum. Factors 30, 111-120.
106. YU, K., EPPS, J., AND CHEN, F. 2011a. Cognitive load evaluation of handwriting using stroke-level features. In Proceedings of the International Conference on Intelligent User Interfaces (IUI'11)
107. YU, K., EPPS, J., AND CHEN, F. 2011b. Cognitive load evaluation with pen orientation and pressure. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST'11)
108. YU, K., EPPS, J., AND CHEN, F. 2011c. Cognitive load measurement with pen orientation and pressure. http://icmi11.forge.nicta.com.au/papers/MMCogEmS2011 Yu.pdf.
109. ZANDER, T. O. AND KOTHE, C. 2011. Towards passive brain-computer interfaces: Applying brain-computer interface technology to human-machine systems in general. J. Neur. Engin. 8