Automatic cognitive load detection from face, physiology, task performance and fusion during affective interference

Interacting with Computers

Volumn 26, Issue 3, 2014, Pages 256-268

  Hussain, M Sazzad ^a,b   Calvo, Rafael A ^a   Chen, Fang ^b  

^a UNIVERSITY OF SYDNEY   (Australia)

^b CSIRO   (Australia)

Author keywords

Affect; Artificial intelligence; Cognitive load measurement; Cognitive science; Data fusion; HCI; Machine learning; Multimodality

Indexed keywords

ARTIFICIAL INTELLIGENCE; DATA FUSION; HUMAN COMPUTER INTERACTION; LEARNING SYSTEMS; PSYCHOPHYSIOLOGY;

AFFECT; COGNITIVE LOAD MEASUREMENT; COGNITIVE SCIENCE; COMBINED CLASSIFIERS; DETECTION ACCURACY; MULTI-MODAL FUSION; MULTI-MODALITY; NON-VERBAL INFORMATION;

FACE RECOGNITION;

EID: 84898433867     PISSN: 09535438     EISSN: None     Source Type: Journal    
DOI: 10.1093/iwc/iwt032     Document Type: Article

References (55)

1. Ark, W.S., Dryer, D.C. and Lu, D.J. (1999, August 22-26) The emotion mouse. Paper Presented at the 8th International Conference on Human-Computer Interaction: Ergonomics and User Interfaces, Munich, Germany
2. Ayres, P. and Gog, T. (2009) Editorial: State of the art research into cognitive load theory. Comput. Hum. Behav., 25, 253-257
3. Beatty, J. and Lucero-Wagoner, B. (2000) The Pupillary System. Handbook of Psychophysiology (2nd edn), pp. 142-162. Cambridge University Press, xiii, 1039
4. Berka, C., Levendowski, D.J., Lumicao, M.N., Yau, A., Davis, G., Zivkovic, V.T. and Craven, P.L. (2007) EEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks. Aviation Space Environ. Med., 78, B231-B244
5. Blair, K.S., Smith, B.W., Mitchell, D.G.V., Morton, J., Vythilingam, M., Pessoa, L., et al. (2007) Modulation of emotion by cognition and cognition by emotion. Neuroimage, 35, 430-440
6. Bradley, M.M., Miccoli, L., Escrig, M.A. and Lang, P.J. (2008) The pupil as a measure of emotional arousal and autonomic activation. Psychophysiology, 45, 602-607
7. Calvo, R.A. and D'Mello, S. (2010) Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Trans. Affect. Comput., 1, 18-37
8. Card, S.K., Moran, T.P. and Newell, A. (1983) The Psychology of Human-Computer Interaction. Lawrence Erlbaum Associates, Hillsdale, NJ.
9. Chen, F., Ruiz, N., Choi, E., Epps, J., Khawaja, M.A., Taib, R. and Wang, Y. (2012) Multimodal behaviour and interaction as indicators of cognitive load.ACMTrans. Interact. Intell. Syst., 2(4), 22:1-22:36
10. Chen, S. and Epps, J. (2012) Automatic classification of eye activity for cognitive load measurement with emotion interference. Comput. Methods Programs Biomed., pii:S0169-2607(12)00283- 0; doi:10.1016/j.cmpb.2012.10.021
11. Cowan, N. (2001) The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behav. Brain Sci., 24, 87-114
12. D'Esposito, M., Postle, B.R., Jonides, J. and Smith, E.E. (1999) The neural substrate and temporal dynamics of interference effects in working memory as revealed by event-related functional MRI. Proc. Natl Acad. Sci., 96, 7514-7519
13. De Tjerk, E.G., Henryk, F.R.A. and Neerincx, M.A. (2010) Adaptive automation based on an object-oriented task model: Implementation and evaluation in a realistic C2 environment. J. Cogn. Eng. Decision Making, 4, 152-182
14. Duric, Z., Gray, W.D., Heishman, R., Li, F., Rosenfeld, A., Schoelles, M.J. andWechsler, H. (2002) Integrating perceptual and cognitive modeling for adaptive and intelligent human-computer interaction. Proc. IEEE, 90, 1272-1289
15. Erthal, F.S., de Oliveira, L., Mocaiber, I., Pereira, M.G., Machado- Pinheiro, W., Volchan, E. and Pessoa, L. (2005) Load-dependent modulation of affective picture processing. Cogn. Affect. Behav. Neurosci., 5, 388-395
16. Gopher, D. and Braune, R. (1984) On the psychophysics of workload: Why bother with subjective measures? Hum. Factors: J. Hum. Factors Ergon. Soc., 26, 519-532
17. Gütl, C., Pivec, M., Trummer, C., García-Barrios, V.M., Mödritscher, F., Pripfl, J. and Umgeher, M. (2005) Adele (adaptive e-learning with eye-tracking): Theoretical background, system architecture and application scenarios. Eur. J. Open Distance E-Learning, 2
18. Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P. and Witten, I.H. (2009) The WEKA data mining software: An update. ACM SIGKDD Explorations Newslett., 11, 10-18
19. Hall, M.A. (1999) Correlation-based Feature Selection for Machine Learning. The University ofWaikato
20. Hockey, G.R.J. (2003) Operator Functional State as a Framework for the Assessment of Performance Degradation Operator Functional State: The Assessment and Prediction of Human Performance Degradation in Complex Tasks, pp. 8-23. IOS, Amsterdam
21. Hollender, N., Hofmann, C., Deneke, M. and Schmitz, B. (2010 Integrating cognitive load theory and concepts of human-computer interaction. Comput. Hum. Behav., 26, 1278-1288
22. Huang, W.D. and Tettegah, S. (2010) Cognitive load and empathy in serious games : A conceptual framework. In R. Van Eck (Ed.) Gaming and Cognition: Theories and Practice from the Learning Sciences, pp. 137-151. Hershey, PA: Information Science Reference. doi:10.4018/978-1-61520-717-6.ch006
23. Hussain, M.S., Monkaresi, H. and Calvo, R.A. (2012, December 5-7) Combining classifiers in multimodal affect detection. Paper Presented at The Australian Data Mining Conference (AusDM 2012), Sydney, Australia
24. Hussain, S., Chen, S., Calvo, R.A. and Chen, F. (2011, November 17) Classification of cognitive load from task performance and multichannel physiology during affective changes. Paper Presented at the MMCogEmS: Inferring Cognitive and Emotional States from Multimodal Measures, ICMI 2011Workshop, Alicante, Spain
25. Immordino-Yang, M.H. and Damasio, A. (2007) We feel, therefore we learn: The relevance of affective and social neuroscience to education. Mind Brain Educ., 1, 3-10
26. Jong, T. (2010) Cognitive load theory, educational research, and instructional design: Some food for thought. Instruct. Sci., 38, 105-134
27. Kalyuga, S. (2011) Cognitive load in adaptive multimedia learning. In Calvo, R.A. and D'Mello, S.K. (eds) New Perspectives on Affect and Learning Technologies, Vol. 3, pp. 203-215. Springer, NewYork
28. Kirschner, P.A., Ayres, P. and Chandler, P. (2011) Contemporary cognitive load theory research: The good, the bad and the ugly. Comput. Hum. Behav., 27, 99-105
29. Klingner, J., Tversky, B. and Hanrahan, P. (2010) Effects of visual and verbal presentation on cognitive load in vigilance, memory, and arithmetic tasks. Psychophysiology, 48, 323-332
30. Kramer, A.F. (1991) Physiological metrics of mental workload: A review of recent progress, pp. 279-328. Navy Personnel Research and Development Center, San Diego, CA, USA
31. Kuncheva, L.I. (2004) Combining pattern classifiers: Methods and algorithms. JohnWiley & Sons, Inc., Hoboken, New Jersey
32. Lang, P.J., Bradley, M.M and Cuthbert, B.N. (2005) International Affective Picture system (IAPS): Technical Manual and Affective Ratings Technical Report A-6. University of Florida, Gainesville, FL.
33. Le, P.N., Ambikairajah, E., Epps, J., Sethu, V. and Choi, E.H.C. (2011) Investigation of spectral centroid features for cognitive load classification. Speech Commun., 53, 540-551
34. Le, P.N., Epps, J., Choi, E.H.C. and Ambikairajah, E. (2010, August 23-26)A study of voice source and vocal tract filter based features in cognitive load classification. Paper Presented at the International Conference on Pattern Recognition (ICPR'10), Istanbul, Turkey
35. Levens, S.M. and Phelps, E.A. (2008) Emotion processing effects on interference resolution in working memory. Emotion, 8, 267-280
36. Monkaresi, H., Calvo, R.A. and Hussain, M.S. (2012, May 21-25) Automatic natural expression recognition using headmovement and skin color features. Paper Presented at the International Conference on Advanced Visual Interfaces, AVI 2012, Capri Island (Naples), Italy
37. Nguyen, T., Li, M., Bass, I. and Sethi, I.K. (2005, December 12-14) Investigation of combining SVM and decision tree for emotion classification. Paper Presented at the Seventh IEEE International Symposium on Multimedia, Irvine, CA, USA
38. Odonnell, R. and Eggemeier, F. (1986) Workload assessment methodology. Handbook Percept. Hum. Perform., 2, 42-41
39. Paas, F., Tuovinen, J.E., Tabbers, H. and Van Gerven, P.W.M. (2003) Cognitive load measurement as a means to advance cognitive load theory. Educ Psychol., 38, 63-71
40. Paas, F.G.W.C. and Van Merriënboer, J.J.G. (1994) Variability of worked examples and transfer of geometrical problemsolving skills: A cognitive-load approach. J. Educ. Psychol., 86, 122-133
41. Pantic, M. and Rothkrantz, L.J.M. (2003) Toward an affectsensitive multimodal human-computer interaction. Proc. IEEE, 91, 1370-1390
42. Picard, R.W., Vyzas, E. and Healey, J. (2001) Toward machine emotional intelligence: Analysis of affective physiological state. IEEE Trans Pattern Anal. Mach. Intell., 23, 1175-1191
43. Russell, J.A. (1980) A circumplex model of affect. J. Pers. Social Psychol., 39, 1161-1178
44. Schultz, R., Peter, C., Blech, M., Voskamp, J. and Urban, B. (2007) Towards detecting cognitive load and emotions in usability studies using the RealEYES framework. Usability and Internationalization. HCI Culture, 4559, 412-421
45. Seewald, A.K. (2003, August 9-15) Towards a theoretical framework for ensemble classification. Paper Presented at the 18th International Joint Conference on Artificial Intelligence, Acapulco, Mexico
46. Sharma, R., Pavlovic, V.I. and Huang, T.S. (1998) Toward multimodal human-computer interface. Proc. IEEE, 86, 853-869
47. Shi, Y., Ruiz, N., Taib, R., Choi, E. and Chen, F. (2007, 28 April- 3 May). Galvanic skin response (GSR) as an index of cognitive load. Paper Presented at the CHI'07 Extended Abstracts on Human Factors in Computing Systems, San Jose, CA, USA
48. Siegle, G.J., Ingram, R.E. and Matt, G.E. (2002)Affective interference: An explanation for negative attention biases in dysphoria? Cogn. Ther. Res., 26, 73-87
49. Sperandio, J.C. (1978) The regulation of working methods as a function of work-load among air traffic controllers. Ergonomics, 21, 195-202
50. Stanners, R.F., Coulter, M., Sweet, A.W. and Murphy, P. (1979) The pupillary response as an indicator of arousal and cognition. Motiv. Emotion, 3, 319-340
51. Tian, Y., Kanade, T. and Cohn, J.F. (2002, May 19-20) Evaluation of Gabor-wavelet-based facial action unit recognition in image sequences of increasing complexity. Paper Presented at the 5th IEEE International Conference on Automatic Face and Gesture Recognition, Washington, DC, USA
52. Utthara, M., Suranjana, S., Sukhendu, D. and Pinaki, C. (2010 A survey of decision fusion and feature fusion strategies for pattern classification. IETE Tech. Rev., 27, 293-307
53. Viola, P. and Jones, M. (2001, December 8-14) Rapid object detection using a boosted cascade of simple features. Paper Presented at the Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2001), Kauai, HI, USA
54. Yerkes, R.M. and Dodson, J.D. (1908) The relation of strength of stimulus to rapidity of habit-formation. J. Comp. Neurol. Psychol., 18, 459-482
55. Yin, B., Chen, F., Ruiz, N. and Ambikairajah, E. (2008, 31 March-04 April). Speech-based cognitive load monitoring system. Paper Presented at the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2008) Las Vegas, Nevada, USA.