Toward a decision-theoretic framework for affect recognition and user assistance

International Journal of Human Computer Studies

Volumn 64, Issue 9, 2006, Pages 847-873

  Liao, Wenhui ^a   Zhang, Weihong ^a   Zhu, Zhiwei ^a   Ji, Qiang ^a   Gray, Wayne D ^a  

^a RENSSELAER POLYTECHNIC INSTITUTE   (United States)

Author keywords

Active sensing; Affective computing; Fatigue recognition; Human computer interaction; Influence diagrams; Stress modeling

Indexed keywords

HUMAN COMPUTER INTERACTION; MATHEMATICAL MODELS; PHYSIOLOGY; PROBABILITY; REAL TIME SYSTEMS;

ACTIVE SENSING; AFFECTIVE COMPUTING; FATIGUE RECOGNITION; HUMAN-COMPUTER INTERACTION; INFLUENCE DIAGRAMS; STRESS MODELING;

DECISION MAKING;

EID: 33745676838     PISSN: 10715819     EISSN: 10959300     Source Type: Journal    
DOI: 10.1016/j.ijhcs.2006.04.001     Document Type: Article

References (76)

1. Anderson J.R., and Lebiere C. The Atomic Components of Thought (1998), Lawrence Erlbaum Associates, Mahwah, NJ
2. Ark, W.S., Dryer, D.C., Lu, D.J., 1999. The emotion mouse. The Eighth International Conference on Human-Computer Interaction: Ergonomics and User Interfaces, vol. I, pp. 818-823.
3. Ball G., and Breeze J. Emotion and personality in a conversational agent. In: Cassel J., Sullivan J., Prevost S., and Churchill E. (Eds). Embodies Conversational Agents (2000), MIT Press, Cambridge, MA 189-219
4. Bauer M., Gymtrasiewicz P.J., and Vassileva J. User Modeling (2001), Springer, New York
5. Beatty J. Task-evoked pupillary responses, processing load, and the structure of processing resources. Psychological Bulletin 91 (1982) 276-292
6. Berthold A., and Jameson A. Interpreting symptoms of cognitive load in speech input. Proceedings of the Seventh International Conference on User Modeling (1999) 235-244
7. Boverie S., Leqellec J.M., and Hirl A. Intelligent systems for video monitoring of vehicle cockpit. International Congress and Exposition ITS. Advanced Controls and Vehicle Navigation Systems (1998) 1-5
8. Breazeal C. Robot in society. friend or appliance? Workshop on Emotion-based Agent Architectures (1999) pp. 18-26
9. Buntine, W., 1994. Operations for learning with graphical models. Journal of AI Research 159-225.
10. Carskadon M.A., and Dement W.C. The multiple sleep latency test. what does it measure?. Sleep 5 (1982) 67-72
11. Cohen, I., Grag, A., Huang, T.S., 2000. Emotion recognition using multilevel-HMM. NIPS Workshop in Affective Computing.
12. Conati C. Probabilistic assessment of user's emotions in educational games. Journal of Applied Artificial Intelligence, Special Issue on Merging Cognition and Affect in HCI 16 (2002) 555-575
13. Cowie R., Douglas-Cowie E., Tsapatsoulis N., Votsis G., Kollias S., Fellenz W., and Taylor J.G. Emotion recognition in human-computer interaction. IEEE Signal Processing Magazine 18 (2001) 32-80
14. Diez F.J. Parameter adjustment in Bayes networks. the generalized noisy or-gate. Proceedings of the Ninth Annual Conference on Uncertainty in Artificial Intelligence (UAI93) (1993) 99-105
15. Dinges, D.F., Mallis, M.M., Maislin, G.M., Powell, J.W., 1998. Evaluation of techniques for ocular measurement as an index of fatigue and the basis for alertness management. NHTSA Report No. DOT HS 808 762, April.
16. Ekman P., and Friesen W. Facial Action Coding System. A Technique for the Measurement of Facial Movement (1978), Consulting Psychologists Press
17. Elliott, C., Rickel, J., Lester, J.C., 1999. Lifelike pedagogical agents and affective computing: an exploratory synthesis. Artificial Intelligence Today 195-211.
18. Empson J. Human Brainwaves. The Psychological Significance of the Electroencephalogram (1986), The Macmillan Press Ltd, New York
19. Gardell B. Worker participation and autonomy. a multilevel approach to democracy at the workplace. International Journal of Health Services 4 (1982) 527-558
20. Grace, R., 2001. Drowsy driver monitor and warning system. International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design, August.
21. Gu, H., Ji, Q., Zhu, Z., 2002. Active facial tracking for fatigue detection. IEEE Workshop on Applications of Computer Vision.
22. Hartley, L., Horberry, T., Mabbott, N., Krueger, G.P., 2000. Review of Fatigue Detection and Prediction Technologies. National Road Transport Commission, ISBN 0 642 54469 7.
23. Hass M.W., and Hettinger L. Adaptive User Interface (2001), Lawrence Erlbaum Associates, Mahwah, NJ
24. Healy, J., Picard, R., 2000. Smartcar: detecting driver stress. The 15th International Conference on Pattern Recognition, pp. 4218-4221.
25. Heishman, R., Duric, Z., Wechsler, H., 2004. Using eye region biometrics to reveal affective and cognitive states. CVPR Workshop on Face Processing in Video.
26. Heitmann A., Guttkuhn R., Trutschel U., and Moore-Ede M. Technologies for the monitoring and prevention of driver fatigue. Proceedings of the First International Driving Symposium on Human Factors in Driving, Assessment, Training and Vehicle Design (2001) 81-86
27. Horvitz, E., 1999. Uncertainty, action, and interaction: in pursuit of mixed-initiative computing. IEEE Intelligent Systems September Issue, 17-20.
28. Horvitz E., Kadie C.M., Paek T., and Hovel D. Models of attention in computing and communications. from principles to applications. Communications of the ACM 46 (2003) 52-59
29. Howard R. Value of information lotteries. IEEE Transactions of Systems Science and Cybernetics 3 1 (1967) 54-60
30. Howard R., and Matheson J. Influence diagrams. Readings on the Principles and Applications of Decision Analysis 2 (1981) 721-762
31. Hudlicka E., and McNeese M.D. Assessment of user affective and belief states for interface adaptation. application to an air force pilot task. User Modeling and User Adapted Interaction 12 (2002) 1-47
32. Ji Q. 3D face pose estimation and tracking from a monocular camera. Image and Vision Computing 20 (2002) 499-511
33. Ji Q., Zhu Z., and Lan P. Real time non-intrusive monitoring and prediction of driver fatigue. IEEE Transactions on Vehicle Technology 53 4 (2004) 1052-1068
34. Jones F., and Bright J. Stress. Myth, Theory and Research (2001), Prentice-Hall, Englewood Cliffs, NJ
35. In: Jordan M.I. (Ed). Learning in Graphical Models (1999), MIT Press, Cambridge, MA
36. Kaapor, A., Mota, S., Picard, R., 2001. Toward a learning companion that recognizes affect. AAAI Fall Symposium: Emotional and Intelligent, North Falmouth, MA, November 2-4.
37. Kaliouby, R.E., Robinson, P., 2004. Real-time inference of complex mental states from facial expressions and head gestures. IEEE Workshop on Real-time Vision for Human-Computer Interaction in Conjunction with IEEE CVPR, Washington, DC, July.
38. Kapoor A., Picard R., and Ivanov Y. Probabilistic combination of multiple modalities to detect interest. IEEE International Conference on Pattern Recognition 3 (2004) 969-972
39. Karasek R.A. Job demands, job decision latitude and mental strain. implications for job design. Administrative Science Quarterly 24 (1979) 285-308
40. Kataoka H., Yoshida H., Saijo A., Yasuda M., and Osumi M. Development of a skin temperature measuring system for non-contact stress evaluation. Proceedings of the 20th Annual International Conference of the IEEE Engineering in Medicine and Biology Society vol. 2 (1998) 940-943
41. Kevin B., and Ann E. Bayesian Artificial Intelligence (2003), Chapman & Hall/CRC, London, Boca Raton, FL
42. Lauritzen S.T. The EM algorithm for graphical association models with missing data. Computational Statistics and Data Analysis 19 (1995) 191-201
43. Lemmer J.F., and Gossink D. Recursive noisy-or: a rule for estimating complex probabilistic causal interactions. IEEE Transactions on Systems, Man, and Cybernetics 34 6 (2004) 2252-2261
44. Li X., and Ji Q. Active affective state detection and user assistance with dynamic Bayesian networks. IEEE Transactions on Systems, Man, and Cybernetics, Part A: Systems and Humans 35 1 (2004) 93-105
45. Machleit K., and Enoglu S. Describing and measuring emotional response to shopping experience. Journal of Business Research 49 (2000) 101-111
46. Maes P., and Schneiderman B. Direct manipulation vs. interface agents. a debate. Interactions IV 6 (1997)
47. Massaro, D.W., 2000. Multimodal emotion perception: analogous to speech processes. ISCA Workshop on Speech and Emotion.
48. Mindtools, 2004. 〈http://www.mindtools.com/stress/understandstress/stressperformance.htm〉.
49. Moriyama T., Saito H., and Ozawa S. Evaluation of the relation between emotional concepts and emotional parameters on speech. IEEE International Conference on Acoustic, Speech, and Signal Processing 2 (1997) 1431-1434
50. Murray R.C., VanLehn K., and Mostow J. Looking ahead to select tutorial actions. a decision-theoretic approach. International Journal of Artificial Intelligence in Education 14 (2004) 235-278
51. Ortony A., Clore G.L., and Collins A. The Cognitive Structure of Emotions (1988), Cambridge University Press, Cambridge, England
52. Pantic, M., Patras, I., Rothkrantz, L. M., 2002. Facial mimics recognition from face profile image sequences. Data and Knowledge Systems Group, Delft University of Technology, Netherlands.
53. Partala T., and Surakka V. Pupil size variation as an indication of affective processing. International Journal of Human-Computer Studies 59 (2003) 185-198
54. Pearl J. Probabilistic Reasoning in Intelligent Systems. Networks of Plausible Inferences (1988), Morgan Kaufmann Publishers, Los Altos, CA
55. Petrushin, V.A., 1999. Emotion in speech: recognition and application to call centers. Conference on Artificial Neural Networks in Engineering (ANNIE'99), St. Louis, November 7-10.
56. Petrushin, V.A., 2000. Emotion recognition in speech signal: experimental study, development, and application. The Sixth International Conference on Spoken Language Processing.
57. Picard R. Affective Computing (1997), Cambridge University Press, Cambridge, England
58. Picard R., Vyzas E., and Healey J. Toward machine emotional intelligence. analysis of affective physiological state. IEEE Transactions on Patter Analysis and Machine Intelligence 23 10 (2001) 1175-1191
59. Qi, Y., Picard, R.W., 2002. Context-sensitive Bayesian classifiers and application to mouse pressure pattern classification. The Proceedings of International Conference on Pattern Recognition, Qué., City, Canada, August.
60. Qi, Y., Reynolds, C., Picard, R.W., 2001. The Bayes point machine for computer-user frustration detection via pressure mouse. The Proceedings of the 2001 Workshop on Perceptive User Interfaces, vol. 15, pp. 1-5.
61. Rani, P., Sarkar, N., Smith, C.A., 2003. Anxiety detection for implicit human-robot collaboration. IEEE International Conference on Systems, Man and Cybernetics 4896-4903.
62. Rimini-Doering, M., Manstetten, D., Altmueller, T., Ladstaetter, U., Mahler, M., 2001. Monitoring driver drowsiness and stress in a driving simulator. First International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design, pp. 58-63.
63. Rosekind, M.R., Co, E.L., Gregory, K.B., Miller, D.L., 2000. Crew factors in flight operations XIII: a survey of fatigue factors in corporate/executive aviation operations. National Aeronautics and Space Administration NASA/TM-2000-209610.
64. Saito, H., Ishiwaka, T., Sakata, M., Okabayashi, S., 1994. Applications of driver's line of sight to automobiles-what can driver's eye tell. Proceedings of 1994 Vehicle Navigation and Information Systems Conference, Yokohama, Japan, pp. 21-26.
65. Scherer K.R. Studying the emotion-antecedent appraisal process. an expert system approach. Cognition and Emotion 7 (1993) 325-355
66. Searle B.J., Bright J.E., and Bochner S. Testing the three-factor model of occupational stress. the impacts of demands, control and social support on a mail sorting task. Work and Stress 13 (1999) 268-279
67. Shachter R. Evaluating influence diagrams. Operations Research 34 6 (1986) 871-882
68. Sherry, P., 2000. Fatigue countermeasures in the railroad industry-past and current developments. Counseling Psychology Program, Intermodal Transportation Institute, University of Denver.
69. Ueno, H., Kaneda, M., Tsukino, M., 1994. Development of drowsiness detection system. Proceedings of 1994 Vehicle Navigation and Information Systems Conference, Yokohama, Japan, August 1994, pp. 15-20.
70. Yeasin M., Bullot B., and Sharma R. From facial expression to level of interest. a spatial-temporal approach. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition vol. 2 (2004) 922-927
71. Zhang N.L., and Poole D. Exploiting causal independence in Bayesian network inference. Journal of Artificial Intelligence Research 5 (1996) 301-328
72. Zhang Y., and Ji Q. Active and dynamic information fusion for facial expression understanding from image sequence. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) 27 5 (2005) 699-714
73. Zhang, Y., Ji, Q., 2005b. Sensor selection for active information fusion. The 20th National Conference on Artificial Intelligence (AAAI-05).
74. Zhu, Z., Ji, Q., 2004. 3D face pose tracking from an uncalibrated monocular camera. IEEE International Conference on Pattern Recognition.
75. Zhu, Z., Ji, Q., 2005. Eye gaze tracking under natural head movements. IEEE International Conference on Computer Vision and Pattern Recognition (CVPR05), San Diego, CA, June 2005.
76. Zhu, Z., Ji, Q., Fujimura, K., Lee, K., 2002. Combining Kalman filtering and mean shift for real time eye tracking under active IR illumination. IEEE International Conference on Pattern Recognition, Que., Canada.