Automated facial expression classification and affect interpretation using infrared measurement of facial skin temperature variations

ACM Transactions on Autonomous and Adaptive Systems

Volumn 1, Issue 1, 2006, Pages 91-113

  Khan, Masood Mehmood ^a   Ingleby, Michael ^b   Ward, Robert D ^b  

^a AMERICAN UNIVERSITY OF SHARJAH   (United Arab Emirates)

^b UNIVERSITY OF HUDDERSFIELD   (United Kingdom)

Author keywords

Automated affect recognition; Facial expression classification; Infrared thermal imaging; Socially intelligent machines

Indexed keywords

AUTOMATED AFFECT RECOGNITION; FACIAL EXPRESSION CLASSIFICATION; SOCIALLY INTELLIGENT MACHINES; THERMAL INTENSITY VALUES (TIV);

AUTONOMOUS AGENTS; CLASSIFICATION (OF INFORMATION); INFRARED IMAGING; INFRARED RADIATION; ROBOTS; THERMAL EFFECTS;

FACE RECOGNITION;

EID: 33748532461     PISSN: 15564665     EISSN: 15564703     Source Type: Journal    
DOI: 10.1145/1152934.1152939     Document Type: Conference Paper

References (55)

1. BALDWIN, J. F., CASE, S. J., AND MARTIN, T. P. 1998. Machine interpretation of facial expressions. BT Technol. J. 16, 156-164.
2. BARTNECK, C. 2001. How convincing is Mr. Data's smile: Affective expressions of machines. User Model. User-Adapt. Interact. 11, 279-295.
3. BELMONT REPORT. 1979. Ethical principles and guidelines for the protection of human subjects of research. United States Government, National Institute of Health, Washington D.C., http://ohsr.od.nih.gov/guidelines/belmont.html.
4. BOLLE, R. M., CONNELL, J. H., PANKANTI, S., RATHA, N. K., AND SENIOR A. W. 2004. Guide to Biometrics. Springer, Verlag, Berlin, Germany.
5. BOULIC, L. E. R. AND THALMANN, D. 1998. Interacting with virtual humans through body actions. IEEE Computer Graphics and Applications 98, 8-11.
6. BROOKS, R. A. 2002. Flesh and Machines: How Robots Will Change Us. Pantheon Books, New York, NY.
7. CANTRONIC SYSTEMS INC. 2001. IR 860 User Manual. Cantronic Systems Inc., Coquitlam, Canada.
8. CANTRONIX SYSTEMS INC. 2003. IR 860 Product Data Sheet. Cantronic Systems Inc., Coquitlam, Canada.
9. COHEN, I., SEBE, N., GARO, A., CHEN, L. S. AND HUANO, T. S. 2003. Facial expression recognition from video sequences: temporal and static modeling. Comput. Vision Image Understand. 91,160-187.
10. COHEN, J. 1977. Statistical Power Analysis for Behavioral Sciences. Academic Press, New York.
11. COSTANZA, M. C. AND AFFIFI, A. A. 1979. Comparison of stopping rules in forward stepwise discriminant analysis. J. Amer. Statist. Assoc. 74, 777-785.
12. CRITCHLEY, D., DALY, E. M., BULLMORE, E. T., WILLIAMS, S. C. R., AMELSVOOHT, T. V., ROBERTSON, D. M., ROWE, A., PHILLIPS, M., MCALONAN, G., HOWLLIN, P., AND MURPHY, D. G. M. 2000. The functional neuroanantomy of social behavior: changes in cerebral blood flow when people with autistic disorder process facial expressions. Brain 123, 203-212.
13. DAUTENHAHN, K. 1995. Getting to know each other-Artificial social intelligence for autonomous robots. Robotics Autonom. Syst. 16, 333-356.
14. DHEW. 1979. Department of Health, Education and Welfare Publication No. (OS) 78-0014. US Government Printing Office, Washington B.C.
15. DONATO, G., BARTLETT, M. S., HAGER J. C., EKMAN P. AND SEJNOWSKI, T. J. 1999. Classifying facial expressions. IEEE Trans. Patt. Anal. Machine Intell. 21, 974-989.
16. DUDA, R. O., HART, P. E. AND STORK, D. G. 2001. Pattern Classification. Wiley Interscience, New York.
17. Ekman, P. 1982. Emotion in the Human Face. Cambridge University Press, Cambridge, UK.
18. EKMAN, P. 1993. Facial expression and emotion. Amer. Psychol. 48, 384-392.
19. EKMAN, P. AND FRIESEN, W. V. 1978. Facial Action Coding System: A Technique for the Measurement off Facial Movement. Consulting Psychology Press, Mountain View, CA.
20. ESSA, I. AND PENTLAND, A. 1997. Coding, analysis, interpretation and recognition of facial expressions. IEEE Trans. Patt. Anal. Machine Intell. 19, 757-763.
21. EVELAND, C. K., SOCOLINSKY, D. A., AND WOLF, L. B. 2003. Tracking human faces in infrared video. Image Vision Comput. 21, 579-590.
22. FASEL, B. AND LUETTIN, J. 2003. Automatic facial expression analysis: a survey. Patt. Recognit. 36, 259-275.
23. FIELD, A. 2000. Discovering Statistics Using SPSS for Windows. Sage Publications, London, UK.
24. GAO, Y., LEUNG, M. K. H., Hui, S. C., AND TANANDA, M. W. 2003. Facial expression recognition form line-based caricatures. IEEE Trans. Syst. Man Cyber. 33, 407-412.
25. GEORGE, D. AND MALLERY, P. 1995. SPSS/PC+ Step by Step: A Simple Guide and Reference. Wadsworth Publishing, London, UK.
26. HUANG, C. AND HUANG, Y 1997. Facial expression recognition using model-based feature extraction and action parameters classification. J. Visual Comm. Image Represent. 8, 278-290.
27. HUBERTY, C. J. 1984. Issues in the use and interpretation of discriminant analysis. Psycholog. Bull. 95, 156-171.
28. HUBERTY, C. J. 1994. Applied Discriminant Analysis. Wiley, New York, NY.
29. JONES, C. H., RING, E. F. J., AND CLARK, R. P. 1988. Medical thermography. In Applications of Thermal Imaging, Burnay, S. G., Williams, T. L. and Jones, C. H. N., Eds. Adam Hilger, Bristol, UK, 156-187.
30. KEARNEY, G. D. AND MCKENZIE, S. 1993. Machine interpretation of emotion: Design of memorybased expert systems for interpreting facial expressions in terms of signaled emotions. Cognitive Science 17, 589-622.
31. KHAN, M. M., WARD, R. D., AND INGLEBY, M. 2004. Automated classification and recognition of facial expressions. In Proceedings of the IEEE Conference on Cybernetics and Intelligent Systems, Singapore, (Dec), 202-206.
32. KHAN, M. M., WARD, R. D., AND INGLEBY, M. 2005. The distinguishing facial expressions by thermal imaging using facial thermal feature points. In Proceedings of the 19th British HCI Group Annual Conference (HCI'OB), Edinburgh, (Sept), L. Mackinnon, O. Bertelsen and N. Bryan-Kinns Eds. The British Computer Society, London, UK. 10-14.
33. KIM, H. K., BANG, S. W., AND KIM S. R. 2004. Emotion recognition system using short-term monitoring of physiological signals. Medical Biological Engin. Comput. 42, 419-427.
34. KLEIN, J., MOON, Y., AND PICARD, R. W. 2002. This computer responds to user frustration: Theory, design and results. Interact. Comput. 14, 119-140.
35. KURSE, P. W. 2001. Uncooled Thermal Imaging: Analysis, Systems and Applications. SPIE Press, Bellingham, WA.
36. LISETTI, C. S. AND SCHIANO, D. J. 2000. Automatic facial expression interpretation: Where humancomputer interaction, artificial intelligence and cognitive science intersect. Pragmatics Cognition 8, 185-235.
37. MATSUZAKI, H. AND MIZOTE, M. 1996. Measurement of facial temperature fluctuations by thermal image analysis. Progress in Biophysics and Molecular Biology 66 Supplement 1, 185-186.
38. MCGIMPSEY, J. G., VAIDYA, A., BIAGIONI, P. A., AND LAMEY, P.-J. 2000. Role of thermography is the assessment of infraorbital nerve injury after malar fractures. British J. Oral Maxillofacial Surg. 38, 581-584.
39. NANAVATI, S., THIEME, M., NANAVATI, R. 2002. Biometrics: Identity Verification in a Networked World. John Wiley & Sons, New York, NY.
40. NOHMAN, D. A., ORTONY, A., AND RUSSELL, D. M. 2003. Affect and machine design: Lessons for the development of autonomous machines. IBM Syst. J. 42, 38-44.
41. OGASAWARA, T., KITAOAWA, Y., OGAWA, T., YAMADA, T., KAWAMURA, Y., AND SANG, K. 2001. MR imaging and thermography of facial angioedema: A case report. Oral Surg, Oral Medicine, Oral Pathol. 92, 473-476.
42. PANTIC, M. AND ROTHKRANTZ, L. J. M. 2000. Automatic analysis of facial expressions: The state of the art. IEEE Trans. Patt. Anal. Machine Understand. 22, 1424-1445.
43. PAVLIDIS, I. 2004. Lie detection using thermal imaging. In Proceedings of SPIE Thermosense XXVI (April), Orlando, FL, The International Society for Optical Engineering, Bellingham, WA, 270-279.
44. PICARD, R. W. 2000. Affective Computing. MIT Press, Cambridge, MA.
45. PIZZAGALLI, D., KOENIG, T., REGARD, M., AND LEHMANN, D. 1998. Faces and emotion: brain electric field sources during covert emotional processing. Neuropsychologia 36, 323-332.
46. POSAMENTIEH, M. T. AND ABOI, H. 2003. Processing faces and facial expressions. Neuropsychology Rev. 13, 113-143.
47. PROKOSKI, F. J. AND IEDEL, R. 1999. Infrared identification of faces and body parts. In Biometrics: Personal Identification in Networked Society. Jain, A. K., Bolle, R. M., and Pankanti, S., Eds. Kulwer Academic Press, Boston, MA, 191-212.
48. REDFORD, P. 2000. Conference Report: Social goals and emotions. Psychologist 13, 290-291.
49. SCHWARZ, S., HOFMANN, M. H., GUTZEN, C., SCHLAX S., AND EMDE, G. V. 2002. VIEWER: A program for visualizing, recording, and analyzing animal behavior. Comput. Methods Programs Biomedicine 67, 55-66.
50. SHARMA, S. 1996. Applied Multivariate Techniques. Wiley, New York, NY.
51. SUGIMOTO, Y., YOSHITOMI, Y., AND TOMITA, S. 2000. A method of detecting transitions of emotional states using a thermal facial image based on a synthesis of facial expressions. Robotics Autonom. Syst. 31, 147-160.
52. SWETS, D. AND WENG, J. 1998. Using discriminant eigenfeatures for image retrieval. IEEE Trans. Patt. Anal. Machine Intell. 20, 39-51.
53. TURNER, J. R. AND THAYER, J. F. 2001. Introduction to Variance Analysis. Sage Publications, London UK.
54. WARD, R. D. AND MARSDEN, P. H. 2004. Affective computing: problems, reactions and intentions. Interacting with Computers 16, 707-713.
55. YOSHITOMI, Y., KIM, S. I., KAWANO, T., AND KITAZOE, T. 2000. Effects of sensor fusion for recognition of emotional states using voice, face image and thermal image of face. In Proceedings of the IEEE International Workshop on Robotics and Human Interactive Communication, Osaka, Japan, (Sept), 178-183.