Machine analysis of facial behaviour: Naturalistic and dynamic behaviour

Philosophical Transactions of the Royal Society B: Biological Sciences

Volumn 364, Issue 1535, 2009, Pages 3505-3513

  Pantic, Maja ^a,b  

^a IMPERIAL COLLEGE LONDON   (United Kingdom)

^b UNIVERSITY OF TWENTE   (Netherlands)

Author keywords

Automatic facial expression analysis; Dynamics of facial behaviour; Naturalistic behaviour

Indexed keywords

AUTOMATION; HUMAN BEHAVIOR; PSYCHOLOGY;

EID: 73549083235     PISSN: 09628436     EISSN: 14712970     Source Type: Journal    
DOI: 10.1098/rstb.2009.0135     Document Type: Article

References (59)

1. Ambadar, Z., Schooler, J. & Cohn, J. F. 2005 Deciphering the enigmatic face: the importance of facial dynamics in interpreting subtle facial expressions. Psychol. Sci. 16, 403-410. (doi:10.1111/j.0956-7976.2005.01548.x) (Pubitemid 40763688)
2. Ambady, N. & Rosenthal, R. 1992 Thin slices of expressive behavior as predictors of interpersonal consequences: a meta-analysis. Psychol. Bull. 111, 256-274. (doi:10.1037/0033-2909.111.2.256)
3. Ashraf, A. B., Lucey, S., Cohn, J. F., Chen, T., Ambadar, Z., Prkachin, K., Solomon, P. & Theobald, B. J. 2007 The painful face: pain expression recognition using active appearance models. Proc. ACM Int. Conf. on Multimodal Interfaces, 9-14.
4. Bartlett, M. S., Viola, P. A., Sejnowski, T. J., Golomb, B. A., Larsen, J., Hager, J. C. & Ekman, P. 1996 Classifying facial actions. Adv. Neural Inf. Process. Syst. 8, 823-829.
5. Bartlett, M. S., Hager, J. C., Ekman, P. & Sejnowski, T. J. 1999 Measuring facial expressions by computer image analysis. Psychophysiology 36, 253-263. (doi:10.1017/S0048577299971664) (Pubitemid 29130001)
6. Bartlett, M. S., Littlewort, G., Frank, M., Lainscsek, C., Fasel, I. & Movellan, J. 2005 Recognizing facial expression: machine learning and application to spontaneous behavior. Proc. IEEE Int. Conf. on Computer Vision and Pattern Recognition, 568-573. (Pubitemid 43897407)
7. Cohn, J. F. & Ekman, P. 2005 Measuring facial actions. In The new handbook of methods in nonverbal behavior research (eds J. A. Harrigan, R. Rosenthal & K. Scherer), pp. 9-64. New York, NY: Oxford University Press.
8. Cohn, J. F. & Schmidt, K. L. 2004 The timing of facial motion in posed and spontaneous smiles. J. Wavelets Multi-Resolut. Inf. Process. 2, 121-132.
9. Cohn, J. F., Reed, L. L., Ambadar, Z., Xiao, J. & Moriyama, T. 2004 Automatic analysis and recognition of brow actions and head motion in spontaneous facial behavior. Proc. IEEE Int. Conf. Syst. Man Cybern. 1, 610-616. (Pubitemid 40415926)
10. Costa, M., Dinsbach, W., Manstead, A. S. R. & Bitti, P. E. R. 2001 Social presence, embarrassment, and nonverbal behaviour. J. Nonverbal Behav. 25, 225-240. (doi:10.1023/A:1012544204986) (Pubitemid 33374576)
11. Cunningham, D. W., Kleiner, M., Wallraven, C. & Bülthoff, H. H. 2004 The components of conversational facial expressions. Proc. ACM Int. Conf. on Applied Perception in Graphics and Visualization, 143-149.
12. Ekman, P. 2003 Darwin, deception, and facial expression. Ann. N Y. Acad. Sci. 1000, 205-221. (doi:10.1196/annals.1280.010)
13. Ekman, P. & Friesen, W. V. 1969 The repertoire of nonverbal behavior. Semiotica 1, 49-98.
14. Ekman, P. & Rosenberg, E. L. (eds) 2005 What the face reveals: basic and applied studies of spontaneous expression using the facial action coding system. Oxford, UK: Oxford University Press.
15. Ekman, P., Friesen, W. V. & Hager, J. C. 2002, Facial action coding system. Salt Lake City, UT: A Human Face.
16. El Kaliouby, R. & Robinson, P. 2004 Real-time inference of complex mental states from facial expressions and head gestures. Proc. IEEE Int. Conf. Comput Vis. Pattern Recognit. 3, 154.
17. Fan, X., Sun, Y. & Yin, B. 2008 Multi-scale dynamic human fatigue detection with feature level fusion. Proc. IEEE Int. Conf. on Automatic Face & Gesture Recognition.
18. Frank, M. G. & Ekman, P. 2004 Appearing truthful generalizes across different deception situations. J. Pers. Soc. Psychol. 86,486-495. (doi:10.1037/0022-3514.86.3.486) (Pubitemid 38397481)
19. Frank, M. G., Ekman, P. & Friesen, W. V. 1993 Behavioural markers and recognizability of die smile of enjoyment. J. Pers. Soc. Psychol. 64, 83-93. (doi:10.1037/0022-3514.64.1.83)
20. Gokturk, S. B., Bouguet, J. Y., Tomasi, C. & Girod, B. 2002 Model-based face tracking for view independent facial expression recognition. Proc. IEEE Int. Conf. on Face and Gesture Recognition, 272-278.
21. Gralewski, L., Campbell, N. & Voak, I. P. 2006 Using a tensor framework for die analysis of facial dynamics. Proc. IEEE Int. Conf. on Face & Gesture Recognition, pp. 217-222. (Pubitemid 44711353)
22. Ioannou, S., Raouzaiou, A., Tzouvaras, V., Mailis, T., Karpouzis, K. & Kollias, S. 2005 Emotion recognition through facial expression analysis based on a neurofuzzy method. J. Neural Netw. 18, 423-435. (doi:10.1016/j. neunet.2005.03.004) (Pubitemid 40922649)
23. Ji, Q., Lan, P. & Looney, C. 2006 A probabilistic framework for modeling and real-time monitoring human fatigue. IEEE Trans. Syst. Man Cybern. A 36, 862-875. (doi: 10.1109/TSMCA.2005.855922)
24. Kanadě, T., Cohn, J. F. & Tian, Y. 2000 Comprehensive database for facial expression analysis. Proc. IEEE Int. Conf. on Automatic Face & Gesture Recognition, pp. 46-53.
25. Kapoor, A., Qi, Y & Picard, R. W. 2003 Fully automatic upper facial action recognition. Proc. IEEE Int. Workshop on Analysis and Modeling of Faces and Gestures.
26. Kapoor, A., Burleson, W. & Picard, R. W 2007 Automatic prediction of frustration. J. Hum. Comput Stud. 65, 724-736.
27. Keltner, D. & Ekman, P. 2000 Facial expression of emotion. In Handbook of emotions (eds M. Lewis & J. M. HavilandJones), pp. 236-249. New York, NY: Guilford Press.
28. Koelstra, S. & Pantic, M. 2008 Non-rigid registration using free-form deformations for recognition of facial actions and dieir temporal dynamics. Proc. IEEE Int. Conf. on Automatic Face and Gesture Recognition.
29. Kotsia, I. & Pitas, I. 2007 Facial expression recognition in image sequences using geometric deformation features and SVM. IEEE Trans. Image Process. 16, 172-187. (doi:10.1109/TIP.2006.884954) (Pubitemid 46437480)
30. Krumhuber, E., Manstead, A. S. R. & Kappas, A. 2007 Temporal aspects of facial displays in person and expression perception: the effects of smile dynamics, head-tilt, and gender. J. Nonverbal Behav. 31, 39-56. (doi:10.1007/s10919-006-0019-x)
31. Li, S. Z. & Jain, A. K. (eds) 2005 Handbook of face recognition. New York, NY: Springer.
32. Lien, J. J. J., Kanade, T., Cohn, J. F. & Li, C. C. 1998 Subtly different facial expression recognition and expression intensity estimation. Proc. IEEE Int. Conf. on Computer Vision & Pattern Recognition, 853-859.
33. Littlewort, G., Bartlett, M. S., Fasel, L., Susskind, J. & Movellan, J. 2006 Dynamics of facial expression extracted automatically from video. J. Image Vis. Comput. 24, 615-625. (doi:10.1016/j.imavis.2005.09.011) (Pubitemid 43867835)
34. Littlewort, G. C., Bartlett, M. S. & Lee, K. 2007 Faces of pain: automated measurement of spontaneous facial expressions of genuine and posed pain. Proc. ACM Int Conf. on Multimodal Interfaces, pp. 15-21.
35. Pantic, M. & Bartlett, M. S. 2007 Machine analysis of facial expressions. In Face recognition (eds K. Delac & M. Grgic), pp. 377-416. Vienna, Austria: I-Tech Education and Publishing.
36. Pantic, M. & Patras, I. 2005 Detecting facial actions and their temporal segments in nearly frontal-view face image sequences. Proc. IEEE Int. Conf. on Systems, Man & Cybernetics, pp. 3358-3363. (doi:10.1109ATSMCB.2005. 859075)
37. Pantic, M. & Patras, I. 2006 Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences. IEEE Trans. Syst. Man Cybern. B 36, 433-449.
38. Pantic, M. & Rothkrantz, L. J. M. 2000 Automatic analysis of facial expressions: die state of the art. IEEE Trans. Pattern Anal. Mach. Intell. 22, 1424-1445. (doi: 10.1109/34.895976) (Pubitemid 32088469)
39. Pantic, M. & Rothkrantz, L. J. M. 2003 Toward an affectsensitive multimodal HCI. Proc. IEEE 91, 1370-1390. (doi:10.1109/JPROC.2003.817122)
40. Pantic, M. & Rothkrantz, L. J. M. 2004 Facial action recognition for facial expression analysis from static face images. IEEE Trans. Syst Man Cybern. B 34, 1449-1461. (doi: 10.1109/TSMCB.2004.825931)
41. Pantic, M., Rothkrantz, L. J. M. & Koppelaar, H. 1998 Automation of non-verbal communication of facial expressions. Proc. Conf. on Euromedia, pp. 86-93.
42. Pantic, M., Sebe, N., Cohn, J. F. & Huang, T. 2005a Affective multimodal human-computer interaction. Proc. ACM Int. Conf. on Multimedia, pp. 669-676
43. Pantic, M., Valstar, M. F., Rademaker, R. & Maat, L. 2005b Web-based database for facial expression analysis. Proc. IEEE Int. Conf. on Multimedia & Expo, pp. 317-321. (Pubitemid 44668868)
44. 2): how far are we from attaining it? J. Autonom. Adaptive Commun. Syst. 1, 168-187. (doi:10.1504/IJAACS.2008.019799)
45. Russell, J. A. & Fernandez-Dols, J. M. (eds) 1997 The psychology of facial expression. New York, NY: Cambridge University Press.
46. Sayette, M. A., Smith, D. W., Breiner, M. J. & Wilson, G. T. 1992 The effect of alcohol on emotional response to a social stressor. J. Stud. Alcohol 53, 541-545.
47. Tian, Y. L., Kanade, T. & Cohn, J. F. 2001 Recognizing action units for facial expression analysis. IEEE Trans. Pattern Anal. Mach. lntell. 23, 97-115. (doi:10.1109/34.908962) (Pubitemid 32254704)
48. Tong, Y., Liao, W. & Ji, Q. 2007 Facial action unit recognition by exploiting their dynamics and semantic relationships. IEEE Trans. Pattern Anal. Mach. Intell. 29, 1683-1699. (doi:10.1109/TPAMI.2007.1094)
49. Valstar, M. F. & Pantic, M. 2006 Biologically vs. logic inspired encoding of facial actions and emotions in video. Proc. IEEE Int. Conf. on Multimedia & Expo, pp. 325-328. (Pubitemid 46679720)
50. Valstar, M. F. & Pantic, M. 2007 Support vector machines and hidden Markov models for modeling facial action temporal dynamics. Lea. Notes Comput. Sci. 4796, 118-127. (doi:10.1007/978-3-540-75773-3-13)
51. Valstar, M. F., Pantic, M., Ambadar, Z. & Cohn, J. F. 2006 Spontaneous versus posed facial behavior: automatic analysis of brow actions. Proc. ACM Int. Conf. on Multimodal Interfaces, pp. 162-170. (Pubitemid 47129473)
52. Valstar, M. F., Gunes, H. & Pantic, M. 2007 How to distinguish posed from spontaneous smiles using geometric features. Proc. ACM Int. Conf. on Multimodal Interfaces, pp. 38-45.
53. Vinciarelli, A., Pantic, M. & Bourlard, H. 2009 Social signal processing: survey of an emerging domain. J. Image Vis. Comput. 27, 1743-1760.
54. Viola, P. & Jones, M. 2004 Robust real-time face detection. J. Comput. Vis. 57, 137-154. (doi:10.1023/B:VISI.0000013087.49260.fb)
55. Williams, A. C. 2002 Facial expression of pain: an evolutionary account. Behav. Brain Sci. 25, 439-488.
56. Yang, M. H., Kriegman, D. J. & Ahuja, N. 2002 Detecting faces in images: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 24, 34-58. (doi: 10.1109/34.982883)
57. Zeng, Z., Fu, Y., Roisman, G. L., Wen, Z., Hu, Y & Huang, T. S. 2006 Spontaneous emotional facial expression detection. J. Multimedia 1, 1-8.
58. Zeng, Z., Pantic, M., Roisman, G. I. & Huang, T. S. 2009 A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans. Pattern Anal. Mach. Intell. 31, 39-58. (doi:10.1109/TPAMI.2008.52)
59. Zhang, Y & Ji, Q. 2005 Active and dynamic information fusion for facial expression understanding from image sequence. IEEE Trans. Pattern Anal. Mach. Intell. 27, 699-714. (doi:10.1109/TPAMI.2005.93) (Pubitemid 40608403)