Multiparty interaction understanding using smart multimodal digital signage

IEEE Transactions on Human-Machine Systems

Volumn 44, Issue 5, 2014, Pages 625-637

  Tung, Tony ^a   Gomez, Randy ^a,b   Kawahara, Tatsuya ^a   Matsuyama, Takashi ^a  

^a KYOTO UNIVERSITY   (Japan)

^b HONDA RESEARCH INSTITUTE JAPAN CO LTD   (Japan)

Author keywords

Human machine system; multimodal interaction dynamics; multiparty interaction; smart digital signage

Indexed keywords

DIGITAL SIGNAGE; HUMAN-MACHINE SYSTEMS; MULTI-MODAL; MULTI-MODAL INTERACTIONS; MULTI-PARTY INTERACTIONS;

EID: 84907215298     PISSN: 21682291     EISSN: None     Source Type: Journal    
DOI: 10.1109/THMS.2014.2326873     Document Type: Article

References (42)

1. L. Chen, R. Rose, Y. Qiao, I. Kimbara, F. Parrill, H. Welji, T. Han, J. Tu, Z. Huang, M. Harper, F. Quek, Y. Xiong, D. McNeill, R. Tuttle, and T. Huang, "Vace multimodalmeeting corpus," Machine Learning for Multimodal Interaction, S. Renals, and S. Bengio Eds. New York, NY, USA: Springer, 2006.
2. F. Pianesi, M. Zancanaro, B. Lepri, and A. Cappelletti, "A multimodal annotated corpus of concensus decision making meetings," Lang. Resources Eval., vol. 41, pp. 409-429, 2007.
3. M. Poel, R. Poppe, and A. Nijholt, "Meeting behavior detection in smart environments: Nonverbal cues that help to obtain natural interaction," in Proc. IEEE Int Conf. Autom. Face Gesture Recog., 2008, pp. 1-6.
4. Y. Sumi, M. Yano, and T. Nishida, "Analysis environment of conversational structure with nonverbal multimodal data," in Proc. Int. Conf. Multimodal Interfaces Workshop Mach. Learn. Multimodal Interact., 2010, pp. 44-1-44-4.
5. R.Gomez and T. Kawahara, "Robust speech recognition based on dereverberation parameter optimization using acoustic model likelihood," IEEE Trans. Audio, Speech Lang. Process., vol. 18, no. 7, pp. 1708-1716, Sep. 2010.
6. T. Matsuyama, S. Nobuhara, T. Takai, and T. Tung, 3D Video and Its Applications. New York, NY, USA: Springer, 2012.
7. H. Kawashima and T. Matsuyama, "Interval-based modeling of human communication dynamics via hybrid dynamical systems," in Proc. NIPS Workshop Modeling Human Commun. Dyn., 2010, pp. 34-37.
8. S. White, "Backchannels across cultures: A study of Americans and Japanese," Lang. Soc., vol. 18, pp. 59-76, 1989.
9. T. Kawahara, T. Iwatate, and K. Takanashi, "Prediction of turn-taking by combining prosodic and eye-gaze information in poster conversations," in Proc. Conf. Interspeech, 2012, pp. 1-4.
10. M. A. Smith and T. Kanade, "Video skimming and characterization through the combination of image and language understanding techniques," in Proc. IEEE Conf. Comput. Vision Pattern Recog., 1997, pp. 775-781.
11. G. Murray, S. Renals, and J. Carletta, "Extractive summarization of meeting recordings," in Proc. Conf. Interspeech, 2005, pp. 775-781.
12. K. Nakadai, T. Lourens, H. G. Okuno, and H. Kitano, "Active audition for humanoid," in Proc. Nat. Conf. Artif. Intell., 2000, pp. 832-839.
13. Hark 2.0. (2013). [Online]. Available: http://winnie.kuis.kyotou. ac.jp/hark
14. L. R. Rabiner, "A tutorial on hidden Markow models and selected applications in speech recognition," Proc. IEEE, vol. 77, no. 2, pp. 257-286, Feb. 1989.
15. C.-D. Liu, Y.-N. Chung, and P.-C. Chung, "An interaction-embedded HMM framework for human behavior understanding: With nursing environments as examples," IEEE Trans. Inf. Technol. Biomed., vol. 14, no. 5, pp. 1236-1246, Sep. 2010.
16. T. Tung, R. Gomez, T. Kawahara, and T. Matsuyama, "Group dynamics and multimodal interaction modeling using a smart digital signage," in Eur. Conf. Comput. Vision Workshop, Springer LNCS, Part I, vol. 7583, 2012, pp. 362-371.
17. T. Tung, R. Gomez, T. Kawahara, and T. Matsuyama, "Multi-party humanmachine interaction using a smart multimodal digital signage," in Proc. Int Conf. Human-Comput. Interact, Springer LNCS, Part IV, vol. 8007, 2013, pp. 408-415.
18. Y. Suzuki, F. Asano, H.-Y. Kim, and T. Sone, "An optimum computergenerated pulse signal suitable for the measurement of very long impulse responses," J. Acoust. Soc. Amer., vol. 97, no. 2, pp. 1119-1123, 1995.
19. H. Sawada, R. Mukai, S. Araki, and S. Makino, "Polar coordinate based nonlinear function for frequency-domain blind source separation," in Proc. IEEE Int Conf. Acoust., Speech, Signal Process., 2002, pp. 1001-1004.
20. H. Nakajima, K. Nakadai, Y. Hasegawa, and H. Tsujino, "Adaptive stepsize parameter control for real world blind source separation," in Proc. IEEE Int Conf. Acoust., Speech, Signal Process., 2008, pp. 149-152.
21. F. Jelinek, Statistical Methods for Speech Recognition. Cambridge, MA, USA: MIT Press, 1997.
22. R. Gomez, T. Kawahara, K. Nakamura, and K. Nakadai, "Multi-party human-robot interaction with distant-talking speech recognition," in Proc. ACM/IEEE Int Conf. Human-Robot Interaction, 2012, pp. 439-446.
23. R. Gomez, K. Nakamura, and K. Nakadai, "Automatic distance compensation for robust voice-based human-computer interaction," Int. J. Comput., Inf., Mechatronics Syst. Sci. Eng., vol. 7, no. 7, pp. 398-407, 2013.
24. J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, and A. Blake, "Real-time human pose recognition in parts from a single depth image," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2011, pp. 1297-1304.
25. T. Tung and T. Matsuyama, "Human motion tracking using a color-based particle filter driven by optical flow," Eur. Conf. Comput. VisionWorkshop, MLVMA, pp. 1-12, 2008.
26. P. Viola and M. Jones, "Robust real-time face detection," Int J. Comput. Vision, vol. 57, no. 2, pp. 137-154, 2004.
27. T. F. Cootes, G. J. Edwards, and C. J. Taylor, "Active appearance models," in Proc. Eur. Conf. Comput. Vis., 1998, pp. 484-498.
28. G. Fanelli, T.Weise, J. Gall, and L. V. Gool, "Real time head pose estimation from consumer depth cameras," in Proc. DAGM - Int. Conf. Pattern Recog., 2011, pp. 101-110.
29. S. Xu, H. Jiang, and F. C. Lau, "User-oriented document summarization through vision-based eye-tracking," in Proc. 13th ACM Int Conf. Intell. User Interfaces, 2009, pp. 7-16.
30. L. Feng,Y. Sugano, T. Okabe, andY. Sato, "Inferring human gaze from appearance via adaptive linear regression," in Proc. IEEE Int Conf. Comput. Vision, 2011, pp. 153-160.
31. T. Tung, S. Nobuhara, and T. Matsuyama, "Complete multi-view reconstruction of dynamic scenes from probabilistic fusion of narrow and wide baseline stereo," in Proc. IEEE Int Conf. Comput. Vision, 2009, pp. 1709- 1716.
32. T. Kawahara, S. Hayashi, and K. Takanashi, "Estimation of interest and comprehension level of audience through multi-modal behaviors in poster conversations," in Proc. Conf. Interspeech, 2013, pp. 1882-1885.
33. G. Doretto, A. Chiuso, Y. Wu, and S. Soatto, "Dynamic textures," Int J. Comput. Vis, vol. 51, no. 2, pp. 91-109, 2003.
34. A. Ravichandran, R. Chaudhry, and R. Vidal, "View-invariant dynamic texture recognition using a bag of dynamical systems," in Proc. IEEE Conf. Comput. Vision Pattern Recog., 2009, pp. 1932-1939.
35. R. Chaudhry, A. Ravichandran, G. Hager, and R. Vidal, "Histograms of oriented optical flow and binet-cauchy kernels on nonlinear dynamical systems for the recognition of human actions," in Proc. IEEE Conf. Comput. Vision Pattern Recog., 2009, pp. 1932-1939.
36. T. Tung and T. Matsuyama, "Intrinsic characterization of dynamic surfaces," in Proc. IEEE Conf. Comput. Vision Pattern Recog., 2013, pp. 233-240.
37. T. Tung and T. Matsuyama, "Timing-based local descriptor for dynamic surfaces," in Proc. IEEE Conf. Comput. Vision Pattern Recog., 2014.
38. A. B. Chan and N. Vasconcelos, "Mixtures of dynamic textures," in Proc. IEEE Int Conf. Comput. Vis., 2005, pp. 641-647.
39. J. Koenderink and A. van Doorn, "Surface shape and curvature scales," Image Vis. Comput., vol. 10, pp. 557-564, 1992.
40. E. de Aguiar, C. Stoll, C. Theobalt, N. Ahmed, H.-P. Seidel, and S. Thrun, "Performance capture from sparsemulti-viewvideo," ACMTrans.Graph., vol. 27, no. 3, pp. 98-1-98-10, 2008.
41. F. David, R. Pisani, and R. Purves, Statistics, 4th ed. New York, NY, USA: Norton, 2007.
42. T. Tung and T. Matsuyama, "Topology dictionary for 3D video understanding," IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 8, pp. 1645-1657, Aug. 2012.