A multidimensional dynamic time warping algorithm for efficient multimodal fusion of asynchronous data streams

Neurocomputing

Volumn 73, Issue 1-3, 2009, Pages 366-380

  Wöllmer, Martin ^a   Al Hames, Marc ^a   Eyben, Florian ^a   Schuller, Björn ^a   Rigoll, Gerhard ^a  

^a TECHNICAL UNIVERSITY OF MUNICH   (Germany)

Author keywords

Asynchronous hidden Markov model; Dynamic time warping; Multimodal data fusion

Indexed keywords

ASYNCHRONOUS DATA; ASYNCHRONOUS DATA STREAM; ASYNCHRONOUS HIDDEN MARKOV MODEL; CLASSIFICATION TASKS; DYNAMIC TIME WARPING; DYNAMIC TIME WARPING ALGORITHMS; EARLY FUSION; FUSION EXPERIMENTS; INPUT STREAMS; MULTI-MODAL; MULTI-MODAL FUSION; MULTIMODAL DATA FUSION; MUTUAL INFORMATIONS;

COMPUTATIONAL COMPLEXITY; DATA FUSION; DECODING; ELECTRONIC DOCUMENT IDENTIFICATION SYSTEMS; HIDDEN MARKOV MODELS; INFORMATION FUSION; WEAVING;

COMPUTATIONAL EFFICIENCY;

ACCURACY; ALGORITHM; ARTICLE; CLASSIFICATION; CONTROLLED STUDY; GESTURE; HIDDEN MARKOV MODEL; INTERMETHOD COMPARISON; MAN MACHINE INTERACTION; MATHEMATICAL COMPUTING; PRIORITY JOURNAL; SPEECH DISCRIMINATION;

EID: 70449526103     PISSN: 09252312     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.neucom.2009.08.005     Document Type: Article

References (77)

1. Ablassmeier M., Poitschke T., and Rigoll G. Eye gaze studies comparing head-up and head-down displays in vehicles. Proceedings of ICME (2007) 2250-2252
2. Al-Hames M., and Rigoll G. Reduced complexity and scaling for asynchronous HMMs in a bimodal input fusion application. IEEE International Conference on Acoustics, Speech and Signal Processing (2006) 757-760
3. Althoff F., McGlaun G., and Lang M.K. Using multimodal interaction to navigate in arbitrary virtual VRML worlds. Proceedings of the 2001 Workshop on Perceptive User Interfaces (2001) 1-8
4. Arsic D., Schuller B., and Rigoll G. Suspicious behaviour detection in public transport by fusion of low-level video descriptors. ICME 2007 (2007) 2018-2021
5. D. Arsic, F. Wallhoff, B. Schuller, G. Rigoll, Vision based online behavior detection using probabilistic multi-stream fusion, in: ICIP 2005, Genova, Italy, 2005, pp. 606-609.
6. Ben-Yacoub S., Abdeljaoued Y., and Mayoraz E. Fusion of face and speech data for person identity verification. IEEE Transactions on Neural Networks 10 (1999) 1065-1074
7. S. Bengio, An asynchronous hidden Markov model for audio-visual speech recognition, in: Advances in Neural Information Processing Systems, vol. 15, 2003.
8. Bengio S. Multimodal authentication using asynchronous HMMs. Proceedings of the IEEE AVBPA (2003) 770-777
9. Y. Bengio, P. Frasconi, An Input Output HMM architecture, in: Advances in Neural Information Processing Systems, vol. 7, 1995, pp. 427-434.
10. Bolt R.A. Put-that-there: voice and gesture at the graphics interface. Proceedings of SIGGRAPH (1980) 262-270
11. Bowyer K.W. Face recognition technology: security versus privacy. IEEE Technology and Society Magazine 23 (2004) 9-19
12. Bregler C., Hild H., Manke S., and Waibel A. Improving connected letter recognition by lipreading. Proceedings of ICASSP'93 (1993) 557-560
13. Chen T., and Rao R.R. Audio-visual integration in multimodal communication. Proceedings of the IEEE (1998) 837-852
14. A. Cheyer, L. Julia, Designing, developing and evaluating multimodal applications, in: Chi'99 (WS Pen/Voice Interface), 1999, pp. 1-4.
15. P.R. Cohen, M. Dalrymple, D.B. Moran, F.C. Pereira, J.W. Sullivan, Synergistic use of direct manipulation and natural language, in: Human Factors in Computing Systems: CHI'89 Conference Proceedings, vol. 20, 1989, pp. 227-233.
16. de Mello R.F., and Gondra I. Multi-dimensional dynamic time warping for image texture similarity. Advances in Artificial Intelligence (2008) 23-32
17. P. Duchnowski, U. Meier, A. Waibel, See me, hear me: integrating automatic speech recognition and lip-reading, in: International Conference on Spoken Language Processing, ICSLP 1994, 1994.
18. S. Dusan, G.J. Gadbois, J. Flanagan, Multimodal interaction on PDAs integrating speech and pen inputs, in: Proceedings of EUROSPEECH 03, 2003, pp. 2225-2228.
19. Fukumoto M., Suenaga Y., and Mase K. Finger-pointer: pointing interface by image processing. Computer Graphics 18 (1994) 633-642
20. Gillick L., and Cox S.J. Some statistical issues in the comparison of speech recognition algorithms. Proceedings of ICASSP (1989) 23-26
21. H. Gunes, M. Piccardi, Affect recognition from face and body: early fusion vs. late fusion, in: IEEE International Conference on Systems, Man and Cybernetics, vol. 4, 2005, pp. 3437-3443.
22. D. Gutchess, M. Trajkovic, E. Cohen-Solal, D. Lyons, A.K. Jain, A background model initialization algorithm for video surveillance, in: Eighth International Conference on Computer Vision (ICCV'01), vol. 1, 2001, p. 733.
23. Hochreiter S., and Schmidhuber J. Long short-term memory. Neural Computation (1997) 1735-1780
24. M.J. Hunt, Some experience in in-car speech recognition, in: IEEE Colloquium on Interactive Spoken Dialogue Systems for Telephony Applications (Ref. No. 1999/209), 1999, pp. 1-9.
25. Itakura F. Minimum prediction residual principle applied to speech recognition. IEEE Transactions on Acoustics, Speech, and Signal Processing 23 (1975) 67-72
26. Jiang F., Yao H., and Yao G. Multilayer architecture in sign language recognition system. Proceedings of the 6th International Conference on Multimodal Interfaces (2004) 352-353
27. R. Kernchen, P.P. Boda, K. Moessner, B. Mrohs, M. Boussard, G. Giuliani, Multimodal user interfaces for context-aware mobile applications, in: IEEE 16th International Symposium on Personal, Indoor and Mobile Radio Communications, vol. 4, 2005, pp. 2268-2273.
28. Ko M.H., West G., Venkatesh S., and Kumar M. Using dynamic time warping for online temporal fusion in multisensor systems. Information Fusion 9 (2008) 370-388
29. Lizhong W., Oviatt S., and Cohen P.R. Multimodal integration-a statistical view. IEEE Transactions on Multimedia 1 (1999) 334-341
30. MacKenzie I.S., and Soukoreff R.W. Text entry for mobile computing: models and methods, theory and practice. Human-Computer Interaction 17 (2002) 147-198
31. H. Misra, H. Bourlard, V. Tyagi, New entropy based combination rules in HMM/ANN multi-stream ASR, in: Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 2, 2003, pp. 741-744.
32. Morguet P. Comparison of approaches to continuous hand gesture recognition for a visual dialogue system. Proceedings of ICASSP (1999) 3549-3552
33. M. Müller, H. Mattes, F. Kurth, An efficient multiscale approach to audio synchronization, in: Proceedings of ISMIR, 2006.
34. Myers C., and Rabiner L. Connected digit recognition using a level-building DTW algorithm. IEEE Transactions on Acoustics, Speech and Signal Processing 29 3 (1981) 351-363
35. Nair N.U., and Sreenivas T.V. Joint decoding of multiple speech patterns for robust speech recognition. IEEE Workshop on Automatic Speech Recognition and Understanding (2007) 9-13
36. Neal J.G., and Shapiro S.C. Intelligent multimedia interface technology. Intelligent User Interfaces (1991) 11-43
37. Nefian A.V., Luhong L., Xiaobo P., Liu X., Mao C., and Murphy K. A coupled HMM for audio-visual speech recognition. IEEE International Conference on Acoustics, Speech and Signal Processing (2002) 2013-2016
38. C. Neti, G. Potamianos, J. Luettin, I. Matthews, D. Vergyri, J. Sison, A. Mashari, J. Zhou, Audio visual speech recognition, in: Final Workshop 2000 Report, 2000.
39. Ney H. The use of a one-stage dynamic programming algorithm for connected word recognition. IEEE Transactions on Acoustics, Speech, and Signal Processing 32 (1984) 263-271
40. Nigay L., and Coutaz J. A generic platform for addressing the multimodal challenge. Proceedings of CHI '95 (1995) 98-105
41. S. Oviatt, Multimodal interface research: a science without borders. in T.H.B. Yuan, X. Tang (Eds.), Proceedings of the 6th International Conference on Spoken Language Processing (ICSLP 2000), vol. 3, Chinese Friendship Publishers, 2000, pp. 1-6.
42. Oviatt S., and Cohen P. Perceptual user interfaces: multimodal interfaces that process what comes naturally. Communications of the ACM 43 (2000) 45-53
43. Oviatt S., Cohen P., Wu L., Duncan L., Suhm B., Bers J., Holzman T., Winograd T., Landay J., Larson J., and Ferro D. Designing the user interface for multimodal speech and pen-based gesture applications: state-of-the-art systems and future directions. Human Computer Interaction 15 (2000) 263-322
44. Oviatt S., Coulston R., Tomko S., Xiao B., Lunsford R., Wesson M., and Carmichael L. Toward a theory of organized multimodal integration patterns during human-computer interaction. Proceedings of the 5th International Conference on Multimodal Interfaces (2003) 44-51
45. Plamondon R., and Srihari S.N. Online and off-line handwriting recognition: a comprehensive survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 22 (2000) 63-84
46. Quattoni A., Wang S., Morency L.P., Collins M., and Darrell T. Hidden conditional random fields. IEEE Transactions on Pattern Analysis and Machine Intelligence 29 (2007) 1848-1853
47. Rabiner L. A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE 77 (1989) 257-286
48. Rabiner L., and Juang B. An introduction to Hidden Markov Models. IEEE ASSP Magazine 3 (1986) 4-16
49. Rabiner L., and Levinson S. Isolated and connected word recognition-theory and selected applications. IEEE Transactions on Communications 29 (1981) 621-659
50. Reiter S., Schuller B., and Rigoll G. Hidden conditional random fields for meeting segmentation. ICME 2007 (2007) 639-642
51. D.A. Reynolds, An overview of automatic speaker recognition technology, in: IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 4, 2002, pp. 4072-4075.
52. N. Roy, G. Baltus, D. Fox, F. Gemperle, J. Goetz, T. Hirsch, D. Margaritis, M. Montemerlo, J. Pineau, J. Schulte, S. Thrun, Towards personal service robots for the elderly, in: Workshop on Interactive Robots and Entertainment (WIRE 2000), 2000.
53. Sakoe H., and Chiba S. Dynamic programming algorithm optimization for spoken word recognition. IEEE Transactions on Acoustics, Speech, and Signal Processing 26 (1978) 43-49
54. J. Schenk, J. Lenz, G. Rigoll, Line-members-a novel feature in on-line whiteboard note recognition, in: Proceedings of 11th International Conference on Frontiers in Handwriting Recognition, 2008.
55. Schomaker L. From handwriting analysis to pen-computer applications. Electronics and Communication Engineering Journal 10 (1998) 93-102
56. Schuller B., Ablassmeier M., Mueller R., Reifinger S., Poitschke T., and Rigoll G. Speech communication and multimodal interfaces. Advanced Man-Machine Interaction (2006), Springer, Berlin, Heidelberg, New York 141-190
57. Schuller B., Mueller R., Hoernler B., Hoethker A., Konosu H., and Rigoll G. Audiovisual recognition of spontaneous interest within conversations. Proceedings of the ACMICMI 2007, 9th Conference on Multimodal Interfaces (2007) 30-37
58. B. Schuller, R. Müller, F. Eyben, J. Gast, B. Hörnler, M. Wöllmer, G. Rigoll, A. Höthker, H. Konosu, Being bored? Recognizing natural interest by extensive audiovisual integration for real-life application, Image and Vision Computing Journal (IMAVIS), 2009
59. Schuller B., Wimmer M., Arsic D., Rigoll G., and Radig B. Audiovisual behaviour modeling by combined feature spaces. Proceedings of ICASSP 2007 (2007) 733-736
60. Schuller B., Wöllmer M., Moosmayr T., and Rigoll G. Speech recognition in noisy environments using a switching linear dynamic model for feature enhancement. Proceedings of Interspeech 2008 (2008) 1789-1792
61. Sergio P., and Oliveira C.L. DTW-based phonetic alignment using multiple acoustic features. Eurospeech 2003 (2003) 309-312
62. Sharma R., Pavlovic V.I., and Huang T.S. Toward multimodal human-computer interface. Proceedings of the IEEE 86 (1998) 853-869 (Special issue on Multimedia signal processing)
63. Sodhi M., Reimer B., Cohen J.L., Vastenburg E., Kaars R., and Kirschenbaum S. On-road driver eye movement tracking using head-mounted devices. Proceedings of the 2002 Symposium on Eye Tracking Research and Applications (2002) 61-68
64. Stettiner Y., Malah D., and Chazan D. Dynamic time warping with path control and non-local cost. Pattern Recognition 3 (1994) 174-177
65. Q. Summerfield, Lipreading and audio-visual speech perception. Philosophical Transactions: Biological Sciences (1992) 71-78.
66. T. Tatschke, Early sensor data fusion techniques for collision mitigation purposes, in: Intelligent Vehicles Symposium, 2006 IEEE, 2006, pp. 445-452.
67. G.A. ten Holt, M.J.T. Reinders, E.A. Hendriks, Multi-dimensional dynamic time warping for gesture recognition, in: Annual Conference on the Advanced School for Computing and Imaging, 2007.
68. Vlachos M., Hadjieleftheriou M., Gunopulos D., and Keogh E. Indexing multidimensional time-series. The VLDB Journal 15 1 (2006) 1-20
69. M.T. Vo, A. Waibel, Multimodal human-computer interaction, in: Proceedings of ISSD '93, Waseda, Japan, 1993.
70. M.T. Vo, C. Wood, Building an application framework for speech and pen input integration in multimodal learning interfaces, in: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, 1996, pp. 3545-3548.
71. Waibel A., Vo M.T., Duchnowski P., and Manke S. Multimodal interfaces. Artificial Intelligence Review 10 (1996) 299-319
72. C. Ware, H.H. Mikaelian, An evaluation of an eye tracker as a device for computer input, in: Proceedings of the CHI + GI 87 Conference on Human Factors in Computing Systems and Graphics Interface, vol. 17, 1987, pp. 183-188.
73. Wimmer M., Schuller B., Arsic D., Radig B., and Rigoll G. Low-level fusion of audio and video features for multi-modal emotion recognition. Proceedings of the 3rd International Conference on Computer Vision Theory and Applications (2008) 145-151
74. Wöllmer M., Eyben F., Reiter S., Schuller B., Cox C., Douglas-Cowie E., and Cowie R. Abandoning emotion classes-towards continuous emotion recognition with modelling of long-range dependencies. Proceedings of Interspeech 2008 (2008) 597-600
75. Zhang D., Gatica-Perez D., Bengio S., McCowan I., and Lathoud G. Modeling individual and group actions in meetings: a two-layer HMM framework. Proceedings of the IEEE CVPR (2004) 117
76. Y. Zhang, S. Levinson, T. Huang, Speaker independent audio-visual speech recognition, in: IEEE International Conference on Multimedia and Expo, vol. 2, 2000, pp. 1073-1076.
77. M. Zobl, M. Geiger, B. Schuller, G. Rigoll, M. Lang, A realtime system for hand-gesture controlled operation of in-car devices, in: Proceedings of ICME 2003, 2003, pp. 541-544.