Fusion in multimodal interactive systems: An HMM-based algorithm for user-induced adaptation

EICS'12 - Proceedings of the 2012 ACM SIGCHI Symposium on Engineering Interactive Computing Systems

Volumn , Issue , 2012, Pages 15-24

  Dumas, Bruno ^a   Signer, Beat ^a   Lalanne, Denis ^b  

^a VRIJE UNIVERSITEIT BRUSSEL   (Belgium)

^b UNIVERSITY OF FRIBOURG   (Switzerland)

Author keywords

HMM based fusion; Multimodal fusion; Multimodal interaction; User interface adaptation

Indexed keywords

ADAPTIVE INTERACTIVE SYSTEMS; INTERACTIVE SYSTEM; INTERFACE ADAPTATION; MULTI-MODAL FUSION; MULTI-MODAL INTERACTIONS; MULTI-MODAL INTERFACES; MULTIMODAL INTERACTIVE SYSTEMS; SEMANTIC LEVELS; TEMPORAL RELATIONSHIPS;

HIDDEN MARKOV MODELS; INTERACTIVE COMPUTER SYSTEMS; USER INTERFACES;

ALGORITHMS;

EID: 84864043640     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2305484.2305490     Document Type: Conference Paper

References (31)

1. Arhippainen, L., Rantakokko, T., and Tähti, M. Navigation with an Adaptive Mobile Map-Application: User Experiences of Gesture- and Context-Sensitiveness. In Proceedings of the 2nd International Conference on Ubiquitous Computing Systems (UCS 2004) (Tokyo, Japan, November 2005), 62-73. (Pubitemid 41435914)
2. Baum, L., Petrie, T., Soules, G., and Weiss, N. A Maximization Technique Occurring in the Statistical Analysis of Probabilistic Functions of Markov Chains. The Annals of Mathematical Statistics 41, 1 (1970), 164-171.
3. Bolt, R. A. "Put-that-there": Voice and Gesture at the Graphics Interface. In Proceedings of the 7th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH 1980) (Seattle, USA, July 1980), 262-270.
4. Bourguet, M.-L. A Toolkit for Creating and Testing Multimodal Interface Designs. In Companion of the 15th Annual Symposium on User Interface Software and Technology (UIST 2002) (Paris, France, October 2002), 29-30.
5. Cesar, P., Vaishnavi, I., Kernchen, R., Meissner, S., Hesselman, C., Boussard, M., Spedalieri, A., Bulterman, D. C., and Gao, B. Multimedia Adaptation in Ubiquitous Environments: Benefits of Structured Multimedia Documents. In Proceedings of the 8th ACM Symposium on Document Engineering (DocEng 2008) (São Paulo, Brazil, September 2008), 275-284.
6. Chai, J., Hong, P., and Zhou, M. A Probabilistic Approach to Reference Resolution in Multimodal User Interfaces. In Proceedings of the 9th International Conference on Intelligent User Interfaces (IUI 2004) (Funchal, Madeira, Portugal, January 2004), 70-77. (Pubitemid 40673704)
7. Chen, Q., Georganas, N. D., and Petriu, E. M. Hand Gesture Recognition Using Haar-Like Features and a Stochastic Context-Free Grammar. IEEE Transactions on Instrumentation and Measurement 57, 8 (August 2008), 1562-1571.
8. Choudhury, T., Clarkson, B., Jebara, T., and Pentland, A. Multimodal Person Recognition using Unconstrained Audio and Video. In Proceedings of the International Conference on Audio- and Video-Based Person Authentication (AVBPA 1999) (Washington DC, USA, 1999), 176-181.
9. Cohen, P. R., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I., Chen, L., and Clow, J. QuickSet: Multimodal Interaction for Simulation Set-Up and Control. In Proceedings of the 5th Conference on Applied Natural Language Processing (ANLC 1997) (Washington DC, USA, April 1997), 20-24.
10. Coutaz, J., Nigay, L., Salber, D., Blandford, A., May, J., and Young, R. M. Four Easy Pieces for Assessing the Usability of Multimodal Interaction: The CARE Properties. In Proceedings of the 5th International Conference on Human-Computer Interaction (Interact 1995) (Lillehammer, Norway, June 1995), 115-120.
11. Dumas, B., Ingold, R., and Lalanne, D. Benchmarking Fusion Engines of Multimodal Interactive Systems. In Proceedings of the 11th International Conference on Multimodal Interfaces (ICMI-MLMI 2009) (Cambridge, USA, November 2009), 169-176.
12. Dumas, B., Lalanne, D., and Ingold, R. HephaisTK: A Toolkit for Rapid Prototyping of Multimodal Interfaces. In Proceedings of the 11th International Conference on Multimodal Interfaces (ICMI-MLMI 2009) (Cambridge, USA, November 2009), 231-232.
13. Dumas, B., Lalanne, D., and Ingold, R. Description Languages for Multimodal Interaction: A Set of Guidelines and its Illustration with SMUIML. Journal on Multimodal User Interfaces: "Special Issue on The Challenges of Engineering Multimodal Interaction" 3, 3 (February 2010), 237-247.
14. Flippo, F., Krebs, A., and Marsic, I. A Framework for Rapid Development of Multimodal Interfaces. In Proceedings of the 5th International Conference on Multimodal Interfaces (ICMI 2003) (Vancouver, Canada, November 2003), 109-116. (Pubitemid 40681859)
15. Forney Jr, G. The Viterbi Algorithm. Proceedings of the IEEE 61, 3 (1973), 268-278.
16. Hoste, L., Dumas, B., and Signer, B. Mudra: A Unified Multimodal Interaction Framework. In Proceedings of the 13th International Conference on Multimodal Interfaces (ICMI 2011) (Alicante, Spain, November 2011), 97-104.
17. Huang, J., Liu, Z., Wang, Y., Chen, Y., and Wong, E. Integration of Multimodal Features for Video Scene Classification Based on HMM. In Proceedings of the 3rd IEEE Workshop on Multimedia Signal Processing (MMSP 1999) (Copenhagen, Denmark, September 1999), 53-58.
18. Johnston, M., Cohen, P., McGee, D., Oviatt, S., Pittman, J., and Smith, I. Unification-Based Multimodal Integration. In Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics (ACL 1997) (Madrid, Spain, July 1997), 281-288.
19. Juang, B., and Rabiner, L. Hidden Markov Models for Speech Recognition. Technometrics 33 (1991), 251-272.
20. König, W. A., Rädle, R., and Reiterer, H. Squidy: A Zoomable Design Environment for Natural User Interfaces. In Proceedings of the 27th International Conference on Human Factors in Computing Systems (CHI 2009) (Boston, USA, April 2009), 4561-4566.
21. Lalanne, D., Nigay, L., Palanque, P., Robinson, P., Vanderdonckt, J., and Ladry, J. Fusion Engines for Multimodal Input: A Survey. In Proceedings of the 11th International Conference on Multimodal Interfaces (ICMI-MLMI 2009) (Cambridge, USA, September 2009), 153-160.
22. López-Jaquero, V., Vanderdonckt, J., Montero, F., and González, P. Towards an Extended Model of User Interface Adaptation: The Isatine Framework. In Engineering Interactive Systems, J. Gulliksen, M. B. Harning, P. Palanque, G. C. Veer, and J. Wesson, Eds., vol. 4940 of LNCS. Springer Verlag, 2008, 374-392.
23. Malinowski, U., Thomas, K., Dieterich, H., and Schneider-Hufschmidt, M. A Taxonomy of Adaptive User Interfaces. In Proceedings of the Conference on People and Computers VII (HCI 1992) (York, United Kingdom, September 1992), 391-414.
24. Octavia, J., Raymaekers, C., and Coninx, K. Adaptation in Virtual Environments: Conceptual Framework and User Models. Multimedia Tools and Applications 54 (2011), 121-142.
25. Oviatt, S. Human-Centered Design Meets Cognitive Load Theory: Designing Interfaces That Help People Think. In Proceedings of the 14th ACM International Conference on Multimedia (ACM MM 2006) (Santa Barbara, USA, October 2006), 871-880. (Pubitemid 47113647)
26. Rabiner, L. A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition. Proceedings of the IEEE 77, 2 (1989), 257-286.
27. Schwarz, J., Hudson, S., Mankoff, J., and Wilson, A. D. A Framework for Robust and Flexible Handling of Inputs with Uncertainty. In Proceedings of the 23nd Symposium on User Interface Software and Technology (UIST 2010) (New York, USA, October 2010), 47-56.
28. Serrano, M., Nigay, L., Lawson, J., Ramsay, A., Murray-Smith, R., and Denef, S. The OpenInterface Framework: A Tool for Multimodal Interaction. In Proceedings of the 26th International Conference on Human Factors in Computing Systems (CHI 2008) (Florence, Italy, April 2008), 3501-3506.
29. Sharma, R., Pavlovic, V., and Huang, T. Toward Multimodal Human-Computer Interface. Proceedings of the IEEE 86, 5 (1998), 853-869. (Pubitemid 128720228)
30. Vo, M., and Wood, C. Building an Application Framework for Speech and Pen Input Integration in Multimodal Learning Interfaces. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 1996) (Atlanta, USA, May 1996), 3545-3548.
31. Wu, L., Oviatt, S., and Cohen, P. From Members to Teams to Committee - A Robust Approach to Gestural and Multimodal Recognition. IEEE Transactions on Neural Networks 13, 4 (2002), 972-982.