Modeling multimodal integration with event logic charts

ICMI'12 - Proceedings of the ACM International Conference on Multimodal Interaction

Volumn , Issue , 2012, Pages 125-132

  Mehlmann, Gregor ^a   André, Elisabeth ^a  

^a UNIVERSITY OF AUGSBURG   (Germany)

Author keywords

Event logic chart; Interaction modeling; Multimodal fusion

Indexed keywords

COMBINED MODELING; DISCRETE INTERACTION; EVENT LOGIC; INCREMENTAL PARSING; INTERACTION MANAGEMENT; INTERACTION MODELING; ITERATIVE DEVELOPMENT; LEVELS OF ABSTRACTION; MULTI-MODAL; MULTI-MODAL FUSION; MULTIMODAL INTEGRATION; MULTIMODAL SYSTEM; MULTIPLE MODALITIES; SEMANTIC CONSTRAINTS; STATE CHARTS; TIGHT COUPLING;

INTERACTIVE COMPUTER SYSTEMS; ITERATIVE METHODS; RAPID PROTOTYPING; SEMANTICS;

VISUAL LANGUAGES;

EID: 84870194030     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2388676.2388705     Document Type: Conference Paper

References (24)

1. J. F. Allen and G. Ferguson. Actions and events in interval temporal logic. Journal of Logic and Computation, 4:531-579, 1994.
2. R. A. Bolt. Put-that-there : Voice and gesture at the graphics interface. In Proceedings of SIGGRAPH '80, pages 262-270. ACM, New York, NY, 1980.
3. J. Brusk, T. Lager, A. Hjalmarsson, and P. Wik. Deal: Dialogue management in scxml for believable game characters. In Proceedings of Future Play '07, pages 137-144. ACM, New York, NY, 2007.
4. B. Carpenter and F. Pereira. Computational Linguistics, volume 19, chapter The Logic of Typed Feature Structures. Cambridge University Press, Cambridge, England, 1992.
5. P. Cohen. Integrated interfaces for decision support with simulation. In Proceedings of WSC '91, pages 1066-1072. IEEE Computer Society Washington, DC, USA, 1991.
6. L. Hoste, B. Dumas, and B. Signer. Mudra: A unified multimodal interaction framework. In Proceedings of ICMI' 11, pages 97-104. ACM, New York, NY, USA, 2011.
7. M. Johnston. Multimodal language processing. In Proc. of the Int. Conf. on Spoken Language Processing, 1998.
8. M. Johnston. Building multimodal applications with emma. In Proceedings of ICMI-MLMI '09, 2009.
9. M. Johnston and S. Bangalore. Finite-state multimodal integration and understanding. Journal of Natural Language Engineering, 11:159-187, 2005.
10. M. Johnston, P. R. Cohen, D. McGee, S. L. Oviatt, J. A. Pittman, and I. Smith. Unification-based multimodal integration. In Proc. of the Association of Computational Linguistice, 1997.
11. D. B. Koons, C. Sparrel, and K. R. Thorisson. Intelligent Multimedia Interfaces, chapter Integrating simultaneous input from spech, gaze and hand gestures, pages 257-276. AAAI, Menlo Park, CA, 1993.
12. D. Lalanne, L. Nigay, P. Palanque, P. Robinson, J. Vanderdonckt, and J. F. Ladry. Fusion engines for multimodal input: A survey. In Proceedings of ICMI' 09, pages 153-160. ACM, New York, NY, USA, 2009.
13. M. E. Latoschik. Designing transition networks for multimodal vr-interactions using a markup language. In Proceedings of ICMI' 02, pages 411-416, 2002.
14. J. C. Martin. TYCOON: Theoretical framework and software tools for multimodal interfaces. AAAI Press, 1998.
15. G. Mehlmann, B. Endras, and E. André. Modeling parallel state charts for multithreaded multimodal dialogues. In Proceedings of ICMI' 11, pages 385-392. ACM, New York, NY, USA, 2011.
16. D. Navarre, P. Palanque, R. Bastide, A. Schyn, M. A. Winckler, L. Nedel, and C. Freitas. A formal description of multimodal interaction techniques for immersive virtual reality applications. In Proceedings of INTERACT '05, volume 3585 of LNCS, pages 170-185. Springer-Verlag, 2005.
17. J. Neal and S. Shapiro. Intelligent Unser Interfaces, chapter Intelligent Multi-Media Interface Technology, pages 45-68. Addison Wesley, New York, 1991.
18. S. Oviatt. Advances in robust multimodal interface design. IEEE Comput. Graph. Appl., 23(5):62-68, September 2003.
19. S. Oviatt. The Human-Computer Interaction Handbook, chapter Multimodal Interfaces, pages 413-432. Mahwah, NJ: Lawrence Erlbaum and Associates, 2008.
20. R. Sharma, V. I. Pavlovi, and T. S. Huang. Toward multimodal human-computer interface. Proceedings of the IEEE, Special Issue on Multimodal Signal Processing, 86:853-860, 1998.
21. D. Traum, A. Leuski, A. Roque, S. Gandhe, D. DeVault, J. Gerten, S. Robinson, and B. Martinovski. Natural language dialogue architectures for tactical questioning characters. In Army Science Conference, 2008.
22. M. T. Voo and C. Wood. Building an application framework for speech and pen input integration in multimodal learning interfaces. In Proc. of the Int. Conf. on Acoustics, Speech and Signal Processing, 1996.
23. J. Wagner, F. Lingenfelser, and E. André. The social signal interpretation framework (ssi) for real time signal processing and recognition. In Proceedings of INTERSPEECH' 11, pages 3245-3248, 2011.
24. J. Wielemaker, T. Schrijvers, M. Triska, and T. Lager. Swi-prolog. CoRR, abs/1011.5332, 2010.