Finite-state multimodal integration and understanding

Natural Language Engineering

Volumn 11, Issue 2, 2005, Pages 159-187

  Johnston, Michael ^a   Bangalore, Srinivas ^a  

^a AT AND T LABS RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL GRAMMARS; FINITE AUTOMATA; GESTURE RECOGNITION; SEMANTICS; SPEECH RECOGNITION; USER INTERFACES;

FINITE STATE LANGUAGE PROCESSING; MULTIMODAL INTERFACES; MULTIMODAL PARSING;

NATURAL LANGUAGE PROCESSING SYSTEMS;

EID: 20444390925     PISSN: 13513249     EISSN: None     Source Type: Journal    
DOI: 10.1017/S1351324904003572     Document Type: Article

References (46)

1. Abney, S. (1991) Parsing by chunks. In: Berwick, R., Abney, S. and Tenny, C., editors, Principle-based Parsing, pp. 257-278. Kluwer.
2. André, E. (2002) Natural language in multimedia/multimodal systems. In: Mitkov, R., editor, Handbook of Computational Linguistics, pp. 650-669. OUP.
3. Bangalore, S. and J. ohnston, M. (2000) Tight-coupling of multimodal language processing with speech recognition. Proceedings of ICSLP, pp. 126-129. Beijing, China.
4. Bangalore, S. (1997) Complexity of Lexical Descriptions and its Relevance to Partial Parsing. PhD thesis, University of Pennsylvania, Philadelphia, PA.
5. Bangalore, S. and Riccardi, G. (2000) Stochastic finite-state models for spoken language machine translation. Proceedings Workshop on Embedded Machine Translation Systems, pp. 52-59.
6. Bellik, Y. (1995) Interface Multimodales: Concepts, Modèles et Architectures. PhD thesis, Paris XI University, France.
7. Bellik, Y. (1997) Media integration in multimodal interfaces. Proceedings of IEEE Workshop on Multimedia Signal Processing, Princeton, NJ.
8. Bellik, Y. and Burger, D. (1994) Multimodal interfaces: New solutions to the problem of computer accessibility for the blind. Proceedings CHI 94, pp. 24-28. Boston, MA.
9. Bellik, Y. and Teil, D. (1993) A multimodal dialogue controller for multimodal user interface management system, application: a multimodal window manager. Proceedings INTERCHI 93, pp. 24-29. Amsterdam, The Netherlands.
10. Bird, S. and Ellison, T. M. (1994) One level phonology: autosegmental representations and rules as finite automata. Computational Linguistics, 20: 55-90.
11. Bolt, R. A. (1980) "Put-That-There": voice and gesture at the graphics interface. Computer Graphics, 14(3): 262-270.
12. Buntschuh, B., Kamm, C., DiFabbrizio, G., Abella, A., Mohri, M., Narayanan, S., Zeljkovic, I., Sharp, R. D., Wright, J., Marcus, S., Shaffer, J., Duncan, R. and Wilpon, J. G. (1998) VPQ: A spoken language interface to large scale directory information. Proceedings of ICSLP, Sydney, Australia.
13. Carbonell, J. (1970) AI in CAI: An artificial intelligence approach to computer assisted instruction. IEEE Trans. Man-Machine Syst. 11 (4): 190-202.
14. Carpenter, R. (1992) The Logic of Typed Feature Structures. Cambridge University Press, England.
15. Cheyer, A. and Julia, L. (1995) Multimodal maps: An agent based approach. Proceedings International Conference on Cooperative Multimodal Communication, pp. 103-114. Eindhoven, The Netherlands.
16. Cohen, P. R., Johnston, M., McGee, D., Oviatt, S. L., Pittman, J., Smith, I., Chen, L. and Clow, LJ. (1998) Multimodal interaction for distributed interactive simulation. In: Maybury, M. and Wahlster, W., editors, Readings in Intelligent Interfaces, pp. 562-571. Morgan Kaufmann.
17. Jacob, R. J. K. (1986) A specification language for direct-manipulation user interfaces. ACM Trans. Graphics, 283-317.
18. Johnson, M. (1998) Finite-state approximation of constraint-based grammars using left-corner grammar transforms. Proceedings COLING-ACL, pp. 619-623. Montreal, Canada.
19. Johnston, M. (2000) Deixis and conjunction in multimodal systems. Proceedings COLING 2000, pp. 362-368. Saarbrücken, Germany.
20. Johnston, M. and Bangalore, S. (2000) Finite-state multimodal parsing and understanding. Proceedings COLING 2000, pp. 369-373. Saarbrücken, Germany.
21. Johnston, M., Bangalore, S., Vasireddy, G., Stent, A., Ehlen, P., Walker, M., Whittaker, S. and Maloor, P. (2002) MATCH: An architecture for multimodal dialog systems. Proceedings ACL-02, pp. 376-383. Philadelphia.
22. Johnston, M. (1998a) Multimodal language processing. Proceedings ICSLP, Sydney, Australia.
23. Johnston, M. (1998b) Unification-based multimodal parsing. Proceedings COLING-ACL, pp. 624-630. Montreal, Canada.
24. Johnston, M., Cohen, P. R., McGee, D., Oviatt, S. L., Pittman, J. A. and Smith, I. (1997) Unification-based multimodal integration. Proceedings 35th ACL, pp. 281-288. Madrid, Spain.
25. Joshi, A. and Hopely, P. (1997) A parser from antiquity. Natural Lang. Eng. 2(4).
26. Kaplan, R. M. and Kay, M. (1994) Regular models of phonological rule systems. Computational Linguistics, 20(3): 331-378.
27. Kay, M. (1987) Non-concatenative finite-state morphology. Proceedings Third Conference of the European Chapter of the Association for Computational Linguistics, pp. 2-10.
28. Kiraz, G. and Grimley-Evans, E. (1998) Multi-Tape Automata for Speech and Language Systems: A Prolog Implementation. In: Wood, D. and Yu, S., editors, Automata Implementation: Lecture Notes in Computer Science 1436, pp. 87-103. Springer.
29. Kornai, A. (1995) Formal Phonology. Garland Publishing, New York.
30. Koskenniemi, K. K. (1984) Two-level morphology: a general computation model for word-form recognition and production. PhD thesis, University of Helsinki.
31. Mohri, M. (1997) Finite-state transducers in language and speech processing. Computational Linguistics, 23(2): 269-312.
32. Mohri, M., Pereira, F. C. N. and Riley, M. (1998) A rational design for a weighted finite-state transducer library: Lecture notes in Computer Science 1436, pp. 144-158. Springer.
33. Neal, J. G. and Shapiro, S. C. (1991) Intelligent multi-media interface technology. In: Sullivan, J. W. and Tyler, S. W., editors, Intelligent User Interfaces, pp. 45-68. ACM Press.
34. Nigay, L. and Coutaz, J. (1993) A design space for multimodal systems: Concurrent processing and data fusion. Proceedings INTERCHI '93, pp. 172-178. ACM Press.
35. Oviatt, S. L. (1997) Multimodal interactive maps: Designing for human performance. Human-Computer Interaction, pp. 93-129.
36. Oviatt, S. L. (1999) Mutual disambiguation of recognition errors in a multimodal architecture. CHI '99, pp. 576-583. ACM Press.
37. Pereira, F. and Shieber, S. (1987) Prolog and Natural Language Analysis. CSLI Lecture Notes, Stanford University Press.
38. Pereira, F. C. N. and Riley, M. D. (1997) Speech recognition by composition of weighted finite automata. In: Roche, E. and Schabes, Y., editors, Finite State Devices for Natural Language Processing, pp. 431-456. MIT Press.
39. Pereira, F. C. N. and Wright, R. (1997) Finite-state approximation of phrase structure grammars. In: Roche, E. and Schabes, Y., editors, Finite State Devices for Natural Language Processing, pp. 149-173. MIT Press.
40. Quine, W. V. O. (1969) Ontological Relativity and Other Essays. Columbia University Press.
41. Riccardi, G., Pieraccini, R. and Bocchieri, E. (1996) Stochastic Automata for Language Modeling. Computer Speech & Lang. 10 (4): 265-293.
42. Roche, E. (1999) Finite state transducers: parsing free and frozen sentences. In: Kornai, A., editor, Extended Finite State Models of Language, pp. 103-120. Cambridge University Press.
43. Rosenberg, A. L. (1964) On n-tape finite state acceptors. FOCS, pp. 76-81.
44. van Noord, G. (1997) FSA utilities: A toolbox to manipulate finite-state automata. In: Wood, D., Raymond, D. and Yu, S., editors, Automata Implementation: Lecture Notes in Computer Science 1260, pp. 87-108. Springer.
45. Vo, M. T. (1998) A Framework and Toolkit for the Construction of Multimodal Learning Interfaces. PhD thesis, Carnegie Mellon University.
46. Wahlster, W. (2002) SmartKom: Fusion and fission of speech, gestures, and facial expressions. Proceedings 1st International Workshop on Man-Machine Symbiotic Systems, pp. 213-225. Kyoto, Japan.