Robust understanding in multimodal interfaces

Computational Linguistics

Volumn 35, Issue 3, 2009, Pages 345-397

  Srinivas Bangalore, ^a   Johnston, Michael ^a  

^a AT AND T LABS RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CLASSIFICATION (OF INFORMATION); COMPUTATIONAL LINGUISTICS; SPEECH TRANSMISSION;

CONVERSATIONAL SYSTEMS; EFFECTIVE MECHANISMS; INTERACTIVE SYSTEM; LANGUAGE PROCESSING; MACHINE TRANSLATIONS; MULTI-MODAL INTERFACES; MULTIMODAL APPLICATION; MULTIMODAL INTEGRATION;

SPEECH RECOGNITION;

EID: 70349521673     PISSN: 08912017     EISSN: 15309312     Source Type: Journal    
DOI: 10.1162/coli.08-022-R2-06-26     Document Type: Article

References (95)

1. Abney, S. P. 1991. Parsing by chunks. In R. Berwick, S. Abney, and C. Tenny, editors, Principle-Based Parsing. IEEE, Los Alamitos, CA, pages 257-278.
2. Ades, A. E. and M. Steedman. 1982. On the order of words. Linguistics and Philosophy, 4:517-558.
3. Alexandersson, J. and T. Becker. 2001. Overlay as the basic operation for discourse processing in a multimodal dialogue system. In Proceedings of the IJCAI Workshop: Knowledge and Reasoning in Practical Dialogue Systems, pages 8-14, Seattle, WA.
4. Allauzen, C., M. Mohri, M. Riley, and B. Roark. 2004. A generalized construction of speech recognition transducers. In International Conference on Acoustics, Speech, and Signal Processing, pages 761-764, Montreal.
5. Allauzen, C., M. Mohri, and B. Roark. 2003. Generalized algorithms for constructing statistical language models. In Proceedings of the Association for Computational Linguistics, pages 40-47, Sapporo.
6. Allauzen, C., M. Riley, J. Schalkwyk, W. Skut, and M. Mohri. 2007. Openfst: A general and efficient weighted finite-state transducer library. In Proceedings of the Ninth International Conference on Implementation and Application of Automata, (CIAA 2007), Lecture Notes in Computer Science Vol. 4783, pages 11-23. Springer, Berlin, Heidelberg.
7. Allen, J., D. Byron, M. Dzikovska, G. Ferguson, L. Galescu, and A. Stent. 2001. Towards conversational human-computer interaction. AI Magazine, 22(4):27-38.
8. André, E. 2002. Natural language in multimedia/multimodal systems. In R. Mitkov, editor, Handbook of Computational Linguistics. Oxford University Press, Oxford, pages 650-669.
9. Bacchiani, M. and B. Roark. 2003. Unsupervised language model adaptation. In Proceedings of the International Conference on Acoustics, Speech, and Signal Processing, pages 224-227, Hong Kong.
10. Bangalore, S. 1997. Complexity of Lexical Descriptions and its Relevance to Partial Parsing. Ph.D. thesis, University of Pennsylvania, Philadelphia, PA.
11. Bangalore, S. and M. Johnston. 2000. Tight-coupling of multimodal language processing with speech recognition. In Proceedings of the International Conference on Spoken Language Processing, pages 126-129, Beijing.
12. Bangalore, S. and M. Johnston. 2004. Balancing data-driven and rule-based approaches in the context of a multimodal conversational system. In Proceedings of the North American Association for Computational Linguistics/Human Language Technology, pages 33-40, Boston, MA.
13. Bangalore, S. and A. K. Joshi. 1999. Supertagging: An approach to almost parsing. Computational Linguistics, 25(2):237-265.
14. Bangalore, S. and G. Riccardi. 2000. Stochastic finite-state models for spoken language machine translation. In Proceedings of the Workshop on Embedded Machine Translation Systems, pages 52-59, Seattle, WA.
15. Bangalore, S. and G. Riccardi. 2002. Stochastic finite-state models of spoken language machine translation. Machine Translation, 17(3):165-184.
16. Bellik, Y. 1995. Interface Multimodales: Concepts, Modèles et Architectures. Ph.D. thesis, University of Paris XI (Orsay), France.
17. Bellik, Y. 1997. Media integration in multimodal interfaces. In Proceedings of the IEEE Workshop on Multimedia Signal Processing, pages 31-36, Princeton, NJ.
18. Beutnagel, M., A. Conkie, J. Schroeter, Y. Stylianou, and A. Syrdal. 1999. The AT & T next-generation TTS. In Joint Meeting of ASA; EAA and DAGA, pages 18-24, Berlin.
19. Boros, M., W. Eckert, F. Gallwitz, G. Gǒrz, G. Hanrieder, and H. Niemann. 1996. Towards understanding spontaneous speech: word accuracy vs. concept accuracy. In Proceedings of the International Conference on Spoken Language Processing, pages 41-44, Philadelphia, PA.
20. Brison, E. and N. Vigouroux. 1993. Multimodal references: A generic fusion process. Technical report, URIT-URA CNRS, Universit Paul Sabatier, Toulouse.
21. Carpenter, R. 1992. The Logic of Typed Feature Structures. Cambridge University Press, Cambridge.
22. Cassell, J. 2001. Embodied conversational agents: Representation and intelligence in user interface. AI Magazine, 22:67-83.
23. Cheyer, A. and L. Julia. 1998. Multimodal Maps: An Agent-Based Approach. Lecture Notes in Computer Science, 1374:103-113.
24. Ciaramella, A. 1993. A prototype performance evaluation report. Technical Report WP8000-D3, Project Esprit 2218, SUNDIAL.
25. Clark, S. and J. Hockenmaier. 2002. Evaluating a wide-coverage CCG parser. In Proceedings of the LREC 2002, Beyond Parseval Workshop, pages 60-66, Las Palmas.
26. Cohen, P. R. 1991. Integrated interfaces for decision support with simulation. In Proceedings of the Winter Simulation Conference, pages 1066-1072, Phoenix, AZ.
27. Cohen, P. R. 1992. The role of natural language in a multimodal interface. In Proceedings of the User Interface Software and Technology, pages 143-149, Monterey, CA.
28. Cohen, P. R., M. Johnston, D. McGee, S. L. Oviatt, J. Clow, and I. Smith. 1998a. The efficiency of multimodal interaction: A case study. In Proceedings of the International Conference on Spoken Language Processing, pages 249-252, Sydney.
29. Cohen, P. R., M. Johnston, D. McGee, S. L. Oviatt, J. Pittman, I. Smith, L. Chen, and J. Clow. 1998b. Multimodal interaction for distributed interactive simulation. In M. Maybury and W. Wahlster, editors, Readings in Intelligent Interfaces. Morgan Kaufmann Publishers, San Francisco, CA, pages 562-571.
30. Dowding, J., J. M. Gawron, D. E. Appelt, J. Bear, L. Cherny, R. Moore, and D. B. Moran. 1993. GEMINI: A natural language system for spoken-language understanding. In Proceedings of the Association for Computational Linguistics, pages 54-61, Columbus, OH.
31. Ehlen, P., M. Johnston, and G. Vasireddy. 2002. Collecting mobile multimodal data for MATCH. In Proceedings of the International Conference on Spoken Language Processing, pages 2557-2560, Denver, CO.
32. Flickinger, D., A. Copestake, and I. Sag. 2000. HPSG analysis of English. In W. Wahlster, editor, Verbmobil: Foundations of Speech-to-Speech Translation. Springer-Verlag, Berlin, pages 254-263.
33. Galescu, L., E. K. Ringger, and J. F. Allen. 1998. Rapid language model development for new task domains. In Proceedings of the ELRA First International Conference on Language Resources and Evaluation (LREC), pages 807-812, Granada.
34. Goffin, V., C. Allauzen, E. Bocchieri, D. Hakkani-Tur, A. Ljolje, S. Parthasarathy, M. Rahim, G. Riccardi, and M. Saraclar. 2005. The AT & T WATSON speech recognizer. In Proceedings of the International Conference on Acoustics, Speech, and Signal Processing, pages 1033-1036, Philadelphia, PA.
35. Gorin, A. L., G. Riccardi, and J. H. Wright. 1997. How May I Help You? Speech Communication, 23(1-2):113-127.
36. Gupta, N., G. Tur, D. Tur, S. Bangalore, G. Riccardi, and M. Rahim. 2004. The AT & T spoken language understanding system. IEEE Transactions on Speech and Audio Processing, 14(1):213-222.
37. Gustafson, J., L. Bell, J. Beskow, J. Boye, R. Carlson, J. Edlund, B. Granström, D. House, and M. Wirn. 2000. Adapt-a multimodal conversational dialogue system in an apartment domain. In International Conference on Spoken Language Processing, pages 134-137, Beijing.
38. Haffner, P., G. Tur, and J. Wright. 2003. Optimizing SVMs for complex call classification. In International Conference on Acoustics, Speech, and Signal Processing, pages 632-635, Hong Kong.
39. Hauptmann, A. 1989. Speech and gesture for graphic image manipulation. In Proceedings of CHI'89, pages 241-245, Austin, TX.
40. Jackendoff, R. 2002. Foundations of Language: Brain, Meaning, Grammar, and Evolution Chapter 9. Oxford University Press, New York.
41. Johnston, M. 1998a. Multimodal language processing. In Proceedings of the International Conference on Spoken Language Processing, pages 893-896, Sydney.
42. Johnston, M. 1998b. Unification-based multimodal parsing. In Proceedings of the Association for Computational Linguistics, pages 624-630, Montreal.
43. Johnston, M. 2000. Deixis and conjunction in multimodal systems. In Proceedings of the International Conference on Computational Linguistics (COLING), pages 362-368, Saarbrücken.
44. Johnston, M., P. Baggia, D. C. Burnett, J. Carter, D. Dahl, G. McCobb, and D. Raggett. 2007. EMMA: Extensible MultiModal Annotation markup language. Technical report, W3C Candidate Recommendation.
45. Johnston, M. and S. Bangalore. 2000. Finite-state multimodal parsing and understanding. In Proceedings of the International Conference on Computational Linguistics (COLING), pages 369-375, Saarbrücken.
46. Johnston, M. and S. Bangalore. 2004. Matchkiosk: A multimodal interactive city guide. In Proceedings of the Association of Computational Linguistics (ACL) Poster and Demonstration Session, pages 222-225, Barcelona.
47. Johnston, M. and S. Bangalore. 2005. Finite-state multimodal integration and understanding. Journal of Natural Language Engineering, 11(2):159-187.
48. Johnston, M., S. Bangalore, A. Stent, G. Vasireddy, and P. Ehlen. 2002a. Multimodal language processing for mobile information access. In Proceedings of the International Conference on Spoken Language Processing, pages 2237-2240, Denver, CO.
49. Johnston, M., S. Bangalore, G. Vasireddy, A. Stent, P. Ehlen, M. Walker, S. Whittaker, and P. Maloor. 2002b. MATCH: An architecture for multimodal dialog systems. In Proceedings of the Association of Computational Linguistics, pages 376-383, Philadelphia, PA.
50. Johnston, M., P. R. Cohen, D. McGee, S. L. Oviatt, J. A. Pittman, and I. Smith. 1997. Unification-based multimodal integration. In Proceedings of the Association of Computational Linguistics, pages 281-288, Madrid.
51. Joshi, A. and P. Hopely. 1997. A parser from antiquity. Journal of Natural Language Engineering, 2(4):6-15.
52. Kanthak, S. and H. Ney. 2004. FSA: An Efficient and Flexible C++ Toolkit for Finite State Automata Using On-Demand Computation. In Proceedings of the Association for Computational Linguistics Conference, pages 510-517, Barcelona.
53. Kaplan, R. M. and M. Kay. 1994. Regular models of phonological rule systems. Computational Linguistics, 20(3):331-378.
54. Kartunnen, L. 1991. Finite-state constraints. In Proceedings of the International Conference on Current Issues in Computational Linguistics, Universiti Sains Malaysia, Penang.
55. Koons, D. B., C. J. Sparrell, and K. R. Thorisson. 1993. Integrating simultaneous input from speech, gaze, and hand gestures. In M. T. Maybury, editor, Intelligent Multimedia Interfaces. AAAI Press/MIT Press, Cambridge, MA, pages 257-276.
56. Koskenniemi, K. K. 1984. Two-level Morphology: A General Computation Model for Word-form Recognition and Production. Ph.D. thesis, University of Helsinki.
57. Larsson, S., P. Bohlin, J. Bos, and D. Traum. 1999. TrindiKit manual. Technical report, TRINDI Deliverable D2.2, Gothenburg University, Sweden.
58. Levenshtein, V. I. 1966. Binary codes capable of correcting deletions, insertion and reversals. Soviet Physics Doklady, 10:707-710.
59. Mohri, M., F. C. N. Pereira, and M. Riley. 1998. A rational design for a weighted finite-state transducer library. Lecture Notes in Computer Science, 1436:144-158.
60. Neal, J. G. and S. C. Shapiro. 1991. Intelligent multi-media interface technology. In J. W. Sullivan and S. W. Tyler, editors, Intelligent User Interfaces. Addison Wesley, New York, pages 45-68.
61. Nederhof, M. J. 1997. Regular approximations of CFLs: A grammatical view. In Proceedings of the International Workshop on Parsing Technology, pages 159-170, Boston, MA.
62. Nishimoto, T., N. Shida, T. Kobayashi, and K. Shirai. 1995. Improving human interface in drawing tool using speech, mouse, and keyboard. In Proceedings of the 4th IEEE International Workshop on Robot and Human Communication, ROMAN95, pages 107-112, Tokyo.
63. Noord, G. 1997. FSA utilities: A toolbox to manipulate finite-state automata. Lecture Notes in Computer Science, 1260:87-108.
64. Och, F. J. and H. Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19-51.
65. Oviatt, S., A. DeAngeli, and K. Kuhn. 1997. Integration and synchronization of input modes during multimodal human-computer interaction. In CHI'97: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 415-422, New York, NY.
66. Oviatt, S. L. 1997. Multimodal interactive maps: Designing for human performance. Human-Computer Interaction, 12(1):93-129.
67. Oviatt, S. L. 1999. Mutual disambiguation of recognition errors in a multimodal architecture. In Proceedings of the Conference on Human Factors in Computing Systems: CHI'99, pages 576-583, Pittsburgh, PA.
68. Papineni, K. A., S. Roukos, and T. R. Ward. 1997. Feature-based language understanding. In Proceedings of European Conference on Speech Communication and Technology, pages 1435-1438, Rhodes.
69. Pereira, F. C. N. and M. D. Riley. 1997. Speech recognition by composition of weighted finite automata. In E. Roche and Y. Schabes, editors, Finite State Devices for Natural Language Processing. MIT Press, Cambridge, MA, USA, pages 431-456.
70. Pieraccini, R. 2004. Spoken language understanding: The research/industry chasm. In HLT-NAACL 2004 Workshop on Spoken Language Understanding for Conversational Systems and Higher Level Linguistic Information for Speech Processing, Boston, MA.
71. Pollard, C. and I. A. Sag. 1994. Head-Driven Phrase Structure Grammar. Center for the Study of Language and Information, University of Chicago Press, IL.
72. Punyakanok, V., D. Roth, and W. Yih. 2005. Generalized inference with multiple semantic role labeling systems shared task paper. In Proceedings of the Annual Conference on Computational Natural Language Learning (CoNLL), pages 181-184, Ann Arbor, MI.
73. Rambow, O., S. Bangalore, T. Butt, A. Nasr, and R. Sproat. 2002. Creating a finite-state parser with application semantics. In Proceedings of the International Conference on Computational Linguistics (COLING 2002), pages 1-5, Taipei.
74. Ramshaw, L. and M. P. Marcus. 1995. Text chunking using transformation-based learning. In Proceedings of the Third Workshop on Very Large Corpora, pages 82-94, Cambridge, MA.
75. Riccardi, G., R. Pieraccini, and E. Bocchieri. 1996. Stochastic automata for language modeling. Computer Speech and Language, 10(4):265-293.
76. Rich, C. and C. Sidner. 1998. COLLAGEN: A collaboration manager for software interface agents. User Modeling and User-Adapted Interaction, 8(3-4):315-350.
77. Ringger, E. K. and J. F. Allen. 1996. A fertility channel model for post-correction of continuous speech recognition. In International Conference on Spoken Language Processing, pages 897-900, Philadelphia, PA.
78. Ristad, E. S. and P. N. Yianilos. 1998. Learning string-edit distance. IEEE Transaction on Pattern Analysis and Machine Intelligence, 20(5):522-532.
79. Roche, E. 1999. Finite-state transducers: parsing free and frozen sentences. In A. Kornai, editor, Extended Finite-State Models of Language. Cambridge University Press, Cambridge, pages 108-120.
80. Rubine, D. 1991. Specifying gestures by example. Computer Graphics, 25(4):329-337.
81. Schapire, R., M. Rochery, M. Rahim, and N. Gupta. 2002. Incorporating prior knowledge in boosting. In Proceedings of the Nineteenth International Conference on Machine Learning, pages 538-545, Sydney.
82. Schapire, R. E. 1999. A brief introduction to boosting. In Proceedings of International Joint Conference on Artificial Intelligence, pages 1401-1406, Stockholm.
83. Souvignier, B. and A. Kellner. 1998. Online adaptation for language models in spoken dialogue systems. In International Conference on Spoken Language Processing, pages 2323-2326, Sydney.
84. Stent, A., J. Dowding, J. Gawron, E. Bratt, and R. Moore. 1999. The CommandTalk spoken dialogue system. In Proceedings of the 27th Annual Meeting of the Association for Computational Linguistics, pages 183-190, College Park, MD.
85. Thompson, C. A. and R. J. Mooney. 2003. Acquiring word-meaning mappings for natural language interfaces. Journal of Artificial Intelligence Research, 18:1-44.
86. van Tichelen, L. 2004. Semantic interpretation for speech recognition. Technical Report W3C.
87. Wahlster, W. 2002. SmartKom: Fusion and fission of speech, gestures, and facial expressions. In Proceedings of the 1st International Workshop on Man-Machine Symbiotic Systems, pages 213-225, Kyoto.
88. Walker, M., R. Passonneau, and J. Boland. 2001. Quantitative and qualitative evaluation of DARPA Communicator spoken dialogue systems. In Proceedings of the 39rd Annual Meeting of the Association for Computational Linguistics (ACL/EACL-2001), pages 515-522, Toulouse.
89. Walker, M. A., S. Whittaker, A. Stent, P. Maloor, J. D. Moore, M. Johnston, and G. Vasireddy. 2004. Generation and evaluation of user tailored responses in multimodal dialogue. Cognitive Science, 28(5):811-840.
90. Walker, M. A., S. J. Whittaker, P. Maloor, J. D. Moore, M. Johnston, and G. Vasireddy. 2002. Speech-Plans: Generating evaluative responses in spoken dialogue. In Proceedings of the International Natural Language Generation Conference, pages 73-80, Ramapo, NY.
91. Wang, Y. and A. Acero. 2003. Combination of CFG and n-gram modeling in semantic grammar learning. In Proceedings of the Eurospeech Conference, pages 2809-2812, Geneva.
92. Ward, W. 1991. Understanding spontaneous speech: The Phoenix system. In Proceedings of the International Conference on Acoustics, Speech, and Signal Processing, pages 365-367, Washington, DC.
93. Wauchope, K. 1994. Eucalyptus: Integrating natural language input with a graphical user interface. Technical Report NRL/FR/5510-94-9711, Naval Research Laboratory, Washington, DC.
94. XTAG. 2001. A lexicalized tree-adjoining grammar for English. Technical report, University of Pennsylvania. Available at www.cis.upenn.edu/~xtag/gramrelease.html.
95. Zettlemoyer, L. S. and M. Collins. 2005. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In Proceedings of the Twenty-First Conference on Uncertainty in Artificial Intelligence (UAI-05), pages 658-66, Arlington, VA.