A study in efficiency and modality usage in multimodal form filling systems

IEEE Transactions on Audio, Speech and Language Processing

Volumn 16, Issue 6, 2008, Pages 1194-1206

  Perakakis, Manolis ^a   Potamianos, Alexandres ^a  

^a TECHNICAL UNIVERSITY OF CRETE   (Greece)

Author keywords

Graphical user interfaces (GUIs); Input modality selection; Mobile multimodal interfaces; Speech communication

Indexed keywords

ADAPTIVE INTERFACE; AUTOMATIC SPEECH RECOGNITION; BELL LABS; BROKEN DOWN; EFFICIENCY GAIN; EMPIRICAL PROBABILITIES; FORM FILLING; INPUT MODALITY SELECTION; INPUT MODES; INTERACTION CONTEXT; INTERACTION STYLES; INTERACTION TIME; MOBILE MULTIMODAL INTERFACES; MULTI-MODAL; MULTIMODAL DIALOGUE; MULTIMODAL DIALOGUE SYSTEMS; MULTIMODAL SYSTEM; SPEECH MODALITY; UNIMODAL; USAGE PATTERNS; USER SATISFACTION; WORKING ENVIRONMENT;

HUMAN COMPUTER INTERACTION; INTERACTIVE COMPUTER SYSTEMS; PERSONAL DIGITAL ASSISTANTS; PROBABILITY DENSITY FUNCTION; SPEECH COMMUNICATION; SPEECH PROCESSING; SPEECH RECOGNITION;

GRAPHICAL USER INTERFACES;

EID: 63449107822     PISSN: 15587916     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASL.2008.2001389     Document Type: Article

References (29)

1. A. Potamianos, E. Ammicht, and E. Fosler-Lussier, "Modality tracking in the multimodal bell labs Communicator," in Proc. Autom. Speech Recognition Understanding Workshop, St. Thomas, Virgin Islands, U.S., 2003, pp. 192-197.
2. S. Oviatt, "Multimodal interfaces," in The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications, J. Jacko and A. Sears, Eds. Mahwah, NJ: Lawrence Erlbaum, 2003, pp. 286-304.
3. D. B. Koons, C. J. Sparrell, and K. R. Thorisson, "Integrating simultaneous input from speech, gaze, and hand gestures," in Proc. Artif. Intell. Workshop Intell. Multimedia Interfaces, Menlo Park, CA, 1993 pp. 257-276.
4. A. Waibel, M. Tue Vo, P. Duchnowski, and S. Manke, "Multimodal interfaces" Artif Intell. Rev., vol. 10, no. 3-4, pp. 299-319, 1996.
5. P. R. Cohen, M. Johnston, D. McGee, S. Oviatt, J. Pittman, I. Smith, L. Chen, and J. Clow, "Quickset: Multimodal interaction for distributed applications," in Proc. ACM Int. Conf. Multimedia, Seattle, WA, 1997 pp. 31-40.
6. M. Johnston, S. Bangalore, G. Vasireddy, A. Stent, P. Ehlen, M. Walker, S. Whittaker, and P. Maloor, "MATCH: An architecture for multimodal dialogue systems," in Proc. 40th Annu. Meeting Assoc. Comput. Linguist., Philadelphia, PA, 2002, pp. 376-383.
7. S. Dusan, G. J. Gadbois, and J. Flanagan, "Multimodal interaction on PDA's integrating speech and pen inputs," in Proc. 8th Eur. Conf. Speech Commun. Technol., Geneva, Switzerland, 2003, pp. 2225-2228.
8. R. A. Bolt, "Put-That-There : Voice and gesture at the graphics interface," in Proc. Comput. Graphics Interactive Tech., Seattle, WA, 1980, pp. 262-270. (Pubitemid 11447032)
9. X. Huang, A. Acero, C. Chelba, L. Deng, D. Duchene, J. Goodman, H. Hon, D. Jacoby, L. Jiang, and R. Loynd, "MiPaD: A next generation PDA prototype," in Proc. Int. Conf. Spoken Lang. Process., Beijing, China, 2000, pp. 33-36.
10. L. Deng, K. Wang, A. Acero, H. W. Hon, J. Droppo, C. Boulis, Y. Y. Wang, D. Jacoby, M. Mahajan, and C. Chelba, "Distributed speech processing in MiPaD's multimodal user interface," IEEE Trans. Speech Audio Process., vol. 10, no. 8, pp. 605-619, 2002.
11. L. Nigay and J. Coutaz, "A design space for multimodal systems: Concurrent processing and data fusion," in Proc. INTERACT'93 and CHI'93 Conf. Human Factors Comput. Syst., Amsterdam, The Netherlands, 1993, pp. 172-178.
12. J. Lai and N. Yankelovich, "Conversational speech interfaces," in The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications. Mahwah, NJ: Lawrence Erlbaum, 2003, pp. 698-713.
13. M. Grasso, D. Ebert, and T. Finin, "The integrality of speech in multimodal interfaces," ACM Trans. Computer-Human Interaction (TOCHI), vol. 5, no. 4, pp. 303-325, 1998.
14. P. Cohen and S. Oviatt, "The role of voice in human-machine communication," in Voice Communication Between Humans and Machines. Washington, DC: National Academy Press, 1994, pp. 34-75.
15. P. Cohen, M. Johnston, D. McGee, S. Oviatt, J. Clow, and I. Smith, "The efficiency of multimodal interaction: A case study," in Proc. 5th Int. Conf. Spoken Lang. Process., Sydney, Australia, 1998, pp. 249-252.
16. B. Suhm, B. Myers, and A. Waibel, "Multimodal error correction for speech user interfaces," ACM Trans. Computer-Human Interaction (TOCHI), vol. 8, no. 1, pp. 60-98, 2001.
17. M. Perakakis, M. Toutoudakis, and A. Potamianos, "Blending speech and visual input in multimodal dialogue systems," in Proc. IEEE/ACM Workshop Spoken Lang. Technol., Aruba, 2006, pp. 142-145.
18. N. O. Bernsen and L. Dybkjaer, "Is speech the right thing for your application?" in Proc. 5fth Int. Conf. Spoken Lang. Process., Sydney, Australia, 1998, pp. 3209-3212.
19. H. Mitchard and J. Winkles, "Experimental comparisons of data entry by automated speech recognition, keyboard, and mouse," Human Factors, vol. 44, no. 2, pp. 198-210, 2002. (Pubitemid 35315394)
20. V. Bilici, E. Krahmer, S. te Riele, and R. Veldhuis, "Preferred modalities in dialogue systems," in Proc. 6th Int. Conf. Spoken Lang. Process., Beijng, China, 2000, pp. 727-730.
21. L. M. Reeves, J. C. Martin, M. McTear, T. V. Raman, K. M. Stanney, H. Su, Q. Y. Wang, J. Lai, J. A. Larson, and S. Oviatt, "Guidelines for multimodal user interface design," Commun. ACM, vol. 47, no. 1, pp. 57-59, 2004.
22. K. Stanney, S. Samman, L. Reeves, K. Hale, W. Buff, C. Bowers, B. Goldiez, D. Nicholson, and S. Lackey, "A paradigm shift in interactive computing: Deriving multimodal design principles from, behavioral and neurological foundations," Proc. Int. J. Human-Comput. Interaction, vol. 17, no. 2, pp. 229-257, 2004. (Pubitemid 39055296)
23. A. Potamianos, E. Fosler-Lussier, E. Ammicht, and M. Perakakis, "Information seeking spoken dialogue systems-part II: Multimodal, dialogue," IEEE Trans. Multimedia, vol. 9, no. 3, pp. 550-566,2007. (Pubitemid 46479787)
24. A. Potamianos, E. Ammicht, and H.-K. Kuo, "Dialogue management, in the bell labs communicator system," in Proc. 6th Int. Conf. Spoken Lang. Process., Beijng, China, 2000, pp. 603-606.
25. Q. Zhou, A. Saad, and S. Abdou, "An enhanced BLSTIP dialogue research platform," in Proc. 6th Int. Conf. Spoken Lang. Process., Beijng, China, 2000, pp. 1061-1064.
26. FreeTTS [Online]. Available: http://freetts.sourceforge.net/docs/
27. M. A. Walker, D. J. Litman, C. A. Kamm, and A. Abella, "PARADISE: A framework for evaluating spoken dialogue agents," in Proc, Assoc. Comput. Linguist. (ACL), Somerset, NJ, 1997, pp. 271-280.
28. N. Beringer, U. Kartal, K. Louka, F. Schiel, and U. Turk, "Promise: A procedure for multimodal interactive system evaluation," in Proc. LREC Workshop Multimodal Resources and Multimodal Syst. Eval., Las Palmas, Spain, 2002, pp. 77-80.
29. J. Sturm, B. Cranen, F. Wang, J. Terken, and I. Bakx, "Effects of prolonged use on the usability of a multimodal form-filling interface," in Spoken Multimodal Human-Comput. Dialogue in Mobile Environments. New York: Springer, 2004, pp. 329-348.