Towards a taxonomy of error-handling strategies in recognition-based multi-modal human-computer interfaces

Signal Processing

Volumn 86, Issue 12, 2006, Pages 3625-3643

  Bourguet, Marie Luce ^a  

^a QUEEN MARY UNIVERSITY OF LONDON   (United Kingdom)

Author keywords

Error handling; Interaction design; Interaction robustness; Multi modal interfaces; Recognition based technology; Taxonomy

Indexed keywords

ERROR HANDLING; INTERACTION DESIGN; INTERACTION ROBUSTNESS; MULTIMODAL INTERFACES; RECOGNITION BASED TECHNOLOGY; TAXONOMY;

CLASSIFICATION (OF INFORMATION); INTERFACES (COMPUTER); MODAL ANALYSIS; ROBUSTNESS (CONTROL SYSTEMS);

HUMAN COMPUTER INTERACTION;

EID: 33749443562     PISSN: 01651684     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.sigpro.2006.02.047     Document Type: Article

References (74)

1. Devine E.G., Gaehde S.A., and Curtis A.C. Comparative evaluation of three continuous speech recognition software packages in the generation of medical reports. J. Am. Med. Inform. Assoc. 7 (2000) 462-468
2. Koester H.H. User performance with speech recognition: a literature review. J. Assist. Technol. 13 2 (2001) 116-1130
3. Entwistle M.S. The performance of automated speech recognition systems under adverse conditions of human exertion. Int. J. Hum.-Comput. Interact. 16 2 (2003) 127-140
4. Furui S. Recent progress in corpus-based spontaneous speech recognition. IEICE Trans. Inf. Systems E 88-D 3 (2005) 366-375
5. L.R.B. Schomaker, User-interface aspects in recognising connected-cursive handwriting, in: Proceedings of the IEE Colloquium on Handwriting and Pen-Based Input, London, UK, March 1994.
6. MacKenzie I.S., and Chang L. A performance comparison of two handwriting recognisers. Interact. Comput. 11 (1999) 283-297
7. J.C. Read, S.J. MacFarlane, C. Casey, Measuring the usability of text input methods for children, in: Proceedings of the HCI2001, Lille, France, September 2001, pp. 559-572.
8. MacKenzie I.S., and Soukoreff R.W. Text entry for mobile computing: models and methods. theory and practice, Huma.-Comput. Interact. 17 (2002) 147-198
9. Plamondon R., and Srihari S.N. On-line and off-line handwriting recognition: a comprehensive survey. IEEE Trans. Pattern Anal. Mach. Intell. 22 1 (2000) 63-84
10. B. Suhm, B. Myers, A. Waibel, Model-based and empirical evaluation of Multi-modal interactive error correction, in: Proceedingss ACM CHI'99, Pittsburgh, PA, May 1999, pp. 584-591.
11. C.-M. Karat, C. Halverson, D.Horn, J. Karat, Patterns of entry and correction in large vocabulary contentious speech recognition systems, in: Proceedingss of the ACM CHI'99, Pittsburgh, PA, May 1999, pp. 568-575.
12. C. Halverson, D. Horn, C. Karat, J. Karat, The beauty of errors: patterns of error correction in desktop speech systems, in: Proceedings of the IFIP INTERACT'99, Edinburgh, Scotland, UK, September 1999, pp. 133-140.
13. Rieger J.M. The effect of automatic speech recognition systems on speaking workload and task efficiency. Disabil. Rehabil. 25 (2003) 224-235
14. J.R. Lewis, Effect of error correction strategy on speech dictation throughput, in: Proceedings of the Human Factors and Ergonomics Society 43rd Annual Meeting, 1999, pp. 457-461.
15. J. Mankoff, S.E. Hudson, G.D. Abowd, Interaction techniques for ambiguity resolution in recognition-based interfaces, in: Proceeding of the ACM UIST'00, San Diego, CA, November 2000, pp. 11-20.
16. N. Yankelovich, G.A. Levow, M. Marx, Designing speechacts: issues in speech user interfaces, Proceesing of the CHI'95, Denver, CO, May 1995, pp. 369-376.
17. Koester H.H. Usage, performance, and satisfaction outcomes for experienced users of automatic speech recognition. J. Rehabil. Res. Dev. 41 5 (2004) 739-754
18. C. Frankish, R. Hull, P. Morgan, Recognition accuracy and user acceptance of pen interfaces, in: Proceedings of the ACM CHI'95, Denver, CO, May 1995, pp. 503-510.
19. J. Mankoff, G.D. Abowd, Error correction techniques for handwriting, speech, and other ambiguous or error prone systems, GVU TechReport GIT-GVU-99-18, June 1999, http://www.cc.gatech.edu/fce/errata/publications/survey
20. D. Goldberg, C. Richardson, Touch-typing with a stylus, in: Proceedings of the INTERCHI'93, Amsterdam, The Netherlands, April 1993, pp. 80-87.
21. I.S. MacKenzie, S.X. Zhang, The immediate usability of Graffiti, in: Proceedings of the Graphics Interface, Kelowna, B C, Canada, May 1997, pp. 120-137.
22. Oviatt S.L., Cohen P.R., and Wang M. Toward interface design for human language technology: modality and structure as determinants of linguistic complexity. Speech Commun 15 (1994) 283-300
23. S. Oviatt, Mutual disambiguation of recognition errors in a Multi-modal architecture, in: Proceedings of the ACM CHI'99, Pittsburgh, PA, May 1999, pp. 576-583.
24. Furnas G.W., Landauer T.K., Gomex L.M., and Dumais S.T. The vocabulary problem in human system communication. Commun. ACM 30 (1987) 964-971
25. N. Dahlbäck, A. Jönsson, L. Ahrenberg, Wizard of oz studies why and how, in: Proceedings of the International Workshop on Intelligent User Interfaces, Orlando, FL, January 1993, pp. 193-200.
26. M.A. Walker, J.F. Di Fabbrizio, C. Mestel,. D. Hindle, What can I say?: evaluating a spoken language interface to email. in: Proceedings of the CHI'98, Los Angeles, CA, April 1998, pp. 582-589.
27. P.A. Heeman, M. Johnston, J. Denney, E. Kaiser, Beyond structured dialogues: factoring out grounding, in: Proceedings of the International Conference on Spoken Language Processing (ICSLP), Sydney, Australia, December 1998, pp. 863-866.
28. M. Turunen, Error handling in speech user interfaces in the context of virtual worlds, in: Proceedings of the ACHCI'98, Tampere, Finland, 1998, pp. 68-75.
29. J. Heer, N. Good, A. Ramirez, M. Davis, J. Mankoff, Presiding Over Accidents: System Direction of Human Action, in: Proceedings of the ACM CHI'04, Vienna, Austria, April 2004, pp. 463-470.
30. Zoltan-Ford E. How to get people to say and type what computers can understand. Int. J. Man-Mach. Stud. 34 4 (1991) 527-547
31. Brennan S.E., and Hulteen E.A. Interaction and feedback in a spoken language system: a theoretical framework. Knowl. Based Syst. 8 2-3 (1995) 143-151
32. Larson J.A. VoiceXML: Introduction to Developing Speech Applications (2002), Prentice Hall, Englewood, CPPS, NJ
33. S. Pan, S. Shen, M. Zhou, K. Houck, Two-way adaptation for robust input interpretation in practical multi-modal conversation systems, in: Proceedings of the IUI'05, San Diego, CA, January 2005, pp. 35-42.
34. Oviatt S. Multi-modal interactive maps: designing for human performance. Hum.-Comput. Interact. 12 (1997) 93-129
35. Oviatt S. Predicting spoken disfluencies during human-computer interaction. Comput. Speech Lang. 9 (1995) 19-35
36. J.L. Coutaz, L. Nigay, D. Salber, A. Blandford, J. May, R.M. Young, Four easy pieces for assessing the usability of multi-modal interaction: the CARE properties, in: Proceedings of the INTERACT'95, Lillehammer, Norway, 1995, pp. 115-120.
37. Potamianos G., Neti C., Gravier G., Garg A., and Senior A.W. Recent advances in the automatic recognition of audio-visual speech. Proc. IEEE 91 9 (2003) 1306-1326
38. Meier U., Stiefelhagen R., Yang J., and Waibel A. Towards unrestricted lipreading. Int. J. Pattern Recogn. Artif. Intell. 14 (2000) 571-585
39. Oviatt S.L., Cohen P.R., Wu L., Vergo J., Duncan L., Suhm B., Bers J., Holzman T., Winograd T., Landay J., Larson J., and Ferro D. Designing the user interface for Multi-modal speech and gesture applications: state-of-the-art systems and research directions. Hum. Comput. Interact. 15 4 (2000) 263-322
40. Abowd G.D., and Mynatt E.D. Chating past, present, and future reseach in ubiquitous computing. ACM Trans. Comput. Huma. Interact. 7 1 (2000) 29-58
41. B.H. Yoshimi, G.S. Pingali, A Multi-modal speaker detection and tracking system for teleconferencing, in: Proceedings of the ACM Conference on Multimedia, Juan-les-Pin, France, December 2002, pp. 427-428.
42. Nock H.J., Iyengar G., and Chalapathy N. Multi-modal processing by finding common cause. Commun. ACM 47 1 (2004) 51-56
43. W. Yue, S. Mu, H. Wang, G. Wang, TGH: A case study of designing natural interaction for mobile guide systems, in: Proceedings of the Mobile HCI'05, Salzburg, Austria, September 2005, pp. 199-206.
44. M.A. Feki, S. Renouard, B. Abdulrazak, G. Chollet, M. Mokhtari, Coupling context awareness and multi-modality in smart homes concept, Lect. Notes Comput. Sci. 3118 (2004) 906-913.
45. Oviatt S.L. Ten myths of Multi-modal interaction. Commun. ACM 42 11 (1999) 74-81
46. M. Marx, C. Schmandt, Putting people first: specifying proper names in speech interfaces, in: Proceedings of the ACM UIST'94, Marina del Rey, CA, November 1994, pp. 30-37.
47. Baber C., and Hone K.S. Modelling error recovery and repair in automatic speech recognition. Int. Journal Man-Mach. Studies 39 3 (1993) 495-515
48. Singh P. The public acquisition of commonsense knowledge. Proceeding of the AAAI Spring Symposium: Acquiring (and Using) Linguistic (and World) Knowledge for Information Access (2002), Palo Alto, CA
49. H. Lieberman, A. Faaborg, W. Daher, J. Espinosa, How to wreck a nice beach you sing calm incense, in: Proceedings of the IUI'05, San Diego, CA, January 2005, pp. 278-280.
50. M.L. Bourguet, Software Design and development of multi-modal interaction, in: Proceedings of the IFIP 18th World Computer Congress Topical Days, Toulouse, France, August 2004, pp. 409-414.
51. Minsky M. A framework for presenting knowledge. In: Winston P.H. (Ed). The Psychology of Computer Vision (1975), McGraw-Hill, New York
52. M. Johnston, Unification-based multi-modal parsing, in: Proceedings of the 36th COLING-ACL, Montreal, Canada, August 1998, pp. 624-630.
53. M.L. Bourguet, A. Ando, Synchronisation of speech and hand gestures during multi-modal human-computer interaction, in: Proceedings of the ACM CHI'98 Vol. 2, Los Angeles, CA, April 1998, pp. 241-242.
54. Bourguet M.L., and Ando A. Speech timing prediction in multi-modal human-computer interaction. Sydney proceeding of the IFIP INTERACT"97 (1997), Australia 453-460
55. Narayanan S. Towards modeling user behavior in human-machine interactions: effect of errors and Emotions. Proceeding of the ISLE Workshop on Dialogue Tagging for Multi-modal Human Computer Interaction (2002), Edinburgh, UK
56. M. Zajicek, J. Hewitt, An investigation into the use of error recovery dialogues in a user interface management system for speech recognition, in: proceedings of the IFIP INTERACT'90, Cambridge, UK, August 1990, pp. 755-760.
57. J. Mankoff, S. Hudson, G.D. Abowd. Providing integrated toolkit-level support for ambiguity in recognition-based interfaces, in: Proceedings of the ACM CHI'00, Hague, The Netherlands, May 2000, pp. 368-375.
58. Suhm B., Myers B., and Waibel A. Multi-modal error correction for speech user interfaces. ACM Trans. Comput.-Hum. Interact. (TOCHI), 8 1 (2001) 60-98
59. T. Igarashi, S. Matsuoka, S. Kawachiya, H. Tanaka, Interactive beautification: a technique for rapid geometric design, in: Proceedings of the ACM UIST'97, Banff, Alta, Canada, October 1997, pp. 105-114.
60. D. Goldberg, A. Goodisman, STYLUS User interfaces for manipulating text, in: Proceedings of the ACM UIST'91, Hilton Head, SC, November 1991, pp. 127-135.
61. Frankish C., Jones D., and Hapeshi K. Decline in accuracy of automatic speech recognition as a function of time on task: fatigue or voice drift? Int. J. Man-Mach. Studies 36 (1992) 797-816
62. B. Suhm and A. Waibel, Exploiting repair context in interactive error recovery, in: Proceedings of the Eurospeech 97, Fifth European Conference on Speech Communication and Technology, Rhodos, Greece, September 1997, pp. 1659-1661.
63. Oviatt S., and van Gent R. Error resolution during multi-modal human-computer interaction. Proceeding of the Fourth International Conference on Spoken Language Processing (1996), Philadelphia, PA 204-207
64. Oviatt S. Taming recognition errors with a Multi-modal interface. Commun. ACM 43 9 (2000) 45-51
65. G. McGlaun, F. Althoff, M. Lang, G. Rigoll, Towards multi-modal error management: experimental evaluation of user strategies in event of faulty application behavior in automotive environments, in: Proceedings of the Seventh World Multiconference on Systemics, Cybernetics, and Informatics SCI, Orlando, FL, July 2003, pp. 462-466.
66. Suhm B. Empirical evaluation of interactive multi-modal error correction. Proceeding of the IEEE Workshop on Automatic Speech Recognition and Understanding (1997), Santa Barbara, CA 583-590
67. Larson K., and Mowatt D. Speech error correction: the story of the alternates list. Int. J. Speech Technol. 6 2 (2003) 183-194
68. W. Huerst, J. Yang, A. Waibel, Interactive error repair for an online handwriting interface, in: Proceedings of the ACM CHI'98 (Poster), Los Angeles, CA, April 1998, pp. 353-354.
69. J. Coutaz, J. Caelen, A taxonomy for multimedia and Multi-modal user interfaces, in: Proceedings of the First ERCIM Workshop on Multi-modal Human-Computer Interaction, Lisbon, Portugal, November 1991.
70. S. Oviatt, A. DeAngeli, K. Kuhn, Integration and synchronization of input modes during multimodal human-computer interaction. in: Proceedings of the ACM CHI '97, Atlanta, GA, March 1997, pp. 415-422.
71. S. Oviatt, R. Coulston, R. Lunsford, When Do We Interact Multi-modally? Cognitive Load and Multi-modal Communication Patterns, Proceeding of the ICMI 2004, State College, PA, October 2004.
72. S. L. Oviatt, Breaking the robustness barrier: recent progress on the design of robust multi-modal systems, M. Zelkowitz (Ed.), Advances in Computers Vol. 56, Academic Press, New York, 2002, pp. 305-341.
73. M.L. Bourguet, Designing, prototyping multi-modal commands, in: Proceedings of the INTERACT'03, Zurich, Switzerland, pp. 717-720.
74. M.L. Bourguet, How finite state machines can be used to build error free multi-modal interaction systems, in: Proceedings of the HCI'03, Vol. 2, Bath, UK, September 2003.