Enabling effective design of multimodal interfaces for speech-to-speech translation system: An empirical study of longitudinal user behaviors over time and user strategies for coping with errors

Computer Speech and Language

Volumn 27, Issue 2, 2013, Pages 554-571

  Shin, Jongho ^a   Georgiou, Panayiotis G ^a   Narayanan, Shrikanth ^a  

^a USC Viterbi School of Engineering ^*  (United States)

Author keywords

HCI; Longitudinal studies; S2S; Speech translation; Speech to speech; User behaviors; User interfaces

Indexed keywords

LONGITUDINAL STUDY; S2S; SPEECH TRANSLATION; SPEECH-TO-SPEECH; USER BEHAVIORS;

BEHAVIORAL RESEARCH; ERRORS; HUMAN COMPUTER INTERACTION; INTERACTIVE COMPUTER SYSTEMS; SPEECH RECOGNITION; USER INTERFACES;

SURVEYS;

EID: 84870239584     PISSN: 08852308     EISSN: 10958363     Source Type: Journal    
DOI: 10.1016/j.csl.2012.02.001     Document Type: Article

References (37)

1. Bach N.; Eck M.; Charoenpornsawat P.; Köhler T.; Stüker S.; Nguyen T.L.; Hsiao R.; Waibel A.; Vogel S.; and Schultz T. The CMU TransTac 2007 eyes-free and hands-free two-way speech-to-speech translation system Proceedings of the International Workshop on Spoken Language Translation 2007
2. Barrows H.S. Simulated (Standardized) Patients and other Human Simulations 1987 Health Sciences Consortium Chapel Hill, NC
3. Belvin R.; May W.; Narayanan S.; Georgiou P.; and Ganjavi S. Creation of a doctor-patient dialogue corpus using standardized patients Proceedings of the International Conference on Language Resources and Evaluation (LREC) Lisbon, Portugal 2004
4. Bernstein A.; and Reinecke K. Culturally adaptive software: moving beyond internationalization Proceedings of the HCI International 2007 2007
5. Bub T.; and Schwinn J. VERBMOBIL: the evolution of a complex large speech-to-speech translation system Proceedings of the International Conference on Spoken Language Processing (ICSLP) 1996
6. Deng L.; Wang Y.; Wang K.; Acero A.; Hon H.; Droppo J.; Boulis C.; Mahajan M.; and Huang X.D. Speech and language processing for multimodal human-computer interaction The Journal of VLSI Signal Processing 36 2-3 2004 161 187
7. Doddington G. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics Proceedings of ARPA Workshop on Human Language Technology 2002
8. Dybkjær L.; Bernsen N.O.; and Minker W. Evaluation and usability of multimodal spoken language dialogue systems Speech Communication 43 1-2 2004 33 54
9. Eckert W.; Levin E.; and Pieraccini R. User modeling for spoken dialogue system evaluation Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) 1997
10. Ettelaie Emil, Georgiou Panayiotis G.; and Narayanan Shrikanth Cross-lingual dialog model for speech-to-speech translation Proceedings of the International Conference on Spoken Language Processing Pittsburgh, PA, September 2006 2006
11. Flanagan J.L. Speech-centric multimodal interfaces IEEE Signal Processing Magazine 21 2004 76 81
12. Gao Y.; Gu L.; Zhou B.; Sarikaya R.; Afify M.; and Kuo H.K. IBM MASTOR SYSTEM: multilingual automatic speech-to-speech translator Proceedings of the First International Workshop on Medical Speech Translation, in conjunction with NAACL/HLT 2006 2006
13. Horvitz E.; and Paek T. Harnessing models of users' goals to mediate clarification dialog in spoken language systems Proceedings of 8th International Conference, UM 2001 2001
14. Jannach D.; and Kreutler G. Personalized user preference elicitation for e-services Proceedings of the 2005 IEEE International Conference on e-Technology, e-Commerce and e-Service (EEE'05) on e-Technology, e-Commerce and e-Service 2005
15. Kao C.; Saleem S.; Prasad R.; Choi F.; Natarajan P.; Stallard D.; Krstovski K.; and Kamali M. Rapid development of an English/Farsi speech-to-speech translation system Proceedings of IWSLT, Hawaii Hawaii 2008
16. Knight K.; and Marcu D. Machine translation in the year 2004 Proceedings of the International Conference on Acoustics, Speech, and Signal Processing (ICASSP) 2005
17. Komatani K.; Ueno S.; Kawahara T.; and Okuno H.G. Flexible guidance generation using user model in spoken dialogue systems Proceedings of the 41st Annual Meeting on Association for Computational Linguistics(ACL2003), vol. 1 2003 256 263
18. Krulwich B. Lifestyle finder: intelligent user profiling using large-scale demographic data AI Magazine 18 2 1997 37 45
19. Larson M.L. Meaning-Based Translation: A Guide to Cross-language Equivalence 2nd ed. 1997 University Press of America
20. Levin E.; Pieraccini R.; and Eckert W. A stochastic model of human-machine interaction for learning dialogstrategies Speech and Audio Processing 8 1 2000 11 23
21. Ma X.; and Cieri C. Corpus support for machine translation at LDC Proceedings of LREC 2006: Fifth International Conference on Language Resources and Evaluation 2006
22. MedicineNet, "Medicinenet.com," 2006.
23. Meeuwesen L.; Schaap C.; and van der Staak C. Verbal analysis of doctor-patient communication Social Science and Medicine 32 10 1991 1143 1150
24. Narayanan S.; Ananthakrishnan S.; Belvin R.; Ettaile E.; Ganjavi S.; and Georgiou P. Transonics: a speech to speech system for English-Persian interactions Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) 2003
25. Onga L.M.L.; de Haesa J.C.J.M.; Hoosa A.M.; and Lammesb F.B. Doctor-patient communication: a review of the literature Social Science and Medicine 40 7 1995 903 918
26. Oviatt S.; Coulston R.; and Lunsford R. When do we interact multimodally? Cognitive load and multimodal communication patterns Proceedings of the Sixth International Conference on Multimodal Interfaces (ICMI) Pennsylvania, USA 2004
27. Oviatt S. Human-centered design meets cognitive load theory: designing interfaces that help people think Proceedings of the 14th Annual ACM International Conference on Multimedia 2006
28. Papineni, K.; Roukos, S.; Ward, T.; Zhu, W.J.; 2001. BLEU: a method for automatic evaluation of machine translation. In IBM Research Report RC22176 (W0109-022).
29. Precoda K.; Zheng J.; Vergyri D.; Franco H.; Richey C.; Kathol A.; and Kajarekar S. IraqComm: a next generation translation system Proceedings of the Interspeech 2007 2841 2844
30. Rich E. Users are individuals: individualizing user models International Journal of Human-Computer Studies 51 2 1999 323 338
31. Sanders G.A.; Bronsart S.; Condon S.; and Schlenoff C. Odds of successful transfer of low-level concepts: a key metric for bidirectional speech-to-speech machine translation in Darpa's transtac program Proceedings of the LREC 2008 2008
32. Scheffler K.; and Young S. Automatic learning of dialogue strategy using dialogue simulation and reinforcement learning Proceedings of the Human Language Technology (HLT) 2002
33. Shin J.; Narayanan S.; Gerber L.; Kazemzadeh A.; and Byrd D. Analysis of user behavior under error conditions in spoken dialogs Proceedings of the International Conference on Spoken Language Processing (ICSLP) 2002
34. Shin J.; Georgiou G.; and Narayanan S. User modeling in a speech translation driven mediated interaction setting Proceedings of the 1st ACM International Workshop on Human-centered Multimedia 2006
35. Walker M.A.; Rudnicky A.; Prasad R.; and Aberdeen J. Darpa communicator: cross-system results for the 2001 evaluation Proceedings of the International Conference on Spoken Language Processing (ICSLP) 2002
36. Young S. Talking to machines (statistically speaking) Proceedings of the International Conference on Spoken Language Processing (ICSLP) 2002
37. Zukerman I.; and Albrech D.W. Predictive statistical models for user modeling User Modeling and User-Adapted Interaction 11 1-2 2001 5 18