Towards best practice in the development and evaluation of speech recognition components of a spoken language dialog system?

Natural Language Engineering

Volumn 6, Issue 3-4, 2000, Pages 305-323

  Lamel, Lori ^a   Minker, Wolfgang ^a   Paroubek, Patrick ^a  

^a CNRS   (France)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84994769694     PISSN: 13513249     EISSN: 14698110     Source Type: Journal    
DOI: 10.1017/s1351324900002515     Document Type: Article

References (59)

1. ETRW (1995) Proc. ESCA Tutorial and Research Workshop on Spoken Dialog Systems, Visgo, Denmark.
2. ETRW (1999) Proc. ESCA Tutorial and Research Workshop on Interactive Dialog in Multi-Modal Systems. Kloster Irsee. Germany.
3. IVTTA (1996) Proc. IEEE 3rd Workshop on Interactive Voice Technology for Telecommunications Applications. Basking Ridge. USA.
4. IVTTA (1998) Proc. IEEE 4th Workshop on Interactive Voice Technology for Telecommuni-cations Applications. Torino. Italy.
5. Pirker, H., Loderer, G. and Trost, H. (1999) Thus Spoke the User to the Wizard. Proc. Eurospeech. Budapest, Hungary, pp. 1171–1174.
6. Price, P. (1990) Evaluation of Spoken Language Systems: The ATIS Domain. Proc. DARPA Workshop on Speech & Natural Language. Hidden Valley, USA.
7. Rosset, S., Bennacef, S. and Lamel, L. (1999) Design Strategies for Spoken Language Dialog Systems. Proc. Eurospeech. Budapest, Hungary, pp. 1535–1538.
8. Schwartz, R., Chow, Y., Roucos, S., Krasner, M. and Makhoul, J. (1984) Improved Hidden Markov Modeling of Phonemes for Continuous Speech Recognition. Proc. IEEE ICASSP, San Diego, 3: 35.6.1-35.6.4.
9. Sparck Jones, K. and Galliers, J. R. (1995) Evaluating Natural Language Processing Systems. Springer-Verlag.
10. Vintsyuk, T. K. (1968) Speech discrimination by dynamic programming. Kibnernetika, 4: 81.
11. Vintsyuk, T. K. (1971) Elements-wise recognition of continuous speech composed of words from a specified dictionary. Cybernetics, 74 133–143.
12. Waibel, A. (1996) Interactive Translation of Converstational Speech. Computer, 27(7): 41–48.
13. Young, S. J., Adda-Dekker, M., Aubert, X., Dugast, C., Gauvain, J. L., Kershaw, D. J., Lamel, L., Leeuwen, D., Pye, D., Robinson, A. J., Steeneken, H. J. M. and Woodland, P. C. (1997) Multilingual Large Vocabulary Speech Recognition: The European SQALE Project. Computer Speech and Language, 11: 73–89.
14. Zeppenfeld, T., Linke, M., Ries, K., Westphal, M. and Waibel, A. (1997) Recognition of Conversational Telephone Speech using the Janus Speech Engine. Proc. IEEE-ICASSP. Abella, A. and Gorin, A. L. (1997) Generating Semantically Consistent Inputs to a Dialog Manager. Proc. Eurospeech. Rhodes, Greece, pp. 1879–1882.
15. Bahl, L. R., Baker, J. K., Cohen, P. S., Dixon, N. R., Jelinek, F., Mercer, R. L. and Silverman, H. F. (1976) Preliminary results on the performance of a system for the automatic recog-nition of continuous speech. Proc. IEEE-ICASSP.
16. Bernsen, N. O. (1099) Working Paper on Dialog Management Evaluation. DISC deliverable D3.10.
17. Bernsen, N. O. and Dybkjaer, L. (1999) DISC Dialogue Engineering Best Practice Method-ology. DISC Deliverable D3.8a.
18. Bernsen, N. O., Dybkjaer, H. and Dybkjaer, L. (1998) Designing Interactive Speech Systems. From First Ideas to User Testing. Springer-Verlag.
19. Bernsen, N. O., Dybkjaer, L. and Heid, U. (1999) Current Practice in the Development and Evaluation of Spoken Language Dialog Systems. Proc. Eurospeech. Budapest, Hungary, pp. 1147–1150.
20. Bippus, R., Fischer, A. and Stahl, V. (1999) Domain Adaptation for Robust Automatic Speech Recognition in Car Environments. Proc. Eurospeech. Budapest. Hungary, pp. 1943–1946.
21. Chase L. (1998) Evaluating Word Confidence Annotation for Speech Recognition Systems. Proc. First International Conference on Language Resources and Evaluation. Granada, Spain, pp. 167–173.
22. Chase, L. Lamel, L. and Paroubek, P. (1999) Guidelines and Testing Protocols for the Development of Speech Recognition Components for SLDSs., DISC Deliverable D2.2.
23. Chow, Y. L., Schwartz, R., Roukos, S., Kimball, O., Price, P., Kubala, F., Dunham, M. O., Krasner, M. and Makhoul, J. (1986) The Role of Word-Dependent Coarticulatory Effects in a Phoneme-Based Speech Recognition System. Proc. IEEE-ICASSP. Tokyo, Japan, pp. 1593–1596.
24. Cohen, J. (1960) A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20: 37–46.
25. Cohen, J. (1968) Weighted kappa: nominal scale agreement with provision for scaled dis-agreement or partial credit. Psychological Bulletin, 70(4): 213–220.
26. Das, S., Lubensky, D. and Wu, C. (1999) Towards Robust Speech Recognition in the Telephony Network Environment - Cellular and Landline Conditions. Proc. Eurospeech. Budapest, Hungary, pp. 1959–1962.
27. Dreyfus-Graf, J. (1949) Sonograph and Sound Mechanics. J. Acoust. Soc. America, 22: 731.
28. Dudley, H. and Balashek, S. (1958) Automatic Recognition of Phonetic Patterns in Speech. J. Acoust. Soc. America, 30: 721.
29. Dybkjaer, L., Bernsen, N. O., Carlson, R., Chase, L., Dahlback, N., Failenschmid, K., Heid, U., Heisterkamp, R, Jonson, A., Kamp, H., Karlsson, I., Kuppevelt, J.V., Lamel, L., Paroubek, P. and Williams, D. (1998) The DISC Approach to Spoken Language Dialog Systems Development and Evaluation. Proc. First International Conference on Language Resources and Evaluation. Granada, Spain, pp. 185–189.
30. Fisher, W. M., Fiscus, J. G. and Martin, A. (1995) Further Studies in Phonological Scoring. Proc. ARPA Spoken Language Workshop. Austin, USA, pp. 181–186.
31. Fraser, N. and Gilbert, N. (1991) Simulating speech systems. Computer. Speech, and Language, 5(1): 81–99.
32. Gauvain, J. L., Gangolf, J. J. and Lamel, L. (1996) Speech Recognition for an Information Kiosk. Proc. ICSLP, Philadelphia. USA.
33. Gerbino, E. and Danieli, M. (1993) Managing dialogue in a continuous speech understanding system. Proc. Eurospeech. Berlin, Germany, pp. 1661–1664.
34. Gilbert, P. (1983) Software Design and Development. Science Research Associates Inc.
35. Gillick, L. and Cox, S. (1989) Statistical Significance Tests for Speech Recognition Algorithms. Proc. IEEE-ICASSP. Glasgow, Scotland, pp. 532–535.
36. Guss, N., Judge, P. C., Port, O. and Wildstram, H. (1998) Let's talk! Business Week. Mc-Graw-Hill. Heid U. (1998) Dialog Engineering Best Practice Model - Outline Skeleton. DISC-Deliverable Dl.l, pp. 1–9.
37. Hunt, M. J. (1988) Evaluating the Performance of Connected Word Speech Recognition Systems. Proc. IEEE-ICASSP. New York, USA, pp. 457–460.
38. International Standard ISO/IEC 9126. (1991) Information technology - Software product evaluation - Quality Characteristics and guidelines for their use. Geneva. International Organization for Standardization. International Electrotechnical Commission.
39. Jelinek, F. (1976) Continuous Speech Recognition by Statistical Methods. Proc. of the IEEE, 64(4): 532–556.
40. Katz, S. (1987) Estimation of probabilities from sparse data for the language model component of a speech recognizer. IEEE Trans. Acoustics, Speech and Signal Processing (ASSP), 35: 400–401.
41. King, M. and Maegaard, B. (1998) Issues in Natural Language System Evaluation. Proceedings of the First International Conference on Linguistic Resources and Evaluation (LREC). Granada, pp. 225–230.
42. Kneser, R. and Ney, H. (1995) Improved backing-off for M-gram language modeling. Proc. ICASSP. Detroit, USA.
43. Krauwer, S. (1999) ELSNET in FP5. Elsnews. 8(3):1.
44. Krippendorff, K. (1980) Content Analysis: An Introduction to Its Methodology. Sage, Beverly Hills, CA.
45. van Kuppevelt, J. and Heid, U. (1999) From a Description of Spoken Language Dialogue Systems to their Evaluation and Best Practice. DISC Deliverable D3.8b.
46. Lamel, L., Rosset, S., Bennacef, S., Bonneau-Maynard, H., Devillers, L. and Gauvain, J. L. (1995) Development of spoken language corpora for travel information. Proc. Eurospeech. Madrid, Spain, pp. 1961–1964.
47. Lamel, L., Gauvain, J. L., Bennacef, S. K., Devillers, L., Foukia, S., Gangolf, J. J. and Rosset, S. (1996) Field Trials of a Telephone Service for Rail Travel Information. Proc. IVTTA.
48. Lamel, L., Bennacef, S., Gauvain, J. L., Dartigues, H. and Temem, J.-N. (1998) User Evaluation of the MASK Kiosk. Proc. ICSLP. Sydney, Australia, pp. 2875–2878.
49. Lamel, L., Chase, L. and Paroubek, P. (1998) Working Paper on Speech Recognition Current Practice. DISC-Deliverable D1.2., pp. 1–26.
50. Lamel, L., Rosset, S., Gauvain, J. L., Bennacef, S. (1998) The Limsi ARISE System. Proc. IVTTA. Torino, Italy, pp. 209–214.
51. Lamel L. (1998) Spoken language dialog system development and evaluation at LIMSI Proc. International Symposium on Spoken Dialog. Sydney, Australia.
52. Lamel, L., Rosset, S., Gauvain, J. L. and Bennacef, S. (1999) The LIMSI Arise System for Train Travel Information. Proc. ICASSP. Phoenix, AZ.
53. Lee, K.-F. (1988) Large-vocabulary speaker-independent continuous speech recognition: The SPHINX system. PhD Thesis. Carnegie-Mellon University, USA.
54. Life, A., Salter, I., Temem, J. N., Bernard, F., Rosset, S., Bennacef, S. and Lamel, L. (1996) Data Collection for the MASK Kiosk: WOz vs. Prototype System. Proc. ICSLP. Philadelphia, USA.
55. Maier, E. (1997) Clarification Dialogs in Verbmobil. Proc. Eurospeech. Rhodes, Greece, pp. 1891–1894.
56. Matrouf, D. and Gauvain, J. L. (1997) Model compensation for noises in training and test data. Proceedings of the IEEE-ICASSP, Munich, Germany, volume II, pp. 831–834.
57. Miksic, A. and Horvat, B. (1997) Subband Echo Cancellation in Automatic Speech Dialog Systems. Proc. Eurospeech. Rhodes, Greece, pp. 2579–2582.
58. den Os, E., Boves, L., Lamel, L. and Baggia, P. (1999) Overview of the ARISE Project. Proceedings of Eurospeech. Budapest, Hungary, pp. 1527–1530.
59. Peckham, J. (1993) A New Generation of Spoken Dialog Systems: Results and Lessons from the SUNDIAL Project. Proc. Eurospeech. Berlin, Germany.