Exploiting latent sematic information in statistical language modeling

Proceedings of the IEEE

Volumn 88, Issue 8, 2000, Pages 1279-1296

  Bellegarda, Jerome R ^a,b  

^a APPLE INC   (United States)

^b IEEE   (United States)

Author keywords

Latent semantic analysis; Multispan integration; N grams; Speech recognition; Statistical language modeling

Indexed keywords

EID: 0000274403     PISSN: 00189219     EISSN: None     Source Type: Journal    
DOI: 10.1109/5.880084     Document Type: Article

References (72)

1. L. R. Bahl, F. Jelinek, and R. L. Mercer, "A maximum likelihood approach to continuous speech recognition."IEEE Trans. Pattern Anal. Machine Intuit., vol. PAM1-5, pp.179-190, Mar. 1983.
2. J. R. Beliegarda, "Context-dependent vector clustering for speech recognition,"InAutomatic Speech and Speaker Recognition: Advanced Topics, C.-H. Lee, F. K. Soong, and K. K. Paliwal, Eds. Norwell, MA: Kluwer, Mar. 1996, ch. 6, pp.133-157.
3. _, "A latent semantic analysis framework for large-span language modeling,"InProc. 5th Eur. Conf. Speech Commun. Techno!.., vol. 3, Rhodes, Greece, Sept. 1997, pp.1451-1454.
4. _, "Data-driven semantic inference for command and control by voice," Apple Computer, Cupcrtino, CA, User Experience Group Tech. Rep., Apr. 1998.
5. _, "Exploiting both local and global constraints for multi-span statistical language modeling,"InProc. 1998 Int. Conf. Acoust., Speech, Signal Processing, vol. 2, Seattle, WA, May 1998, pp.677-680.
6. _, "A multi-span language modeling framework for Sarge vocabulary speech recognition,"IEEE Trans. Speech Audio Processing, vol. 6, pp.456-467, Sept. 1998.
7. _, "Multi-span statistical language modeling for large vocabulary speech recognition,"InProc. Int. Conf. Spoken Language Proc., Sydney, Australia, Dec. 1998, pp.2395-2399.
8. _, "Speech recognition experiments using multi-span statistical language modeling,"InProc. 1999 Int. Conf. Acoust., Speech, Signal Processing, vol. 2, Phoenix, AZ, Mar. 1999, pp.717-720.
9. _, "Context scope selection in multi-span statistical language modeling,"InProc. 6th Eur. Conf. Speech Commun. TechnoL, vol. 5, Budapest, Hungary, Sept. 1999, pp.2163-2166.
10. _, "Large vocabulary speech recognition with multi-span statistical language models,"IEEE Trans. Speecli Audio Processing; vol. 8, pp.76-84, Jan. 2000.
11. _, "Robustness in statistical language modeling: Review and perspectives,"InRobustness in Speech and Language Processing, G. J. M. van Noord and J. C. Junqua, Eds. Dortrecht, The Netherlands: Kluwer, to be published.
12. J. R. Beliegarda, J. W. Butzberger, Y. L. Chow, N. B. Coccaro, and D. Naik, "A novel word clustering algorithm based on latent semantic analysis,'' in Proc. 1996 Int. Conf. Acoust., Speech, Signal Processing, Atlanta, GA, May 1996, pp.1172-1175.
13. J. R. Beliegarda and K. E. A. Silverntan, "Toward unconstrained command and control: Data-driven semantic inference,"InProc. Int. Conf. Spoken Language Proc., Beijing, China, Oct. 2000.
14. M. W. Berry, "Large-scale sparse singular value computations,"Int. J. Supercotnp. Appl., voi. 6, no. 1, pp.13-49, 1992.
15. M. W. Berry, S. T. Dumais, and G. W. O'Brien, "Using linear algebra for intelligent information retrieval." SI A W Rev., vol. 37, no. 4, pp.573-595, 1995.
16. M. Berry and A. Sameh, "An overview of parallel algorithms for the singular value and dense symmetric eigenvalue problems," J. Comput. Appl. Math., vol. 27, pp.191-213, 1989.
17. J W. Byrne, private communication, Nov. 1997.
18. B. Carpenter and J. Chu-Carroll, "Natural language call routing: A robust, self-organized approach,"InProc. Int. Conf. Spoken Language Proc., Sydney, Australia, Dec. 1998, pp.2059-2062.
19. C. Chelba, D. Engle, F. Jelinek, v. Jimenez, S. Khudanpur, L. Mangu, H. Priniz, E. S. Risîad. R. Rosenfeld, A. Stolcke. and D. Wu, "Structure and performance of a dependency language model,"InProc. 5th Eur. Conf. Speech Commun. Technol. Rhodes, Greece, Sept. 1997, vol. 5, pp.2775-2778.
20. C. Chelba and F. Jelinek, "Recognition performance of a structured language model,"InProc. 6th Eur. Conf. Speech Commun. TechnoJ., vol. 4. Budapest, Hungary, Sept. 1999, pp.1567-1570.
21. S. Chert, "Building probabilistic models for natural language," Ph.D. dissertation. Harvard Univ., Cambridge, MA, 1996.
22. J. Chu-Carroll and B. Carpenter, "Dialog management in vector-based call routing,"InProc. Conf. Assoc. Comput. Linguistics ACLICOLING, Montreal, Canada, 1998, pp.256-262.
23. P. R. Clarkson and A. J. Robinson, "Language model adaptation using mixtures and an exponentially decaying cache,"InProc. 1997 In.'. Conf. Acottst., Speech, Signal Processing, vol. 1, Munich, Germany, May 1997, pp.799-802.
24. J N. Coccaro and D. Jurafsky, "Toward better integration of semantic predictors in statistical language modeling,"InProc. Int. Conf. Spoken Language Proc., Sydney, Australia, Dec. 1998, pp.2403-2406.
25. J. K. Cullum and R. A. Willoughby, Lanczos Algorithms for Large Symmetric Eigenvalue Computations-vol. 1 Theory. Boston, MA: Backhäuser, 1985, ch. 5.
26. R. De Mori, "Recognizing and using knowledge structures in dialog systems,"InProc. AM. Speech Recog. Understanding Workshop, Keystone, CO, Dec. 1999, pp.297-306.
27. S. Deerwester, S. T. Dumais, G. W. Fumas, T. K. Landauer, and R. Harsliman, "Indexing by latent semantic analysis," J. Amer. Soc. Inform. Sei., vol. 41, pp.391-407, 1990.
28. S. Delia Pietra, v. Delia Pietra, R. Mercer, and S. Roukos, "Adaptive language model estimation using minimum discrimination estimation,"InProc. 1992 Int. Conf. Acoust., Speech, Signal Processing, vol. 1, San Francisco, CA, Apr. 1992, pp.633-636.
29. S. T. Durnais, "Improving the retrieval of information from external sources," Behav. Res. Methods, Instrum., Comput., vol. 23, no. 2, pp.229-236, 1991.
30. _, "Latent semantic indexing (LSi) and TREC-2,"InProc. 2nd Text Retrieval Conf. (TREC-2), D. Harman, Ed., 1994, N1ST Pub. 500-215, pp.105-116.
31. VI. Federico and R. De Mori, "Language modeling,"InSpoken Dialogues with Computers, R. De Mori, Ed. London, U.K.: Academic, 1998, ch. 7, pp.199-230.
32. P. W. Foltz and S. T. Dumais, "Personalized information delivery: An analysis of information filtering methods," Commun. ACM, vol. 35, no. 12, pp.51-60, 1992.
33. P. N. Garner, "On topic identification and dialogue move recognition," Comput. Speech Lang., vol. 11, no. 4, pp.275-306, 1997.
34. D. Gildea and T. Hofmann, "Topic-based language modeling using EM,"InProc. 6th Eur. Conf. Speech Commun. Techno!., vol. 5, Budapest, Hungary, Sept. 1999, pp.2167-2170.
35. G. Golub and C. Van Loan, Matrix Computations, 2nd ed. Baltimore, MD: Johns Hopkins, 1989.
36. Y. Gotoh and S. Renais, "Document space models using latent semantic analysis,"InProc. 5th Eur. Conf. Speech Commun. Technol., vol. 3, Rhodes. Greece, Sept. 1997, pp.1443-1448.
37. T. Hofmann, "Probabilistic latent semantic analysis,"InProc. 15th Conf. Uncertainty In AI. Stockholm, Sweden, Jul 1999.
38. _, "Probabilistic topic maps: navigating through large text collections,"' in Lecture Notes in Computer Science. Heidelberg, Germany: Springer-Verlag, July 1999, pp.161-172. no. 1642.
39. R. lyer and M. Ostendorf. "Modeling long distance dependencies in language: Topic mixtures versus dynamic cache models,"IEEE Trans. Speech Audio Processing, vol. 7, pp.30-39, Jan. 1999.
40. F. Jelinek. "The development of an experimental discrete dictation recognizer," Proc. IEEE, vol. 73, pp.1616-1624. Nov. 1985.
41. _, "Self-organized language modeling for speech recognition,"InReadings m Speech Recognition, A. Waibel and K. F. Lee, Eds. New York: Morgan Kaufmann, 1990, pp.450-506.
42. F. Jelinek and C. Chelba, "Putting language into language modeling,"InPmc. 6th Ear. Conf. Speech Commun. Techno!., vOL i, Budapest, Hungary, Sept. 1999, pp.KN1-KN5.
43. D. Jurafsky, C. Wooters, J. Segal, A. Stölcke, E. Fosler, G. Tajchman, und N. Morgan, "Using a stochastic context-free grammar as a language model for speech recognition,"InProc. 1995 Int. Conf. Ac oust.. Speech, Signal Processing, vol. 1, Detroit, MI, May 1995, pp.189-192.
44. S. Khudanpur, "Putting language back into language modeling," presented at the WcrKshop-2000 Spoken Lang. Reco. Understanding, Summit. NJ, Feb. 2000.
45. R. Kneser, "Statistical language modeling using a variable context,"InProc. Int. Conf. Spoken Language Proc., Philadelphia, PA, Oct. 1996. pp.494-497.
46. F. Kubala, J. R. Beliegarda, J. R. Cohen, D. Pallett, D. B. Paul, M. Phillips, R. Rajasekaran, F. Richardson, M. Riley, R. Rosenreld, R. Roth, ana M. \\eintraub, "The hub and spoke paradigm for CSR evaluation,"InProc. ARPA Speech and datura! Language Workshop, Mar. 1994, pp.40-44.
47. J R. Kühn and R. De Mori, "A cache-based natural language method for speech recognition,"IEEE Trans. Faltern Anal. Machine Intt'll., vol. 12, pp.570-582, June 1990.
48. J. D. Lafferty and B. Suhni, "Cluster expansion and iterative scaling for maximum entropy language models,"InMaximum Entropy and Bayesian Methods, K. Hanson and R. Silver, Eds. Norwell, MA: Kluwer, 1995.
49. T. K. Landauer and S. T. Dumais, "Solution to Plato's problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge," Psvch. Rei:, vol. 104, no. 2, pp.211-240, 1997.
50. T. K. Landauer, D. Laham, B. Render, and M. E. Schreiner, "How well can passage meaning be derived without using word order: A comparison of latent semantic analysis and humans,"InProc. Cognit. Science Soc., 1998.
51. R. Lau, R. Rosenreld, and S. Roukos, "Trigger-based language models: A maximum entropy approach,"InProc. 1993 Int. Conf. Acou.it., Speech, Signal Processing, Minneapolis, MN, May 1993, pp.1145-48.
52. C.-H. Lcc, private communication, Dec. 1999.
53. H. Key, U. Essen, and R. Kneser, "On structuring probabilistic dependences in stochastic language modeling," Ccmput. Speech Lang., vol. 8, pp.1-38, 1994.
54. T. Niesler and P. Woodland, "A variable-length category-based -V-gram language model,"InProc. 1996 Int. Conf. Acottst., Speech. Signal Processing, Atlanta, GA, May 1996, pp.1164-1167.
55. C. H. Papadimilrtou, P. Raghavan, H. Tamaki, and S. Vempala, "Latent semantic indexing: A probabilistic analysis,"in Proc. 17th ACM Syrup. Princip. Database Syst., Seattle, WA, 1998.
56. F. C. Pereira, Y. Singer, and N. Tishby, "Beyond word n-Grams," Coftipul. Linguistics, vol.' 22, June 1996.
57. L. R. Rabiner, B. H. Juang, and C.-H. Lee, "An overview of automatic speech recognition," mAutonuitic Speech and Speaker Recognition: Advanced Topics, C.-H. Lee, F. K. Soong, and K. K. Paliwal, Eds. Boston, MA: Kluwer, 1996, ch. 1, pp.1-30.
58. R. Rosenfeld, "The CMU statistical language modeling toolkit and its use in the 1994 ARPA CSR evaluation;1 in Proc. ARPA Speech and Natural Language Workshop, Mar. 1994.
59. _, "Optimizing, lexical and .V-gram coverage via judicious use of linguistic data,"InProc. 4th Eur. Conf. Speech Comnuim. Technol., Madrid, Spain, Sept. 1995, pp.1763-1766.
60. _, "A maximum entropy approach to adaptive statistical language modeling,"InCompt.it. Speech Lang., July 1996, vol. 10, pp.187-228.
61. _, "Two decades of statistical language modeling: Where do we go from here,"/Vor. IEEE, vol. 88, pp.1270-1278,Aug. 2000.
62. R. Rosenfeld, L. Wasserman, C. Cai, and X. J. Zhu, "Interactive feature induction and logistic regression for whole sentence exponential "language models,"InProc. Ant. Speech Recog. Understanding Workshop, Keystone, CO, Dec. 1999, pp.231-236.
63. J S. Roukos, "Language representation,"InSuney of the State of the An in Human Language Technology, R. Coie, Ed. Cambridge, U.K.: Cambridge Univ. Press, 1997, ch. 6.
64. R. Schwanz, T. Iinai, F. Kubala, L, Nguyen, and J. Makhoul, "A maximum likelihood model for topic classification of broadcast news,"InProc. 5th Eur. Conf. Speech Commun. TechnoL, vol. 5, Rhodes, Greece, Sept. 1997, pp.1455-1458.
65. R- E. Story, "An explanation of the effectiveness of latent semantic indexing by means of a Bayesian regression model,"Inform. Process. Manage., vol. 32, no. 3, pp.329-344, 1996.
66. J. Wu and S. Khudanpur, "Combining nonlocal, syntactic and -V-gram dependencies in language modeling,"InProc. 6th Eur. Conf. Speech Commun. TechnoL, vol. 5, Budapest, Hungary, Sept. 1999, pp.2179-2182.
67. D. H. Younger, "Recognition and parsing of context-free languages intime A"ü,"Inform. Control, vol. I, pp.198-208. 1967.
68. R. Zhang. E. Black, and A. Finch, "L sing detailed linguistic structure in language modeling,"InProc. 6th Eur. Conf. Speech Commun. Techno!., vol. 4, Budapest, Hungary, Sept. 1999, pp.1815-1818.
69. X. J. Zhu, S. F. Chen, and R. Rosenfeld, "Linguistic features for whole sentence maximum entropy language models,"InProc. 6th Kin: Conf. Speech Commun Techno!.., vol. 4, Budapest, Hungary, Sept. 1999. pp.1807-1810.
70. V. Zue, J. Glass, D. Goodine, H. Uung, M. Phillips, J. Polifroni, and S. Seneff, "Integration of speech recognition and natural language processing in the MIT voyager system,"InProc. 1991 IEF.E In!. Conf. Acousl., Speech, Signal Processing, Toronto, Canada, May 1991, pp.713-716.
71. C. H. Papadimitrioii, P. Raghavan, H. Tamaki, and S. Vempala, "Latent semantic indexing: A probabilistic analysis," J. Comp. Syst. Sei., to be published.
72. J R. Rosenfeld, "Two decades of statistical language modeling: Where do we go from here," presented at the Workshop-2000 Spoken Lang. Reco. Understanding, Summit, NJ. Feb. 2000.