A Review of Statistical Language Processing Techniques

Artificial Intelligence Review

Volumn 12, Issue 5, 1998, Pages 347-391

  Mcmahon, John ^a   Smith, F Jack ^a  

^a QUEEN'S UNIVERSITY BELFAST   (United Kingdom)

Author keywords

N gram models; Natural language processing; Probability; Statistics

Indexed keywords

COMPUTATIONAL LINGUISTICS; INFORMATION THEORY; MATHEMATICAL MODELS; PROBABILITY; RANDOM PROCESSES; STATISTICAL METHODS;

STATISTICAL LANGUAGE PROCESSING TECHNIQUES;

NATURAL LANGUAGE PROCESSING SYSTEMS;

EID: 0032187759     PISSN: 02692821     EISSN: None     Source Type: Journal    
DOI: 10.1023/A:1006517723917     Document Type: Review

References (120)

1. Antonisse, Hendrik James (1991). A Grammar-Based Genetic Algorithm. Foundations of Genetic Algorithms: 193-204.
2. Atick, Joseph J. & Redlich, A. Norman (1990). Towards a Theory of Early Visual Processing. Neural Computation 2: 308-320.
3. Attneave, Fred (1954). Some Informational Aspects of Visual Perception. Psychological Review 61(3): 183-193.
4. Badalamenti, Anthony F., Langs, Robert J. & Robinson James (1994). Lawful Systems Dynamics in How Poets Choose Their Words. Behavioral Science 39.
5. Bahl, Lalit R., Brown, Peter F., DeSouza, Peter V. & Mercer, Robert L. (July, 1989). A Tree-Based Statistical Language Model for Natural Language Speech Recognition. IEEE Transactions on Acoustics, Speech and Signal Processing 37(7): 1001-1008.
6. Barry, Alwyn (September 1993). The Emergence of High Level Structure in Classifier Systems - a Proposal. In Cowie, R. & Owens M. (eds.) Proceedings of the Sixth Irish Conference on Artificial Intelligence and Cognitive Science, 185-196.
7. Basili, Roberto, Pazienza, Teresa & Velardi, Paolo (1992). Combining NLP and Statistical Techniques for Lexical Acquisition. In Probabilistic Approaches to Natural Language. American Association for Artificial Intelligence, AAAI Press, Technical report FS-92-05.
8. Basili, Roberto, Pazienza Teresa & Velardi, Paolo (1993). What Can Be Learned from Raw Texts? Machine Translation 8: 147-173.
9. Baum, Leonard E., Petrie, Ted, Soules, George & Weiss, Norman (1970). A Maximization Technique Occurring in the Statistical Analyses of Probabilistic Functions of Markov Chains. The Annals of Mathematical Statistics 41(1): 164-171.
10. Beckwith, Richard, Fellbaum, Christiane, Gross, Derek & Miller, George A. (1991). WordNet: A Lexical Database Organized on Psycholinguistic Principles. In Zernik, Uri (ed.) Lexical Acquisition: Exploiting On-Line Resources to Build a Lexicon, ch. 9, 211-232. Lawrence Erlbaum Associates.
11. Berwick, Robert C. (1986). Learning from Positive-Only Examples - the Subset Principle and Three Case Studies. In Carbonell. J. C., Michalski, R. S. & Mitchell, T. M. (eds.) Machine Learning: An Artificial Intelligence Approach (Volume 2). Morgan Kaufmann Publishers.
12. Bod, Rens (1995). Enriching Language With Statistics: Performance Models of Natural Language. ILLC-publications.
13. Bouchaffra, Djamel & Rouault, Jacques (1992). A Nonstationary Hidden Markov Model with a Hard Capture of Observations: Application to the Problem of Morphological Ambiguities. In Probabilistic Approaches to Natural Language. American Association for Artificial Intelligence, AAAI Press. Technical report FS-92-05.
14. Brent, Michael (1993). Minimal Generative Explanations: A Middle Ground Between Neurons and Triggers. In Proceedings of the Fifteenth Meeting of the Cognitive Science Society.
15. Brill, Eric & Marcus, Mitch (1992). Tagging an Unfamiliar Text with Minimal Human Supervision. In Probabilistic Approaches to Natural Language. American Association for Artificial Intelligence, AAAI Press. Technical report FS-92-05.
16. Brill, Eric, Magerman, David, Marcus, Mitchell & Santorini, Beatrice (1990). Deducing Linguistic Structure from the Statistics of Large Corpora. In Proceedings of the DARPA Speech and Natural Language Workshop.
17. Brill, Eric (1994). Some Advances in Transformation-Based Part of Speech Tagging. In Proceedings of the 12th National Conference on Artificial Intelligence, AAAI-94.
18. Brooks, Rodney A. (1991). Intelligence without Reason. In IJCAI-91.
19. Brown, P. F., Cocke, J., Della Pietra, V., Delia Pietra, S., Jelinek, F., Mercer, R. L. & Roosin, P. S. (1990). A Statistical Approach to Machine Translation. Computational Linguistics 16: 79-85.
20. Brown, Peter F., Della Pietra, Vincent, De Souza, Peter, Lai, Jennifer C. & Mercer, Robert (1992). Class-Based n -Gram Models of Natural Language. Computational Linguistics 18(4): 467-479.
21. Burger, John D. & Connolly, Dennis (1992). Probabilistic Resolution of Anaphoric Reference. In Probabilistic Approaches to Natural Language. American Association for Artificial Intelligence, AAAI Press. Technical report FS-92-05.
22. Carroll, Glenn & Charniak, Eugene (1992). Learning Probabilistic Dependence Grammars from Labelled Text. In Probabilistic Approaches to Natural Language. American Association for Artificial Intelligence, AAAI Press. Technical report FS-92-05.
23. Chater, Nick (1994). Neural Networks: The New Statistical Models of Mind. In Proceedings of the 1993 Neural Computation and Psychology Workshop. UCL Press: London.
24. Cheng, Bing & Titterington, D. M. (1994). Neural Networks: A Review from a Statistical Perspective. Statistical Science 9(1): 2-54.
25. Chomsky, Noam (1957). Syntactic Structures. The Hague: Mouton.
26. Church, Kenneth W. & Gale, William A. (1991). A Comparison of the Enhanced Good-Turing and Deleted Estimation Methods for Estimating Probabilities of English Bigrams. Computer Speech and Language 5: 19-54.
27. Church, Kenneth W. & Gale, Willliam A. (1995). Poisson Mixtures. Natural Language Engineering 1(2): 163-190.
28. Church, Kenneth Ward & Hanks, Patrick (1989). Word Association Norms, Mutual Information and Lexicography. In Proceedings of the 27th Annual Conference of the Association of Computational Linguistics, 76-82.
29. Church, Kenneth W. & Mercer, Robert L. (1993). Introduction to the Special Issue on Computational Linguistics Using Large Corpora. Computational Linguistics 19(1): 1-23.
30. Church, Kenneth W., Gale, William A., Hanks, Patrick & Hindle, Donald (1991). Using Statistics in Lexical Analysis. In Zernik, Uri (ed.) Lexical Acquisition: Exploiting On-Line Resources to Build a Lexicon, ch. 6, 115-164. Lawrence Erlbaum Associates.
31. Church, Kenneth Ward (1988). A Stochastic Parts Program and Noun Phrase Parser for Unrestricted Text. In Second Conference on Applied Natural Language Processing.
32. Cover, Thomas M. & and Thomas, Joy A. (1991). Elements of Information Theory. John Wiley and Sons.
33. de Saussure, Ferdinand (1983). Course in General Linguistics. London: Duckworth.
34. Dennett, Daniel (1991). Consciousness Explained. London: Allen Lane.
35. Derouault, Anne-Marie & Merialdo, Bernard (November 1986). Natural Language Modelling for Phoneme-to-Text Transcription. I.E.E.E. Transactions on Pattern Analysis and Machine Intelligence, PAMI-8(6).
36. Dunning, Ted (1993). Accurate Methods for the Statistics of Surprise and Coincidence. Computational Linguistics 19(1): 61-74.
37. Egedi, D., Palmer, M., Park, H. S. & Joshi, A. K. (October 1994). Korean to English Translation Using Synchronous TAGs. In Proceedings of the First Conference of the Association for Machine Translation in the Americas, 48-55. Columbia, Maryland.
38. Elman, Jeffrey L. (1990). Finding Structure in Time. Cognitive Science 14: 179-211.
39. Elman, Jeffrey L. (1991). Incremental Learning, or the Importance of Starting Small. Technical Report 9101, Center for Research in Language, U.C.S.D.
40. Elworthy, David (October 1994). Does baum-welch Re-Estimation Help Taggers? In Proceedings of the Fourteenth ACL Conference on Applied Natural Language Processing, ANLP-94, 53-58.
41. Faulk, R. D. & Gustavson, F. Goertzel (1990). Segmenting Discrete Data Representing Continuous Speech Input. I.B.M. Systems Journal 29(2).
42. Finch, Steven & Chater, Nick (1992). Bootstrapping Syntactic Categories Using Statistical Methods. In Daelemans, Walter & Powers, David (eds.) Background and Experiments in Machine Learning of Natural Language, 229-235. Institute for Language Technology and AI.
43. Finch, Steven & Chater, Nick (1994). Learning Syntactic Categories: A Statistical Approach. In Oaksford M. & Brown, G. D. A. (eds.) Neurodynamics and Psychology, ch. 12. Academic Press.
44. Firth, J. R. (1968). A Synopsis of Linguistic Theory 1930 - 1955. In Palmer, F. (ed.) Selected Papers of J. R. Firth. Longman.
45. Fisher, David & Riloff, Ellen (1992). Applying Statistical Methods to Small Corpora: Benefitting from a Limited Domain. In Probabilistic Approaches to Natural Language. American Association for Artificial Intelligence, AAAI Press. Technical report FS-92-05.
46. Gale, William A. & Church, Kenneth W. (1990). Poor Estimates of Context Are Worse Than None. In Proceedings of the DARPA Speech and Natural Language Workshop, 283-287.
47. Gale, William A., Church, Kenneth W. & Yarowsky, David (1992). Work on Statistical Methods for Word Sense Disambiguation. In Probabilistic Approaches to Natural Language. American Association for Artificial Intelligence, AAAI Press. Technical report FS-92-05.
48. Gaussier, Eric & Lange, Jean-Marc (1992). Towards Bilingual Terminology. In Proceedings of the 19th International Conference of the Association for Literary and Linguistic Computing and the 12th International Conference on Computers and the Humanities, 121-124.
49. Gold, M. (1967). Language Identification in the Limit. Information and Control 10: 447-474.
50. Goldberg, D. E. (1989). Genetic Algorithms in Search Optimization and Machine Learning. Addison Wesley.
51. Gonzalez, Rafael C. & Thomason, Michael G. (1978). Syntactic Pattern Recognition. Addison Wesley.
52. Good, I. J. (December 1953). The Population Frequencies of Species and the Estimation of Population Parameters. Biometrika 40: 237-264.
53. Gorin, A. L., Levinson, S. E., Gertner, A. N. & Goldman, E. (1991). Adaptive Acquisition of Language. Computer Speech and Language 5: 101-132.
54. Grefenstette, Gregory (1992). Finding Semantic Similarity in Raw Text: The Deese Antonyms. In Probabilistic Approaches to Natural Language. American Association for Artificial Intelligence, AAAI Press. Technical report FS-92-05.
55. Harris, Zellig S. (1951). Structural Linguistics. Phoenix Books.
56. Holland, J. H. (1975). Adaptation in Natural and Artificial Systems. University of Michigan Press.
57. Holland, J. H. (1986). Escaping Brittleness: The Possibilities of General Purpose Learning Algorithms Applied to Parallel Rule-Bases Systems. In Carbonell, J. G., Michalski, R. S. & Mitchell, T. M. (eds.) Machine Learning II. Morgan Kaufmann.
58. Hughes, John & Atwell, Eric (1994). The Automated Evaluation of Inferred Word Classifications. In Eleventh European Conference on Artificial Intelligence.
59. Hull, Jonathan J. (1992). Combining Syntactic Knowledge and Visual Text Recognition: A Hidden Markov Model for Part of Speech Tagging in a Word Recognition Algorithm. In Probabilistic Approaches to Natural Language. American Association for Artificial Intelligence, AAAI Press. Technical report FS-92-05.
60. Jelinek, Frederick, Merialdo, B., Roukos, S. & Strauss, M. (February 1991). A Dynamic Language Model for Speech Recognition. In Proceedings of the Speech and Language DARPA Workshop, 293-295.
61. Jelinek, Frederick (April 1976). Continuous Speech Recognition by Statistical Methods. Proceedings of the I.E.E.E. 64(4).
62. Jelinek, Frederick (1985). The Development of an Experimental Discrete Dictation Recogniser. Proceedings of the I.E.E.E. 73(11).
63. Jelinek, Frederick (1990). Self-Organized Language Modelling for Speech Recognition. In Waibel & Lee (eds.) Readings in Speech Recognition. San Mateo, California: Morgan Kaufmann.
64. Jones, Daniel (1992). Virtual Machine Translation. In Probabilistic Approaches to Natural Language. American Association for Artificial Intelligence, AAAI Press. Technical report FS-92-05.
65. Jordan, M. (1986). Attractor Dynamics and Parallelism in a Connectionist Sequential Machine. In Proceedings of the Eighth Annual Conference of the Cognitive Science Society, 531-546. Lawrence Erlbaum Associates.
66. Juola, Patrick, Hall, Chris & Boggs, Adam (1994). Corpus-Based Morphological Segmentation by Entropy Changes. In Monaghan, A. I. C. (ed.) Third Conference on the Cognitive Science of Natural Language Processing. Dublin City University.
67. Juola, Patrick (1994). A Psycholinguistic Approach to Corpus-Based Machine Translation. In Monaghan, A. I. C. (ed.) Third Conference on the Cognitive Science of Natural Language Processing. Dublin City University.
68. Juola, Patrick (1995). Corpus-Based Acquisition of Grammars and Transfer Functions for Machine Translation. Technical report CU-CS-756-95, Department of Computer Science, University of Colorado at Boulder.
69. Katz, Slava M. (March 1987). Estimation of Probabilities for Sparse Data for the Language Model Component of a Speech Recogniser. I.E.E.E. Transactions on Acoustics, Speech and Signal Processing, ASSP-35(3): 400-401.
70. Kiss, George R. (1973). Grammatical Word Classes: A Learning Process and Its Simulation. Psychology of Learning and Motivation 7: 1-41.
71. Kneser, Reinhard & Ney, Hermann (1993). Forming Word Classes by Statistical Clustering for Statistical Language Modelling. In Köhler, R. & Rieger, B. B. (eds.) Contributions to Quantitative Linguistics, 221-226. Kluwer Academic Publishers.
72. Kolmogorov, A. N. (1964). Three Approaches to the Quantitative Definition of Information. Problems in Information Transmission 1: 4-7.
73. Koza, J. R. (1989). Hierarchical Genetic Algorithms that Operate on Populations of Computer Programs. In Proceedings of the Eleventh International Joint Conference on Artificial Intelligence, 768-780.
74. Kozima, Hideki (1993). Text Segmentation Based on Similarity Between Words. In Proceedings of the Association for Computational Linguistics.
75. Krotov, Alexander, Gaizauskas, Robert & Wilks, Yorick (1994). Acquiring a Stochastic Context-Free Grammar from the Penn Treebank. In Monaghan, A. I. C. (ed.) Third Conference on the Cognitive Science of Natural Language Processing. Dublin City University.
76. Kuhn, Ronald & De Mori, Renato (June 1990). A Cache-Based Natural Language Model for Speech Recognition. I.E.E.E. Transactions on Pattern Analysis and Machine Intelligence 12(6): 570-583.
77. Kupiec, Julian (1992). Robust Part-of-Speech Tagging Using a Hidden Markov Model. Computer Speech and Language 6: 225-242.
78. Lau, Raymond, Rosenfeld, Ronald & Roukos, Salim (March 1993). Adaptive Language Modelling Using the Maximum Entropy Principle. In ARPA Workshop on Human Language Technology, 108-113. Princeton.
79. Liberman, Mark K. (1991). The Trend Towards Statistical Models in Natural Language Processing. In Klein, E. & Veltman, F. (eds.) Natural Language and Speech. Berlin: Springer Verlag.
80. Liddy, Elizabeth D. & Paik, Woojin (1992). Statistically-Guided Word Sense Disambiguation. In Probabilistic Approaches to Natural Language. American Association for Artificial Intelligence, AAAI Press. Technical report FS-92-05.
81. Linsker, Ralph (1988). Self-Organization in a Perceptual Network. I.E.E.E. Computer 21(3): 105-117.
82. Magerman, David M. (February 1994). Natural Language Parsing as Statistical Pattern Recognition. PhD thesis, Stanford University Computer Science Department.
83. Makhoul, John, Jelinek, Fred, Rabiner, Larry, Weinstein, Clifford & Zue, Victor (1990). Spoken Language Systems. Annual Review of Computer Science 4: 481-501.
84. Manderick, Bernard (1992). The Genetic Algorithm. In Background and Experiments in Machine Learning of Natural Language.
85. McGee-Wood, Mary (1993). Categorial Grammars. London: Routledge.
86. McMahon, John & Smith, F. J. (1996). Improving Statistical Language Model Performance with Automatically Generated Word Hierarchies. Computational Linguistics 22(2): 217-247.
87. Miller, George A. & Charles, Walter G. (1991). Contextual Correlates of Semantic Similarity. Language and Cognitive Processes 6(1): 1-28.
88. Miller, George A. (1951). Language and Communication. New York: McGraw-Hill.
89. Nagao, Makoto (1984). A Framework of a Mechanical Translation Between Japanese and English by Analogy Principle. In Elithorn A. & Barnerji, R. (eds.) Artificial and Human Intelligence, 173-180. North-Holland.
90. Naumann, Sven & Schrepp, Jürgen (1992). Inductive Learning of Reversible Grammars. In Daelemans, Walter & Powers, David (eds.) Background and Experiments in Machine Learning of Natural Language, 237-243. Institute for Language Technology and AI.
91. Ney, Hermann, Essen, Ute & Kneser, Reinhard (1994). On Structuring Probabilistic Dependencies in Stochastic Language Modelling. Computer Speech and Language 8: 1-38.
92. Nicholis, John S. & Katsikas, Anastassis A. (1993). Chaotic Dynamics of Linguistic-Like Processes at the Syntactical and Semantic Levels: In Pursuit of a Multifractal Attractor. In Patterns, Information and Chaos in Neuronal Systems. World Scientific.
93. Niyogi, Partha & Berwick, Robert C. (March 1995). A Note on Zipf's Law, Natural Languages, and Noncoding DNA Regions. Technical Report Memo 118, M.I.T. CBCL. In the Computation and Language archive.
94. O'Boyle, Peter, Owens, Marie & Smith, F. J. (1994). A Weighted Average N-Gram Model of Natural Language. Computer Speech and Language 8: 337-349.
95. O'Boyle, Peter, McMahon, John & Smith, F. J. (December 1995). Combining a Multi-Level Class Hierarchy with Weighted Average Function-Based Smoothing. In IEEE Workshop on Automatic Speech recognition, 9/1-9/7. Snowbird, Utah.
96. Oncina, J., Castellanos, A., Vidal, E. & Jimémez, V. (1994). Corpus-Based Machine Translation Through Subsequential Transducers. In Monaghan, A. I. C. (ed.) Third Conference on the Cognitive Science of Natural Language Processing. Dublin City University.
97. Pereira, Fernando & Tishby, Naftali (1992). Distributed Similarity, Phase Transitions and Hierarchical Clustering. In Probabilistic Approaches to Natural Language. American Association for Artificial Intelligence, AAAI Press. Technical report FS-92-05.
98. Plumbley, M. D. (1993). Information Theory and Neural Network Learning Algorithms. In Orchard, Gerry (ed.) Neural Computing - Research and Applications, 145-155. Institute of Physics Publishing. Proceedings of the Second Irish Neural Networks Conference.
99. Powers, David & Daelemans, Walter (1992). SHOE: The Extraction of Hierarchical Structure for Machine Learning of Natural Language (Project Summary). In Background and Experiments in Machine Learning of Natural Language, 125-161.
100. Rabiner, L. R. & Juang. B. J. (January 1986). An Introduction to Hidden Markov Models. I.E.E.E. A.S.S.P. Magazine, 4-16.
101. Ramsay, Allan (July 1994). Linguistics: The Cognitive Science of Natural Language. In Third Conference on the Cognitive Science of Natural Language Processing. Dublin City University.
102. Rao, P. Srinivasa, Monkowski, Michael D. & Roukos, Salim (May 1995). Language Model Adaptation Via Minimum Discrimination Information. In ICASSP 95, 161-164. IEEE.
103. Redington, Martin, Chater, Nick & Finch, Steven (1993). Distributional Information and the Acquisition of Linguistic Categories: A Statistical Approach. In Proceedings of the Fifteenth Annual Conference of the Cognitive Science Society.
104. Reilly, Ronan (1992). A Connectionist Technique for On-Line Parsing. In The Cognitive Science of Natural Language Processing.
105. Reilly, Ronan (1992). An Exploration of Clause Boundary Effects in Simple Recurrent Network Representations. In The Second Irish Neural Netowrks Conference.
106. Resnik, Philip S. (1992). Probabilistic Tree-Adjoining Grammar as a Framework for Statistical Natural Language Processing. In Proceedings of COLING-92. Nantes.
107. Resnik, Philip S. (December 1993). Selection and Information: A Class-Based Approach to Lexical Relationships. PhD thesis, Computer and Information Science, University of Pennsylvania. Institute for Research in Cognitive Science Report I.R.C.S.-93-42.
108. Rohlicek, Jan Robin, Chow, Yen-Lu & Roukos, Salim (1988). Statistical Language Modelling Using a Small Corpus from an Application Domain. In IEEE International Conference on Acoustics, Speech and Signal Processing, 267-270.
109. Sampson, Geoffrey (1987). Evidence Against the Grammatical-Ungrammatical Distinction. In Meijs, Wilem (ed.) Corpus Linguistics and Beyond: Proceedings of the 7th International Conference on English Language Research on Computerized Corpora, 219-226. Rodopi, Amsterdam.
110. Scholtes, J. C. (1992). Resolving Linguistic Ambiguities with a Neural Data Oriented Parsing (DOP) System. In Background and Experiments in Machine Learning of Natural Language, 279-282.
111. Schütze, Hinrich (1993). Part-of-Speech Induction from Scratch. In Proceedings of the Association for Computational Linguistics 31, 251-258.
112. Shannon, C. E. (1951). Prediction and Entropy of Printed English. Bell System Technical Journal.
113. Solomon, D. & McGee-Wood, M. (December 1993). Unified Lexicon and Grammar. In Collingham, Russell J. (ed.) Workshop on the Unified Lexicon.
114. Somers, H., McLean, I. & Jones, D. (1994). Experiments in Multilingual Example-Based Generation. In Monaghan, A. I. C. (ed.) Third Conference on the Cognitive Science of Natural Language Processing. Dublin City University.
115. Sutcliffe, Richard F. E., McElligott, Annette & O'Néill, G. (1993). Irish-English Lexical Translation Using Distributed Semantic Representations. In Cowie, R. & Owens, M. (eds.) Artificial Intelligence and Cognitive Science.
116. Tanenhaus, Michael K. (1988). Psycholinguistics: An Overview. In Newmeyer, Frederick J. (ed.) Linguistics: The Cambridge Survey, volume III, ch. 1, 1-37. Cambridge University Press.
117. Wolff, J. Gerard (1991). Towards a Theory of Cognition and Computing. Ellis Horwood.
118. Wyard, P. J. & Nightingale, C. (1992). Grammar Recognition by a Single Layer Higher Order Neural Net. B. T. Technological Journal 10(3).
119. Zernik, Uri (1991). Introduction. In Zernik, Uri (ed.) Lexical Acquisition: Exploiting On-Line Resources to Build a Lexicon, ch. 1, 1-26. Lawrence Erlbaum Associates.
120. Zipf, George K. (1949). Human Behaviour and the Principle of Least Effort. Addison-Wesley.