Covariance in unsupervised learning of probabilistic grammars

Journal of Machine Learning Research

Volumn 11, Issue , 2010, Pages 3017-3051

  Cohen, Shay B ^a   Smith, Noah A ^a  

^a Carnegie Mellon University   (United States)

Author keywords

Bayesian inference; Dependency grammar induction; Logistic normal distribution; Variational inference

Indexed keywords

BAYESIAN; BAYESIAN INFERENCE; DEPENDENCY GRAMMAR; DIRECTLY MODEL; DIRICHLET PRIOR; INFERENCE ALGORITHM; LEARNING SETTINGS; LINGUISTIC PROPERTIES; NATURAL LANGUAGE TEXT; NATURAL LANGUAGES; PARALLEL DATA; PERFORMANCE IMPROVEMENTS; PROBABILISTIC GRAMMARS; SEQUENTIAL DATA; TREEBANKS; VARIATIONAL INFERENCE;

BAYESIAN NETWORKS; COMPUTATIONAL GRAMMARS; INFERENCE ENGINES; LEARNING ALGORITHMS;

NORMAL DISTRIBUTION;

EID: 79551472048     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (83)

1. S. Afonso, E. Bick, R. Haber, and D. Santos. Floresta sinta(c)tica: A treebank for Portuguese. In Proceedings of LREC, 2002.
2. A. Ahmed and E. Xing. On tight approximate inference of the logistic normal topic admixture model. In Proceedings of AISTATS, 2007.
3. J. Aitchison. The Statistical Analysis of Compositional Data. Chapman and Hall, London, 1986.
4. H. Alshawi and A. L. Buchsbaum. Head automata and bilingual tiling: Translation with minimal representations. In Proceedings of ACL, 1996.
5. D. Angluin. Learning regular sets from queries and counterexamples. Information and Computation, pages 87-106, 1988.
6. N. B. Atalay, K. Oflazer, and B. Say. The annotation process in the Turkish treebank. In Proceedings of LINC, 2003.
7. J. Baker. Trainable grammars for speech recognition. In The 97th meeting of the Acoustical Society of America, 1979.
8. A. Banerjee. On Bayesian bounds. In Proceedings of ICML, 2006.
9. T. Berg-Kirkpatrick and D. Klein. Phylogenetic grammar induction. In Proceedings of ACL, 2010.
10. T. Berg-Kirkpatrick, A. Bouchard-Cote, J. DeNero, and D. Klein. Unsupervised learning with features. In Proceedings of NAACL, 2010.
11. D. M. Blei and J. D. Lafferty. Correlated topic models. In Proceedings of NIPS, 2006.
12. D. M. Blei, A. Ng, and M. Jordan. Latent Dirichlet allocation. Journal of Machine Learning Research, 3:993-1022, 2003.
13. J. L. Boyd-Graber and D. M. Blei. Syntactic topic models. CoRR, abs/1002.4665, 2010.
14. P. F. Brown, V. J. Della Pietra, P. V. deSouza, J. C. Lai, and R. L. Mercer. Class-based n-gram models of natural language. Computational Linguistics, 1990.
15. S. Buchholz and E. Marsi. CoNLL-X shared task on multilingual dependency parsing. In Proceedings of CoNLL, 2006.
16. D. Burkett and D. Klein. Two languages are better than one (for syntactic parsing). In Proceedings of EMNLP, 2008.
17. G. Carroll and E. Charniak. Two experiments on learning probabilistic dependency grammars from corpora. Technical report, Brown University, 1992.
18. E. Charniak and M. Johnson. Coarse-to-fine n-best parsing and maxent discriminative reranking. In Proceedings of ACL, 2005.
19. S. F. Chen. Bayesian grammar induction for language modeling. In Proceedings of ACL, 1995.
20. D. Chiang. A hierarchical phrase-based model for statistical machine translation. In Proceedings of ACL, 2005.
21. A. Clark and F. Thollard. PAC-learnability of probabilistic deterministic finite state automata. Journal of Machine Learning Research, 5:473-497, 2004.
22. A. Clark, R. Eyraud, and A. Habrard. A polynomial algorithm for the inference of context free languages. In Proceedings of ICGI, 2008.
23. S. B. Cohen and N. A. Smith. Shared logistic normal distributions for soft parameter tying in unsupervised grammar induction. In Proceedings of NAACL-HLT, 2009.
24. S. B. Cohen and N. A. Smith. Empirical risk minimization with approximations of probabilistic grammars. In NIPS, 2010.
25. S. B. Cohen, K. Gimpel, and N. A. Smith. Logistic normal priors for unsupervised probabilistic grammar induction. In NIPS, 2008.
26. S. B. Cohen, D. M. Blei, and N. A. Smith. Variational inference for adaptor grammars. In Proceedings of NAACL, 2010.
27. T. Cohn, S. Goldwater, and P. Blunsom. Inducing compact but accurate tree-substitution grammars. In Proceedings of HLT-NAACL, 2009.
28. M. Collins. Head-Driven Statistical Models for Natural Language Parsing. PhD thesis, U. Penn., 1999.
29. M. Collins. Head-driven statistical models for natural language processing. Computational Linguistics, 29:589-637, 2003. (Pubitemid 39056498)
30. I. Dagan. Two languages are more informative than one. In Proceedings of ACL, 1991.
31. D. Das, N. Schneider, D. Chen, and N. A. Smith. Probabilistic frame-semantic parsing. In Proceedings of ACL, 2010.
32. C. de la Higuera. A bibliographical study of grammatical inference. Pattern Recognition, 38:1332-1348, 2005.
33. J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. In Proceedings of OSDI, 2004.
34. A. Dempster, N. Laird, and D. Rubin. Maximum likelihood estimation from incomplete data via the EM algorithm. Journal of the Royal Statistical Society B, 39:1-38, 1977.
35. Y. Ding and M. Palmer. Machine translation using probabilistic synchronous dependency insertion grammars. In Proceedings of ACL, 2005.
36. J. Eisner. Bilexical grammars and a cubic-time probabilistic parser. In Proceedings of IWPT, 1997.
37. J. R. Finkel, T. Grenager, and C. D. Manning. The infinite tree. In Proceedings of ACL, 2007.
38. H. Gaifman. Dependency systems and phrase-structure systems. Information and Control, 8, 1965.
39. K. Ganchev, J. Gillenwater, and B. Taskar. Dependency grammar induction via bitext projection constraints. In Proceedings of ACL, 2009.
40. J. Gillenwater, K. Ganchev, J. Graça, F. Pereira, and B. Taskar. Sparsity in dependency grammar induction. In Proceedings of ACL, 2010.
41. K. Gimpel and N. A. Smith. Feature-rich translation by quasi-synchronous lattice parsing. In Proceedings of EMNLP, 2009.
42. S. Goldwater. Nonparametric Bayesian models of lexical acquisition. PhD thesis, Brown University, 2006.
43. S. Goldwater and T. L. Griffiths. A fully Bayesian approach to unsupervised part-of-speech tagging. In Proceedings of ACL, 2007.
44. A. Haghighi, P. Liang, T. Berg-Kirkpatrick, and D. Klein. Learning bilingual lexicons from monolingual corpora. In Proceedings of ACL, 2008.
45. J. Hajič, A. Böhmová, E. Hajičová, and B. Vidová Hldaká. The Prague dependency treebank: A three-level annotation scenario. Treebanks: Building and Using Parsed Corpora, pages 103-127, 2000.
46. W. P. Headden, M. Johnson, and D. McClosky. Improving unsupervised dependency parsing with richer contexts and smoothing. In Proceedings of NAACL-HLT, 2009.
47. G. E. Hinton. Products of experts. In Proceedings of ICANN, 1999.
48. R. Hwa, P. Resnik, A. Weinberg, C. Cabezas, and O. Kolak. Bootstrapping parsers via syntactic projection across parallel texts. Journal of Natural Language Engineering, 11(3):311-25, 2005.
49. R. Johansson and P. Nugues. LTH: Semantic structure extraction using nonprojective dependency trees. In Proceedings of SemEval, 2007.
50. M. Johnson. Why doesn't EM find good HMM POS-taggers? In Proceedings EMNLP-CoNLL, 2007.
51. M. Johnson, T. L. Griffiths, and S. Goldwater. Adaptor grammars: A framework for specifying compositional nonparameteric Bayesian models. In NIPS, 2006.
52. M. Johnson, T. L. Griffiths, and S. Goldwater. Bayesian inference for PCFGs via Markov chain Monte Carlo. In Proceedings of NAACL, 2007.
53. Y. Kawata and J. Bartels. Stylebook for the Japanese treebank in VERBMOBIL. Technical Report Verbmobil-Report 240, Seminar für Sprachwissenschaft, Univerisität Tübingen, 2000.
54. D. Klein and C. D. Manning. A generative constituent-context model for improved grammar induction. In Proceedings of ACL, 2002.
55. D. Klein and C. D. Manning. Accurate unlexicalized parsing. In Proceedings of ACL, pages 423-430, 2003.
56. D. Klein and C. D. Manning. Corpus-based induction of syntactic structure: Models of dependency and constituency. In Proceedings of ACL, 2004.
57. K. Kurihara and T. Sato. Variational Bayesian grammar induction for natural language. In Proceedings of ICGI, 2006.
58. K. Lari and S. J. Young. The estimation of stochastic context-free grammars using the inside-outside algorithm. Computer Speech and Language, 4:35-56, 1990.
59. P. Liang, S. Petrov, M. Jordan, and D. Klein. The infinite PCFG using hierarchical Dirichlet processes. In Proceedings of EMNLP, 2007.
60. D. Lin. A path-based transfer model for machine translation. In Proceedings of COLING, 2004.
61. M. P.Marcus, B. Santorini, andM. A.Marcinkiewicz. Building a large annotated corpus of English: The Penn treebank. Computational Linguistics, 19:313-330, 1993.
62. D. McAllester. PAC-Bayesian model averaging. Machine Learning Journal, 5:5-21, 2003.
63. D. Mimno, H. Wallach, and A. McCallum. Gibbs sampling for logistic normal topic models with graph-based priors. In In Proceedings of NIPS Workshop on Analyzing Graphs, 2008.
64. J. Nivre, J. Hall, S. Kübler, R. McDonald, J. Nilsson, S. Riedel, and D. Yuret. The CoNLL 2007 shared task on dependency parsing. In Proceedings of the CoNLL Shared Task, EMNLP-CoNLL, 2007.
65. K. Oflazer, B. Say, D. Z. Hakkani-Tür, and G. Tür. Building a Turkish treebank. In A. Abeille, editor, Building and Exploiting Syntactically-Annotated Corpora. Kluwer, 2003.
66. F. C. N. Pereira and Y. Schabes. Inside-outside reestimation from partially bracketed corpora. In Proceedings of ACL, 1992.
67. H. Raiffa and R. Schaifer. Applied Statistical Decision Theory. Wiley-Interscience, 1961.
68. M. Seeger. PAC-Bayesian generalization bounds for Gaussian processes. Journal ofMachine Learning Research, 3:233-269, 2002.
69. D. A. Smith and N. A. Smith. Bilingual parsing with factored estimation: Using English to parse Korean. In Proceedings of EMNLP, 2004.
70. N. A. Smith. Novel Estimation Methods for Unsupervised Discovery of Latent Structure in Natural Language Text. PhD thesis, Johns Hopkins University, 2006.
71. N. A. Smith and J. Eisner. Guiding unsupervised grammar induction using contrastive estimation. In Proceedings of IJCAI Workshop on Grammatical Inference Applications, 2005.
72. N. A. Smith and J. Eisner. Annealing structural bias in multilingual weighted grammar induction. In Proceedings of COLING-ACL, 2006.
73. B. Snyder and R. Barzilay. Unsupervised multilingual learning for morphological segmentation. In Proceedings of ACL, 2008.
74. V. Spitkovsky, H. Alshawi, and D. Jurafsky. From baby steps to leapfrog: How "less is more" in unsupervised dependency parsing. In Proceedings of NAACL, 2010a.
75. V. I. Spitkovsky, H. Alshawi, D. Jurafsky, and C. D. Manning. Viterbi training improves unsupervised dependency parsing. In Proceedings of CoNLL, 2010b.
76. V. I. Spitkovsky, D. Jurafsky, and H. Alshawi. Profiting from mark-up: Hyper-text annotations for guided parsing. In Proceedings of ACL, 2010c.
77. L. Tesnière. Élément de Syntaxe Structurale. Klincksieck, 1959.
78. K. Toutanova and M. Johnson. A Bayesian LDA-based model for semi-supervised part-of-speech tagging. In Proceedings of NIPS, 2007.
79. M. J. Wainwright and M. I. Jordan. Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning, 1:1-305, 2008.
80. M. Wang, N. A. Smith, and T. Mitamura. What is the Jeopardy model? a quasi-synchronous grammar for question answering. In Proceedings of EMNLP, 2007.
81. D.Wu. Stochastic inversion transduction grammars and bilingual parsing of parallel corpora. Computational Linguistics, 23(3):377-404, 1997.
82. N. Xue, F. Xia, F.-D. Chiou, and M. Palmer. The Penn Chinese Treebank: Phrase structure annotation of a large corpus. Natural Language Engineering, 10(4):1-30, 2004.
83. D. Yarowsky, G. Ngai, and R. Wicentoswki. Inducing multilingual text analysis tools via robust projection across aligned corpora. In Proceedings of HLT, 2001.