Empirical Risk Minimization for Probabilistic Grammars: Sample Complexity and Hardness of Learning

Computational Linguistics

Volumn 38, Issue 3, 2012, Pages 479-526

  Cohen, Shay B ^a   Smith, Noah A ^b  

^a Department of Earth and Environmental Sciences   (United States)

^b Carnegie Mellon University ^*  (United States)

Author keywords

[No Author keywords available]

Indexed keywords

MAXIMUM PRINCIPLE;

APPROXIMATE ALGORITHMS; EMPIRICAL RISK MINIMIZATION; EXPECTATION - MAXIMIZATIONS; HARDNESS OF LEARNING; PROBABILISTIC GRAMMARS; SAMPLE COMPLEXITY BOUNDS; STRUCTURAL RISK MINIMIZATION; UNDERLYING DISTRIBUTION;

PROBABILITY DISTRIBUTIONS;

EID: 84865284030     PISSN: 08912017     EISSN: 15309312     Source Type: Journal    
DOI: 10.1162/COLI_a_00092     Document Type: Article

References (56)

1. Abe, N., J. Takeuchi, and M. Warmuth. 1991. Polynomial learnability of probabilistic concepts with respect to the Kullback-Leiber divergence. In Proceedings of the Conference on Learning Theory, pages 277-289.
2. Abe, N. and M. Warmuth. 1992. On the computational complexity of approximating distributions by probabilistic automata. Machine Learning, 2:205-260.
3. Angluin, D. 1987. Learning regular sets from queries and counterexamples. Information and Computation, 75:87-106.
4. Anthony, M. and P. L. Bartlett. 1999.Neural Network Learning: Theoretical Foundations. Cambridge University Press.
5. Balcan, M. and A. Blum. 2010. A discriminative model for semi-supervised learning. Journal of the Association for Computing Machinery, 57(3):1-46.
6. Balle, B., A. Quattoni, and X. Carreras. 2011. A spectral learning algorithm for finite state transducers. In Proceedings of the European Conference on Machine Learning/the Principles and Practice of Knowledge Discovery in Databases, pages 156-171.
7. Bane, M., J. Riggle, and M. Sonderegger. 2010. The VC dimension of constraint-based grammars. Lingua, 120(5):1194-1208.
8. Bartlett, P., O. Bousquet, and S. Mendelson. 2005. Local Rademacher complexities. Annals of Statistics, 33(4):1497-1537.
9. Bishop, C. M. 2006. Pattern Recognition and Machine Learning. Springer, Berlin.
10. Boyd, S. and L. Vandenberghe. 2004. Convex Optimization. Cambridge University Press.
11. Carrasco, R. 1997. Accurate computation of the relative entropy between stochastic regular grammars. Theoretical Informatics and Applications, 31(5):437-444.
12. Carroll, G. and E. Charniak. 1992. Two experiments on learning probabilistic dependency grammars from corpora. Technical report, Brown University, Providence, RI.
13. Charniak, E. 1993. Statistical Language Learning. MIT Press, Cambridge, MA.
14. Charniak, E. and M. Johnson. 2005. Coarse-to-fine n-best parsing and maxent discriminative reranking. In Proceedings of the Association for Computational Linguistics, pages 173-180.
15. Chi, Z. 1999. Statistical properties of probabilistic context-free grammars. Computational Linguistics, 25(1):131-160.
16. Clark, A., R. Eyraud, and A. Habrard. 2008. A polynomial algorithm for the inference of context free languages. In Proceedings of the International Colloquium on Grammatical Inference, pages 29-42.
17. Clark, A. and S. Lappin. 2010. Unsupervised learning and grammar induction. In Alexander Clark, Chris Fox, and Shalom Lappin, editors, The Handbook of Computational Linguistics and Natural Language Processing. Wiley-Blackwell, London, pages 197-220.
18. Clark, A. and F. Thollard. 2004. PAC-learnability of probabilistic deterministic finite state automata. Journal of Machine Learning Research, 5:473-497.
19. Cohen, S. B. and N. A. Smith. 2010a. Covariance in unsupervised learning of probabilistic grammars. Journal of Machine Learning Research, 11:3017-3051.
20. Cohen, S. B. and N. A. Smith. 2010b. Empirical risk minimization with approximations of probabilistic grammars. In Proceedings of the Advances in Neural Information Processing Systems, pages 424-432.
21. Cohen, S. B. and N. A. Smith. 2010c. Viterbi training for PCFGs: Hardness results and competitiveness of uniform initialization. In Proceedings of the Association for Computational Linguistics, pages 1502-1511.
22. Collins, M. 2003. Head-driven statistical models for natural language processing. Computational Linguistics, 29:589-637.
23. Collins, M. 2004. Parameter estimation for statistical parsing models: Theory and practice of distribution-free methods. In H. Bunt, J. Carroll, and G. Satta, Text, Speech and Language Technology (New Developments in Parsing Technology). Kluwer, Dordrecht, pages 19-55.
24. Corazza, A. and G. Satta. 2006. Cross-entropy and estimation of probabilistic context-free grammars. In Proceedings of the North American Chapter of the Association for Computational Linguistics, pages 335-342.
25. Cover, T. M. and J. A. Thomas. 1991. Elements of Information Theory. Wiley, London.
26. Dasgupta, S. 1997. The sample complexity of learning fixed-structure bayesian networks. Machine Learning, 29(2-3):165-180.
27. de la Higuera, C. 2005. A bibliographical study of grammatical inference. Pattern Recognition, 38:1332-1348.
28. Dempster, A., N. Laird, and D. Rubin. 1977. Maximum likelihood estimation from incomplete data via the EM algorithm. Journal of the Royal Statistical Society B, 39:1-38.
29. Gildea, D. 2010. Optimal parsing strategies for linear context-free rewriting systems. In Proceedings of the North American Chapter of the Association for Computational Linguistics, pages 769-776.
30. Gómez-Rodríguez, C. and G. Satta. 2009. An optimal-time binarization algorithm for linear context-free rewriting systems with fan-out two. In Proceedings of the Association for Computational Linguistics-International Joint Conference on Natural Language Processing, pages 985-993.
31. Grenander, U. 1981. Abstract Inference. Wiley, New York.
32. Haussler, D. 1992. Decision-theoretic generalizations of the PAC model for neural net and other learning applications. Information and Computation, 100:78-150.
33. Hsu, D., S. M. Kakade, and T. Zhang. 2009. A spectral algorithm for learning hidden Markov models. In Proceedings of the Conference on Learning Theory.
34. Ishigami, Y. and S. Tani. 1993. The VC-dimensions of finite automata with n states. In Proceedings of Algorithmic Learning Theory, pages 328-341.
35. Ishigami, Y. and S. Tani. 1997. VC-dimensions of finite automata and commutative finite automata with k letters and n states. Applied Mathematics, 74(3):229-240.
36. Jaeger, H. 1999. Observable operator models for discrete stochastic time series. Neural Computation, 12:1371-1398.
37. Kearns, M. and L. Valiant. 1989. Cryptographic limitations on learning Boolean formulae and finite automata. In Proceedings of the 21st Association for Computing Machinery Symposium on the Theory of Computing, pages 433-444.
38. Kearns, M. J. and U. V. Vazirani. 1994. An Introduction to Computational Learning Theory. MIT Press, Cambridge, MA.
39. Klein, D. and C. D. Manning. 2004. Corpus-based induction of syntactic structure: Models of dependency and constituency. In Proceedings of the Association for Computational Linguistics, pages 478-487.
40. Koltchinskii, V. 2006. Local Rademacher complexities and oracle inequalities in risk minimization. The Annals of Statistics, 34(6):2593-2656.
41. Leermakers, R. 1989. How to cover a grammar. In Proceedings of the Association for Computational Linguistics, pages 135-142.
42. Manning, C. D. and H. Schütze. 1999. Foundations of Statistical Natural Language Processing. MIT Press, Cambridge, MA.
43. Massart, P. 2000. Some applications of concentration inequalities to statistics. Annales de la Faculté des Sciences de Toulouse, IX(2):245-303.
44. Nijholt, A. 1980. Context-Free Grammars: Covers, Normal Forms, and Parsing (volume 93 of Lecture Notes in Computer Science). Springer-Verlag, Berlin.
45. Palmer, N. and P. W. Goldberg. 2007. PAC-learnability of probabilistic deterministic finite state automata in terms of variation distance. In Proceedings of Algorithmic Learning Theory, pages 157-170.
46. Pereira, F. C. N. and Y. Schabes. 1992. Inside-outside reestimation from partially bracketed corpora. In Proceedings of the Association for Computational Linguistics, pages 128-135.
47. Pitt, L. 1989. Inductive inference, DFAs, and computational complexity. Analogical and Inductive Inference, 397:18-44.
48. Pollard, D. 1984. Convergence of Stochastic Processes. Springer-Verlag, New York.
49. Ron, D. 1995. Automata Learning and Its Applications. Ph.D. thesis, Hebrew University of Jerusalem.
50. Ron, D., Y. Singer, and N. Tishby. 1998. On the learnability and usage of acyclic probabilistic finite automata. Journal of Computer and System Sciences, 56(2):133-152.
51. Shalev-Shwartz, S., O. Shamir, K. Sridharan, and N. Srebro. 2009. Learnability and stability in the general learning setting. In Proceedings of the Conference on Learning Theory.
52. Sipser, M. 2006. Introduction to the Theory of Computation, Second Edition. Thomson Course Technology, Boston, MA.
53. Talagrand, M. 1994. Sharper bounds for Gaussian and empirical processes. Annals of Probability, 22:28-76.
54. Terwijn, S. A. 2002. On the learnability of hidden Markov models. In P. Adriaans, H. Fernow, & M. van Zaane. Grammatical Inference: Algorithms and Applications (Lecture Notes in Computer Science). Springer, Berlin, pages 344-348.
55. Tsybakov, A. 2004. Optimal aggregation of classifiers in statistical learning. The Annals of Statistics, 32(1):135-166.
56. Vapnik, V. N. 1998. Statistical Learning Theory. Wiley-Interscience, New York.