Iterative scaling and coordinate descent methods for maximum entropy models

Journal of Machine Learning Research

Volumn 11, Issue , 2010, Pages 815-848

  Huang, Fang Lan ^a   Hsieh, Cho Jui ^a   Chang, Kai Wei ^a   Lin, Chih Jen ^a  

^a NATIONAL TAIWAN UNIVERSITY   (Taiwan)

Author keywords

Coordinate descent; Iterative scaling; Maximum entropy; Natural language processing; Optimization

Indexed keywords

CONVERGENCE RESULTS; COORDINATE DESCENT; LINEAR SVM; MAXIMUM ENTROPY; MAXIMUM ENTROPY MODELS; NATURAL LANGUAGE PROCESSING; SCALING METHOD; UNIFIED FRAMEWORK;

COMPUTATIONAL COMPLEXITY; COMPUTATIONAL LINGUISTICS; ECHO SUPPRESSION; ENTROPY; NATURAL LANGUAGE PROCESSING SYSTEMS;

ITERATIVE METHODS;

EID: 77949528697     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (35)

1. Galen Andrew and Jianfeng Gao. Scalable training of L1-regularized log-linear models. In Proceedings of the Twenty Fourth International Conference on Machine Learning (ICML), 2007.
2. Tom M. Apostol. Mathematical Analysis. Addison-Wesley, second edition, 1974.
3. Jason Baldridge, Tom Morton, and Gann Bierner. OpenNLP package, 2001. URL http://opennlp.sourceforge.net/.
4. Adam L. Berger, Vincent J. Della Pietra, and Stephen A. Della Pietra. A maximum entropy approach to natural language processing. Computational Linguistics, 22(1):39-71, 1996.
5. Dimitri P. Bertsekas. Nonlinear Programming. Athena Scientific, Belmont, MA 02178-9998, second edition, 1999.
6. Léon Bottou. Stochastic learning. In Olivier Bousquet and Ulrike von Luxburg, editors, Advanced Lectures on Machine Learning, Lecture Notes in Artificial Intelligence, LNAI 3176, pages 146-168. Springer Verlag, 2004.
7. Kai-Wei Chang, Cho-Jui Hsieh, and Chih-Jen Lin. Coordinate descent method for large-scale L2-loss linear SVM. Journal of Machine Learning Research, 9:1369-1398, 2008. URL http://www.csie.ntu.edu.tw/cjlin/papers/cdl2.pdf.
8. Stanley F. Chen and Ronald Rosenfeld. A survey of smoothing techniques for ME models. IEEE Transactions on Speech and Audio Processing, 8(1):37-50, January 2000.
9. Michael Collins, Robert E. Schapire, and Yoram Singer. Logistic regression, AdaBoost and Bregman distances. Machine Learning, 48(1-3):253-285, 2002.
10. Michael Collins, Amir Globerson, Terry Koo, Xavier Carreras, and Peter Bartlett. Exponentiated gradient algorithms for conditional random fields and max-margin Markov networks. Journal of Machine Learning Research, 9:1775-1822, 2008.
11. John N. Darroch and Douglas Ratcliff. Generalized iterative scaling for log-linear models. The Annals of Mathematical Statistics, 43(5):1470-1480, 1972.
12. Hal Daumé, III. Notes on CG and LM-BFGS optimization of logistic regression. 2004. URL http://www.cs.utah.edu/hal/megam/.
13. Stephen Della Pietra, Vincent Della Pietra, and John Lafferty. Inducing features of random fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(4):380-393, 1997.
14. Miroslav Dudík, Steven J. Phillips, and Robert E. Schapire. Performance guarantees for regularized maximum entropy density estimation. In Proceedings of the 17th Annual Conference on Computational Learning Theory, pages 655-662, New York, 2004. ACM press.
15. Roger Fletcher. Practical Methods of Optimization. John Wiley and Sons, 1987.
16. Jerome Friedman, Trevor Hastie, and Robert Tibshirani. Regularization paths for generalized linear models via coordinate descent. 2008.
17. Jianfeng Gao, Galen Andrew, Mark Johnson, and Kristina Toutanova. A comparative study of parameter estimation methods statistical natural language processing. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics (ACL), pages 824-831, 2007.
18. Alexandar Genkin, David D. Lewis, and David Madigan. Large-scale Bayesian logistic regression for text categorization. Technometrics, 49(3):291-304, 2007.
19. Joshua Goodman. Sequential conditional generalized iterative scaling. In Proceedings of the 40th Annual Meeting of the Association of Computational Linguistics (ACL), pages 9-16, 2002.
20. Luigi Grippo and Marco Sciandrone. Globally convergent block-coordinate techniques for unconstrained optimization. Optimization Methods and Software, 10:587-637, 1999.
21. Fang-Lan Huang, Cho-Jui Hsieh, Kai-Wei Chang, and Chih-Jen Lin. Iterative scaling and coordinate descent methods for maximum entropy. In Proceedings of the 47th Annual Meeting of the Association of Computational Linguistics (ACL), 2009. Short paper.
22. Rong Jin, Rong Yan, Jian Zhang, and Alex G. Hauptmann. A faster iterative scaling algorithm for conditional exponential model. In Proceedings of the Twentieth International Conference on Machine Learning (ICML), 2003.
23. S. Sathiya Keerthi, Kaibo Duan, Shirish Shevade, and Aun Neow Poo. A fast dual algorithm for kernel logistic regression. Machine Learning, 61:151-165, 2005.
24. Kwangmoo Koh, Seung-Jean Kim, and Stephen Boyd. An interior-point method for large-scale l1-regularized logistic regression. Journal of Machine Learning Research, 8:1519-1555, 2007. URL http://www.stanford.edu/boyd/l1-logistic-reg. html.
25. Kenneth Lange, David R. Hunter, and Ilsoon Yang. Optimization transfer using surrogate objective functions. Journal of Computational and Graphical Statistics, 9(1):1-20, March 2000.
26. Chih-Jen Lin, Ruby C. Weng, and S. Sathiya Keerthi. Trust region Newton method for largescale logistic regression. Journal of Machine Learning Research, 9:627-650, 2008. URL http://www.csie.ntu.edu.tw/cjlin/papers/logistic.pdf.
27. Dong C. Liu and Jorge Nocedal. On the limited memory BFGS method for large scale optimization. Mathematical Programming, 45(1):503-528, 1989.
28. Zhi-Quan Luo and Paul Tseng. On the convergence of coordinate descent method for convex differentiable minimization. Journal of Optimization Theory and Applications, 72(1):7-35, 1992.
29. Robert Malouf. A comparison of algorithms for maximum entropy parameter estimation. In Proceedings of the 6th conference on Natural language learning, pages 1-7. Association for Computational Linguistics, 2002.
30. Ryan McDonald and Fernando Pereira. Online learning of approximate dependency parsing algorithms. In Proceedings of 11th Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 81-88, 2006.
31. Thomas P. Minka. A comparison of numerical optimizers for logistic regression, 2003. URL http://research.microsoft.com/minka/papers/logreg/.
32. Adwait Ratnaparkhi. Maximum Entropy Models For Natural Language Ambiguity Resolution. PhD thesis, University of Pennsylvania, 1998.
33. Nicol N. Schraudolph, Jin Yu, and Simon Gunter. A stochastic quasi-Newton method for online convex optimization. In Proceedings of the 11th International Conference Artificial Intelligence and Statistics (AISTATS), pages 433-440, 2007.
34. S.V.N. Vishwanathan, Nicol N. Schraudolph, Mark W. Schmidt, and Kevin Murphy. Accelerated training of conditional random fields with stochastic gradient methods. In Proceedings of the 23rd International Conference on Machine Learning (ICML), pages 969-976, 2006.
35. Zhihua Zhang, James T. Kwok, and Dit-Yan Yeung. Surrogate maximization/minimization algorithms and extensions. Machine Learning, 69(1):1-33, October 2007.