Periodic step-size adaptation in second-order gradient descent for single-pass on-line structured learning

Machine Learning

Volumn 77, Issue 2-3, 2009, Pages 195-224

  Hsu, Chun Nan ^a   Huang, Han Shen ^a   Chang, Yu Ming ^a   Lee, Yuh Jye ^b  

^a INSTITUTE OF INFORMATION SCIENCE   (Taiwan)

^b NATIONAL TAIWAN UNIVERSITY OF SCIENCE AND TECHNOLOGY   (Taiwan)

Author keywords

Conditional random fields; Convolutional neural networks; On line learning; Sequence labeling; Stochastic gradient descent

Indexed keywords

CLASSIFICATION TASKS; CONDITIONAL RANDOM FIELD; CONVOLUTIONAL NEURAL NETWORK; GENERALIZATION PERFORMANCE; GRADIENT DESCENT; HESSIAN MATRICES; HIGH-DIMENSIONAL FEATURE SPACE; JACOBIANS; LARGE-SCALE SEQUENCES; LINEAR RELATION; LINEAR SUPPORT VECTOR MACHINES; LOSS FUNCTIONS; MAPPING FUNCTIONS; ON-LINE SETTING; ONLINE LEARNING; SECOND ORDERS; SINGLE PASS; STOCHASTIC GRADIENT DESCENT; STRUCTURED LEARNING; STRUCTURED PREDICTION; TRAINING EXAMPLE;

COMMUNICATION CHANNELS (INFORMATION THEORY); CONTENT BASED RETRIEVAL; CONVOLUTION; INVERSE PROBLEMS; LABELING; PEELING; STOCHASTIC SYSTEMS; TIME VARYING NETWORKS;

NEURAL NETWORKS;

EID: 72449129112     PISSN: 08856125     EISSN: 15730565     Source Type: Journal    
DOI: 10.1007/s10994-009-5142-6     Document Type: Article

References (52)

1. S.-I. Amari 1998 Natural gradient works efficiently in learning Neural Computation 10 251 276 10.1162/089976698300017746
2. Becker, S., & LeCun, Y. (1988). Improving the convergence of back-propagation learning with second-order methods. In Proceedings of the 1988 connectionist models summer school (pp. 29-37). San Mateo: Morgan Kaufman.
3. Benveniste, A., Metivier, M., & Priouret, P. (1990). Adaptive algorithms and stochastic approximations. Berlin: Springer.
4. Bordes, A., Bottou, L., Gallinari, P., & Weston, J. (2007). Solving multiclass support vector machines with LaRank. In Proceedings of the 24th international conference on machine learning (ICML'07) (pp. 89-96). New York: ACM. (Pubitemid 47275053)
5. Bottou, L. (2007). The tradeoffs of large-scale learning. Tutorial, the 21st annual conference on neural information processing systems (NIPS 2007), Vancouver, BC, Canada. http://leon.bottou.org/talks/largescale.
6. Bottou, L., & Bousquet, O. (2008). The tradeoffs of large scale learning. In Advances in neural information processing systems, 20 (NIPS 2007). Cambridge: MIT Press.
7. Bottou, L., & LeCun, Y. (2004). Large scale online learning. In S. Thrun, L. Saul, & B. Schölkopf (Eds.), Advances in neural information processing systems 16 (NIPS 2003). Cambridge: MIT Press.
8. L. Bottou Y. LeCun 2005 On-line learning for very large data sets Applied Stochastic Models in Business and Industry 21 2 137 151 1091.68063 10.1002/asmb.538 2137546 (Pubitemid 40541359)
9. Bottou, L., Chapelle, O., DeCoste, D., & Weston, J. (Eds.) (2007). Large scale kernel machines. Cambridge: MIT Press.
10. Burden, R. L., & Faires, D. (1988). Numerical analysis. PWS-KENT Pub. Co.
11. Chang, C.-C., & Lin, C.-J. (2001). LIBSVM: a library for support vector machines. Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm.
12. Chen, S. F., & Rosenfeld, R. (1999). A Gaussian prior for smoothing maximum entropy models. Technical Report CMU-CS-99-108, School of Computer Science, Carnegie Mellon University, PA, USA, Pittsburgh.
13. M. Collins A. Globerson T. Koo X. Carreras P. L. Bartlett 2008 Exponentiated gradient algorithms for conditional random fields and max-margin Markov networks Journal of Machine Learning Research 9 1775 1822 2438824
14. A. Dempster N. Laird D. Rubin 1977 Maximum likelihood from incomplete data via the EM algorithm Journal of the Royal Statistical Society, Series B 39 1 1 38 0364.62022 501537
15. Fahlmann, S. E. (1988). An empirical study of learning speed in back-propagation networks. Technical Report CMU-CS-88-162, Carnegie Mellon University.
16. C. Fraley 1999 On computing the largest fraction of missing information for the EM algorithm and the worst linear function for data augmentation Computational Statistics and Data Analysis 31 1 13 26 1063.62507 10.1016/S0167-9473(99)00003-1 1705719 (Pubitemid 29520103)
17. Hesterberg, T. (2005). Staggered Aitken acceleration for EM. In Proceedings of the statistical computing section of the American statistical association, Minneapolis, Minnesota, USA.
18. Hsu, C.-N., Huang, H.-S., & Yang, B.-H. (2006). Global and componentwise extrapolation for accelerating data mining from large incomplete data sets with the EM algorithm. In Proceedings of the sixth IEEE international conference on data mining (ICDM'06) (pp. 265-274), Hong Kong, China.
19. C.-N. Hsu Y.-M. Chang C.-J. Kuo Y.-S. Lin H.-S. Huang I.-F. Chuang 2008 Integrating high dimensional bi-directional parsing models for gene mention tagging Bioinformatics 24 13 i286 i294 10.1093/bioinformatics/btn183 Proceedings of ISMB-2008 (Pubitemid 351911684)
20. Huang, H.-S., Yang, B.-H., & Hsu, C.-N. (2005). Triple-jump acceleration for the EM algorithm. In Proceedings of the fifth IEEE international conference on data mining (ICDM'05) (pp. 649-652).
21. H.-S. Huang B.-H. Yang Y.-M. Chang C.-N. Hsu 2009 Global and componentwise extrapolations for accelerating training of Bayesian networks and conditional random fields Data Mining and Knowledge Discovery 19 1 58 91 10.1007/s10618-009-0128-3
22. Joachims, T. (1998). Making large-scale svm learning practical. In B. Schölkoph, C. J. C. Burges, & A. J. Smola (Eds.), Advances in kernel methods: support vector learning (Chap. 11, pp. 169-184). Cambridge: MIT Press.
23. Joachims, T. (2006). Training linear SVMs in linear time. In Proceedings of the 12th ACM SIGKDD international conference on knowledge discovery and data mining (KDD'06) (pp. 217-226). New York: ACM.
24. Kudo, T. (2006). CRF++: Yet another CRF toolkit. Available under LGPL from the following URL: http://crfpp.sourceforge.net/.
25. Lafferty, J., McCallum, A., & Pereira, F. (2001). Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of 18th international conference on machine learning (ICML'03) (pp. 282-289).
26. LeCun, Y., & Cortes, C. (1998). The MNIST database of handwritten digits. http://yann.lecun.com/exdb/mnist/.
27. LeCun, Y., Bottou, L., Orr, G. B., & Muller, K.-R. (1998a). Efficient backprop. In G. Orr & K. Muller (Eds.), Neural networks: tricks of the trade. Berlin: Springer.
28. Y. LeCun L. Bottou Y. Bengio P. Haffner 1998 Gradient-based learning applied to document recognition Proceedings of the IEEE 86 11 2278 2324 10.1109/5.726791
29. LeCun, Y., Haffner, P., Bottou, L., & Bengio, Y. (1999). Object recognition with gradient-based learning. In Shape, contour and grouping in computer vision (p. 319). London: Springer.
30. LeRoux, N., Manzagol, P.-A., & Bengio, Y. (2008). Topmoumoute online natural gradient algorithm. In Advances in neural information processing systems, 20 (NIPS 2007). Cambridge: MIT Press.
31. Malouf, R. (2002). A comparison of algorithms for maximum entropy parameter estimation. In Proceedings of the sixth conference on natural language learning (CoNLL-2002) (pp. 49-55).
32. McLachlan, G. J., & Krishnan, T. (1997). The EM algorithm and extensions. Wiley series in probability and statistics. New York: Wiley-Interscience.
33. X.-L. Meng D. B. Rubin 1994 On the global and componentwise rates of convergence of the EM algorithm Linear Algebra and Its Applications 199 413 425 0818.65153 10.1016/0024-3795(94)90363-8 1274429
34. Murata, N. (1998). A statistical study of on-line learning. In On-line learning in neural networks (pp. 63-92). Cambridge: Cambridge University Press.
35. N. Murata S.-I. Amari 1999 Statistical analysis of learning dynamics Signal Processing 74 1 3 28 0922.68094 10.1016/S0165-1684(98)00206-0
36. Nocedal, J., & Wright, S. J. (1999). Numerical optimization. Berlin: Springer.
37. Ortega, J. M., & Rheinboldt, W. C. (1970). Iterative solution of nonlinear equations in several variables. San Diego: Academic Press.
38. Salakhutdinov, R., & Roweis, S. (2003). Adaptive overrelaxed bound optimization methods. In Proceedings of the twentieth international conference on machine learning (ICML'03) (pp. 664-671).
39. Schafer, J. L. (1997). Analysis of incomplete multivariate data. New York: Chapman and Hall.
40. Schraudolph, N. N., & Graepel, T. (2002). Conjugate directions for stochastic gradient descent. In Proceedings of international conference on artificial neural networks (ICANN'02) (pp. 1351-1358).
41. Schraudolph, N. N., Yu, J., & Günter, S. (2007). A stochastic quasi-newton method for online convex optimization. In Proceedings of the 11th international conference on artificial intelligence and statistics (AISTATS 2007), San Juan, Puerto Rico.
42. Settles, B. (2004). Biomedical named entity recognition using conditional random fields and novel feature sets. In Proceedings of the joint workshop on natural language processing in biomedicine and its applications (JNLPBA-2004) (pp. 104-107).
43. Sha, F., & Pereira, F. (2003). Shallow parsing with conditional random fields. In Proceedings of human language technology, the North American chapter of the association for computational linguistics (NAACL'03) (pp. 213-220).
44. Shalev-Shwartz, S., Singer, Y., & Srebro, N. (2007). Pegasos: Primal Estimated sub-GrAdient SOlver for SVM. In ICML'07: Proceedings of the 24th international conference on machine learning (pp. 807-814). New York: ACM. (Pubitemid 47275141)
45. Spall, J. C. (2003). Introduction to stochastic search and optimization: estimation, simulation and control. New York: Wiley-Interscience.
46. Taskar, B. (2005). Learning structured prediction models: a large margin approach. Ph.D. thesis, Stanford University, Stanford, CA, USA. Adviser-Daphne Koller.
47. Taskar, B., Chatalbashev, V., Koller, D., & Guestrin, C. (2005). Learning structured prediction models: a large margin approach. In Proceedings of the 22nd international conference on machine learning (ICML'05) (pp. 896-903). New York: ACM.
48. Traub, J. (1964). Iterative methods for the solution of equations. Englewood Cliffs: Prentice Hall.
49. I. Tsochantaridis T. Joachims T. Hofmann Y. Altun 2005 Large margin methods for structured and interdependent output variables Journal of Machine Learning Research 6 1453 1484 2249862
50. Vishwanathan, S., Schraudolph, N. N., Schmidt, M. W., & Murphy, K. P. (2006a). Accelerated training of conditional random fields with stochastic gradient methods. In Proceedings of the 23rd international conference on machine learning (ICML'06), Pittsburgh, PA, USA.
51. S. Vishwanathan N. N. Schraudolph A. J. Smola 2006 Step size adaptation in reproducing kernel Hilbert space Journal of Machine Learning Research 7 1107 1133 2274399
52. Widrow, B., & Hoff, M. E. (1960). Adaptive switching circuits. In 1960 IRE WESCON convention record (pp. 96-104), New York.