Distributed optimization of deeply nested systems

Journal of Machine Learning Research

Volumn 33, Issue , 2014, Pages 10-19

  Carreira Perpiñán, Miguel Á ^a   Wang, Weiran ^a  

^a Merced   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; CONSTRAINED OPTIMIZATION; CONSTRAINT THEORY; NONLINEAR PROGRAMMING; OBJECT RECOGNITION;

ALTERNATING OPTIMIZATIONS; CONSTRAINED PROBLEM; DISTRIBUTED OPTIMIZATION; INTELLIGENT PROCESSING; NONCONVEX OPTIMIZATION; NONLINEAR OPTIMIZERS; NONLINEAR PROCESSING; PARAMETER DERIVATIVES;

PARAMETER ESTIMATION;

EID: 84955486808     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Conference Paper

References (35)

1. Y. Bengio and Y. LeCun. Scaling learning algorithms toward AI. In L. Bottou, O. Chapelle, D. DeCoste, and J. Weston, editors, Large-Scale Kernel Machines, Neural Information Processing Series, pages 321-360. MIT Press, 2007.
2. Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle. Greedy layer-wise training of deep networks. In B. Schölkopf, J. Platt, and T. Hofmann, editors, Advances in Neural Information Processing Systems (NIPS), Volume 19, pages 153-160. MIT Press, Cambridge, MA, 2007.
3. J. Bergstra and Y. Bengio. Random search for hyperparameter optimization. J. Machine Learning Research, 13:281-305, 2012.
4. C. M. Bishop. Pattern Recognition and Machine Learning. Springer Series in Information Science and Statistics. Springer-Verlag, Berlin, 2006.
5. S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning, 3(1):1-122, 2011.
6. M. Á. Carreira-Perpiñán and Z. Lu. Dimensionality reduction by unsupervised regression. In Proc. of the 2008 IEEE Computer Society Conf. Computer Vision and Pattern Recognition (CVPR'08), Anchorage, AK, June 23-28 2008.
7. M. Á. Carreira-Perpiñán and Z. Lu. Parametric dimensionality reduction by unsupervised regression. In Proc. of the 2010 IEEE Computer Society Conf. Computer Vision and Pattern Recognition (CVPR'10), pages 1895-1902, San Francisco, CA, June 13-18 2010.
8. M. Á. Carreira-Perpiñán and Z. Lu. Manifold learning and missing data recovery through unsupervised regression. In Proc. of the 12th IEEE Int. Conf. Data Mining (ICDM 2011), pages 1014-1019, Vancouver, BC, Dec. 11-14 2011.
9. M. Á. Carreira-Perpiñán and W. Wang. Distributed optimization of deeply nested systems. Unpublished manuscript, arXiv:1212.5921, Dec. 24 2012.
10. E. Castillo, B. Guijarro-Berdiñas, O. Fontenla-Romero, and A. Alonso-Betanzos. A very fast learning method for neural networks based on sensitivity analysis. J. Machine Learning Research, 7:1159-1182, July 2006.
11. L. Deng, D. Yu, and J. Platt. Scalable stacking and learning for building deep architectures. In Proc. of the IEEE Int. Conf. Acoustics, Speech and Sig. Proc. (ICASSP'12), pages 2133-2136, Kyoto, Japan, Mar. 25-30 2012.
12. D. Erhan, P. A. Manzagol, Y. Bengio, S. Bengio, and P. Vincent. The difficulty of training deep architectures and the effect of unsupervised pre-training. In Proc. of the 12th Int. Workshop on Artificial Intelligence and Statistics (AISTATS 2009), pages 153-160, Clearwater Beach, FL, Mar. 21-24 2009.
13. A. Gifi. Nonlinear Multivariate Analysis. John Wiley & Sons, 1990.
14. T. Grossman, R. Meir, and E. Domany. Learning by choice of internal representations. Complex Systems, 2(5):555-575, 1988.
15. T. J. Hastie, R. J. Tibshirani, and J. H. Friedman. The Elements of Statistical Learning - Data Mining, Inference and Prediction. Springer Series in Statistics. Springer-Verlag, second edition, 2009.
16. G. E. Hinton and R. R. Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313 (5786):504-507, July 28 2006.
17. K. Kavukcuoglu, M. Ranzato, and Y. LeCun. Fast inference in sparse coding algorithms with applications to object recognition. Technical Report CBLL-TR-2008-12-01, Dept. of Computer Science, New York University, Dec. 4 2008.
18. A. Krogh, C. J. Thorbergsson, and J. A. Hertz. A cost function for internal representations. In D. S. Touretzky, editor, Advances in Neural Information Processing Systems (NIPS), Volume 2, pages 733-740. Morgan Kaufmann, San Mateo, CA, 1990.
19. N. Lawrence. Probabilistic non-linear principal component analysis with Gaussian process latent variable models. J. Machine Learning Research, 6:1783-1816, Nov. 2005.
20. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proc. IEEE, 86(11):2278-2324, Nov. 1998.
21. S. Ma, C. Ji, and J. Farmer. An efficient EM-based training algorithm for feedforward neural networks. Neural Networks, 10(2):243-256, Mar. 1997.
22. S. A. Nene, S. K. Nayar, and H. Murase. Columbia object image library (COIL-20). Technical Report CUCS-005-96, Dept. of Computer Science, Columbia University, Feb. 1996.
23. J. Nocedal and S. J. Wright. Numerical Optimization. Springer Series in Operations Research and Financial Engineering. Springer-Verlag, New York, second edition, 2006.
24. B. A. Olshausen and D. J. Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381(6583):607-609, June 13 1996.
25. G. B. Orr and K.-R. Müller, editors. Neural Networks: Tricks of the Trade, Volume 1524 of Lecture Notes in Computer Science. Springer-Verlag, Berlin, 1998.
26. M. Ranzato, C. Poultney, S. Chopra, and Y. LeCun. Efficient learning of sparse representations with an energy-based model. In B. Schölkopf, J. Platt, and T. Hofmann, editors, Advances in Neural Information Processing Systems (NIPS), Volume 19, pages 1137-1144. MIT Press, Cambridge, MA, 2007.
27. R. T. Rockafellar. Monotone operators and the proximal point algorithm. SIAM J. Control and Optim., 14(5): 877-898, 1976.
28. T. Rögnvaldsson. On Langevin updating in multilayer perceptrons. Neural Computation, 6(5):916-926, Sept. 1994.
29. R. Rohwer. The 'moving targets' training algorithm. In D. S. Touretzky, editor, Advances in Neural Information Processing Systems (NIPS), Volume 2, pages 558-565. Morgan Kaufmann, San Mateo, CA, 1990.
30. D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning representations by back-propagating errors. Nature, 323:533-536, 1986.
31. D. Saad and E. Marom. Learning by choice of internal representations: An energy minimization approach. Complex Systems, 4(1):107-118, 1990.
32. G. Saon and J.-T. Chien. Large-vocabulary continuous speech recognition systems: A look at some recent advances. IEEE Signal Processing Magazine, 29(6):18-33, Nov. 2012.
33. T. Serre, L. Wolf, S. Bileschi, M. Riesenhuber, and T. Poggio. Robust object recognition with cortex-like mechanisms. IEEE Trans. Pattern Analysis and Machine Intelligence, 29(3):411-426, Mar. 2007.
34. S. Tan and M. L. Mavrovouniotis. Reducing data dimensionality through optimizing neural network inputs. AIChE Journal, 41(6):1471-1479, June 1995.
35. W. Wang and M. Á. Carreira-Perpiñán. Nonlinear low-dimensional regression using auxiliary coordinates. In N. Lawrence and M. Girolami, editors, Proc. of the 15th Int. Workshop on Artificial Intelligence and Statistics (AISTATS 2012), pages 1295-1304, La Palma, Canary Islands, Spain, Apr. 21-23 2012.