Faster training of very deep networks via p-norm gates

Proceedings - International Conference on Pattern Recognition

Volumn 0, Issue , 2016, Pages 3542-3547

  Pham, Trang ^a   Tran, Truyen ^a   Phung, Dinh ^a   Venkatesh, Svetha ^a  

^a DEAKIN UNIVERSITY   (Australia)

Author keywords

[No Author keywords available]

Indexed keywords

DEEP NEURAL NETWORKS; NETWORK LAYERS; PATTERN RECOGNITION;

CONTRIBUTING FACTOR; CONTROLLABLE FLOW; FEEDFORWARD MODEL; HIGHWAY NETWORKS; LEARNING PROCESS; MACHINE TRANSLATIONS; RECURRENT MODELS; SENSORY INFORMATION;

LONG SHORT-TERM MEMORY;

EID: 85019136679     PISSN: 10514651     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICPR.2016.7900183     Document Type: Conference Paper

References (21)

1. K. He, X. Zhang, S. Ren, and J. Sun, "Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, " in Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 1026-1034.
2. G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.-r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath et al., "Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups, " Signal Processing Magazine, IEEE, vol. 29, no. 6, pp. 82-97, 2012.
3. I. Sutskever, O. Vinyals, and Q. V. Le, "Sequence to sequence learning with neural networks, " in Advances in Neural Information Processing Systems, 2014, pp. 3104-3112.
4. A. Kumar, O. Irsoy, J. Su, J. Bradbury, R. English, B. Pierce, P. Ondruska, I. Gulrajani, and R. Socher, "Ask me anything: Dynamic memory networks for natural language processing, " arXiv preprint arXiv:1506. 07285, 2015.
5. Y. Bengio, "Learning deep architectures for AI, " Foundations and trendsR in Machine Learning, vol. 2, no. 1, pp. 1-127, 2009.
6. A. Choromanska, M. Henaff, M. Mathieu, G. B. Arous, and Y. LeCun, "The loss surfaces of multilayer networks, " in International Conference on Artificial Intelligence and Statistics, 2015, pp. 192-204.
7. H. Larochelle, Y. Bengio, J. Louradour, and P. Lamblin, "Exploring strategies for training deep neural networks, " The Journal of Machine Learning Research, vol. 10, pp. 1-40, 2009.
8. S. Hochreiter, Y. Bengio, P. Frasconi, and J. Schmidhuber, "Gradient flow in recurrent nets: the difficulty of learning long-term dependencies, " A field guide to dynamical recurrent neural networks. IEEE Press, 2001.
9. X. Glorot, A. Bordes, and Y. Bengio, "Deep sparse rectifier networks, " in Proceedings of the 14th International Conference on Artificial Intelligence and Statistics. JMLR W&CP Volume, vol. 15, 2011, pp. 315-323.
10. I. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio, "Maxout networks, " in Proceedings of The 30th International Conference on Machine Learning, 2013, pp. 1319-1327.
11. S. Hochreiter and J. Schmidhuber, "Long short-term memory, " Neural computation, vol. 9, no. 8, pp. 1735-1780, 1997.
12. A. Graves, "Generating sequences with recurrent neural networks, " arXiv preprint arXiv:1308. 0850, 2013.
13. R. K. Srivastava, K. Greff, and J. Schmidhuber, "Training very deep networks, " in Advances in Neural Information Processing Systems, 2015, pp. 2368-2376.
14. K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition, " arXiv preprint arXiv:1512. 03385, 2015.
15. K. Cho, B. van Merriënboer, D. Bahdanau, and Y. Bengio, "On the properties of neural machine translation: Encoder-decoder approaches, " arXiv preprint arXiv:1409. 1259, 2014.
16. K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, "Learning phrase representations using rnn encoder-decoder for statistical machine translation, " in EMNLP, 2014, pp. 1724-1734.
17. J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, "Empirical evaluation of gated recurrent neural networks on sequence modeling, " arXiv preprint arXiv:1412. 3555, 2014.
18. T. Mikolov, I. Sutskever, A. Deoras, H.-S. Le, S. Kombrink, and J. Cernocky, "Subword language modeling with neural networks, " preprint (http://www. fit. vutbr. cz/imikolov/rnnlm/char. pdf), 2012.
19. Y. Kim, Y. Jernite, D. Sontag, and A. M. Rush, "Character-aware neural language models, " arXiv preprint arXiv:1508. 06615, 2015.
20. T. Pham, T. Tran, D. Phung, and S. Venkatesh, "Deepcare: A deep dynamic memory model for predictive medicine, " arXiv preprint arXiv:1602. 00357, 2016.
21. D. Krueger and R. Memisevic, "Regularizing RNNs by Stabilizing Activations, " arXiv preprint arXiv:1511. 08400, 2015.