Learning activation functions to improve deep neural networks

3rd International Conference on Learning Representations, ICLR 2015 - Workshop Track Proceedings

Volumn , Issue , 2015, Pages

  Agostinelli, Forest ^a   Hoffman, Matthew ^b   Sadowski, Peter ^a   Baldi, Pierre ^a  

^a Irvine   (United States)

^b ADOBE RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BENCHMARKING; BOSONS; CHEMICAL ACTIVATION; GRADIENT METHODS; NETWORK ARCHITECTURE; NEURAL NETWORKS; PIECEWISE LINEAR TECHNIQUES;

ACTIVATION FUNCTIONS; ADAPTIVE ACTIVATION FUNCTION; GRADIENT DESCENT; HIGGS BOSON DECAY; LINEAR UNITS; NONLINEAR ACTIVATION FUNCTIONS; PIECEWISE LINEAR ACTIVATIONS; STATE-OF-THE-ART PERFORMANCE;

DEEP NEURAL NETWORKS;

EID: 85083953189     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (26)

1. Baldi, P, Sadowski, P, and Whiteson, D. Searching for exotic particles in high-energy physics with deep learning. Nature Communications, 5, 2014.
2. − searches with deep learning. Physics Review Letters, 2015. In press.
3. Cho, Youngmin and Saul, Lawrence. Large margin classification in infinite neural networks. Neural Computation, 22(10), 2010.
4. Cowan, Glen, Cranmer, Kyle, Gross, Eilam, and Vitells, Ofer. Asymptotic formulae for likelihood-based tests of new physics. Eur.Phys.J., C71:1554, 2011. doi: 10.1140/epjc/s10052-011-1554-0.
5. Di Lena, P., Nagata, K., and Baldi, P. Deep architectures for protein contact map prediction. Bioinformatics, 28:2449–2457, 2012. doi: 10.1093/bioinformatics/bts475. First published online: July 30, 2012.
6. Glorot, Xavier, Bordes, Antoine, and Bengio, Yoshua. Deep sparse rectifier networks. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics. JMLR W&CP Volume, volume 15, pp. 315–323, 2011.
7. Goodfellow, Ian J, Warde-Farley, David, Mirza, Mehdi, Courville, Aaron, and Bengio, Yoshua. Maxout networks. arXiv preprint arXiv:1302.4389, 2013.
8. Gulcehre, Caglar, Cho, Kyunghyun, Pascanu, Razvan, and Bengio, Yoshua. Learned-norm pooling for deep feedforward and recurrent neural networks. In Machine Learning and Knowledge Discovery in Databases, pp. 530–546. Springer, 2014.
9. Hannun, Awni Y., Case, Carl, Casper, Jared, Catanzaro, Bryan C., Diamos, Greg, Elsen, Erich, Prenger, Ryan, Satheesh, Sanjeev, Sengupta, Shubho, Coates, Adam, and Ng, Andrew Y. Deep speech: Scaling up end-to-end speech recognition. CoRR, abs/1412.5567, 2014. URL http: //arxiv.org/abs/1412.5567.
10. Hinton, Geoffrey E, Srivastava, Nitish, Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan R. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012.
11. Hornik, Kurt, Stinchcombe, Maxwell, and White, Halbert. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359–366, 1989.
12. Jarrett, Kevin, Kavukcuoglu, Koray, Ranzato, M, and LeCun, Yann. What is the best multi-stage architecture for object recognition? In Computer Vision, 2009 IEEE 12th International Conference on, pp. 2146–2153. IEEE, 2009.
13. Jia, Yangqing, Shelhamer, Evan, Donahue, Jeff, Karayev, Sergey, Long, Jonathan, Girshick, Ross, Guadarrama, Sergio, and Darrell, Trevor. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014.
14. Krizhevsky, Alex and Hinton, Geoffrey. Learning multiple layers of features from tiny images. Computer Science Department, University of Toronto, Tech. Rep, 2009.
15. Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097–1105, 2012.
16. Lee, Chen-Yu, Xie, Saining, Gallagher, Patrick, Zhang, Zhengyou, and Tu, Zhuowen. Deeply-supervised nets. arXiv preprint arXiv:1409.5185, 2014.
17. Lin, Min, Chen, Qiang, and Yan, Shuicheng. Network in network. arXiv preprint arXiv:1312.4400, 2013.
18. Lusci, Alessandro, Pollastri, Gianluca, and Baldi, Pierre. Deep architectures and deep learning in chemoinformatics: the prediction of aqueous solubility for drug-like molecules. Journal of chemical information and modeling, 53(7):1563–1575, 2013.
19. Maas, Andrew L, Hannun, Awni Y, and Ng, Andrew Y. Rectifier nonlinearities improve neural network acoustic models. In Proc. ICML, volume 30, 2013.
20. Snoek, Jasper, Larochelle, Hugo, and Adams, Ryan P. Practical bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems, pp. 2951–2959, 2012.
21. Springenberg, Jost Tobias and Riedmiller, Martin. Improving deep neural networks with probabilistic maxout units. arXiv preprint arXiv:1312.6116, 2013.
22. Srivastava, Nitish, Hinton, Geoffrey, Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958, 2014.
23. Stollenga, Marijn F, Masci, Jonathan, Gomez, Faustino, and Schmidhuber, Jürgen. Deep networks with internal selective attention through feedback connections. In Advances in Neural Information Processing Systems, pp. 3545–3553, 2014.
24. Turner, Andrew James and Miller, Julian Francis. Neuroevolution: Evolving heterogeneous artificial neural networks. Evolutionary Intelligence, pp. 1–20, 2014.
25. Wang, Qi and JaJa, Joseph. From maxout to channel-out: Encoding information on sparse pathways. arXiv preprint arXiv:1312.1909, 2013.
26. Yao, Xin. Evolving artificial neural networks. Proceedings of the IEEE, 87(9):1423–1447, 1999.