Improved dropout for shallow and deep learning

Advances in Neural Information Processing Systems

Volumn , Issue , 2016, Pages 2531-2539

  Li, Zhe ^a   Gong, Boqing ^b   Yang, Tianbao ^a  

^a UNIVERSITY OF IOWA   (United States)

^b UNIVERSITY OF CENTRAL FLORIDA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DEEP NEURAL NETWORKS; NEURAL NETWORKS; NEURONS; OPTIMIZATION; PROBABILITY; PROBABILITY DISTRIBUTIONS; SAMPLING;

BENCHMARK DATASETS; DATA DISTRIBUTION; FASTER CONVERGENCE; LOGISTIC REGRESSIONS; MULTINOMIAL DISTRIBUTIONS; PREDICTION PERFORMANCE; SECOND ORDER STATISTICS; STOCHASTIC OPTIMIZATIONS;

DEEP LEARNING;

EID: 85018878306     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (23)

1. Jimmy Ba and Brendan Frey. Adaptive dropout for training deep neural networks. In Advances in Neural Information Processing Systems, pages 3084-3092, 2013.
2. Pierre Baldi and Peter J Sadowski. Understanding dropout. In Advances in Neural Information Processing Systems, pages 2814-2822, 2013.
3. Benjamin Graham, Jeremy Reizenstein, and Leigh Robinson. Efficient batchwise dropout training using submatrices. CoRR, abs/1502.02478, 2015.
4. Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012.
5. Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
6. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.
7. Diederik P. Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameterization trick. CoRR, abs/1506.02557, 2015.
8. Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images, 2009.
9. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097-1105, 2012.
10. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.
11. Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, page 4. Granada, Spain, 2011.
12. Behnam Neyshabur, Ruslan R Salakhutdinov, and Nati Srebro. Path-sgd: Path-normalized optimization in deep neural networks. In Advances in Neural Information Processing Systems, pages 2413-2421, 2015.
13. Marc'Aurelio Ranzato, Alex Krizhevsky, and Geoffrey E. Hinton. Factored 3-way restricted boltzmann machines for modeling natural images. In AISTATS, pages 621-628, 2010.
14. Shai Shalev-Shwartz, Ohad Shamir, Nathan Srebro, and Karthik Sridharan. Stochastic convex optimization. In The 22nd Conference on Learning Theory (COLT), 2009.
15. Nathan Srebro, Karthik Sridharan, and Ambuj Tewari. Smoothness, low noise and fast rates. In Advances in Neural Information Processing Systems 23 (NIPS), pages 2199-2207, 2010.
16. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929-1958, 2014.
17. Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In Proceedings of the 30th international conference on machine learning (ICML-13), pages 1139-1147, 2013.
18. Stefan Wager, Sida Wang, and Percy S Liang. Dropout training as adaptive regularization. In Advances in Neural Information Processing Systems, pages 351-359, 2013.
19. Li Wan, Matthew Zeiler, Sixin Zhang, Yann L Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In Proceedings of the 30th International Conference on Machine Learning (ICML-13), pages 1058-1066, 2013.
20. Sida Wang and Christopher Manning. Fast dropout training. In Proceedings of the 30th International Conference on Machine Learning (ICML-13), pages 118-126, 2013.
21. Sida I Wang, Mengqiu Wang, Stefan Wager, Percy Liang, and Christopher D Manning. Feature noising for log-linear structured prediction. In EMNLP, pages 1170-1179, 2013.
22. Sixin Zhang, Anna Choromanska, and Yann LeCun. Deep learning with elastic averaging sgd. arXiv preprint arXiv:1412.6651, 2014.
23. Jingwei Zhuo, Jun Zhu, and Bo Zhang. Adaptive dropout rates for learning with corrupted features. In IJCAI, pages 4126-4133, 2015.