The dropout learning algorithm

Artificial Intelligence

Volumn 210, Issue 1, 2014, Pages 78-122

  Baldi, Pierre ^a   Sadowski, Peter ^a  

^a Irvine   (United States)

Author keywords

Backpropagation; Ensemble; Geometric mean; Machine learning; Neural networks; Regularization; Sparse representations; Stochastic gradient descent; Stochastic neurons; Variance minimization

Indexed keywords

BACKPROPAGATION; COMPUTATIONAL FLUID DYNAMICS; LEARNING ALGORITHMS; LEARNING SYSTEMS; NEURAL NETWORKS; TIME VARYING NETWORKS;

ENSEMBLE; GEOMETRIC MEAN; REGULARIZATION; SPARSE REPRESENTATION; STOCHASTIC GRADIENT DESCENT; STOCHASTIC NEURONS; VARIANCE MINIMIZATION;

LINEAR NETWORKS;

EID: 84896507538     PISSN: 00043702     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.artint.2014.02.004     Document Type: Article

References (43)

1. J. Aldaz Self improvement of the inequality between arithmetic and geometric means J. Math. Inequal. 3 2 2009 213 216
2. J. Aldaz Sharp bounds for the difference between the arithmetic and geometric means arXiv preprint arXiv:1203.4454 2012
3. N. Alon, and J.H. Spencer The Probabilistic Method 2004 John Wiley & Sons
4. H. Alzer A new refinement of the arithmetic mean geometric mean inequality J. Math. 27 3 1997
5. H. Alzer Some inequalities for arithmetic and geometric means Proc. R. Soc. Edinb., Sect. A, Math. 129 02 1999 221 228
6. G. An The effects of adding noise during backpropagation training on a generalization performance Neural Comput. 8 3 1996 643 674
7. J. Ba, and B. Frey Adaptive dropout for training deep neural networks C. Burges, L. Bottou, M. Welling, Z. Ghahramani, K. Weinberger, Advances in Neural Information Processing Systems vol. 26 2013 3084 3092
8. P. Baldi, and K. Hornik Neural networks and principal component analysis: Learning from examples without local minima Neural Netw. 2 1 1988 53 58
9. P. Baldi, and K. Hornik Learning in linear networks: a survey IEEE Trans. Neural Netw. 6 4 1994 837 858
10. P. Baldi, and P.J. Sadowski Understanding dropout Advances in Neural Information Processing Systems vol. 26 2013 2814 2822
11. E.F. Beckenbach, and R. Bellman Inequalities 1965 Springer-Verlag Berlin
12. C.M. Bishop Training with noise is equivalent to Tikhonov regularization Neural Comput. 7 1 1995 108 116
13. L. Bottou Online algorithms and stochastic approximations D. Saad, Online Learning and Neural Networks 1998 Cambridge University Press Cambridge, UK
14. L. Bottou Stochastic learning O. Bousquet, U. von Luxburg, Advanced Lectures on Machine Learning Lecture Notes in Artificial Intelligence vol. 3176 2004 Springer Verlag Berlin 146 168
15. S.R. Bowling, M.T. Khasawneh, S. Kaewkuekool, and B.R. Cho A logistic approximation to the cumulative normal distribution J. Ind. Eng. Manag. 2 1 2009 114 127
16. S. Boyd, and L. Vandenberghe Convex Optimization 2004 Cambridge University Press
17. L. Breiman Bagging predictors Mach. Learn. 24 2 1996 123 140
18. C. Carr, and M. Konishi A circuit for detection of interaural time differences in the brain stem of the barn owl J. Neurosci. 10 10 1990 3227 3246
19. C.E. Carr, and M. Konishi Axonal delay lines for time measurement in the owl's brainstem Proc. Natl. Acad. Sci. 85 21 1988 8311 8315
20. D. Cartwright, and M. Field A refinement of the arithmetic mean-geometric mean inequality Proc. Am. Math. Soc. 1978 36 38
21. D.D.R. Cox The Analysis of Binary Data, vol. 32 1989 CRC Press
22. P. Diaconis Bayesian numerical analysis Statistical Decision Theory and Related Topics IV vol. 1 1988 163 175
23. R.O. Duda, P.E. Hart, and D.G. Stork Pattern Classification second ed. 2000 Wiley New York, NY
24. D. Gardner Noise modulation of synaptic weights in a biological neural network Neural Netw. 2 1 1989 69 76
25. S.J. Hanson A stochastic version of the delta rule Physica D 42 1 1990 265 272
26. G. Harnischfeger, G. Neuweiler, and P. Schlegel Interaural time and intensity coding in superior olivary complex and inferior colliculus of the echolocating bat molossus ater J. Neurophysiol. 53 1 1985 89 109
27. G. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R.R. Salakhutdinov Improving neural networks by preventing co-adaptation of feature detectors http://arxiv.org/abs/1207.0580 2012
28. N. Levinson Generalization of an inequality of Ky Fan J. Math. Anal. Appl. 8 1 1964 133 134
29. L. Maaten, M. Chen, S. Tyree, and K.Q. Weinberger Learning with marginalized corrupted features Proceedings of the 30th International Conference on Machine Learning (ICML-13) 2013 410 418
30. K. Matsuoka Noise injection into inputs in back-propagation learning IEEE Trans. Syst. Man Cybern. 22 3 1992 436 440
31. A.M. Mercer Improved upper and lower bounds for the difference an-gn J. Math. 31 2 2001
32. P.R. Mercer Refined arithmetic, geometric and harmonic mean inequalities J. Math. 33 4 2003
33. M. Mitzenmacher, and E. Upfal Probability and Computing: Randomized Algorithms and Probabilistic Analysis 2005 Cambridge University Press
34. A.F. Murray, and P.J. Edwards Enhanced mlp performance and fault tolerance resulting from synaptic weight noise during training IEEE Trans. Neural Netw. 5 5 1994 792 802
35. E. Neuman, and J. Sándor On the Ky Fan inequality and related inequalities i Math. Inequal. Appl 5 2002 49 56
36. E. Neuman, and J. Sandor On the Ky Fan inequality and related inequalities ii Bull. Aust. Math. Soc. 72 1 2005 87 108
37. Y. Raviv, and N. Intrator Bootstrapping with noise: An effective regularization technique Connect. Sci. 8 3-4 1996 355 372
38. H. Robbins, and D. Siegmund A convergence theorem for non negative almost supermartingales and some applications Optimizing Methods in Statistics 1971 233 257
39. R.T. Rockafellar Convex Analysis, vol. 28 1997 Princeton University Press
40. D. Rumelhart, G. Hintont, and R. Williams Learning representations by back-propagating errors Nature 323 6088 1986 533 536
41. D.J. Spiegelhalter, and S.L. Lauritzen Sequential updating of conditional probabilities on directed graphical structures Networks 20 5 1990 579 605
42. P. Vincent, H. Larochelle, Y. Bengio, and P. Manzagol Extracting and composing robust features with denoising autoencoders Proceedings of the 25th International Conference on Machine Learning 2008 ACM 1096 1103
43. S. Wager, S. Wang, and P. Liang Dropout training as adaptive regularization C. Burges, L. Bottou, M. Welling, Z. Ghahramani, K. Weinberger, Advances in Neural Information Processing Systems vol. 26 2013 351 359