Noise-enhanced convolutional neural networks

Neural Networks

Volumn 78, Issue , 2016, Pages 15-23

  Audhkhasi, Kartik ^a   Osoba, Osonde ^a   Kosko, Bart ^a  

^a University of Southern California   (United States)

Author keywords

Backpropagation; Convolutional neural network; Expectation maximization algorithm; Noise injection; Sampling from big data sets; Stochastic resonance

Indexed keywords

BACKPROPAGATION; BACKPROPAGATION ALGORITHMS; BIG DATA; CLASSIFICATION (OF INFORMATION); COMPLEX NETWORKS; CONVOLUTION; GEOMETRY; IMAGE SEGMENTATION; ITERATIVE METHODS; MAGNETIC RESONANCE; MAXIMUM PRINCIPLE; NEURAL NETWORKS; STOCHASTIC SYSTEMS;

BACKPROPAGATION TRAINING; CLASSIFICATION ERRORS; CONVOLUTIONAL NEURAL NETWORK; EXPECTATION-MAXIMIZATION ALGORITHMS; GENERALIZED EXPECTATION MAXIMIZATION; LINEAR TIME COMPLEXITY; NOISE INJECTION; STOCHASTIC RESONANCES;

ALGORITHMS;

ACCURACY; ALGORITHM; ARTICLE; CONVOLUTIONAL NEURAL NETWORK; ENTROPY; HUMAN; KERNEL METHOD; NERVE CELL NETWORK; NOISE; PRIORITY JOURNAL; RANDOM SAMPLE; SIMULATION; SUPPORT VECTOR MACHINE; ARTIFICIAL NEURAL NETWORK; LEARNING; NORMAL DISTRIBUTION; PHYSIOLOGY; RANDOMIZATION; STATISTICAL MODEL;

ALGORITHMS; LEARNING; LIKELIHOOD FUNCTIONS; NEURAL NETWORKS (COMPUTER); NORMAL DISTRIBUTION; RANDOM ALLOCATION;

EID: 84949871128     PISSN: 08936080     EISSN: 18792782     Source Type: Journal    
DOI: 10.1016/j.neunet.2015.09.014     Document Type: Article

References (42)

1. Audhkhasi K., Osoba O., Kosko B. Noise benefits in backpropagation and deep bidirectional pre-training. The international joint conference on neural networks, IJCNN 2013, 1-8. IEEE.
2. Audhkhasi, K., Osoba, O., & Kosko, B. (2013b). Noisy hidden Markov models for speech recognition. In Proc. IJCNN (pp. 1-8).
3. Audhkhasi, K., Osoba, O., & Kosko, B. (2016). Noise can speed backpropagation learning and deep bidirectional pre-training (in review).
4. Bishop C.M. Pattern recognition and machine learning. Vol. 1 2006, Springer.
5. Bulsara A.R., Gammaitoni L. Tuning in to noise. Physics Today 1996, 39-45.
6. Cireşan D.C., Meier U., Masci J., Gambardella L.M., Schmidhuber J. Flexible, high performance convolutional neural networks for image classification. Proceedings of the twenty-second international joint conference on artificial intelligence. Vol. 2 2011, 1237-1242. AAAI Press.
7. Cover Thomas M., Thomas Joy A. Elements of information theory 2012, Wiley-Interscience.
8. Dempster Arthur P., Laird Nan M., Rubin Donald B. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society. Series B. Statistical Methodology 1977, 1-38.
9. Franzke B., Kosko B. Noise can speed convergence in Markov chains. Physical Review E 2011, 84(4).
10. Franzke Brandon, Kosko Bart Using noise to speed up Markov chain Monte Carlo estimation. Procedia Computer Science 2015, 53:113-120.
11. Fukushima K. Neocognitron: A hierarchical neural network capable of visual pattern recognition. Neural Networks 1988, 1(2):119-130.
12. Gammaitoni L. Stochastic resonance in multi-threshold systems. Physics Letters A 1995, 208:315-322.
13. Hubel D.H., Wiesel T.N. Receptive fields of single neurones in the cat's striate cortex. The Journal of Physiology 1959, 148(3):574-591.
14. Jain A.K. Data clustering: 50 years beyond K-means. Pattern Recognition Letters 2010, 31(8):651-666.
15. Kosko B. Noise 2006, Viking.
16. Krizhevsky, A., Sutskever, I., & Hinton, G.E. (2012). Imagenet classification with deep convolutional neural networks. In NIPS, Vol. 1 (p. 4).
17. Kullback Solomon, Leibler Richard A. On information and sufficiency. The Annals of Mathematical Statistics 1951, 22(1):79-86.
18. Kung S.Y. Kernel methods and machine learning 2014, Cambridge University Press.
19. Lawrence S., Giles C.L., Tsoi A.C., Back A.D. Face recognition: A convolutional neural-network approach. IEEE Transactions on Neural Networks 1997, 8(1):98-113.
20. Le, Q., Sarlós, T., & Smola, A. (2013). Fastfood-approximating kernel expansions in loglinear time. In Proc. ICML.
21. LeCun Y., Bengio Y., Hinton G. Deep learning. Nature 2015, 521:436-444.
22. LeCun, Y., Boser, B., Denker, J.S., Henderson, D., Howard, R.E., & Hubbard, W. et al. (1990). Handwritten digit recognition with a back-propagation network. In Proc. NIPS.
23. LeCun Y., Bottou L., Bengio Y., Haffner P. Gradient-based learning applied to document recognition. Proceedings of the IEEE 1998, 86(11):2278-2324.
24. Matsugu M., Mori K., Mitari Y., Kaneda Y. Subject independent facial expression recognition with robust face detection using a convolutional neural network. Neural Networks 2003, 16(5):555-559.
25. Mitaim Sanya, Kosko Bart Noise-benefit forbidden-interval theorems for threshold signal detectors based on cross correlations. Physical Review E 2014, 90(5).
26. Oakes David Direct calculation of the information matrix via the EM. Journal of the Royal Statistical Society. Series B (Statistical Methodology) 1999, 61(2):479-482.
27. Osoba O., Kosko B. Noise-enhanced clustering and competitive learning algorithms. Neural Networks 2013.
28. Osoba O., Kosko B. The noisy expectation-maximization algorithm for multiplicative noise injection. Fluctuation and Noise Letters 2016.
29. Osoba O., Mitaim S., Kosko B. Noise benefits in the expectation-maximization algorithm: NEM theorems and models. The international joint conference on neural networks (IJCNN) 2011, 3178-3183. IEEE.
30. Osoba O., Mitaim S., Kosko B. The noisy expectation-maximization algorithm. Fluctuation and Noise Letters 2013, 12(03).
31. Palm, Rasmus Berg (2014). DeepLearn Toolbox. https://github.com/rasmusbergpalm/DeepLearnToolbox.
32. Patel, A., & Kosko, B. (2007). Levy noise benefits in neural signal detection. In IEEE international conference on acoustics, speech and signal processing, 2007. ICASSP 2007, Vol. 3 (pp. III-1413-III-1416).
33. Patel A., Kosko B. Stochastic resonance in continuous and spiking neurons with Levy noise. IEEE Transactions on Neural Networks 2008, 19(12):1993-2008.
34. Patel A., Kosko B. Error-probability noise benefits in threshold neural signal detection. Neural Networks 2009, 22(5):697-706.
35. Patel A., Kosko B. Optimal mean-square noise benefits in quantizer-array linear estimation. IEEE Signal Processing Letters 2010, 17(12):1005-1009.
36. Patel A., Kosko B. Noise benefits in quantizer-array correlation detection and watermark decoding. IEEE Transactions on Signal Processing 2011, 59(2):488-505.
37. Rumelhart D.E., Hinton G.E., Williams R.J. Learning representations by back-propagating errors. Nature 1986, 323(6088):533-536.
38. Simard, P., Steinkraus, D., & Platt, J.C. (2003). Best practices for convolutional neural networks applied to visual document analysis. In ICDAR, Vol. 3 (pp. 958-962).
39. Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. ArXiv Preprint arxiv:1409.1556.
40. Slavakis K., Giannakis G., Mateos G. Modeling and optimization for big data analytics: (statistical) learning tools for our era of data deluge. IEEE Signal Processing Magazine 2014, 31(5):18-31.
41. Szarvas M., Yoshizawa A., Yamamoto M., Ogata J. Pedestrian detection with convolutional neural networks. Proceedings of the IEEE intelligent vehicles symposium 2005, 224-229. IEEE.
42. Wilde M., Kosko B. Quantum forbidden-interval theorems for stochastic resonance. Journal of Physics A: Mathematical and Theoretical 2009, 42(46).