Enhancing performance of the backpropagation algorithm via sparse response regularization

Neurocomputing

Volumn 153, Issue , 2015, Pages 20-40

  Zhang, Jiangshe ^a   Ji, Nannan ^a   Liu, Junmin ^a   Pan, Jiyuan ^b   Meng, Deyu ^a  

^a XI'AN JIAOTONG UNIVERSITY   (China)

^b 20th Institute of China Electronics Technology Group Corporation   (China)

Author keywords

Backpropagation; Feed forward artificial neural network; Human nervous system; Regularization

Indexed keywords

ENERGY CONSERVATION; FEEDFORWARD NEURAL NETWORKS;

CONVERGENCE RATES; FEED-FORWARD ARTIFICIAL NEURAL NETWORKS; GENERALIZATION CAPABILITY; HUMAN NERVOUS SYSTEM; HUMAN NEURONS; REGULARIZATION; SPARSE REPRESENTATION; TRAINING STRATEGY;

BACKPROPAGATION;

ARTICLE; ARTIFICIAL NEURAL NETWORK; BACK PROPAGATION; ENERGY CONSERVATION; FEED FORWARD ARTIFICIAL NEURAL NETWORK; KERNEL METHOD; LAPLACE ALGORITHM; LEARNING ALGORITHM; NERVOUS SYSTEM FUNCTION; POSITIVE FEEDBACK; PROBLEM SOLVING; SPARSE RESPONSE BACK PROPAGATION; SYSTEMATIC ERROR; VALIDATION PROCESS;

EID: 84922243719     PISSN: 09252312     EISSN: 18728286     Source Type: Journal    
DOI: 10.1016/j.neucom.2014.11.055     Document Type: Article

References (56)

1. F. Fleureta, T. Li, C. Dubouta, E.K. Wamplerd, S. Yantisd, D. Gemanc, Comparing machines and humans on a visual categorization test, in: Proceedings of the National Academy of Sciences, 2011, pp. 17621-17625.
2. Chang K.C., Hong T.P., Tseng S.S. Machine learning by imitating human learning. Mind. Mach. 1996, 6:203-228.
3. Bengio Y. Learning deep architectures for AI. Found. Trends Mach. Learn. 2009, 2:1-127.
4. Aizenberg I., Paliy D., Zurada J., Astola J. Blur identification by multilayer neural network based on multi-valued neurons. IEEE Trans. Neural Netw. 2008, 19:883-898.
5. Marchant J.A., Onyango C.M. Comparison of a Bayesian classifier with a multilayer feed-forward neural network using the example of plant/weed/soil discrimination. Comput. Electron. Agr. 2003, 39:3-22.
6. Luo X., Patton A.D., Singh C. Real power transfer capability calculations using multi-layer feed-forward neural networks. IEEE Trans. Power Syst. 2000, 15:903-908.
7. Hastie T., Tibshirani R. Discriminant adaptive nearest neighbor classification. IEEE Trans. Pattern Anal. Mach. Intell. 1995, 18:607-616.
8. E.P. Xing, A.Y. Ng, M.I. Jordan, S. Russell, Distance metric learning with application to clustering with side-information, in: S. Becker, S. Thrun, K. Obermayer (Eds.), Advances in Neural Information Processing Systems, vol. 15, 2003, MIT Press, Cambridge, MA, pp. 505-512.
9. N. Shental, T. Hertz, D. Weinshall, M. Pavel, Adjustment learning and relevant component analysis, in: Proceedings of the European Conference on Computer Vision, Springer-Verlag, London, UK, 2002, pp. 776-792.
10. Shrivastava S., Singh M.P. Performance evaluation of feed-forward neural network with soft computing techniques for hand written English alphabets. Appl. Soft Comput. 2011, 11:1156-1182.
11. H. Su, G. Li, D. Yu, F. Seide, Error back propagation for sequence training of context-dependent deep networks for conversational speech transcription, in: Proceedings of the International Conference on Acoustics, Speech, and Signal Processing, Vancouver, BC, Canada, 2013, pp. 6664-6668.
12. G. Heigold, V. Vanhoucke, A. Senior, P. Nguyen, M. Ranzato, M. Devin, J. Dean, Multilingual acoustic models using distributed deep neural networks, in: Proceedings of the International Conference on Acoustics, Speech, and Signal Processing, Vancouver, BC, Canada, 2013, pp. 8619-8623.
13. Zhang J., Morris A. A sequential learning approach for single hidden layer neural networks. Neural Netw. 1997, 11:65-80.
14. Moody J., Anstsaking P.J. The dependence identification neural network construction algorithm. IEEE Trans. Neural Netw. 1996, 7:13-15.
15. Setiono R., Hui L.C.K. Use of a quasi-Newton method in a feedforward neural network construction algorithm. IEEE Trans. Neural Netw. 1995, 6:237-277.
16. Parekh R., Yang J.H., Honavar V. Constructive neural-network learning algorithms for pattern classification. IEEE Trans. Neural Netw. 2000, 11:436-451.
17. Gallant S.I. Neural Network Learning and Expert Systems 1993, MIT Press, Cambridge, MA.
18. Honavor V., Uhr V.L. Generative learning structures for generalized connectionist networks. Inf. Sci. 1993, 70:75-108.
19. Kwok T.Y., Yeung D.Y. Objective functions for training new hidden units in constructive neural networks. IEEE Trans. Neural Netw. 1997, 8:1131-1148.
20. S.E. Fahlman, C. Lebiere, The Cascade-Correlation Learning Architecture. Carnegie Mellon University, Technical Report CMU-CS-90-100, 1991.
21. Prechelt L. Investigation of the Casor family of learning algorithms. Neural Netw. 1997, 10:885-896.
22. T.Y. Kwok, D.Y. Yeung, Bayesian regularization in constructive neural networks, in: Proceedings of the International Conference on Artificial Neural Networks, Bochum, Germany, 1996, pp. 557-562.
23. LeCun Y., Denker J.S., Solla S.A. Optimal brain damage. Advances in Neural Information Processing Systems 1990, 598-605. Morgan Kaufmann, San Mateo, CA. D.S. Touretsky (Ed.).
24. Hassibi B., Stork D.G. Second-order derivatives for network pruning. optimal brain surgeon. Advances in Neural Information Processing Systems 1993, 164-171. Morgan Kaufmann, San Mateo, CA. S.J. Hanson, J.D. Cowan, C.L. Giles (Eds.).
25. Reed R. Pruning algorithms-a review. IEEE Trans. Neural Netw. 1991, 2:47-55.
26. Girosi F., Jones M., Poggio T. Regularization theory and neural network architecture. Neural Comput. 1995, 7:219-269.
27. Williams P. Bayesian regularization and pruning using a Laplace prior. Neural Comput. 1995, 7:117-143.
28. A.S. Weigend, D.E. Rumelhart, B.A. Huberman, Generalization by weight-elimination with application to forecasting, in: Advances in Neural Information Processing Systems, vol. 3, Morgan Kaufmann, San Mateo, CA, 1990, pp. 875-882.
29. Nowlan S.J., Hinton G.E. Simplifying neural networks by soft weight-sharing. Neural Comput. 1992, 4:473-493.
30. Ishikawa M. Structural learning with forgetting. Neural Netw. 1996, 9:509-521.
31. Chauvin Y. A backpropagation algorithm with optimal use of hidden units. Advances in Neural Information Processing Systems 1989, vol. 1:519-526. Morgan Kaufmann, San Mateo, CA.
32. Mrázová I., Wang D.H. Improved generalization of neural classifiers with enforced internal representation. Neurocomputing 2007, 70:2940-2952.
33. Hirasawa K. Enhancing the generalization ability of neural networks through controlling the hidden layers. Appl. Soft Comput. 2009, 9:401-414.
34. G. Morris, A. Nevet, H. Bergman, Anatomical funneling, sparse connectivity and redundancy reduction in the neural networks of the basal ganglia, J. Physiol.-Paris (2003) 581-589.
35. Barlow H.B. Single units and sensation. a neuron doctrine for perceptual psychology?. Perception 1972, 1:371-394.
36. Olshausen B.A., Field D.J. Sparse coding of sensory inputs. Curr. Opin. Neurobiol. 2004, 14:481-487.
37. H. Lee, C. Ekanadham, A. Ng, Sparse deep belief net model for visual area V2, in: Advances in Neural Information Processing Systems, 2008, pp. 873-880.
38. Lee H., Grosse R., Ranganath R., Ng A.Y. Unsupervised learning of hierarchical representations with convolutional deep belief networks. Commun. ACM 2011, 54:95-103.
39. M. Ranzato, C. Poultney, S. Chopra, Y. LeCun, Efficient learning of sparse representations with an energy-based model, in: Advances in Neural Information Processing Systems, 2007, pp. 1137-1144.
40. J. Mairal, F. Bach, J. Ponce, G. Sapiro, A. Zisserman, Supervised dictionary learning, in: Advances in Neural Information Processing Systems, 2009, pp. 1033-1040.
41. H. Luo, R. Shen, C. Niu, C. Ullrich, Sparse group restricted Boltzmann machines, in: Proceedings of the 25th AAAI Conference on Artificial Intelligence, 2011, pp. 429-434.
42. Thom M., Palm G. Sparse activity and sparse connectivity in supervised learning. J. Mach. Learn. Res. 2013, 14:1091-1143.
43. M.C. Mozer, P. Smolensky, Skeletonization: a technique for trimming the fat from a network via relevance assessment, in: Advances in Neural Information Processing Systems, 1989, pp. 107-115.
44. Lauret P., Fock E., Mara T.A. A node pruning algorithm based on a Fourier amplitude sensitivity test method. IEEE Trans. Neural Netw. 2006, 17:273-293.
45. Olshausen B., Field D. Emergence of simple-cell receptive field properties by learning sparse code for natural images. Nature 1996, 607-609.
46. Candes E., Tao T. Decoding by linear programming. IEEE Trans. Inf. Theory 2005, 15:4203-4215.
47. Donoho D. Compressed sensing. IEEE Trans. IEEE Trans. Inf. Theory 2006, 52:1289-1306.
48. G.H. Mohimani, M. Babaie-Zadeh, C. Jutten, Fast sparse representation based on smoothed L0 norm, in: Proceedings of the Seventh International Conference on Independent Component Analysis and Signal Separation, 2007, pp. 389-396.
49. L. Mancera, J. Portilla, L0-norm-based sparse representation through alternate projections, in: Proceedings of the IEEE Conference on Image Processing, 2006, pp. 2089-2092.
50. P. Favaro, S. Osher, S. Soatto, L. Vese, 3d shape from anisotropic diffusion, in: IEEE Conference on Computer Vision and Pattern Recognition, 2003, pp. 179-186.
51. V. Nair, G.E. Hinton, 3D object recognition with deep belief nets, in: Advances in Neural Information Processing Systems, 2009, pp. 1339-1347.
52. C.L. Blake, C.J. Merz, UCI repository of machine learning databases, Department of Information and Computer Science, University of California, Irvine, CA [Online], 1998. http://www.ics.uci.edu/mlern/Machine-Learning.html.
53. Tipping M.E. Sparse Bayesian learning and the relevance vector machine. J. Mach. Learn. Res. 2001, 1:211-244.
54. Cortes C., Vapnik V. Support-vector networks. Mach. Learn. 1995, 20:273-297.
55. Krishnapuram B., Carin L., Figueiredo M., Hartemink A. Sparse multinomial logistic regression. fast algorithms, and generalization bounds. IEEE Trans. Pattern Anal. Mach. Intell. 2005, 27:957-968.
56. X. Glorot, A. Bordes, Y. Bengio, Deep sparse rectifier neural networks, in: Advances in Neural Information Processing Systems, 2007, pp. 1137-1144.