Towards adaptive learning with improved convergence of deep belief networks on graphics processing units

Pattern Recognition

Volumn 47, Issue 1, 2014, Pages 114-127

  Lopes, Noel ^a,b   Ribeiro, Bernardete ^a,c  

^a UNIVERSITY OF COIMBRA   (Portugal)

^b POLYTECHNIC INSTITUTE OF GUARDA   (Portugal)

^c UNIVERSITY OF COIMBRA   (Portugal)

Author keywords

Adaptive step size; Contrastive divergence; Deep belief networks; Deep learning; GPU computing; Restricted Boltzmann machines

Indexed keywords

ADAPTIVE STEP SIZE; CONTRASTIVE DIVERGENCE; DEEP BELIEF NETWORKS; DEEP LEARNING; GPU COMPUTING; RESTRICTED BOLTZMANN MACHINE;

ALGORITHMS; BAYESIAN NETWORKS; COMPUTER GRAPHICS; PROGRAM PROCESSORS;

COMPUTER GRAPHICS EQUIPMENT;

EID: 84885018698     PISSN: 00313203     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.patcog.2013.06.029     Document Type: Article

References (34)

1. J. Markoff Giant steps in teaching computers to think like us: 'neural nets' mimic the ways human minds listen, see and execute International Herald Tribune 24-25 November 2012 1 8
2. H. Larochelle, D. Erhan, A. Courville, J. Bergstra, Y. Bengio, An empirical evaluation of deep architectures on problems with many factors of variation, in: Proceedings of the 24th International Conference on Machine Learning, 2007, pp. 473-480.
3. N.L. Roux, and Y. Bengio Representational power of restricted Boltzmann machines and deep belief networks Neural Computation 20 6 2008 1631 1649
4. Y. Bengio Learning deep architectures for AI Foundations and Trends in Machine Learning 2 1 2009 1 127
5. H. Lee, R. Grosse, R. Ranganath, A.Y. Ng, Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations, in: Proceedings of the 26th International Conference on Machine Learning, 2009, pp. 609-616.
6. D. Yu, and L. Deng Deep learning and its applications to signal and information processing IEEE Signal Processing Magazine 28 1 2011 145 154
7. N.L. Roux, and Y. Bengio Deep belief networks are compact universal approximators Neural Computation 22 8 2010 2192 2207
8. G.E. Hinton, S. Osindero, and Y.-W. Teh A fast learning algorithm for deep belief nets Neural Computation 18 7 2006 1527 1554 (Pubitemid 44024729)
9. K. Swersky, B. Chen, B. Marlin, N. de Freitas, A tutorial on stochastic approximation algorithms for training restricted Boltzmann machines and deep belief nets, in: Information Theory and Applications Workshop, 2010, pp. 1-10.
10. N. Lopes, B. Ribeiro, Improving convergence of restricted Boltzmann machines via a learning adaptive step size, in: Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications, Lecture Notes in Computer Science, vol. 7441, Springer, Berlin/Heidelberg, 2012, pp. 511-518.
11. N. Lopes, B. Ribeiro, J. Gonçalves, Restricted Boltzmann machines and deep belief networks on multi-core processors, in: The 2012 International Joint Conference on Neural Networks (IJCNN), 2012.
12. S.K. Kim, P.L. McMahon, K. Olukotun, A large-scale architecture for restricted Boltzmann machines, in: 18th IEEE Annual International Symposium on Field-Programmable Custom Computing Machines, 2010.
13. D. Ly, and P. Chow A high-performance FPGA architecture for restricted Boltzmann machines IEEE Transactions on Neural Networks 21 11 2010 1780 1792
14. R. Raina, A. Madhavan, A.Y. Ng, Large-scale deep unsupervised learning using graphics processors, in: Proceedings of the 26th International Conference on Machine Learning, 2009, pp. 873-880.
15. D.L. Ly, V. Paprotski, D. Yen, Neural Networks on GPUs: Restricted Boltzmann Machines, Technical Report, University of Toronto, 2009.
16. D. Steinkraus, I. Buck, P.Y. Simard, Using GPUs for machine learning algorithms, in: Proceedings of the 8th International Conference on Document Analysis and Recognition, vol. 2, 2005, pp. 1115-1120. (Pubitemid 46442497)
17. M. Garland, and D.B. Kirk Understanding throughput-oriented architectures Communications of the ACM 53 11 2010 58 66
18. B. Catanzaro, N. Sundaram, K. Keutzer, Fast support vector machine training and classification on graphics processors, in: Proceedings of the 25th International Conference on Machine Learning, 2008, pp. 104-111.
19. G.E. Hinton, A Practical Guide to Training Restricted Boltzmann Machines, Technical Report, Department of Computer Science, University of Toronto, 2010.
20. M. Ranzato, Y. Boureau, Y. LeCun, Sparse feature learning for deep belief networks, in: Advances in Neural Information Processing Systems (NIPS 2007), vol. 20, 2007, pp. 1185-1192.
21. M.A. Carreira-Perpiñán, G.E. Hinton, On contrastive divergence learning, in: Proceedings of the 10th International Workshop on Artificial Intelligence and Statistics (AISTATS 2005), 2005, pp. 33-40.
22. G.E. Hinton Training products of experts by minimizing contrastive divergence Neural Computation 14 8 2002 1771 1800
23. J. Srinivas, G.S. Prafulla, and P. Premchand Speaker independent vowel recognition using backpropagation neural network on master-slave architecture International Journal of Computer Applications 48 3 2012 45 49
24. N. Lopes, and B. Ribeiro An evaluation of multiple feed-forward networks on GPUs International Journal of Neural Systems 21 1 2011 31 47
25. Z. Zainuddin, N. Mahat, and Y.A. Hassan Improving the convergence of the backpropagation algorithm using local adaptive techniques International Journal of Computational Intelligence 2005 172 175
26. F.M. Silva, L.B. Almeida, Acceleration techniques for the backpropagation algorithm, in: Proceedings of the EURASIP Workshop on Neural Networks, Lecture Notes in Computer Science, vol. 412, Springer Verlag, 1990.
27. L.B. Almeida Handbook of Neural Computation 1997 Oxford University Press (Chapter C1.2 Multilayer perceptrons, pp. C1.2:1-C1.2:30)
28. N. Lopes, and B. Ribeiro GPUMLib an efficient open-source GPU machine learning library International Journal of Computer Information Systems and Industrial Management Applications 3 2011 355 362
29. N. Lopes, B. Ribeiro, R. Quintas, GPUMLib: a new library to combine machine learning algorithms with graphics processing units, in: Proceedings of the 10th International Conference on Hybrid Intelligent Systems, 2010, pp. 229-232.
30. J.D. Owens, M. Houston, D. Luebke, S. Green, J.E. Stone, and J.C. Phillips GPU computing Proceedings of the IEEE 96 5 2008 879 899
31. T. Hey, S. Tansley, and K. Tolle The Fourth Paradigm Data-Intensive Scientific Discovery 2009 Microsoft Research
32. T.R. Halfhill, Looking Beyond Graphics, Technical Report, In-Stat, 2009.
33. S. Ryoo, C.I. Rodrigues, S.S. Baghsorkhi, S.S. Stone, D.B. Kirk, W.W. Hwu, Optimization principles and application performance evaluation of a multithreaded GPU using CUDA, in: Proceedings of the 13th ACM Symposium on Principles and Practice of Parallel Programming, 2008, pp. 73-82.
34. N. Lopes, and B. Ribeiro An efficient gradient-based learning algorithm applied to neural networks with selective actuation neurons Neural, Parallel and Scientific Computations 11 2003 253 272 (Pubitemid 37141689)