Deep unsupervised learning on a desktop pc: A primer for cognitive scientists

Frontiers in Psychology

Volumn 4, Issue MAY, 2013, Pages

  Testolin, Alberto ^a   Stoianov, Ivilin ^a   De Filippo De Grazia, Michele ^a   Zorzi, Marco ^a,b  

^a UNIVERSITY OF PADOVA   (Italy)

^b IRCCS San Camillo Neurorehabilitation Hospital   (Italy)

Author keywords

Cognitive modeling; Computer cluster; Deep neural networks; GPUs; Hierarchical generative models; MPI; Parallel computing architectures; Unsupervised learning

Indexed keywords

EID: 84878913949     PISSN: None     EISSN: 16641078     Source Type: Journal    
DOI: 10.3389/fpsyg.2013.00251     Document Type: Article

References (47)

1. Ackley, D., Hinton, G. E., and Sejnowski, T. J. (1985). A learning algorithm for Boltzmann machines. Cogn. Sci. 9, 147-169.
2. Aisa, B., Mingus, B., and O'Reilly, R. (2008). The emergent neural modeling system. Neural Netw. 21, 1146-1152.
3. Bengio, Y. (2009). Learning Deep Architectures for AI. Foundations and trends® in Machine Learning 2, 1-127.
4. Bengio, Y. (2011). Deep learning of representations for unsupervised and transfer learning. Proc. Int. Conf. Mach. Learn. Appl. 7, 1-20.
5. Bengio, Y., and Lamblin, P. (2007). Greedy layer-wise training of deep networks. Adv. Neural Inf. Process Syst. 19, 153-170.
6. Ciresan, D. C., Meier, U., Gambardella, L. M., and Schmidhuber, J. (2010). Deep big simple neural nets excel on handwritten digit recognition. Neural Comput. 22, 3207-3220.
7. Clark, A. (2012). Whatever next? Predictive brains, situated agents, and the future of cognitive science. Behav. Brain Sci. 1-86.
8. De Filippo De Grazia, M., Cutini, S., Lisi, M., and Zorzi, M. (2012a). Space coding for sensorimotor transformations can emerge through unsupervised learning. Cogn. Process. 13, S141-S146.
9. De Filippo De Grazia, M., Stoianov, I., and Zorzi, M. (2012b). "Parallelization of deep networks," in European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, Belgium: ESANN.
10. Dean, J., Corrado, G. S., Monga, R., Chen, K., Devin, M., Le, Q. V., et al. (2012). Large scale distributed deep networks. Adv. Neural Inf. Process Syst. 25, 1232-1240.
11. Fernández, J., Anguita, M., Ros, E., and Bernier, J. L. (2006). SCE Toolboxes for the development of high-level parallel applications. Lect. Notes Comput. Sci. 1, 518-525.
12. Hertz, J. A., Krogh, A. S., and Palmer, R. G. (1991). Introduction to the Theory of Neural Computation, Redwood City: Westview press.
13. Hinton, G. E. (2002). Training products of experts by minimizing contrastive divergence. Neural Comput. 14, 1771-1800.
14. Hinton, G. E. (2007). Learning multiple layers of representation. Trends Cogn. Sci. (Regul. Ed.) 11, 428-434.
15. Hinton, G. E. (2010a). A Practical Guide to Training Restricted Boltzmann Machines. Technical Report UTML TR 2010-003, University of Toronto 9, :1.
16. Hinton, G. E. (2010b). Learning to represent visual input. Philos. Trans. R. Soc. Lond. B Biol. Sci. 365, 177-184.
17. Hinton, G. E., and Ghahramani, Z. (1997). Generative models for discovering sparse distributed representations. Philos. Trans. R. Soc. Lond. B Biol. Sci. 352, 1177-1190.
18. Hinton, G. E., and Osindero, S. (2006). A fast learning algorithm for deep belief nets. Neural Comput. 18, 1527-1554.
19. Hinton, G. E., and Salakhutdinov, R. (2006). Reducing the dimensionality of data with neural networks. Science 313, 504-507.
20. Honglak, L., Ekanadham, C., and Ng, A. Y. (2008). Sparse deep belief net models for visual area V2. Adv. Neural Inf. Process Syst. 20, 873-880.
21. Huang, Y., and Rao, R. P. N. (2011). Predictive coding. Wiley interdisciplinary reviews. Cogn. Sci. 2, 580-593.
22. Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. Adv. Neural Inf. Process Syst. 25, 1106-1114.
23. Larochelle, H., Bengio, Y., Louradour, J., and Lamblin, P. (2009). Exploring strategies for training deep neural networks. J. Mach. Learn. Res. 10, 1-40.
24. Le, Q. V., Ranzato, M. A., Monga, R., Devin, M., Chen, K., Corrado, G. S., et al. (2012). "Building high-level features using large scale unsupervised learning," in International Conference on Machine Learning, New York.
25. LeCun, Y., and Bottou, L. (1998). Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278-2324.
26. Lee, H., Grosse, R., Ranganath, R., and Ng, A. Y. (2009). "Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations," in International Conference on Machine Learning (New York, NY: ACM Press), 609-616.
27. Lee, V., Kim, C., Chhugani, J., and Deisher, M. (2010). Debunking the 100X GPU vs. CPU myth: an evaluation of throughput computing on CPU and GPU. Comput. Architect. News 38, 451-460.
28. Margaris, A., Souravlas, S., Kotsialos, E., and Roumeliotis, M. (2007). Design and implementation of parallel counter propagation networks using MPI. Informatica 18, 79-102.
29. Mohamed, A., Dahl, G. E., and Hinton, G. E. (2012). Acoustic modeling using deep belief networks. IEEE Trans. Audio Speech Lang. Processing 20, 14-22.
30. Mutch, J., Knoblich, U., and Poggio, T. (2010). CNS: a GPU-based framework for simulating cortically-organized networks. Technical Report MIT-CSAIL-TR-2010-013, Cambridge: Massachusetts Institute of Technology.
31. Nickolls, J., Buck, I., Garland, M., and Skadron, K. (2008). Scalable parallel programming with CUDA. Queue 6, 40.
32. Olshausen, B., and Field, D. (1996). Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381, 607-609.
33. O'Reilly, R. C. (1998). Six principles for biologically based computational models of cortical cognition. Trends Cogn. Sci. (Regul. Ed.) 2, 455-462.
34. Owens, J., and Houston, M. (2008). GPU computing. Proc. IEEE 96, 879-899.
35. Pan, S. J., and Yang, Q. (2010). A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22, 1345-1359.
36. Plesser, H. E., Eppler, J. M., Morrison, A., Diesmann, M., and Gewaltig, M. O. (2007). Efficient parallel simulation of large-scale neuronal networks on clusters of multiprocessor computers. Euro-Par 2007 Parallel Processing. Lect. Notes Comput. Sci. 4641, 672-681.
37. Raina, R., Madhavan, A., and Ng, A. Y. (2009). "Large-scale deep unsupervised learning using graphics processors," in International Conference on Machine Learning (New York, NY: ACM Press), 1-8.
38. Rao, R. P. N., and Ballard, D. H. (1999). Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nat. Neurosci. 2, 79-87.
39. Salakhutdinov, R., and Hinton, G. (2012). A better way to pretrain deep Boltzmann machines. Adv. Neural Inf. Process Syst. 25, 2456-2464.
40. Share, D. L. (1995). Phonological recoding and self-teaching: sine qua non of reading acquisition. Cognition 55, 151-218; discussion 219-226.
41. Sharma, G., and Martin, J. (2008). MATLAB®: a language for parallel computing. Int. J. Parallel Program. 37, 3-36.
42. Smolensky, P. (1986). "Information processing in dynamical systems: foundations of harmony theory," in Parallel Distributed Processing. Explorations in the Microstructure of Cognition, Vol. 1, eds D. E. Rumelhart, and J. L. McClelland (Cambridge: MIT Press), 1, 194-281.
43. Stoianov, I., and Zorzi, M. (2012). Emergence of a "visual number sense" in hierarchical generative models. Nat. Neurosci. 15, 194-196.
44. Tieleman, T. (2010). Gnumpy: an easy way to use GPU boards in Python. Technical Report UTML TR 2010-002, University of Toronto, Toronto.
45. Wilson, D. R., and Martinez, T. R. (2003). The general inefficiency of batch training for gradient descent learning. Neural Netw. 16, 1429-1451.
46. Yu, D., Hinton, G. E., Morgan, N., Chien, J., and Sagayama, S. (2012). Introduction to the special section on deep learning for speech and language processing. IEEE Trans. Audio Speech Lang. Processing 20, 4.
47. Zorzi, M. (2010). The connectionist dual process (CDP) approach to modelling reading aloud. Eur. J. Cogn. Psychol. 22, 836-860.