Where do features come from?

Cognitive Science

Volumn 38, Issue 6, 2014, Pages 1078-1101

  Hinton, Geoffrey ^a  

^a UNIVERSITY OF TORONTO   (Canada)

Author keywords

Backpropagation; Boltzmann machines; Contrastive divergence; Deep learning; Distributed representations; Learning features; Learning graphical models; Variational learning

Indexed keywords

ARTIFICIAL INTELLIGENCE; ARTIFICIAL NEURAL NETWORK; BIOLOGICAL MODEL; COMPUTER SIMULATION; HUMAN; LEARNING;

ARTIFICIAL INTELLIGENCE; COMPUTER SIMULATION; HUMANS; LEARNING; MODELS, NEUROLOGICAL; NEURAL NETWORKS (COMPUTER);

EID: 84906221749     PISSN: 03640213     EISSN: None     Source Type: Journal    
DOI: 10.1111/cogs.12049     Document Type: Article

References (56)

1. Baum, L. E. (1972). An inequality and associated maximization technique in statistical estimation for probabilistic functions of markov processes. Inequalities, 3, 1-8.
2. Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5, 157-166.
3. Buesing, L., Bill, J., Nessler, B., & Maass, W. (2011). Neural dynamics as sampling: A model for stochastic computation in recurrent networks of spiking neurons. PLoS Computational Biology, 7. 10.1371/journal.pcbi.1002211.
4. Crick, F., & Mitchison, G. (1986). The function of dream sleep. Nature, 304, 111-114.
5. Dahl, G. E., Ranzato, M., Mohamed, A., & Hinton, G. E. (2010). Phone recognition with the mean-covariance restricted Boltzmann machine. In Advances in neural information processing systems, Vol. 24 (pp. 469-477).
6. DeMers, D., & Cottrell, G. W. (1993). Nonlinear dimensionality reduction. In C. Hanson & C. Giles (Eds.), Advances in neural information processing systems, Vol. 5 (pp. 580-587). San Mateo, CA: Morgan Kaufmann.
7. Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society B, 39, 1-38.
8. Elman, J. L. (1990). Finding structure in time. Cognitive Science, 14(2), 179-211.
9. Erhan, D., Bengio, Y., Courville, A., Manzagol, P., Vincent, P., & Bengio, S. (2010). Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research, 11, 625-660.
10. Friston, K., Kilner, J., & Harrison, L. (2006). A free energy principle for the brain. Journal of Physiology, 100, 70-87.
11. Hecht-Nielsen, R. (1995). Replicator neural networks for universal optimal source coding. Science, 269, 1860-1863.
12. Heckerman, D. (1986). Probabilistic interpretations for mycin's certainty factors. In L. Kanal & J. Lemmer (Eds.), Uncertainty in artificial intelligence (pp. 167-196). New York: North-Holland.
13. Hinton, G. E. (1989). Connectionist learning procedures. Artificial Intelligence, 40, 185-234.
14. Hinton, G. E. (2002). Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8), 1711-1800.
15. Hinton, G. E. (2010). Learning to represent visual input. Philosophical Transactions of the Royal Society B, 365, 177-184.
16. Hinton, G. E., Dayan, P., Frey, B. J., & Neal, R. (1995). The wake-sleep algorithm for self-organizing neural networks. Science, 268, 1158-1161.
17. Hinton, G. E., Krizhevsky, A., & Wang, S. (2011). Transforming auto-encoders. In T. Hontela, W. Duch, M. Girolami, & S. Kashki (Eds.), ICANN-11: International conference on artificial neural networks (pp. 44-51). Helsinki: Springer.
18. Hinton, G. E., Osindero, S., & Teh, Y. W. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18(7), 1527-1554.
19. Hinton, G. E., & Salakhutdinov, R. R. (2006). Reducing the dimensionality of data with neural networks. Science, 313, 504-507.
20. Hinton, G. E., & Sejnowski, T. J. (1986). Learning and relearning in Boltzmann machines. In D. E. Rumelhart & J. L. McClelland (Eds.), Parallel distributed processing: Volume 1: Foundations (pp. 282-317). Cambridge, MA: MIT Press.
21. Hinton, G. E., & Zemel, R. S. (1994). Autoencoders, minimum description length, and Helmholtz free energy. Advances in Neural Information Processing Systems, 6, 3-10.
22. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9, 1735-1780.
23. Horn, B. K. P. (1977). Understanding image intensities. Artificial Intelligence, 8, 201-231.
24. Jordan, M. I., Ghahramani, Z., Jaakkola, T. S., & Saul, L. K. (1999). An introduction to variational methods for graphical models. In M. I. Jordan (Ed.), Learning in graphical models (pp. 105-161). Cambridge, MA: MIT Press.
25. Lauritzen, S. L., & Spiegelhalter, D. J. (1988). Local computations with probabilities on graphical structures and their application to expert systems. Journal of the Royal Statistical Society B, 50, 157-224.
26. LeCun, Y. (1985). Une procédure d'apprentissage pour réseau a seuil asymmetrique (a learning scheme for asymmetric threshold networks). In F. Fogelman (Ed.), Proceedings of cognitiva, (pp. 599-604). Paris, France.
27. LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278-2324.
28. Lee, H., Grosse, R., Ranganath, R., & Ng, A. (2009). Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In L. Bottou & M. Littman (Eds.), Proceedings of the 26th international conference on machine learning (pp. 609-616). Montreal: ACM.
29. MacKay, D. J. C. (2003). Information theory, inference, and learning algorithms. Cambridge, England: Cambridge University Press.
30. Markram, H., Joachim, L., Frotscher, M., & Sakmann, B. (1997). Regulation of synaptic efficacy by coincidence of postsynaptic APs and EPSPs. Science, 275, 213-215.
31. Martens, J. (2010). Deep learning via Hessian-free optimization. In J. Furnkranz & T. Joachims (Eds.), Proceedings of the 27th international conference on machine learning (ICML) (pp. 735-742). Haifa, Israel: Omnipress.
32. McClelland, J. L., & Rumelhart, D. E. (1981). An interactive activation model of context effects in letter perception: Part 1. An account of basic findings. Psychological Review, 88, 375-407.
33. Nair, V., & Hinton, G. E. (2010). Rectified linear units improve restricted Boltzmann machines. In J. Furnkranz & T. Joachims (Eds.), Proceedings of the 27th international conference on machine learning (pp. 807-814). Haifa, Israel: Omnipress.
34. Neal, R. M. (1992). Connectionist learning of belief networks. Artificial Intelligence, 56(1), 71-113.
35. Neal, R. M. (1994). Bayesian learning for neural networks. PhD thesis, Department of Computer Science, University of Toronto.
36. Neal, R. M., & Hinton, G. E. (1998). A new view of the EM algorithm that justifies incremental, sparse and other variants. In M. I. Jordan (Ed.), Learning in graphical models (pp. 355-368). Dordrecht, The Netherlands: Kluwer Academic Publishers.
37. O'Keefe, J., & Recce, M. L. (1993). Phase relationship between hippocampal place units and the EEG theta rhythm. Hippocampus, 3, 317-330.
38. Pearl, J. (1988). Probabilistic inference in intelligent systems: Networks of plausible inference. San Mateo, CA: Morgan Kaufmann.
39. Ranzato, M., Boureau, Y., & LeCun, Y. (2007). Sparse feature learning for deep belief networks. In B. Scholkopf, J. Platt, & T. Huffman (Eds.), Advances in neural information processing systems, Vol. 20 (pp. 1185-1192). San Mateo, CA: Morgan Kaufmann.
40. Rasmussen, C. E., & Williams, C. K. I. (2006). Gaussian processes for machine learning. Cambridge, MA: MIT Press.
41. Reichert, D. P., Series, P., & Storkey, A. J. (2010). Hallucinations in Charles Bonnet syndrome induced by homeostasis: A deep Boltzmann machine model. In J. Lufferty & C. Williams (Eds.), Advances in neural information processing systems, Vol. 23 (pp. 2020-2028). San Mateo, CA: Morgan Kaufmann.
42. Rifai, S., Vincent, P., Muller, X., Glorot, X., & Bengio, Y. (2011). Contracting autoencoders: Explicit invariance during feature extraction. In L. Getoor & T. Scheffer (Eds.), Proceedings of the 28th international conference on machine learning (pp. 833-840). New York: ACM.
43. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986a). Distributed representations. In D. E. Rumelhart & J. L. McClelland (Eds.), Parallel distributed processing: Volume 1: Foundations (pp. 77-109). Cambridge, MA: MIT Press.
44. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986b). Learning representations by back-propagating errors. Nature, 323, 533-536.
45. Salakhutdinov, R. R., & Hinton, G. E. (2012). An efficient learning procedure for deep Boltzmann machines. Neural Computation, 24, 1967-2006.
46. Saul, L. K., Jaakkola, T., & Jordan, M. (1996). Mean field theory for sigmoid belief networks. Journal of Artificial Intelligence Research, 4, 61-76.
47. Smolensky, P. (1986). Information processing in dynamical systems: Foundations of harmony theory. In D. E. Rumelhart & J. L. McClelland (Eds.), Parallel distributed processing: Volume 1: Foundations (pp. 194-281). Cambridge, MA: MIT Press.
48. Taylor, G. W., Hinton, G. E., & Roweis, S. (2011). Two distributed-state models for generating high-dimensional time series. Journal of Machine Learning Research, 12, 1025-1068.
49. Tenenbaum, J. B., Griffiths, T. L., & Kemp, C. (2006). Theory-based Bayesian models of inductive learning and reasoning. Trends in Cognitive Sciences, 10, 309-318.
50. Tieleman, T. (2008). Training restricted Boltzmann machines using approximations to the likelihood gradient. In A. McCallum & S. Roweis (Eds.), Proceedings of the 25th international conference on machine learning (pp. 1064-1071). New York: ACM.
51. Tieleman, T., & Hinton, G. E. (2009). Using fast weights to improve persistent contrastive divergence. In L. Bottou M. Littman (Eds.), Proceedings of the 26th international conference on machine learning (pp. 1033-1040). New York: ACM.
52. Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., & Manzagol, P.-A. (2010). Stacked denoising auto-encoders: Learning useful representations in a deep network with a local denoising criterion. Journal of Machine Learning Research, 11, 3371-3408.
53. Waibel, A., Hanazawa, T., Hinton, G. E., Shikano, K., & Lang, K. J. (1989). Phoneme recognition using time-delay neural networks. IEEE Transactions on Acoustics, Speech, and Signal Processing, 31, 328-339.
54. Welling, M., Rosen-Zvi, M., & Hinton, G. E. (2005). Exponential family harmoniums with an application to information retrieval. In L. Saul, Y. Weiss, & L. Bottou (Eds.), Advances in neural information processing systems (pp. 1481-1488). Cambridge, MA: MIT Press.
55. Werbos, P. J. (1974). Beyond regression: New tools for prediction and analysis in the behavioral sciences. PhD thesis, Harvard University.
56. Yao, X. (1999). Evolving artificial neural Networks. Proceedings of the IEEE, 87, 1423-1447.