Varieties of Helmholtz machine

Neural Networks

Volumn 9, Issue 8, 1996, Pages 1385-1403

  Dayan, Peter ^a   Hinton, Geoffrey E ^a  

^a NONE

Author keywords

expectation maximization; feedback connections; unsupervised learning

Indexed keywords

BRAIN MODELS; DATA PROCESSING; LEARNING ALGORITHMS; MATHEMATICAL MODELS; NEURAL NETWORKS; NEUROPHYSIOLOGY; PHYSIOLOGICAL MODELS; PROBABILITY DENSITY FUNCTION;

CORTICAL INFORMATION PROCESSING; FEEDBACK CONNECTIONS; HELMHOLTZ MACHINE;

LEARNING SYSTEMS;

ARTIFICIAL NEURAL NETWORK; LEARNING; PRIORITY JOURNAL; REINFORCEMENT; REVIEW;

EID: 0030297038     PISSN: 08936080     EISSN: None     Source Type: Journal    
DOI: 10.1016/S0893-6080(96)00009-3     Document Type: Article

References (65)

1. Amari S. The EM algorithm and information geometry in neural network learning. Neural Computation. 7:1995;13-18.
2. Antonini A., Stryker M.P. Development of individual geniculocortical arbors in cat striate cortex and effects of binocular impulse blockade. Journal of Neuroscience. 13:1993;3549-3573.
3. Artola A., Brocher S., Singer W. Different voltage-dependent thresholds for inducing long-term depression and long-term potentiation in slices of rate visual cortex. Nature. 347:1990;69-72.
4. Barlow H.B. Unsupervised learning. Neural Computation. 1:1989;295-311.
5. Barto A.G., Anandan Pattern recognizing stochastic learning automata. IEEE Transactions on Systems, Man and Cybernetics. 15:1985;360-374.
6. Barto, A. G., Sutton, R. S., & Watkins, C. J. C. H. (1989). Learning and sequential decision making. In M. Gabriel & J. Moore (Eds.), Learning and computational neuroscience: Foundations of adaptive networks. Cambridge, MA: MIT Press, Bradford Books.
7. Baum L.E., Eagon J.A. An inequality with applications to statistical estimation for probabilistic functions of Markov processes and to a model for ecology. Bulletin of the American Mathematical Society. 73:1967;360-363.
8. Baum L.E., Petrie E., Soules G., Weiss N. A maximisation technique occurring in the statistical analysis of probabilistic functions of Markov Chains. Annals of Mathematics and Statistics. 41(1):1970;164-171.
9. Becker S., Hinton G.E. A self-organizing neural network that discovers surfaces in random-dot stereograms. Nature. 355:1992;161-163.
10. Ben Yishai R., Baror R.L., Sompolinsky H. Theory of orientation tuning in visual cortex. Proceedings of the National Academy of Sciences. 92:1995;3844-3848.
11. Carpenter G., Grossberg S. A massively parallel architecture for a self-organizing neural pattern recognition machine. Computer Vision, Graphics and Image Processing. 37:1987;54-115.
12. Dayan P., Zemel R.S. Competition and multiple cause models. Neural Computation. 7:1995;565-579.
13. Dayan P., Hinton G.E., Neal R.M., Zemel R.S. The Helmholtz machine. Neural Computation. 7:1995;889-904.
14. Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm Proceedings of the Royal Statistical Society, B-39, 1-38.
15. Douglas R.J., Martin K.A., Whitteridge D. A canonical microcircuit for neocortex. Neural Computation. 1:1989;480-488.
16. Everitt, B. S. (1984). An introduction to latent variable models. London: Chapman and Hall.
17. Felleman D.J., Van Essen D.C. Distributed hierarchical processing in the primate cerebral cortex. Cerebral Cortex. 1:1991;1-47.
18. Friedlander M.J., Martin K.A.C. Development of Y-axon innervation of cortical area 18 in the cat. Journal of Physiology. 416:1989;183-213.
19. Ghahramani, Z., & Jordan, M. I. (1984). Supervised learning from incomplete data via an EM approach. In J. D. Cowan G. Tesauro, & J. Alspector (Eds.), Advances in neural information processing systems 6 (pp. 120-127). San Mateo, CA: Morgan Kaufmann.
20. Ghahramani, Z., & Jordan, M. I. (1995). Factorial hidden Markov models. Computational Cognitive Science Technical Report 9502, MIT.
21. Hancock P.J.B., Smith L.S., Phillisp W.A. A biologically supported error-correcting learning rule. Neural Computation. 3:1991;210-212.
22. Hebb, D. O. (1949). The organization of behavior. New York: John Wiley.
23. Hinton G.E. Deterministic Boltzmann learning performs steepest descent in weight-space. Neural Computation. 1:1986;143-150.
24. Hinton, G. E., & Sejnowski, T. J. (1986). Learning and relearning in Boltzmann machines. In D. E. Rumelhart, J. L. McClelland, & the PDP research group (Eds.), Parallel distributed processing: explorations in the microstructure of cognition. Volume 1: Foundations (pp. 282-317). Cambridge, MA: MIT Press.
25. Hinton, G. E., & van Camp, D. (1993). Keeping neural networks simple by minimizing the description length of the weights. In Proceedings of the Sixth Annual Conference on Computational Learning Theory (pp. 5-13). Santa Cruz, CA.
26. Hinton, G. E., & Zemel, R. S. (1994). Autoencoders, minimum description length and Helmholtz free energy. In J. D. Cowan, G. Tesauro, & J. Alspector (Eds), Advances in neural information processing systems 6 (pp. 3-10). San Mateo, CA: Morgan Kaufmann.
27. Hinton G.E., Dayan P., Frey B.J., Neal R.M. The wake-sleep algorithm for unsupervised neural networks. Science. 268:1995;1158-1160.
28. Jaakkola, T., Saul, L. K., & Jordan, M. I. (1996). Fast learning by bounding likelihoods in sigmoid belief nets. Advances in Neural Information Processing Systems, 8, in press.
29. Jordan, M. I., & Xu, L. (1993). Convergence results for the EM approach to mixtures of experts architectures. CBCL Memo 87, Centre for Biological and Computational Learning, MIT.
30. Kawato M., Hayakama H., Inui T. A forward-inverse optics model of reciprocal connections between visual cortical areas. Network. 4:1993;415-422.
31. Keeler, J.D., Rumelhart, D.E., & Leow, W.K. (1991). Integrated segmentation and recognition of hand-printed numerals. In R.P. Lippmann, J. Moody, & D.S. Touretzky (Eds.), Advances in neural information processing systems 3 (pp. 557-563). San Mateo, CA: Morgan Kaufmann.
32. Kohonen T. Self-organized formation of topologically correct feature maps. Biological Cybernetics. 43:1982;59-69.
33. Konorski, J. (1948). Conditioned reflexes and neuron organization. Cambridge: Cambridge University Press.
34. Luttrell, S.P. (1995). A componential self-organizing neural network. Submitted to Advances in Neural Information Processing Systems, 8.
35. MacKay, D.M. (1956). The epistemological problem for automata. In C.E. Shannon, & J. McCarthy (Eds.), Automata studies (pp. 235-252). Princeton, NJ: Princeton University Press.
36. Metropolis N., Rosenbluth A.W., Rosenbluth M.N., Teller A.H., Teller E. Equation for state calculations by fast computing machines. Journal of Chemical Physics. 21:1953;1087-1092.
37. Miller E.K., Li L., Desimone R. A neural mechanism for working and recognition memory in inferior temporal cortex. Science. 254:1991;1377-1379.
38. Montague, P.R., Dayan, P., Nowlan, S.J., Pouget, A., & Sejnowski, T.J. (1993). Using aperiodic reinforcement for directed self-organization. In C.L. Giles, S. J. Hanson, & J.D. Cowan (Eds.), Advances in neural information processing systems 5 (pp. 969-977). San Mateo: CA: Morgan Kaufmann.
39. Mumford, D. (1994). Neuronal architectures for pattern-theoretic problems. In C. Koch, & J. Davis (Eds.), Large-scale theories of the cortex (pp. 125-152). Cambridge, MA: MIT Press.
40. Neal R.M. Connectionist learning of belief networks. Artificial Intelligence. 56:1992;71-113.
41. Neal, R. M. (1993). Probabilistic inference using Markov chain Monte Carlo methods. Technical Report CRG-TR-93-1. Department of Computer Science, University of Toronto.
42. Neal, R.M., Dayan, P., & Hinton, G.E. (in preparation). Factor analysis using delta-rule wake-sleep learning.
43. Neal, R.M., & Hinton, G.E. (1994). A new view of the EM algorithm that justifies incremental and other variants. Submitted to Biometrika.
44. Pearl, J. (1988). Probabilistic reasoning in intelligent systems: Networks of plausible inference. San Mateo, CA: Morgan Kaufmann.
45. Pece, A.E.C. (1992). Redundancy reduction of a Gabor representation: a possible computational role for feedback from primary visual cortex to lateral geniculate nucleus. In I. Aleksander, & J. Taylor, (Eds.), Artificial neural networks 2 (pp. 865-868). Amsterdam: Elsevier.
46. Peterson C., Anderson J.R. A mean field theory learning algorithm for neural networks. Complex Systems. 1:1987;995-1019.
47. Rescorla, R.A., & Wagner, A.R. (1972). A theory of Pavlovian conditioning: The effectiveness of reinforcement and non-reinforcement. In A.H. Black, & W.F. Prokasy (Eds.), Classical conditioning II: Current research and theory (pp. 64-69). New York: Appleton-Century Crofts.
48. Rissanen, J. (1989). Stochastic complexity in statistical inquiry. Singapore: World Scientific.
49. Rubin D.B., Thayer D.T. EM algorithms for ML factor analysis. Psychometrika. 47:1982;69-76.
50. Rumelhart D.E., Hinton G.E., Williams R.J. Learning internal representations by back-propagating errors. Nature. 323:1986;533-536.
51. Rumelhart, D.E., Durbin, R., Golden, R., & Chauvin, Y. (1995). Backpropagation: The basic theory. In Y. Chauvin, & D.E. Rumelhart (Eds.), Backpropagation: Theory, architectures and applications (pp. 1-34). Hillsdale, NJ: Erlbaum Associates.
52. Saul, L., & Jordan, M.I. (1995). Boltzmann chains and hidden Markov models. In G. Tesauro, D.S. Touretzky, & T. Leen (Eds.), Advances in neural information processing systems 7. San Mateo, CA: Morgan Kaufmann.
53. Saul, L.K., Jaakkola, T., & Jordan, M.I. (1995). Mean field theory for sigmoid belief networks. Computational Cognitive Science Technical Report 9501, MIT.
54. Saund E. A multiple cause mixture model for unsupervised learning. Neural Computation. 7:1994;51-71.
55. 3) time complexity learning algorithm for fully recurrent continually running networks. Neural Computation. 4:1992;243-248.
56. Somers, D.C., Nelson, S.B., & Sur, M. (1995). An emergent model of orientation selectivity in cat visual cortical simple cells. Journal of Neuroscience (in press).
57. Sutton R.S. Learning to predict by the methods of temporal difference. Machine Learning. 3:1988;9-44.
58. Sutton R. Dyna, an integrated architecture for learning, planning and reacting. Working Notes of the 1991 AAAI Spring Symposium on Integrated Intelligent Architecture and SIGART Bulletin. 2:1991;160-163.
59. Tresp, V., Ahmand, S., & Neunier, R. (1994). Training neural networks with deficient data. In J.D. Cowan, G. Tesauro, & J. Alspector (Eds.), Advances in neural information processing systems 6 (pp. 128-135). San Mateo, CA: Morgan Kaufmann.
60. Widrow, B., & Stearns, S.D. (1985). Adaptive signal processing. Englewood Cliffs, NJ: Prentice-Hall.
61. Williams, C.K.I., & Hinton, G. (1991). Mean field networks that learn to disriminate temporally distorted strings. In D. Touretzky, J. Elman, & T. Sejnowlski (Eds.), Connectionist models: Proceedings of the 1990 Summer School (pp. 18-22). San Mateo, CA: Morgan Kaufmann.
62. Williams R.J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning. 8:1992;229-256.
63. Williams R.J., Zipser D. A learning algorithm for continually running fully recurrent neural networks. Neural Computation. 1:1989;270-280.
64. Xu, L., & Jordan, M.I. (1995). On convergence properties of the EM algorithm for Gaussian mixtures. CBCL Paper; 111, Centre for Biological and Computational Learning, MIT.
65. Zemel, R.S. (1994). A minimum description length framework for unsupervised learning. PhD Dissertation, Computer Science, University of Toronto, Canada.