Generalization in interactive networks: The benefits of inhibitory competition and Hebbian learning

Neural Computation

Volumn 13, Issue 6, 2001, Pages 1199-1241

  O'Reilly, Randall C ^a  

^a UNIVERSITY OF COLORADO   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHM; ANIMAL; BIOLOGICAL MODEL; COMPUTER SIMULATION; HUMAN; LEARNING; NERVE CELL NETWORK; PHYSIOLOGY; PSYCHOLOGICAL MODEL; REVIEW;

ALGORITHMS; ANIMALS; COMPUTER SIMULATION; HUMANS; LEARNING; MODELS, NEUROLOGICAL; MODELS, PSYCHOLOGICAL; NERVE NET;

EID: 0035380627     PISSN: 08997667     EISSN: None     Source Type: Journal    
DOI: 10.1162/08997660152002834     Document Type: Review

References (78)

1. Ackley, D. H., Hinton, G. E., & Sejnowski, T. J. (1985). A learning algorithm for Boltzmann machines. Cognitive Science, 9, 147-169.
2. Almeida, L. B. (1987). A learning rule for asynchronous perceptrons with feedback in a combinatorial environment. In M. Caudil & C. Butler (Eds.), Proceedings of the IEEE First International Conference on Neural Networks (pp. 609-618). San Diego, CA.
3. Artola, A., Brocher, S., & Singer, W. (1990). Different voltage-dependent thresholds for inducing long-term depression and long-term potentiation in slices of rat visual cortex. Nature, 347, 69-72.
4. Barlow, H. B. (1989). Unsupervised learning. Neural Computation, 1, 295-311.
5. Bear, M. F. (1996). A synaptic basis for memory storage in the cerebral cortex. Proceedings of the National Academy of Sciences, 93, 3453.
6. Bear, M. F., & Malenka, R. C. (1994). Synaptic plasticity: LTP and LTD. Current Opinion in Neurobiology, 4, 389-399.
7. Brousse, O. (1993). Generativity and systematicity in neural network combinatorial learning (Tech. Rep. No. CU-CS-676-93). Boulder, CO: University of Colorado at Boulder, Department of Computer Science.
8. Crick, F. H. C. (1989). The recent excitement about neural networks. Nature, 337, 129-132.
9. Dayan, P., & Zemel, R. S. (1995). Competition and multiple cause models. Neural Computation, 7, 565-579.
10. Douglas, R. J., & Martin, K. A. C. (1990). Neocortex. In G. M. Shepherd (Ed.), The synaptic organization of the brain (pp. 389-438). Oxford: Oxford University Press.
11. Felleman, D. J., & Van Essen, D. C. (1991). Distributed hierarchical processing in the primate cerebral cortex. Cerebral Cortex, 1, 1-47.
12. Field, D. J. (1994). What is the goal of sensory coding? Neural Computation, 6(4), 559-601.
13. Földiák, P. (1990). Forming sparse representations by local anti-Hebbian learning. Biological Cybernetics, 64(2), 165-170.
14. Fukai, T., & Tanaka, S. (1997). A simple neural network exhibiting selective activation of neuronal ensembles: From winner-take-all to winners-share-all. Neural Computation, 9, 77-97.
15. Gabbot, P. L. A., & Somogyi, P. (1986). Quantitative distribution of GABA-immunoreactive neurons in the visual cortex (area 17) of the cat. Experimental Brain Research, 61, 323-331.
16. Geman, S., Bienenstock, E. L., & Doursat, R. (1992). Neural networks and the bias/variance dilemma. Neural Computation, 4, 1-58.
17. Glushko, R. J. (1979). The organization and activation of orthographic knowledge in reading aloud. Journal fo Experimental Psychology, 5, 674-691.
18. Goodhill, G. J., & Barrow, H. G. (1994). The role of weight normalization in competitive learning. Neural Computation, 6, 255-269.
19. Grinvald, A., Frostig, R. D., & Lieke, E. (1988). Optical imaging of neuronal activity. Physiological Review, 68, 1285-1366.
20. Grossberg, S. (1976). Adaptive pattern classification and universal recoding I: Parallel development and coding of neural feature detectors. Biological Cybernetics, 23, 121-134.
21. Harm, M. W., & Seidenberg, M. S. (1999). Phonology, reading acquisition, and dyslexia: Insights from connectionist models. Psychological Review, 106, 491-528.
22. Hinton, G. E. (1989a). Connectionist learning procedures. Artificial Intelligence, 40, 185-234.
23. Hinton, G. E. (1989b). Deterministic Boltzmann learning performs steepest descent in weight-space. Neural Computation, 1, 143-150.
24. Hinton, G. E., & Ghahramani, Z. (1997). Generative models for discovering sparse distributed representations. Philosophical Transactions of the Royal Society (London) B, 352, 1177-1190.
25. Hinton, G. E., & McClelland, J. L. (1988). Learning representations by recirculation. In D. Z. Anderson (Ed.), Neural information processing systems, 1987 (pp. 358-366). New York: American Institute of Physics.
26. Hopfield, J. J. (1982). Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79, 2554-2558.
27. Hopfield, J. J. (1984). Neurons with graded response have collective computational properties like those of two-state neurons. Proceedings of the National Academy of Sciences, 81, 3088-3092.
28. Kohonen, T. (1984). Self-organization and associative memory. New York: Springer-Verlag.
29. Levitt, J. B., Lewis, D. A., Yoshioka, T., & Lund, J. S. (1993). Topography of pyramidal neuron intrinsic connections in macaque monkey prefrontal cortex (areas 9 and 46). Journal of Comparative Neurology, 338, 360-376.
30. Lisman, J. E. (1989). A mechanism for the Hebb and the anti-Hebb processes underlying learning and memory. Proceedings of the National Academy of Sciences, 86, 9574-9578.
31. Lisman, J. (1994). The CaM Kinase II hypothesis for the storage of synaptic memory. Trends in Neurosciences, 17, 406.
32. Majani, E., Erlarson, R., & Abu-Mostafa, Y. (1989). The induction of multiscale temporal structure. In D. S. Touretzky (Ed.), Advances in neural information processing systems, 1 (pp. 634-642). San Mateo, CA: Morgan Kaufmann.
33. Marcus, G. F. (1998). Rethinking eliminative connectionism. Cognitive Psychology, 37, 243.
34. Marr, D. (1971). Simple memory: A theory for archicortex. Philosophical Transactions of the Royal Society (London) B, 262, 23-81.
35. McCann, R. S., & Besner, D. (1987). Reading pseudohomophones: Implications for models of pronunciation and the locus of word-frequency effects in word naming. Journal of Experimental Psychology, 13, 14-24.
36. McClelland, J. L., McNaughton, B. L., & O'Reilly, R. C. (1995). Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory. Psychological Review, 102, 419-457.
37. McClelland, J. L., & Rumelhart, D. E. (1981). An interactive activation model of context effects in letter perception: Part 1. An account of basic findings. Psychological Review, 88(5), 375-407.
38. Miller, K. D., Keller, J. B., & Stryker, M. P. (1989). Ocular dominance column development: Analysis and simulation. Science, 245, 605-615.
39. Miller, K. D., & MacKay, D. J. C. (1994). The role of constraints in Hebbian learning. Neural Computation, 6, 100-126.
40. Movellan, J. R. (1990). Contrastive Hebbian learning in the continuous Hopfield model. In D. S. Touretzky, G. E. Hinton, & T. J. Sejnowski (Eds.), Proceedings of the 1989 Connectionist Models Summer School (pp. 10-17). San Mateo, CA: Morgan Kaufman.
41. Noelle, D. C., & Cottrell, G. W. (1996). In search of articulated attractors. In G. W. Cottrell (Ed.), Proceedings of the 18th Annual Conference of the Cognitive Science Society (pp. 329-334). Hillsdale, NJ: Erlbaum.
42. Noelle, D. C., & Zimdars, A. L. (1999). Methods for learning articulated attractors over internal representations. In M. Hahn, & S. C. Stoness (Eds.), Proceedings of the 21st Annual Conference of the Cognitive Science Society (pp. 480-485). Hillsdale, NJ: Erlbaum.
43. Nowlan, S. J. (1990). Maximum likelihood competitive learning. In D. S. Touretzky (Ed.), Advances in neural information processing systems, 2 (pp. 574-582). San Mateo, CA: Morgan Kaufmann.
44. Oja, E. (1982). A simplified neuron model as a principal component analyzer. Journal of Mathematical Biology, 15, 267-273.
45. Olshausen, B. A., & Field, D. J. (1996). Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381, 607.
46. O'Reilly, R. C. (1996a). Biologically plausible error-driven learning using local activation differences: The generalized recirculation algorithm. Neural Computation, 8(5), 895-938.
47. O'Reilly, R. C. (1996b). The Leabra model of neural interactions and learning in the neocortex. Unpublished doctoral dissertation, Carnegie Mellon University, Pittsburgh, PA.
48. O'Reilly, R. C. (1998). Six principles for biologically-based computational models of cortical cognition. Trends in Cognitive Sciences, 2(11), 455-462.
49. O'Reilly, R. C., & McClelland, J. L. (1994). Hippocampal conjunctive encoding, storage, and recall: Avoiding a tradeoff. Hippocampus, 4(6), 661-682.
50. O'Reilly, R. C., & Munakata, Y. (2000). Computational explorations in cognitive neuroscience: Understanding the mind by simulating the brain. Cambridge, MA: MIT Press.
51. O'Reilly, R. C., & Rudy, J. W. (2000). Computational principles of learning in the neocortex and hippocampus. Hippocampus, 10, 389-397.
52. O'Reilly, R. C., & Rudy, J. W. (in press). Conjunctive representations in learning and memory: Principles of cortical and hippocampal function. Psychological Review.
53. Pineda, F. J. (1987a). Generalization of backpropagation to recurrent and higher order neural networks. In D. Z. Anderson (Ed.), Proceedings of the IEEE Conference on Neural Information Processing Systems (pp. 602-611). New York: IEEE.
54. Pineda, F. J. (1987b). Generalization of backpropagation to recurrent neural networks. Physical Review Letters, 18, 2229-2232.
55. Pineda, F. J. (1988). Dynamics and architecture for neural computation. Journal of Complexity, 4, 216-245.
56. Pinker, S., & Prince, A. (1988). On language and connectionism: Analysis of a parallel distributed processing model of language acquisition. Cognition, 28, 73-193.
57. Plaut, D. C., & McClelland, J. L. (1993). Generalization with componential attractors: Word and nonword reading in an attractor network. Proceedings of the 15th Annual Conference of the Cognitive Science Society (pp. 824-829). Hillsdale, NJ: Erlbaum.
58. Plaut, D. C., McClelland, J. L., Seidenberg, M. S., & Patterson, K. E. (1996). Understanding normal and impaired word reading: Computational principles in quasi-regular domains. Psychological Review, 103, 56-115.
59. Plunkett, K., & Marchman, V. A. (1996). Learning from a connectionist model of the acquisition of the English past tense. Cognition, 61, 299.
60. Poggio, T., & Girosi, F. (1990). Regularization algorithms for learning that are equivalent to multilayer networks. Science, 247, 978-982.
61. Rubinstein, Y. D., & Hastie, T. (1997). Discriminative vs. informative learning. In D. Heckerman, H. Mannila, & D. Pregibon (Eds.), Proceedings of the Third International Conference on Knowledge Discovery and Data Mining (pp. 49-53). Menlo Park, CA: AAAI Press.
62. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning internal representations by error propagation. In D. E. Rumelhart, J. L. McClelland, & PDP Research Group (Eds.), Parallel distributed processing. Vol. 1: Foundations (pp. 318-362). Cambridge, MA: MIT Press.
63. Rumelhart, D. E., & McClelland, J. L. (1986). On learning the past tenses of English verbs. In J. L. McClelland, D. E. Rumelhart, & PDP Research Group (Eds.), Parallel distributed processing. Vol. 2: Psychological and biological models (pp. 216-271). Cambridge, MA: MIT Press.
64. Rumelhart, D. E., & Zipser, D. (1986). Feature discovery by competitive learning. In D. E. Rumelhart, J. L. McClelland, & PDP Research Group (Eds.), Parallel distributed processing. Vol. 1: Foundations (pp. 151-193). Cambridge, MA: MIT Press.
65. Saund, E. (1995). A multiple cause mixture model for unsupervised learning. Neural Computation, 7, 51-71.
66. Seidenberg, M. S., & McClelland, J. L. (1989). A distributed, developmental model of word recognition and naming. Psychological Review, 96, 523-568.
67. Smolensky, P. (1986). Information processing in dynamical systems: Foundations of harmony theory. In D. E. Rumelhart, J. L. McClelland, & PDP Research Group (Eds.), Parallel distributed processing. Vol. 1: Foundations (pp. 282-317). Cambridge, MA: MIT Press.
68. Taraban, R., & McClelland, J. L. (1987). Conspiracy effects in word pronunciation. Journal of Memory and language, 26, 608-631.
69. Vapnik, V. N., & Chervonenkis, A. (1971). On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability and Its Applications, 16, 264-280.
70. Vecera, S. P., & O'Reilly, R. C. (1998). Figure-ground organization and object recognition processes: An interactive account. Journal of Experimental Psychology: Human Perception and Performance, 24, 441-462.
71. Weigend, A. (1994). On overfitting and the effective number of hidden units. In M. C. Mozer, P. Smolensky, & A. S. Weigend (Eds.), Proceedings of the 1993 Connectionist Models Summer School (pp. 335-342). Hillsdale, NJ: Erlbaum.
72. Weigend, A. S., Rumelhart, D. E., & Huberman, B. A. (1991). Generalization by weight-elimination with application to forecasting. In R. P. Lippmann, J. E. Moody, & D. S. Touretzky (Eds.), Advances in neural information processing systems, 3 (pp. 875-882). San Mateo, CA: Morgan Kaufmann.
73. White, E. L. (1989). Cortical circuits: Synaptic organization of the cerebral cortex, structure, function, and theory. Boston: Birkhäuser.
74. Wolpert, D. H. (1992). On the connection between in-sample testing and generalization error. Complex Systems, 6, 47-94.
75. Wolpert, D. H. (1996a). The existence of a priori distinctions between learning algorithms. Neural Computation, 8, 1391-1420.
76. Wolpert, D. H. (1996b). The lack of a priori distinctions between learning algorithms. Neural Computation, 8, 1341-1390.
77. Zemel, R. S. (1993). A minimum description length framework for unsupervised learning. Unpublished doctoral dissertation, University of Toronto, Canada.
78. Zipser, D., & Andersen, R. A. (1988). A backpropagation programmed network that simulates response properties of a subset of posterior parietal neurons. Nature, 331, 679-684.