Sequential learning in neural networks: A review and a discussion of pseudorehearsal based methods

Intelligent Data Analysis

Volumn 8, Issue 3, 2004, Pages 301-322

  Robins, Anthony ^a  

^a UNIVERSITY OF OTAGO   (New Zealand)

Author keywords

catastrophic forgetting; consolidation; pseudorehearsal; rehearsal; sequential learning

Indexed keywords

CONSOLIDATION; NEURAL NETWORKS;

CATASTROPHIC FORGETTING; CATASTROPHIC FORGETTING PROBLEM; EFFECTIVE SOLUTION; PSEUDOREHEARSAL; REHEARSAL; SEQUENTIAL LEARNING;

BACKPROPAGATION;

EID: 13144253819     PISSN: 1088467X     EISSN: 15714128     Source Type: Journal    
DOI: 10.3233/ida-2004-8306     Document Type: Review

References (68)

1. B. Ans and S. Rousset, Avoiding catastrophic forgetting by coupling two reverberating neural networks, C. R. Acad. Sci. Paris, Life Sciences 320 (1997), 989-997.
2. B. Ans and S. Rousset, Neural Networks With a Self-refreshing Memory: Knowledge transfer in Sequential Learning Tasks without Catastrophic Forgetting, Connection Science 12 (2000), 1-19.
3. B. Ans, S. Rousset, R.M. French and S. Musca, Preventing catastrophic interference in multiple-sequence learning using coupled reverberating Elman networks, Proceedings of the 24th Annual Conference of the Cognitive Science Society, Earlbaum, NJ, 2002, pp. 71-76.
4. B. Ans, S. Rousset, S. Musca and R.M. French, Self-refreshing memory in artificial neural networks: Learning temporal sequences without catastrophic forgetting, Submitted to Cognitive Science (2003).
5. N. Burgess, J.L. Shapiro and M.A. Moore, Neural network models of list learning, Network 2 (1991), 399-422.
6. L. Bottu and V. Vapnik, Local Learning Algorithms, Neural Computation 4 (1992), 888-900.
7. G. Carpenter and S. Grossberg, A massively parallel architecture for a self-organizing neural pattern recognition machine, Computer Vision, Graphics, and Image Processing 37 (1987), 54-115.
8. G. Carpenter and S. Grossberg, The ART of adaptive pattern recognition by a self-organizing neural network, Computer 21 (1988), 77-88.
9. G.A. Christos, Investigation of the Crick-Mitchison reverse-learning dream sleep hypothesis in a dynamical setting, Neural Networks 9 (1996), 427-434.
10. F. Crick and G. Mitchison, The function of dream sleep, Nature 304 (1983), 111-114.
11. R. Duda and P. Hart, Pattern Classification and Scene Analysis, Wiley, New York, 1973.
12. T. Eastman, Catastrophic Interference in Constructive Neural Networks, MSc Thesis (in progress), Otago University, New Zealand, 2003.
13. S. Fahlman and C. Lebiere, The cascade-correlation learning architecture, in: Advances in Neural Information Processing Systems II, D. Touretzky, ed., Morgan Kauffman, San Mateo, 1990, pp. 524-532.
14. J.A. Feldman and D.H. Ballard, Connectionist models and their properties, Cognitive Science 6 (1982), 205-254.
15. M. Frean, A thermal perceptron learning rule, Neural Computation 4 (1992), 946-957.
16. M. Frean and A. Robins, Catastrophic forgetting in simple networks: An analysis of the pseudorehearsal solution, Network: Computation in Neural Systems 10 (1999), 227-236.
17. R.M. French, Semi-distributed representations and catastrophic forgetting in connectionist networks, Connection Science 4(3&4) (1992), 365-377.
18. R.M. French, Dynamically constraining connectionist networks to produce distributed, orthogonal representations to reduce catastrophic interference, Proceedings of the 16th Annual Cognitive Science Society Conference, Earlbaum, Hillsdale NJ, 1994, pp. 335-340.
19. R.M. French, Interactive connectionist networks: An approach to the sensitivity-stability dilemma, Connection Science 9 (1997), 353-380.
20. R.M. French, Selective memory loss in aphasics: An insight from pseudo-recurrent connectionist networks, in: Connectionist representations: Proceedings of the fourth annual computation and psychology workshop, J. Bullanaria, G. Houghton and D. Glasspool, eds, Springer Verlag, Berlin, 1997, pp. 183-195.
21. R.M. French, Catastrophic forgetting in connectionist networks: causes, consequences and solutions, Trends in Cognitive Science 3(4) (1999), 128-135.
22. R.M. French, B. Ans and S. Rousset, Pseudopatterns and dual-network memory models: advantages and shortcomings, in: Connectionist Models of Learning, Development and Evolution, R.M. French and J.P. Sougńe, eds, Springer, Berlin, 2001, pp. 13-22.
23. R.M. French and N. Chater, Using noise to compute error surfaces in connectionist networks: a novel means of reducing catastrophic forgetting, Neural Computation 14 (2002), 1755-1769.
24. R.M. French and D. Mareschal, Could category-specific semantic deficits reflect differences in the distribution of features within a unified semantic memory? Proceedings of the 20th Annual Conference of the Cognitive Science Society, Lawrence Earlbaum, New Jersey, 1998, pp. 374-379.
25. E. Gardner, Maximum storage capacity in neural networks, Europhysics Letters 4 (1987), 257-270.
26. E. Gardner, N. Stroud and D. Wallace, Training with noise and the storage of correlated patterns in a neural network model, Journal of Physics A: Mathematics and General 22 (1989), 2019-2030.
27. S. Grossberg, Competitive learning: From interactive activation to adaptive resonance, Cognitive Science 11 (1987), 23-63.
28. S. Haykin, Neural Networks A Comprehensive Foundation (Second edition), Prentice-Hall, New Jersey, 1999.
29. J. Hertz, A. Krough and R.G. Palmer, Introduction to the Theory of Neural Computation, Addison-Wesley, Redwood City CA, 1991.
30. P.A. Hetherington and M.S. Seidenberg, Is there catastrophic interference in connectionist networks? in Proceedings of the Eleventh Annual Conference of the Cognitive Science Society, Lawrence Earlbaum, Hillsdale NJ, 1989, pp. 26-33.
31. G.E. Hinton and D.C. Plaut, Using Fast Weights to Deblur Old Memories, Proceedings of the 9th Annual Conference of the Cognitive Science Society, Earlbaum, Hillsdale NJ, 1987, pp. 177-186.
32. J.J. Hopfield, Neural networks and physical systems with emergent collective computational abilities, Proceedings of the National Academy of Sciences, USA 79 (1982), 2554-2558.
33. J.J. Hopfield, D.I. Feinstein and R.G. Palmer, Unlearning has a stabilising effect in collective memories, Nature 304 (1983), 158-159.
34. C.A. Kortge, Episodic memory in connectionist networks, Proceedings of the 12th Annual Conference of the Cognitive Science Society, Lawrence Earlbaum, Hillsdale NJ, 1990, pp. 764-771.
35. J.K. Kruschke, ALCOVE: An Exemplar-based Connectionist Model of Category Learning, Psychological Review 99 (1992), 22-44.
36. S. Lewandowsky, Gradual unlearning and catastrophic interference: A comparison of distributed architectures, in: Relating Theory and Data: Essays on Human Memory in Honour of Bennet B. Murdok, Lawrence Earlbaum, W.E. Hockley and S. Lewandowsky, eds, Hillsdale NJ, 1991, pp. 445-476.
37. S. Lewandowsky and S. Li, Catastrophic interference in neural networks: causes, solutions, and data, in: New Perspectives on Interference and Inhibition in Cognition, F.N. Dempster and C. Brainerd, eds, Academic Press, San Diego, 1995, pp. 329-361.
38. S. McCallum and A. Robins Mechanisms for memory consolidation, New Zealand Journal of Computing 7 (1999), 13-20.
39. J.L. McClelland, B.L. McNaughton and R.C. O'Reilly, Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory, Psychological Review 102(3) (1995), 419-457.
40. M. McCloskey and N.J. Cohen, Catastrophic interference in connectionist networks: The sequential learning problem, in: The Psychology of Learning and Motivation, (Vol. 23), G.H. Bower, ed., Academic Press, New York, 1989, pp. 109-164.
41. K. McRae and P.A. Hetherington, Catastrophic interference is eliminated in pretrained networks, Proceedings of the Fifteenth Annual Meeting of the Cognitive Science Society, Lawrence Earlbaum, Hillsdale NJ, 1993, pp. 723-728.
42. M. Meeter, Control of consolidation in neural networks: Avoiding runaway effects, Submitted to Connection Science, 2003.
43. J. Moody and C. Darken, Fast Learning in Networks of Locally-Tuned Processing Units, Neural Computation 1 (1989), 281-294.
44. P.M. Murphy and D.W. Aha, UCI Repository of Machine Learning Databases [http://www.ics.uci.edu/mlearn/MLRepository.html], University of California, Department of Information and Computer Science, Irvine CA, 1994.
45. J. Murre, Learning and Categorization in Modular Neural Networks, Earlbaum, Hillsdale NJ, 1992.
46. J. Murre, The effects of pattern presentation on interference in back propagation networks, in Proceedings of the 14th Annual Conference of the Cognitive Science Society, Earlbaum, Hillsdale NJ, 1992, pp. 54-59.
47. J.P. Nadal, G. Toulouse, J.P. Changeux and S. Dehaene, Networks of formal neurons and memory palimpsets, Europhysics Letters 1 (1986), 535-542.
48. R. Ratcliff, Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions, Psychological Review 97 (1990), 285-308.
49. R. Ratcliff, S. Clark and R. Shiffrin, The list-strength effect: Data and discussion, Journal of Experimental Psychology: Learning, Memory and Cognition 16 (1990), 163-178.
50. R. Ratcliff and B. Murdock, Retrieval processes in recognition memory, Psychological Review 83 (1976), 190-214.
51. A. Robins, Catastrophic forgetting, rehearsal, and pseudorehearsal, Connection Science 7 (1995), 123-146.
52. A. Robins, Consolidation in neural networks and in the sleeping brain, Connection Science 8 (1996), 259-275.
53. A. Robins, Maintaining stability during new learning in neural networks, in Proceedings of the IEEE International Conference on Systems, Man, & Cybernetics (SMC97), IEEE Computer Society Press, Los Alamos, 1997, pp. 3013-3018.
54. A. Robins, Towards a framework for consolidation and transfer effects in neural network models, Dynamical cognitive science: Proceedings of the Fourth Australasian Cognitive Science Conference, CD-ROM, University of Newcastle, Australia, 1997.
55. A. Robins, Solutions to the catastrophic forgetting problem, Proceedings of The 20th Annual Conference of the Cognitive Science Society, Lawrence Earlbaum, New Jersey, 1998, pp. 899-904.
56. A. Robins and M. Frean, Local learning algorithms for sequential tasks in neural networks, Advanced Computational Intelligence 2 (1998), 221-227.
57. A. Robins and M. Frean, Learning and generalisation in a stable network, Progress in Connectionist-based Information Systems: Proceedings of the 1997 Conference on Neural Information Processing and Intelligent Information Systems, Springer-Verlag, Singapore, 1998, pp. 314-317.
58. A. Robins and S. McCallum, Catastrophic forgetting and the pseudorehearsal solution in Hopfield type networks, Connection Science 10 (1998), 121-135.
59. A. Robins and S. McCallum, The consolidation of learning during sleep: Comparing the pseudorehearsal and unlearning accounts, Neural Networks 12 (1999), 1191-1206.
60. A. Robins and S. McCallum, A robust method for distinguishing between learned and spurious attractors, Submitted to Neural Networks (2003).
61. N.E. Sharkey and A.J.C. Sharkey, Understanding Catastrophic Interference in Neural Nets, Technical Report CS-94-4, Department of Computer Science, University of Sheffield U.K., 1994.
62. N.E. Sharkey and A.J.C. Sharkey, Interference and Discrimination in Neural Net Memory, in: Connectionist Models of Memory and Language, J. Levy, D. Bairaktaris, J. Bullinaria and P. Cairns, eds, UCL Press, London, 1994.
63. N. Sharkey and A. Sharkey, An analysis of catastrophic interference, Connection Science 7 (1995), 301-329.
64. R. Shiffrin, R. Ratcliff and S. Clark, The list-strength effect II: Theoretical mechanisms, Journal of Experimental Psychology: Learning, Memory and Cognition 16 (1990), 179-195.
65. D. Silver, Selective transfer of neural network task knowledge, PhD thesis, University of Western Ontario, 2000.
66. D. Silver and R. Mercer, The task rehearsal method of sequential learning, Technical Report #517, Department of Computer Science, University of Western Ontario, 1998.
67. J. van Hemmen, Hebbian learning, its correlation catastrophe, and unlearning, Network 8 (1997), V1-V17.
68. S. Wimbauer, N. Klemmer and J. van Hemmen, Universality of unlearning, Neural Networks 7 (1994), 261-270.