Ensemble learning in fixed expansion layer networks for mitigating catastrophic forgetting

IEEE Transactions on Neural Networks and Learning Systems

Volumn 24, Issue 10, 2013, Pages 1623-1634

  Coop, Robert ^a   Mishtal, Aaron ^a   Arel, Itamar ^a  

^a University of Tennessee Knoxville   (United States)

Author keywords

Catastrophic forgetting; nonstationary inputs; sparse encoding neural networks

Indexed keywords

CATASTROPHIC FORGETTING; CLASSIFICATION TASKS; ENSEMBLE LEARNING; HIDDEN LAYERS; LAYER NETWORKS; NONSTATIONARY; REGRESSION PROBLEM; SYSTEM CONTROL;

ARTIFICIAL INTELLIGENCE; ENCODING (SYMBOLS); PLASTICITY TESTING;

FEEDFORWARD NEURAL NETWORKS;

ALGORITHM; ARTIFICIAL NEURAL NETWORK; COMPUTER SIMULATION; FEEDBACK SYSTEM; LEARNING; MEMORY;

ALGORITHMS; COMPUTER SIMULATION; FEEDBACK; LEARNING; MEMORY; NEURAL NETWORKS (COMPUTER);

EID: 84885190141     PISSN: 2162237X     EISSN: 21622388     Source Type: Journal    
DOI: 10.1109/TNNLS.2013.2264952     Document Type: Article

References (28)

1. B. Ans and S. Rousset, Avoiding catastrophic forgetting by coupling two reverberating neural networks Comptes Rendus l?Académie Sci., Ser. III, Sci. Vie, vol. 320, no. 12, pp. 989-997, 1997.
2. B. Ans, S. Rousset, R. M. French, and S. Musca, Self-refreshing memory in artificial neural networks: Learning temporal sequences without catastrophic forgetting Connect. Sci., vol. 16, no. 2, pp. 71-99, 2004.
3. L. Breiman, Bagging predictors Mach. Learn., vol. 24, no. 2, pp. 123-140, Aug. 1996.
4. G. Brown and J. Wyatt, Negative correlation learning and the ambiguity family of ensemble methods in Multiple Classifier Systems. New York, NY, USA: Springer-Verlag, 2003, p. 161.
5. R. Elwell and R. Polikar, Incremental learning of concept drift in nonstationary environments IEEE Trans. Neural Netw., vol. 22, no. 10, pp. 1517-1531, Oct. 2011.
6. D. M. Endres and J. E. Schindelin, A new metric for probability distributions IEEE Trans. Inf. Theory, vol. 49, no. 7, pp. 1858-1860, Jul. 2003.
7. M. Frean and A. Robins, Catastrophic forgetting in simple networks: An analysis of the pseudorehearsal solution Netw., Comput. Neural Syst., vol. 10, no. 3, pp. 227-236, 1999.
8. R. M. French, Using semi-Distributed representations to overcome catastrophic forgetting in connectionist networks Connect. Sci., vol. 4, nos. 3-4, pp. 365-378, 1992.
9. R. M. French, Catastrophic interference in connectionist networks: Can it be predicted, can it be prevented? in Advances in Neural Information Processing Systems. Cambridge, MA, USA: MIT Press, 1994, p. 1176.
10. R. M. French, Using pseudo-recurrent connectionist networks to solve the problem of sequential learning in Proc. 19th Annu. Cognit. Sci. Soc. Conf., 1997, pp. 1-6.
11. R. M. French and A. Ferrara, Modeling time perception in rats: Evidence for catastrophic interference in animal learning in Proc. 21st Annu. Conf. Cognit. Sci. Conf., 1999, pp. 173-178.
12. M. Hattori, Dual-network memory model using a chaotic neural network in Proc. Int. Joint. Conf. Neural Netw., 2010, pp. 1-5.
13. G. E. Hinton and D. C. Plaut, Using fast weights to deblur old memories in Proc. 9th Annu. Conf. Cognit. Sci. Soc., 1987, pp. 177-186.
14. Y. Lecun and C. Cortes. (1998). The MNIST Database of Handwritten Digits[Online]. Available: Http://yann.lecun.com/exdb/mnist/
15. H. Lee, A. Battle, R. Raina, and A. Y. Ng, Efficient sparse coding algorithms in Advances in Neural Information Processing Systems, vol. 19. Cambridge, MA, USA: MIT Press, 2007, p. 801.
16. H. Lejeune, A. Ferrara, F. Simons, and J.H. Wearden, Adjusting to changes in the time of reinforcement: Peak interval transitions in rats J. Experim. Psychol., Animal Behavior Process., vol. 23, no. 2, pp. 211-231, 1997.
17. J. P. Levy and D. Bairaktaris, Connectionist dual-weight architectures Lang. Cognit. Process., vol. 10, nos. 3-4, pp. 265-283, 1995.
18. J. Lin, Divergence measures based on the shannon entropy IEEE Trans. Inf. Theory, vol. 37, no. 1, pp. 145-151, Jan. 1991.
19. M. McCloskey and N. J. Cohen, Catastrophic interference in connectionist networks: The sequential learning problem Psychol. Learn. Motivat., vol. 24, pp. 109-165, Jan. 1989.
20. O.-m. Moe-helgesen and H. Stranden, Catastophic forgetting in neural networks Dept. Comput. & Information Sci., Norwegian Univ. Science & Technology (NTNU), Trondheim, Norway, Tech. Rep., 2005.
21. M. D. Muhlbaier, A. Topalis, and R. Polikar, Learn++ .NC: Combining ensemble of classifiers with dynamically weighted consult-and-vote for efficient incremental learning of new classes IEEE Trans. Neural Netw., vol. 20, no. 1, pp. 152-168, Jan. 2009.
22. I. Mukherjee, C. Rudin, and R. Schapire, The rate of convergence of AdaBoost in Proc. 24th Annu. COLT, 2011, pp. 537-558.
23. N. C. Oza, R. Polikar, J. Kittler, and F. Roli, Multiple classifier systems in Proc. 6th Int. Workshop MCS, Seaside, CA, USA, Jun. 2005.
24. R. Ratcliff, Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions Psychol. Rev., vol. 97, no. 2, pp. 285-308, 1990.
25. A. Robins, Catastrophic forgetting in neural networks: The role of rehearsal mechanisms in Proc. 1st New Zealand Int. Two-stream Conf. Artif. Neural Netw. Expert Syst., 1993, pp. 65-68.
26. A. Robins, Catastrophic forgetting, rehearsal and pseudorehearsal Connect. Sci., vol. 7, no. 2, pp. 123-146, 1995.
27. S. Wang, K. Tang, and X. Yao, Diversity exploration and negative correlation learning on imbalanced data sets in Proc. Int. Joint Conf. Neural Netw., Jun. 2009, pp. 3259-3266.
28. S. Wang and X. Yao, The effectiveness of a new negative correlation learning algorithm for classification ensembles in Proc. IEEE Int. Conf. Data Mining Workshops, Dec. 2010, pp. 1013-1020.