Measuring catastrophic forgetting in neural networks

32nd AAAI Conference on Artificial Intelligence, AAAI 2018

Volumn , Issue , 2018, Pages 3390-3398

  Kemker, Ronald ^a   McClure, Marc ^a   Abitino, Angelina ^b   Hayes, Tyler L ^a   Kanan, Christopher ^a  

^a ROCHESTER INSTITUTE OF TECHNOLOGY   (United States)

^b SWARTHMORE COLLEGE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; BIRDS;

CATASTROPHIC FORGETTING; CATASTROPHIC FORGETTING PROBLEM; DIFFERENT MECHANISMS; FLOWER RECOGNITION; INCREMENTAL TRAINING; MACHINE PERCEPTION; OPTIMAL PERFORMANCE; STATE-OF-THE-ART SYSTEM;

DEEP NEURAL NETWORKS;

EID: 85057759997     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (37)

1. Abraham, W. C., and Robins, A. 2005. Memory retention-the synaptic stability versus plasticity dilemma. Trends in Neurosciences 28(2):73-78.
2. Abu-El-Haija, S.; Kothari, N.; Lee, J.; et al. 2016. Youtube-8m: A large-scale video classification benchmark. arXiv:1609.08675.
3. Coop, R.; Mishtal, A.; and Arel, I. 2013. Ensemble learning in fixed expansion layer networks for mitigating catastrophic forgetting. IEEE Trans. on Neural Networks and Learning Systems 24(10):1623-1634.
4. Dai, W.; Yang, Q.; Xue, G.-R.; and Yu, Y. 2007. Boosting for transfer learning. In ICML, 193-200. ACM.
5. Draelos, T. J.; Miner, N. E.; Lamb, C. C.; Vineyard, C. M.; Carlson, K. D.; James, C. D.; and Aimone, J. B. 2016. Neu-rogenesis deep learning. arXiv:1612.03770.
6. Eich, J. M. 1982. A composite holographic associative recall model. Psych. Review 89(6):627.
7. Fernando, C.; Banarse, D.; Blundell, C.; Zwols, Y.; Ha, D.; Rusu, A. A.; Pritzel, A.; and Wierstra, D. 2017. Path-net: Evolution channels gradient descent in super neural networks. arXiv:1701.08734.
8. French, R. M. 1997. Pseudo-recurrent connectionist networks: An approach to the 'sensitivity-stability' dilemma. Connection Science 9(4):353-380.
9. French, R. M. 1999. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences 3(4):128-135.
10. Gemmeke, J. F.; Ellis, D. P. W.; Freedman, D.; Jansen, A.; Lawrence, W.; Moore, R. C.; Plakal, M.; and Ritter, M. 2017. Audio set: An ontology and human-labeled dataset for audio events. In ICASSP.
11. Gepperth, A., and Karaoguz, C. 2016. A bio-inspired incremental learning architecture for applied perceptual problems. Cognitive Computation 8(5):924-934.
12. Goodfellow, I. J.; Mirza, M.; Xiao, D.; Courville, A.; and Bengio, Y. 2013. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv:1312.6211.
13. Goodrich, B., and Arel, I. 2014. Unsupervised neuron selection for mitigating catastrophic forgetting in neural networks. In IEEE 57th Int. Midwest Symposium on Circuits and Systems (MWSCAS), 2014, 997-1000. IEEE.
14. He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In CVPR, 770-778.
15. Hershey, S.; Chaudhuri, S.; Ellis, D. P.; et al. 2017. Cnn architectures for large-scale audio classification. In ICASSP.
16. Hinton, G. E., and Plaut, D. C. 1987. Using fast weights to deblur old memories. In Proc. of the Ninth Annual Conference of the Cognitive Science Society, 177-186.
17. Kafle, K., and Kanan, C. 2017. Visual question answering: Datasets, algorithms, and future challenges. Computer Vision and Image Understanding.
18. Kanerva, P. 1988. Sparse distributed memory. MIT press.
19. Khilari, P., and Bhope, V. 2015. A review on speech to text conversion methods. Int. J. of Advanced Research in Computer Engineering and Technology 4:3067-3072.
20. Kirkpatrick, J.; Pascanu, R.; Rabinowitz, N.; Veness, J.; et al. 2017. Overcoming catastrophic forgetting in neural networks. Proc. of the National Academy of Sciences 201611835.
21. Kruschke, J. K. 1992. Alcove: An exemplar-based connectionist model of category learning. Psych. review 99(1):22.
22. Long, J.; Shelhamer, E.; and Darrell, T. 2015. Fully convolutional networks for semantic segmentation. In CVPR, 3431-3440.
23. McCloskey, M., and Cohen, N. J. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. Psych. of Learning & Motivation 24:109-165.
24. Mnih, V.; Kavukcuoglu, K.; Silver, D.; Graves, A.; Antonoglou, I.; Wierstra, D.; and Riedmiller, M. 2013. Playing atari with deep reinforcement learning. In NIPS Deep Learning Workshop.
25. Murdock, B. B. 1983. A distributed memory model for serial-order information. Psych. Review 90(4):316.
26. Murre, J. M. 2014. Learning and categorization in modular neural networks. Psych. Press.
27. Polikar, R.; Upda, L.; Upda, S. S.; and Honavar, V. 2001. Learn++: An incremental learning algorithm for supervised neural networks. IEEE Trans. on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 31(4):497-508.
28. Ren, B.; Wang, H.; Li, J.; and Gao, H. 2017. Life-long learning based on dynamic combination model. Applied Soft Computing 56:398-404.
29. Robins, A. 1995. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science 7(2):123-146.
30. Russakovsky, O.; Deng, J.; Su, H.; Krause, J.; Satheesh, S.; Ma, S.; Huang, Z.; Karpathy, A.; Khosla, A.; Bernstein, M.; et al. 2015. Imagenet large scale visual recognition challenge. IJCV 115(3):211-252.
31. Schroff, F.; Kalenichenko, D.; and Philbin, J. 2015. Facenet: A unified embedding for face recognition and clustering. In CVPR, 815-823.
32. Sharkey, N. E., and Sharkey, A. J. 1995. An analysis of catastrophic interference. Connection Science 7:301-329.
33. Soltoggio, A.; Stanley, K. O.; and Risi, S. 2017. Born to learn: the inspiration, progress, and future of evolved plastic artificial neural networks. arXiv:1703.10371.
34. Triki, A. R.; Aljundi, R.; Blaschko, M. B.; and Tuytelaars, T. 2017. Encoder based lifelong learning. arXiv:1704.01920.
35. Wah, C.; Branson, S.; Welinder, P.; Perona, P.; and Belongie, S. 2011. The caltech-ucsd birds-200-2011 dataset. Tech Report: CNS-TR-2011-001.
36. Wang, H.; Fan, W.; Yu, P. S.; and Han, J. 2003. Mining concept-drifting data streams using ensemble classifiers. 226-235. ACM.
37. Yu, L., and Liu, H. 2003. Feature selection for high-dimensional data: A fast correlation-based filter solution. In ICML, 856-863.