An empirical investigation of catastrophic forgetting in gradient-based neural networks

2nd International Conference on Learning Representations, ICLR 2014 - Conference Track Proceedings

Volumn , Issue , 2014, Pages

  Goodfellow, Ian J ^a   Mirza, Mehdi ^a   Xiao, Da ^a   Courville, Aaron ^a   Bengio, Yoshua ^a  

^a NONE

Author keywords

[No Author keywords available]

Indexed keywords

CHEMICAL ACTIVATION;

ACTIVATION FUNCTIONS; CATASTROPHIC FORGETTING; CATASTROPHIC FORGETTING PROBLEM; EMPIRICAL INVESTIGATION; GRADIENT-BASED NEURAL NETWORKS; MACHINE LEARNING MODELS; TRADE-OFF CURVES; TRAINING ALGORITHMS;

MACHINE LEARNING;

EID: 85083951527     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (18)

1. Bastien, Frédéric, Lamblin, Pascal, Pascanu, Razvan, Bergstra, James, Goodfellow, Ian J., Bergeron, Arnaud, Bouchard, Nicolas, and Bengio, Yoshua. Theano: new features and speed improvements. Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop, 2012.
2. Bergstra, James and Bengio, Yoshua. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13:281–305, February 2012.
3. Bergstra, James, Breuleux, Olivier, Bastien, Frédéric, Lamblin, Pascal, Pascanu, Razvan, Desjardins, Guillaume, Turian, Joseph, Warde-Farley, David, and Bengio, Yoshua. Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy), June 2010. Oral Presentation.
4. Blitzer, John, Dredze, Mark, and Pereira, Fernando. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In ACL’07, pp. 440–447, 2007.
5. Glorot, Xavier, Bordes, Antoine, and Bengio, Yoshua. Deep sparse rectifier neural networks. In JMLR W&CP: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (AISTATS 2011), April 2011a.
6. Glorot, Xavier, Bordes, Antoine, and Bengio, Yoshua. Domain adaptation for large-scale sentiment classification: A deep learning approach. In Proceedings of the Twenty-eight International Conference on Machine Learning (ICML’11), volume 27, pp. 97–110, June 2011b.
7. Goodfellow, Ian J., Warde-Farley, David, Lamblin, Pascal, Dumoulin, Vincent, Mirza, Mehdi, Pascanu, Razvan, Bergstra, James, Bastien, Frédéric, and Bengio, Yoshua. Pylearn2: a machine learning research library. arXiv preprint arXiv:1308.4214, 2013a.
8. Goodfellow, Ian J., Warde-Farley, David, Mirza, Mehdi, Courville, Aaron, and Bengio, Yoshua. Maxout networks. In Dasgupta, Sanjoy and McAllester, David (eds.), Proceedings of the 30th International Conference on Machine Learning (ICML’13), pp. 13191327. ACM, 2013b. URL http://icml.cc/2013/.
9. Hinton, Geoffrey E., Srivastava, Nitish, Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan. Improving neural networks by preventing co-adaptation of feature detectors. Technical report, arXiv:1207.0580, 2012.
10. Jarrett, Kevin, Kavukcuoglu, Koray, Ranzato, Marc'Aurelio, and LeCun, Yann. What is the best multi-stage architecture for object recognition? In Proc. International Conference on Computer Vision (ICCV’09), pp. 2146–2153. IEEE, 2009.
11. Martens, James and Sutskever, Ilya. Learning recurrent neural networks with Hessian-free optimization. In Proc. ICML’2011. ACM, 2011.
12. McClelland, J. L., McNaughton, B. L., and O’Reilly, R. C. Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory. Psychological Review, 102:419–457, 1995.
13. McClelland, James L.
14. McCloskey, M. and Cohen, N. J. Catastrophic interference in connectionist networks: The sequential learning problem. In Bower, G. H. (ed.), The Psychology of Learning and Motivation, Vol. 24, pp. 109–164. Academic Press, San Diego, CA, 1989.
15. Ratcliff, R. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2):285–308, April 1990. ISSN 0033-295X. URL http://view.ncbi.nlm.nih.gov/pubmed/2186426.
16. Srebro, Nathan and Shraibman, Adi. Rank, trace-norm and max-norm. In Proceedings of the 18th Annual Conference on Learning Theory, pp. 545–560. Springer-Verlag, 2005.
17. Srivastava, Nitish. Improving neural networks with dropout. Master’s thesis, U. Toronto, 2013.
18. Srivastava, Rupesh K, Masci, Jonathan, Kazerounian, Sohrob, Gomez, Faustino, and Schmidhuber, Jürgen. Compete to compute. In Burges, C.J.C., Bottou, L., Welling, M., Ghahramani, Z., and Weinberger, K.Q. (eds.), Advances in Neural Information Processing Systems 26, pp. 2310–2318. 2013. URL http://media.nips.cc/nipsbooks/nipspapers/paper_files/nips26/1109.pdf.