Continual learning through synaptic intelligence

34th International Conference on Machine Learning, ICML 2017

Volumn 8, Issue , 2017, Pages 6072-6082

  Zenke, Friedemann ^a   Poole, Ben ^a   Ganguli, Surya ^a  

^a STANFORD UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPLEX NETWORKS; COMPUTATIONAL EFFICIENCY; MACHINERY; NEURAL NETWORKS;

BIOLOGICAL COMPLEXITY; BIOLOGICAL NEURAL NETWORKS; CLASSIFICATION TASKS; CONTINUAL LEARNING; DATA DISTRIBUTION; DIVERSE APPLICATIONS; MOLECULAR MACHINERY; TASK RELEVANT;

DEEP LEARNING;

EID: 85048533047     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (27)

1. Benna, Marcus K. and Fusi, Stefano. Computational principles of synaptic memory consolidation. Nat Neurosci, advance online publication, October 2016. ISSN 1097-6256. doi:10.1038/nn.4401.
2. Choy, Min Chee, Srinivasan, Dipti, and Cheu, Ruey Long. Neural networks for continuous online learning and control. IEEE Trans Neural Netw, 17(6): 1511-1531, November 2006. ISSN 1045-9227. doi:10.1109/TNN.2006.881710.
3. Donahue, Jeff, Jia, Yangqing, Vinyals, Oriol, Hoffman, Judy, Zhang, Ning, Tzeng, Eric, and Darrell, Trevor. Decaf: A deep convolutional activation feature for generic visual recognition. In International Conference in Machine Learning (ICML), 2014.
4. Fukushima, Kunihiko and Miyake, Sci. Neocognitron: A Self-Organizing Neural Network Model for a Mechanism of Visual Pattern Recognition. In Competition and Cooperation in Neural Nets, pp. 267-285. Springer, Berlin, Heidelberg, 1982. doi:10.1007/978-3-642-46466-9.18.
5. Fusi, Stefano, Drew, Patrick J., and Abbott, Larry F. Cascade models of synaptically stored memories. Neuron, 45(4): 599-611, February 2005. ISSN 0896-6273. doi:10.1016/j.neuron.2005.02.001.
6. Goodfellow, Ian J., Mirza, Mehdi, Xiao, Da, Courville, Aaron, and Bengio, Yoshua. An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks. arXiv.1312.6211 [cs, stat], December 2013. arXiv: 1312.6211.
7. Hinton, Geoffrey, Vinyals, Oriol, and Dean, Jeff. Distilling the knowledge in a neural network. NIPS Deep Learning and Representation Learning Workshop, 2014.
8. Jung, Heechul, Ju, Jeongwoo, Jung, Minju, and Kim, Junmo. Less-forgetting Learning in Deep Neural Networks. arXiv:1607.00122 [cs], July 2016. arXiv: 1607.00122.
9. Kingma, Diederik and Ba, Jimmy. Adam: A Method for Stochastic Optimization. arXiv: 1412.6980 [cs], December 2014. arXiv: 1412.6980.
10. Kirkpatrick, James, Pascanu, Razvan, Rabinowitz, Neil, Veness, Joel, Desjardins, Guillaume, Rusu, Andrei A., Milan, Kieran, Quan, John, Ramalho, Tiago, Grabska-Barwinska, Agnieszka, Hassabis, Demis, Clopath, Claudia, Kumaran, Dharshan, and Hadsell, Raia. Overcoming catastrophic forgetting in neural networks. PNAS, pp. 201611835, March 2017. ISSN 0027-8424, 1091-6490. doi:10.1073/pnas.1611835114.
11. Krizhevsky, Alex and Hinton, Geoffrey. Learning multiple layers of features from tiny images. 2009.
12. Lahiri, Subhaneil and Ganguli, Surya. A memory frontier for complex synapses. In Advances in Neural Information Processing Systems, Volume 26, pp. 1034-1042, Tahoe, USA, 2013. Curran Associates, Inc.
13. LeCun, Yann, Cortes, Corinna, and Burges, Christopher JC. The MNIST database of handwritten digits. 1998.
14. LeCun, Yann, Bengio, Yoshua, and Hinton, Geoffrey. Deep learning. Nature, 521(7553): 436-444, May 2015. ISSN 0028-0836. doi:10.1038/nature14539.
15. Li, Zhizhong and Hoiem, Derek. Learning without forgetting. In European Conference on Computer Vision, pp. 614-629. Springer, 2016.
16. Martens, James. Second-order optimization for neural networks. PhD thesis, University of Toronto, 2016.
17. Montgomery, Johanna M. and Madison, Daniel V. State-Dependent Heterogeneity in Synaptic Depression between Pyramidal Cell Pairs. Neuron, 33(5): 765-777, February 2002. ISSN 0896-6273. doi:10.1016/S0896-6273(02)00606-2.
18. Pascanu, Razvan and Bengio, Yoshua. Revisiting natural gradient for deep networks. arXiv preprint arXiv: 1301.3584, 2013.
19. Razavian, Ali Sharif, Azizpour, Hossein, Sullivan, Josephine, and Carlsson, Stefan. Cnn features off-the-shelf: an astounding baseline for recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 806-813, 2014.
20. Redondo, Roger L. and Morris, Richard G. M. Making memories last: the synaptic tagging and capture hypothesis. Nat Rev Neurosci, 12(1): 17-30, January 2011. ISSN 1471-003X. doi:10.1038/nrn2963.
21. Rosenblatt, Frank. The perceptron: A probabilistic model for information storage and organization in the brain. Psychological review, 65(6): 386, 1958.
22. Rusu, Andrei A., Rabinowitz, Neil C, Desjardins, Guillaume, Soyer, Hubert, Kirkpatrick, James, Kavukcuoglu, Koray, Pascanu, Razvan, and Hadsell, Raia. Progressive Neural Networks. arXiv: 1606.04671[cs], June 2016. arXiv: 1606.04671.
23. Srivastava, Rupesh K, Masci, Jonathan, Kazerounian, Sohrob, Gomez, Faustino, and Schmidhuber, Juergen. Compete to Compute. In Burges, C. J. C, Bottou, L., Welling, M., Ghahramani, Z., and Weinberger, K. Q. (eds.), Advances in Neural Information Processing Systems 26, pp. 2310-2318. Curran Associates, Inc., 2013.
24. Yosinski, Jason, Clune, Jeff, Bengio, Yoshua, and Lipson, Hod. How transferable are features in deep neural networks? In Advances in neural information processing systems, pp. 3320-3328, 2014.
25. Zenke, Friedemann, Agnes, Everton J., and Gerstner, Wulfram. Diverse synaptic plasticity mechanisms orchestrated to form and retrieve memories in spiking neural networks. Nat Commun, 6, April 2015. doi:10.1038/ncomms7922.
26. Zhou, Qiang, Tao, Huizhong W., and Poo, Mu-Ming. Reversal and Stabilization of Synaptic Modifications in a Developing Visual System. Science, 300(5627): 1953-1957, June 2003. doi:10.1126/science.1082212.
27. Ziegler, Lorric, Zenke, Friedemann, Kastner, David B., and Gerstner, Wulfram. Synaptic Consolidation: From Synapses to Behavioral Modeling. J Neurosci, 35(3): 1319-1334, January 2015. ISSN 0270-6474, 1529-2401. doi:10.1523/JNEUROSCI.3989-14.2015.