Net2Net: Accelerating learning via knowledge transfer

4th International Conference on Learning Representations, ICLR 2016 - Conference Track Proceedings

Volumn , Issue , 2016, Pages

  Chen, Tianqi ^a   Goodfellow, Ian ^a   Shlens, Jonathon ^a  

^a GOOGLE INC   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

KNOWLEDGE MANAGEMENT;

DESIGN PROCESS; KNOWLEDGE TRANSFER; KNOWLEDGE TRANSFER MECHANISMS; PRE-TRAINING; REAL-WORLD; STATE OF THE ART; WORK-FLOWS;

NEURAL NETWORKS;

EID: 85083953532     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (20)

1. Abadi, Martín, Agarwal, Ashish, Barham, Paul, Brevdo, Eugene, Chen, Zhifeng, Citro, Craig, Corrado, Greg S., Davis, Andy, Dean, Jeffrey, Devin, Matthieu, Ghemawat, Sanjay, Goodfellow, Ian, Harp, Andrew, Irving, Geoffrey, Isard, Michael, Jia, Yangqing, Jozefowicz, Rafal, Kaiser, Lukasz, Kudlur, Manjunath, Leven-berg, Josh, Mané, Dan, Monga, Rajat, Moore, Sherry, Murray, Derek, Olah, Chris, Schuster, Mike, Shlens, Jonathon, Steiner, Benoit, Sutskever, Ilya, Talwar, Kunal, Tucker, Paul, Vanhoucke, Vincent, Vasudevan, Vijay, Viégas, Fernanda, Vinyals, Oriol, Warden, Pete, Wattenberg, Martin, Wicke, Martin, Yu, Yuan, and Zheng, Xiaoqiang. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL http://tensorflow.org/. Software available from tensorflow.org.
2. Buciluǎ, Cristian, Caruana, Rich, and Niculescu-Mizil, Alexandru. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535–541. ACM, 2006.
3. Fahlman, Scott E. and Lebiere, Christian. The cascade-correlation learning architecture. pp. 524–532, Denver, CO, 1990. Morgan Kaufmann, San Mateo.
4. Glorot, X., Bordes, A., and Bengio, Y. Deep sparse rectifier neural networks. In AISTATS’2011, 2011.
5. Goodfellow, Ian J., Warde-Farley, David, Mirza, Mehdi, Courville, Aaron, and Bengio, Yoshua. Maxout networks. In ICML’2013, 2013.
6. Gutstein, Steven, Fuentes, Olac, and Freudenthal, Eric. Knowledge transfer in deep convolutional neural nets. International Journal on Artificial Intelligence Tools, 17(03):555–567, 2008.
7. Hinton, Geoffrey, Vinyals, Oriol, and Dean, Jeff. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
8. Hinton, Geoffrey E., Osindero, Simon, and Teh, Yee Whye. A fast learning algorithm for deep belief nets. Neural Computation, 18:1527–1554, 2006.
9. Ioffe, Sergey and Szegedy, Christian. Batch normalization: Accelerating deep network training by reducing internal covariate shift. 2015.
10. Jarrett, Kevin, Kavukcuoglu, Koray, Ranzato, Marc’Aurelio, and LeCun, Yann. What is the best multi-stage architecture for object recognition? In Proc. International Conference on Computer Vision (ICCV’09), pp. 2146–2153. IEEE, 2009.
11. Mahayri, Amjad, Ballas, Nicolas, and Courville, Aaron. FitNets and batch normalization. Technical report, (unpublished), 2015.
12. Mitchell, T., Cohen, W., Hruschka, E., Talukdar, P., Betteridge, J., Carlson, A., Dalvi, B., Gardner, M., Kisiel, B., Krishnamurthy, J., Lao, N., Mazaitis, K., Mohamed, T., Nakashole, N., Platanios, E., Ritter, A., Samadi, M., Settles, B., Wang, R., Wijaya, D., Gupta, A., Chen, X., Saparov, A., Greaves, M., and Welling, J. Never-ending learning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence (AAAI-15), 2015.
13. Romero, Adriana, Ballas, Nicolas, Ebrahimi Kahou, Samira, Chassang, Antoine, Gatta, Carlo, and Bengio, Yoshua. FitNets: Hints for thin deep nets. Technical Report Arxiv report 1412.6550, arXiv, 2014.
14. Silver, DL, Yang, Q, and Li, L. Lifelong machine learning systems: Beyond learning algorithms. In AAAI Spring Symposium-Technical Report, 2013.
15. Simonyan, Karen and Zisserman, Andrew. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
16. Socher, Richard, Bauer, John, Manning, Christopher D., and Ng, Andrew Y. Parsing with compositional vector grammars. In In Proceedings of the ACL conference, 2013.
17. Srivastava, Nitish, Hinton, Geoffrey, Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15: 1929–1958, 2014. URL http://jmlr.org/papers/v15/srivastava14a.html.
18. Szegedy, Christian, Liu, Wei, Jia, Yangqing, Sermanet, Pierre, Reed, Scott, Anguelov, Dragomir, Erhan, Du-mitru, Vanhoucke, Vincent, and Rabinovich, Andrew. Going deeper with convolutions. Technical report, arXiv:1409.4842, 2014.
19. Thrun, Sebastian. Lifelong learning: A case study. Technical Report CMU-CS-95-208, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA 15213, November 1995.
20. Tieleman, T and Hinton, G. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 4, 2012.