A theoretical framework for deep transfer learning

Information and Inference

Volumn 5, Issue 2, 2016, Pages 159-209

  Galanti, Tomer ^a   Wolf, Lior ^a   Hazan, Tamir ^b  

^a TEL AVIV UNIVERSITY   (Israel)

^b TECHNION ISRAEL INSTITUTE OF TECHNOLOGY   (Israel)

Author keywords

Deep learning; PAC learning; PAC Bayesian; Transfer learning

Indexed keywords

EID: 85054903248     PISSN: None     EISSN: 20498772     Source Type: Journal    
DOI: 10.1093/imaiai/iaw008     Document Type: Article

References (46)

1. Ando, R. K., Zhang, T. & Bartlett, P. (2005) A framework for learning predictive structures from multiple tasks and unlabeled data. J. Mach. Learn. Res., 6, 1817-1853.
2. Baxter, J. (2000) A model of inductive bias learning. J. Artif. Intell. Res. (JAIR), 12, 149-198.
3. Ben-David, S., Blitzer, J., Crammer, K. & Pereira, F. (2007) Analysis of representations for domain adaptation. Adv. Neural Inf. Process. Syst., 19, 137.
4. Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F. & Vaughan, J. W. (2010a) A theory of learning from different domains. Mach. Learn., 79, 151-175.
5. Ben-David, S., Lu, T., Luu, T. & Pál, D. (2010b) Impossibility theorems for domain adaptation. International Conference on Artificial Intelligence and Statistics, pp. 129-136.
6. Blumer, A., Ehrenfeucht, A., Haussler, D. & Warmuth, M. K. (1989) Learnability and the vapnikchervonenkis dimension. J. ACM, 36, 929-965.
7. Bousquet, O. & Elisseeff, A. (2002) Stability and generalization. J. Mach. Learn. Res., 2, 499-526.
8. Cortes, C., Mohri, M., Riley, M. & Rostamizadeh, A. (2008) Sample selection bias correction theory. Algorithmic Learning Theory. Berlin, Heidelberg, New York: Springer, pp. 38-53.
9. Crammer, K., Kearns, M. & Wortman, J. (2008) Learning from multiple sources. J. Mach. Learn. Res., 9, 1757-1774.
10. Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E. & Darrell, T. (2013) Decaf: A deep convolutional activation feature for generic visual recognition. CoRR, abs/1310.1531.
11. Fei-Fei, L., Fergus, R. & Perona, P. (2006) One-shot learning of object categories. IEEE Trans. Pattern Anal. Mach. Intell., 28, 594-611.
12. Girshick, R., Donahue, J., Darrell, T. & Malik, J. (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on. IEEE, pp. 580-587.
13. Hardt, M., Recht, B. & Singer, Y. (2015) Train faster, generalize better: Stability of stochastic gradient descent. ArXiv e-prints, September.
14. Kakade, S. M. & Tewari, A. (2008) VC dimension of multilayer neural networks, range queries. Lecture notes.
15. Kifer, D., Ben-David, S. & Gehrke, J. (2004) Detecting change in data streams. Proceedings of the 30th International Conference on Very Large Data Bases-Volume 30. VLDB Endowment, pp. 180-191.
16. Krizhevsky, A., Sutskever, I. & Hinton, G. E. (2012) Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, pp. 1097-1105.
17. Kuzborskij, I. & Orabona, F. (2013) Stability and hypothesis transfer learning. Proceedings of The 30th International Conference on Machine Learning, pp. 942-950.
18. Kuzborskij, I., Orabona, F. & Caputo, B. (2013) From n to n + 1: Multiclass transfer incremental learning. 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp. 3358-3365.
19. Li, X. & Bilmes, J. (2007) A Bayesian divergence prior for classiffier adaptation. International Conference on Artificial Intelligence and Statistics, pp. 275-282.
20. Mansour, Y., Mohri, M. & Rostamizadeh, A. (2009a) Domain adaptation: Learning bounds and algorithms. COLT 2009-The 22nd Conference on Learning Theory, Montreal, Quebec, Canada, June 18-21.
21. Mansour, Y., Mohri, M. & Rostamizadeh, A. (2009b) Domain adaptation with multiple sources. Advances in Neural Information Processing Systems, pp. 1041-1048.
22. McAllester, D. A. (1998) Some PAC-Bayesian theorems. Proceedings of the Eleventh Annual Conference on Computational Learning Theory, Madison, WI, 1998. ACM, pp. 230-234.
23. McAllester, D. (2013) A PAC-Bayesian tutorial with A dropout bound. CoRR, abs/1307.2118.
24. Mukherjee, S., Niyogi, P., Poggio, T. & Rifkin, R. (2002) Statistical learning: Stability is sufficient for generalization and necessary and sufficient for consistency of empirical risk minimization. Technical Report, Advances in Computational Mathematics.
25. Orabona, F., Castellini, C., Caputo, B., Fiorilla, A. E. & Sandini, G. (2009) Model adaptation with leastsquares SVM for adaptive hand prosthetics. IEEE International Conference on Robotics and Automation, 2009. ICRA'09. IEEE, pp. 2897-2903.
26. Pentina, A. & Lampert, C. H. (2014) A pac-bayesian bound for lifelong learning. Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, June 21-26, pp. 991-999.
27. Poggio, T., Anselmi, F. & Rosasco, L. (2015) On invariance and selectivity in representation learning. Inf. Inference: J. IMA. 11, 2635-2670.
28. Razavian, A. S., Azizpour, H., Sullivan, J. & Carlsson, S. (2014) CNN features off-the-shelf: An astounding baseline for recognition. CoRR, abs/1403.6382.
29. Rumelhart, D. E., Hinton, G. E. & Williams, R. J. (1988) Learning representations by back-propagating errors. Neurocomputing: Foundations of Research. Cambridge, MA, USA: MIT Press, pp. 696-699.
30. Sauer, N. (1972) On the density of families of sets. J. Comb. Theory Ser. A, 13, 145-147.
31. Shalev-Shwartz, S. & Ben-David, S. (2014) Understanding Machine Learning: From Theory to Algorithms. New York, NY, USA: Cambridge University Press.
32. Shalev-Shwartz, S., Shamir, O., Srebro, N. & Sridharan, K. (2010) Learnability, stability and uniform convergence. JMLR.
33. Sukhbaatar, S. & Fergus, R. (2014) Learning from noisy labels with deep neural networks. CoRR, abs/1406.2080.
34. Taigman, Y., Yang, M., Ranzato, M. & Wolf, L. (2014) Deepface: Closing the gap to human-level performance in face verification. Conference on Computer Vision and Pattern Recognition (CVPR).
35. Taigman, Y., Yang, M., Ranzato, M. & Wolf, L. (2015) Web-scale training for face identification. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2746-2754.
36. Tishby, N., Pereira, F. C. N. & Bialek, W. (2000) The information bottleneck method. CoRR, physics/0004057.
37. Tishby, N. & Zaslavsky, N. (2015) Deep learning and the information bottleneck principle. 2015 IEEE Information Theory Workshop, ITW 2015, Jerusalem, Israel, April 26-May 1, pp. 1-5.
38. Tommasi, T., Orabona, F. & Caputo, B. (2010) Safety in numbers: Learning categories from few examples with multi model knowledge transfer. 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp. 3081-3088.
39. Tommasi, T., Orabona, F. & Caputo, B. (2014) Learning categories from few examples with multi model knowledge transfer. IEEE Trans. Pattern Anal. Mach. Intell., 36, 928-941.
40. Valiant, L. G. (1984) A theory of the learnable. Commun. ACM, 27, 1134-1142.
41. Vapnik, V. (1998) Statistical Learning Theory. Wiley.
42. Vapnik, V. N. (1995) The Nature of Statistical Learning Theory. New York, NY, USA: Springer-Verlag New York, Inc.
43. Vapnik, V. N. & Chervonenkis, A. Ya. (1971) On the uniform convergence of relative frequencies of events to their probabilities. Theory Probab. Appl., 16, 264-280.
44. Yang, Y. & Hospedales, T. M. (2014) A unified perspective on multi-domain and multi-task learning. CoRR, abs/1412.7489.
45. Yang, J., Yan, R. & Hauptmann, A. G. (2007) Cross-domain video concept detection using adaptive svms. Proceedings of the 15th International Conference on Multimedia. ACM, pp. 188-197.
46. Yosinski, J., Clune, J., Bengio, Y. & Lipson, H. (2014) How transferable are features in deep neural networks? CoRR, abs/1411.1792.