Algorithm-Dependent Generalization Bounds for Multi-Task Learning

IEEE Transactions on Pattern Analysis and Machine Intelligence

Volumn 39, Issue 2, 2017, Pages 227-241

  Liu, Tongliang ^a   Tao, Dacheng ^a   Song, Mingli ^b   Maybank, Stephen J ^c  

^a UNIVERSITY OF TECHNOLOGY SYDNEY   (Australia)

^b ZHEJIANG UNIVERSITY   (China)

^c UNIVERSITY OF LONDON   (United Kingdom)

Author keywords

generalization; inductive bias; learning theory; learning to learn; Multi Task learning; regularization; stability

Indexed keywords

CONVERGENCE OF NUMERICAL METHODS; LEARNING SYSTEMS; LINEARIZATION; PARAMETER ESTIMATION;

GENERALIZATION; INDUCTIVE BIAS; LEARNING THEORY; LEARNING TO LEARN; MULTITASK LEARNING; REGULARIZATION;

LEARNING ALGORITHMS;

EID: 85009732680     PISSN: 01628828     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPAMI.2016.2544314     Document Type: Article

References (54)

1. R. Caruna, "Multitask learning: A knowledge-based source of inductive bias, " in Proc. Int. Conf. Mach. Learn., 1993, pp. 41-48.
2. X. Wang, C. Zhang, and Z. Zhang, "Boosted multi-Task learning for face verification with applications to web image and video search, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2009, pp. 142-149.
3. Y. Zhang and D.-Y. Yeung, "Multi-Task warped gaussian process for personalized age estimation, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2010, pp. 2622-2629.
4. T. Zhang, B. Ghanem, S. Liu, and N. Ahuja, "Robust visual tracking via multi-Task sparse learning, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2012, pp. 2042-2049.
5. G. Pillonetto, F. Dinuzzo, and G. De Nicolao, "Bayesian online multitask learning of Gaussian processes, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 2, pp. 193-205, Feb. 2010.
6. R. Collobert and J. Weston, "A unified architecture for natural language processing: Deep neural networks with multitask learning, " in Proc. 25th Int. Conf. Mach. Learn., 2008, pp. 160-167.
7. A. Argyriou, T. Evgeniou, and M. Pontil, "Convex multi-Task feature learning, " Mach. Learn., vol. 73, no. 3, pp. 243-272, 2008.
8. P. Gong, J. Zhou, W. Fan, and J. Ye, "Efficient multi-Task feature learning with calibration, " in Proc. 20th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2014, pp. 761-770.
9. X.-L. Zhang, "Convex discriminative multitask clustering, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 37, no. 1, pp. 28-40, Jan. 2015.
10. J. Chen, L. Tang, J. Liu, and J. Ye, "A convex formulation for learning a shared predictive structure from multiple tasks, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 5, pp. 1025-1038, May 2013.
11. J. Baxter, "A model of inductive bias learning, " J. Artif. Intell. Res., vol. 12, no. 1, pp. 149-198, 2000.
12. S. Ben-David and R. Schuller, "Exploiting task relatedness for multiple task learning, " in Proc. 16th Annu. Conf. Computational Learn. Theory 7th Kernel Workshop, 2003, pp. 567-580.
13. R. K. Ando and T. Zhang, "A framework for learning predictive structures from multiple tasks and unlabeled data, " J. Mach. Learn. Res., vol. 6, pp. 1817-1853, 2005.
14. A. Maurer, "Bounds for linear multi-Task learning, " J. Mach. Learn. Res., vol. 7, pp. 117-139, 2006.
15. A. Maurer, "Transfer bounds for linear feature learning, " Mach. Learn., vol. 75, no. 3, pp. 327-350, 2009.
16. A. Maurer, M. Pontil, and B. Romera-Paredes, "Sparse coding for multitask and transfer learning, " in Proc. 30th Int. Conf. Mach. Learn., 2013, pp. 343-351.
17. C. A. Micchelli and M. Pontil, "Kernels for multi-Task learning, " in Proc. Adv. Neural Inf. Process. Syst., 2004, pp. 921-928.
18. J. Shawe-Taylor, P. L. Bartlett, R. C. Williamson, and M. Anthony, "Structural risk minimization over data-dependent hierarchies, " IEEE Trans. Inf. Theory, vol. 44, no. 5, pp. 1926-1940, Sep. 1998.
19. V. Vapnik, The Nature of Statistical Learning Theory. New York, NY, USA: Springer, 2000.
20. P. Bartlett, S. Kulkarni, and S. Posner, "Covering numbers for realvalued function classes, " IEEE Trans. Inf. Theory, vol. 43, no. 5, pp. 1721-1724, Sep. 1997.
21. T. Zhang, "Covering number bounds of certain regularized linear function classes, " J. Mach. Learn. Res., vol. 2, pp. 527-550, 2002.
22. D.-X. Zhou, "Capacity of reproducing kernel spaces in learning theory, " IEEE Trans. Inf. Theory, vol. 49, no. 7, pp. 1743-1752, Jul. 2003.
23. Y. Guo, P. Bartlett, J. Shawe-Taylor, and R. Williamson, "Covering numbers for support vector machines, " IEEE Trans. Inf. Theory, vol. 48, no. 1, pp. 239-250, Jan. 2002.
24. V. Koltchinskii, "Rademacher penalties and structural risk minimization, " IEEE Trans. Inf. Theory, vol. 47, no. 5, pp. 1902-1914, Jul. 2001.
25. P. L. Bartlett and S. Mendelson, "Rademacher and Gaussian complexities: Risk bounds and structural results, " J. Mach. Learn. Res., vol. 3, pp. 463-482, 2003.
26. O. Bousquet and A. Elisseeff, "Stability and generalization, " J. Mach. Learn. Res., vol. 2, pp. 499-526, 2002.
27. Y. Zhang, "Multi-Task learning and algorithmic stability, " in Proc. 29th AAAI Conf. Artif. Intell., 2015, pp. 3181-3187.
28. J. Audiffren and H. Kadri, "Stability of multi-Task kernel regression algorithms, " in Proc. ACML, 2013, pp. 1-16.
29. A. Elisseeff and M. Pontil, "Leave-one-out error and stability of learning algorithms with applications, " NATO Sci. Series Sub Series iii Comput. Syst. Sci., vol. 190, pp. 111-130, 2003.
30. I. Kuzborskij and F. Orabona, "Stability and hypothesis transfer learning, " in Proc. 30th Int. Conf. Mach. Learn., 2013, pp. 942-950.
31. A. Argyriou, M. Pontil, Y. Ying, and C. A. Micchelli, "A spectral regularization framework for multi-Task structure learning, " in Proc. Adv. Neural Inf. Process. Syst. 20, 2007, pp. 25-31.
32. T. K. Pong, P. Tseng, S. Ji, and J. Ye, "Trace norm regularization: Reformulations, algorithms, and multi-Task learning, " SIAM J. Optimization, vol. 20, no. 6, pp. 3465-3489, 2010.
33. M. Pontil and A. Maurer, "Excess risk bounds for multitask learning with trace norm regularization, " in Proc. COLT, 2013, pp. 55-76.
34. K. Lounici, M. Pontil, A. B. Tsybakov, and S. van de Geer, "Taking advantage of sparsity in multi-Task learning, " in Proc. COLT, 2009, pp. 73-82.
35. T. Evgeniou and M. Pontil, "Regularized multi-Task learning, " in Proc. 10th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2004, pp. 109-117.
36. J. Chen, L. Tang, J. Liu, and J. Ye, "A convex formulation for learning shared structures from multiple tasks, " in Proc. 26th Annu. Int. Conf. Mach. Learn., 2009, pp. 137-144.
37. P. Rai and H. Daume, "Infinite predictor subspace models for multitask learning, " in Proc. 13th Int. Conf. Artif. Intell. Statist., 2010, pp. 613-620.
38. J. Zhou, J. Chen, and J. Ye, "Clustered multi-Task learning via alternating structure optimization, " in Proc. Adv. Neural Inf. Process. Syst. 24, 2011, pp. 702-710.
39. A. Kumar and H. Daume, "Learning task grouping and overlap in multi-Task learning, " in Proc. 29th Int. Conf. Mach. Learn., 2012, pp. 1383-1390.
40. B. Lin, S. Yang, C. Zhang, J. Ye, and X. He, "Multi-Task vector field learning, " in Proc. Adv. Neural Inf. Process. Syst. 25, 2012, pp. 296-304.
41. Q. Liu, X. Liao, H. Carin, J. Stack, and L. Carin, "Semisupervised multitask learning, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 6, p. 1074-1086, Jun. 2009.
42. M. Kearns and D. Ron, "Algorithmic stability and sanity-check bounds for leave-one-out cross-validation, " Neural Comput., vol. 11, no. 6, pp. 1427-1453, 1999.
43. S. Shalev-Shwartz, O. Shamir, N. Srebro, and K. Sridharan, "Learnability, stability and uniform convergence, " J. Mach. Learn. Res., vol. 11, pp. 2635-2670, 2010.
44. P. Massart, "Some applications of concentration inequalities to statistics, " in Proc. Annales de la Faculte des Sci. de Toulouse, 2000, pp. 245-303.
45. N. Sauer, "On the density of families of sets, " J. Combinatorial Theory, Series A, vol. 13, no. 1, pp. 145-147, 1972.
46. R. M. Dudley, "The sizes of compact subsets of hilbert space and continuity of Gaussian processes, " J. Functional Anal., vol. 1, no. 3, pp. 290-330, 1967.
47. E. Hazan and S. Kale, "Beyond the regret minimization barrier: An optimal algorithm for stochastic strongly-convex optimization, " in Proc. COLT, 2011, pp. 421-436.
48. A. Rakhlin, O. Shamir, and K. Sridharan, "Making gradient descent optimal for strongly convex stochastic optimization, " in Proc. 29th Int. Conf. Mach. Learn., 2012, pp. 449-456.
49. K. I. Tsianos and M. G. Rabbat, "Distributed strongly convex optimization, " in Proc. Annu. Allerton Conf. Commun., Control, Comput., 2012, pp. 593-600.
50. S. M. Kakade and A. Tewari, "On the generalization ability of online strongly convex programming algorithms, " in Proc. 30th Int. Conf. Mach. Learn., 2009, pp. 441-449.
51. M. Mohri, A. Rostamizadeh, and A. Talwalkar, Foundations of Machine Learning. Cambridge, MA, USA: MIT Press, 2012.
52. C. McDiarmid, "Concentration, " in Probabilistic Methods for Algorithmic Discrete Mathematics. New Yor, NY, USA: Springer, 1998, pp. 195-248.
53. W. Hoeffding, "Probability inequalities for sums of bounded random variables, " J. Am. Statist. Assoc., vol. 58, no. 301, pp. 13-30, 1963.
54. I. Pinelis, "Optimum bounds for the distributions of martingales in Banach spaces, " Ann. Probability, vol. 22, pp. 1679-1706, 1994.