A convex formulation for learning a shared predictive structure from multiple tasks

IEEE Transactions on Pattern Analysis and Machine Intelligence

Volumn 35, Issue 5, 2013, Pages 1025-1035

  Chen, Jianhui ^a   Tang, Lei ^b   Liu, Jun ^c   Ye, Jieping ^d  

^a GE GLOBAL RESEARCH   (United States)

^b WalmartLabs   (United States)

^c SIEMENS CORPORATE RESEARCH   (United States)

^d Arizona State University   (United States)

Author keywords

accelerated projected gradient; alternating structure optimization; Multitask learning; shared predictive structure

Indexed keywords

ALTERNATING STRUCTURE OPTIMIZATIONS (ASO); EFFECTIVENESS AND EFFICIENCIES; GENERALIZATION PERFORMANCE; MULTITASK LEARNING; PREDICTIVE STRUCTURES; PROJECTED GRADIENT; SEMI-DEFINITE PROGRAM (SDP); STRUCTURE OPTIMIZATION;

ALGORITHMS; GENE EXPRESSION; OPTIMAL SYSTEMS;

STRUCTURAL OPTIMIZATION;

EID: 84875472724     PISSN: 01628828     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPAMI.2012.189     Document Type: Article

References (54)

1. C.M. Bishop, Pattern Recognition and Machine Learning. Springer, 2006.
2. T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, 2009.
3. R. Caruana, "Multitask Learning," Machine Learning, vol. 28, no. 1, pp. 41-75, 1997. (Pubitemid 127507169)
4. B. Heisele, T. Serre, M. Pontil, T. Vetter, and T. Poggio, "Categorization by Learning and Combining Object Parts," Proc. Advances in Neural Information Processing System, 2001.
5. R.K. Ando and T. Zhang, "A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data," J. Machine Learning Research, vol. 6, pp. 1817-1853, 2005.
6. Y. Xue, X. Liao, L. Carin, and B. Krishnapuram, "Multi-Task Learning for Classification with Dirichlet Process Priors," J. Machine Learning Research, vol. 8, pp. 35-63, 2007. (Pubitemid 46155123)
7. S. Yu, V. Tresp, and K. Yu, "Robust Multi-Task Learning with t-Processes," Proc. Int'l Conf. Machine Learning, 2007.
8. A. Argyriou, T. Evgeniou, and M. Pontil, "Convex Multi-Task Feature Learning," Machine Learning, vol. 73, no. 3, pp. 243-272, 2008.
9. J. Zhou, J. Chen, and J. Ye, MALSAR: Multi-tAsk Learning via StructurAl Regularization. Arizona State Univ., 2012.
10. R.K. Ando, "BioCreative II Gene Mention Tagging System at IBM Watson," Proc. Second BioCreative Challenge Evaluation Workshop, 2007.
11. J. Bi, T. Xiong, S. Yu, M. Dundar, and R.B. Rao, "An Improved Multi-Task Learning Approach with Applications in Medical Diagnosis," Proc. European Conf. Machine Learning, 2008.
12. O. Chapelle, P. Shivaswamy, S. Vadrevu, K. Weinberger, Y. Zhang, and B. Tseng, "Boosted Multi-Task Learning," Machine Learning, vol. 85, pp. 149-173, 2011.
13. A. Quattoni, M. Collins, and T. Darrell, "Learning Visual Representations Using Images with Captions," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2007.
14. J. Li, Y. Tian, T. Huang, and W. Gao, "Probabilistic Multi-Task Learning for Visual Saliency Estimation in Video," Int'l J. Computer Vision, vol. 90, no. 2, pp. 150-165, 2010.
15. S. Thrun and J. O'Sullivan, "Discovering Structure in Multiple Learning Tasks: The TC Algorithm," Proc. Int'l Conf. Machine Learning, 1996.
16. J. Baxter, "A Model of Inductive Bias Learning," J. Artificial Intelligence Research, vol. 12, pp. 149-198, 2000.
17. B. Bakker and T. Heskes, "Task Clustering and Gating for Bayesian Multitask Learning," J. Machine Learning Research, vol. 4, pp. 83-99, 2003.
18. N.D. Lawrence and J.C. Platt, "Learning to Learn with the Informative Vector Machine," Proc. Int'l Conf. Machine Learning, 2004.
19. A. Schwaighofer, V. Tresp, and K. Yu, "Learning Gaussian Process Kernels via Hierarchical Bayes," Proc. Advances in Neural Information Processing Systems, 2004.
20. K. Yu, V. Tresp, and A. Schwaighofer, "Learning Gaussian Processes from Multiple Tasks," Proc. Int'l Conf. Machine Learning, 2005.
21. J. Zhang, Z. Ghahramani, and Y. Yang, "Learning Multiple Related Tasks Using Latent Independent Component Analysis," Proc. Advances in Neural Information Processing System, 2005.
22. L. Jacob, F. Bach, and J.-P. Vert, "Clustered Multi-Task Learning: A Convex Formulation," Proc. Advances in Neural Information Processing System, 2008.
23. T. Evgeniou and M. Pontil, "Regularized Multi-Task Learning," Proc. ACM SIGKDD Int'l Conf. Knowledge Discovery and Data Mining, 2004.
24. T. Evgeniou, C.A. Micchelli, and M. Pontil, "Learning Multiple Tasks with Kernel Methods," J. Machine Learning Research, vol. 6, pp. 615-637, 2005.
25. T. Jebara, "Multi-Task Feature and Kernel Selection for SVMs" Proc. Int'l Conf. Machine Learning, 2004.
26. G. Obozinski, B. Taskar, and M.I. Jordan, "Multi-Task Feature Selection," Proc. ICML Workshop Structural Knowledge Transfer for Machine Learning, 2006.
27. A. Argyriou, T. Evgeniou, and M. Pontil, "Multi-Task Feature Learning," Proc. Advances in Neural Information Processing System, 2006.
28. B. Recht, M. Fazel, and P.A. Parrilo, "Guaranteed Minimum Rank Solutions to Linear Matrix Equations via Nuclear Norm Minimization," SIAM Rev., vol. 52, no. 3, pp. 471-501, 2010.
29. S. Ji and J. Ye, "An Accelerated Gradient Method for Trace Norm Minimization," Proc. Int'l Conf. Machine Learning, 2009.
30. T.K. Pong, P. Tseng, S. Ji, and J. Ye, "Trace Norm Regularization: Reformulations, Algorithms, and Multi-Task Learning," SIAM J. Optimization, vol. 20, pp. 3465-3489, 2009.
31. J. Zhou, J. Chen, and J. Ye, "Clustered Multi-Task Learning via Alternating Structure Optimization," Proc. Advances in Neural Information Processing System, 2011.
32. D.P. Bertsekas, Nonlinear Programming. Athena Scientific, 1999.
33. A. Nemirovski, Efficient Methods in Convex Programming, Springer, 1995.
34. Y. Nesterov, Introductory Lectures on Convex Programming. Kluwer Academic Publishers, 1998.
35. N. Ueda and K. Saito, "Parametric Mixture Models for Multi-Labeled Text," Proc. Advances in Neural Information Processing System, 2002.
36. http://www.flyexpress.net/2012.
37. M.L. Overton and R.S. Womersley, "Optimality Conditions and Duality Theory for Minimizing Sums of the Largest Eigenvalues of Symmetric Matrics," Math. Programming, vol. 62, nos. 1-3, pp. 321-357, 1993.
38. S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge Univ. Press, 2004.
39. A. Argyriou, C.A. Micchelli, M. Pontil, and Y. Ying, "A Spectral Regularization Framework for Multi-Task Structure Learning," Proc. Advances in Neural Information Processing System, 2007.
40. G.H. Golub and C.F. Van Loan, Matrix Computations. Johns Hopkins Univ. Press, 1996.
41. J.F. Sturm, "Using SeDuMi 1.02, a MATLAB Toolbox for Optimization over Symmetric Cones," Optimization Methods and Software, vols. 11/12, pp. 625-653, 1999.
42. C.-C. Chang and C.-J. Lin, "LIBSVM: A Library for Support Vector Machines," ACM Trans. Intelligent Systems and Technology, vol. 2, pp. 27:1-27:27, 2011.
43. P. Brucker, "An O(n) Algorithm for Quadratic Knapsack Problems," Operations Research Letters, vol. 3, no. 3, pp. 163-166, 1984.
44. J.-J. Moreau, "Proximité et Dualité dans un Espace Hilbertien," Bull. Soc. Math. France, vol. 93, pp. 273-299, 1965.
45. J. Liu, S. Ji, and J. Ye, SLEP: Sparse Learning with Efficient Projections. Arizona State Univ., 2009.
46. F. Bach, R. Jenatton, J. Mairal, and G. Obozinski, "Convex Optimization with Sparsity-Inducing Norms," Optimization for Machine Learning, S. Sra, S. Nowozin, and S.J. Wright, eds., MIT Press, 2011.
47. J. Chen, J. Liu, and J. Ye, "Learning Incoherent Sparse and Low-Rank Patterns from Multiple Tasks," Proc. 16th ACM SIGKDD Int'l Conf. Knowledge Discovery and Data Mining, 2010.
48. D.D. Lewis, Y. Yang, T.G. Rose, and F. Li, "RCV1: A New Benchmark Collection for Text Categorization Research," J. Machine Learning Research, vol. 5, pp. 361-397, 2004.
49. D.D. Lewis, "Evaluating Text Categorization," Proc. Speech and Natural Language Workshop, 1991.
50. C.C. Fowlkes, C.L.L. Hendriks, S.V. Keränen, G.H. Weber, O. Rü bel, M.-Y. Huang, S. Chatoor, A.H. DePace, L. Simirenko, C. Henriquez, A. Beaton, R. Weiszmann, S. Celniker, B. Hamann, D.W. Knowles, M.D. Biggin, M.B. Eisen, and J. Malik, "A Quantitative Spatiotemporal Atlas of Gene Expression in the Drosophila Blastoderm," Cell, vol. 133, no. 2, pp. 364-374, 2008. (Pubitemid 351508300)
51. E. Lécuyer, H. Yoshida, N. Parthasarathy, C. Alm, T. Babak, T. Cerovina, T.R. Hughes, P. Tomancak, and H.M. Krause, "Global Analysis of mRNA Localization Reveals a Prominent Role in Organizing Cellular Architecture and Function," Cell, vol. 131, no. 1, pp. 174-187, 2007. (Pubitemid 47498530)
52. P. Tomancak, A. Beaton, R. Weiszmann, E. Kwan, S. Shu, S.E. Lewis, S. Richards, M. Ashburner, V. Hartenstein, S.E. Celniker, and G.M. Rubin, "Systematic Determination of Patterns of Gene Expression during Drosophila Embryogenesis," Genome Biology, vol. 3, no. 12, 2002.
53. S. Ji, L. Yuan, Y.-X. Li, Z.-H. Zhou, S. Kumar, and J. Ye, "Drosophila Gene Expression Pattern Annotation Using Sparse Features and Term-Term Interactions," Proc. 15th ACM SIGKDD Int'l Conf. Knowledge Discovery and Data Mining, 2009.
54. D.G. Lowe, "Distinctive Image Features from Scale-Invariant Keypoints," Int'l J. Computer Vision, vol. 60, no. 2, pp. 91-110, 2004.