Worst case linear discriminant analysis as scalable semidefinite feasibility problems

IEEE Transactions on Image Processing

Volumn 24, Issue 8, 2015, Pages 2382-2392

  Li, Hui ^a   Shen, Chunhua ^a,b   Van Den Hengel, Anton ^a,b   Shi, Qinfeng ^a  

^a UNIVERSITY OF ADELAIDE   (Australia)

^b AUSTRALIAN CENTRE FOR ROBOTIC VISION   (Australia)

Author keywords

Dimensionality Reduction; Semidefinite Programming; Worst Case Linear Discriminant Analysis

Indexed keywords

DISCRIMINANT ANALYSIS; NEWTON-RAPHSON METHOD;

DIMENSIONALITY REDUCTION; EIGEN DECOMPOSITION; FEASIBILITY PROBLEM; LINEAR DISCRIMINANT ANALYSIS; OPTIMIZATION METHOD; OPTIMIZATION PROBLEMS; QUASI-NEWTON METHODS; SEMI-DEFINITE PROGRAMMING;

OPTIMIZATION;

EID: 84929484948     PISSN: 10577149     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIP.2015.2401511     Document Type: Article

References (50)

1. N. Shental, T. Hertz, D. Weinshall, and M. Pavel, "Adjustment learning and relevant component analysis," in Proc. 7th Eur. Conf. Comp. Vis., 2002, pp. 776-790.
2. K. Q. Weinberger, J. Blitzer, and L. K. Saul, "Distance metric learning for large margin nearest neighbor classification," in Advances in Neural Information Processing Systems, vol. 18. Cambridge, MA, USA: MIT Press, 2006, pp. 1473-1481.
3. L.-F. Chen, H.-Y. M. Liao, M.-T. Ko, J.-C. Lin, and G.-J. Yu, "A new LDA-based face recognition system which can solve the small sample size problem," Pattern Recognit., vol. 33, no. 10, pp. 1713-1726, 2000.
4. J. Ye, "Characterization of a family of algorithms for generalized discriminant analysis on undersampled problems," J. Mach. Learn. Res., vol. 6, no. 4, pp. 483-502, Apr. 2005.
5. Y. Zhang and D.-Y. Yeung, "Worst-case linear discriminant analysis," in Advances in Neural Information Processing Systems, vol. 23. Cambridge, MA, USA: MIT Press, 2010, pp. 2568-2576.
6. Q. Gao, J. Liu, J. Zhang, J. Hou, and X. Yang, "Enhanced Fisher discriminant criterion for image recognition," Pattern Recognit., vol. 45, no. 10, pp. 3717-3724, 2012.
7. D. Tao, X. Li, X. Wu, and S. J. Maybank, "Geometric mean for subspace selection," IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 2, pp. 260-274, Feb. 2009.
8. W. Bian and D. Tao, "Harmonic mean for subspace selection," in Proc. 19th Int. Conf. Pattern Recognit., Dec. 2008, pp. 1-4.
9. W. Bian and D. Tao, "Max-min distance analysis by using sequential SDP relaxation for dimension reduction," IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 5, pp. 1037-1050, May 2011.
10. W. Bian and D. Tao, "Asymptotic generalization bound of Fisher's linear discriminant analysis," IEEE Trans. Pattern Anal. Mach. Intell., vol. 36, no. 12, pp. 2325-2337, 2014.
11. C. Shen, J. Kim, and L. Wang, "A scalable dual approach to semidefinite metric learning," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2011, pp. 2601-2608.
12. J. Ye and T. Xiong, "Computational and theoretical analysis of null space and orthogonal linear discriminant analysis," J. Mach. Learn. Res., vol. 7, pp. 1183-1204, Jul. 2006.
13. Y. Jia, F. Nie, and C. Zhang, "Trace ratio problem revisited," IEEE Trans. Neural Netw., vol. 20, no. 4, pp. 729-735, Apr. 2009.
14. Y.-F. Guo, S.-J. Li, J.-Y. Yang, T.-T. Shu, and L.-D. Wu, "A generalized Foley-Sammon transform based on generalized Fisher discriminant criterion and its application to face recognition," Pattern Recognit. Lett., vol. 24, nos. 1-3, pp. 147-158, 2003.
15. H. Wang, S. Yan, D. Xu, X. Tang, and T. Huang, "Trace ratio vs. ratio trace for dimensionality reduction," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2007, pp. 1-8.
16. H. Shen, K. Diepold, and K. Hüper, "A geometric revisit to the trace quotient problem," in Proc. 19th Int. Symp. Math. Theory Netw. Syst., 2010, pp. 1-4.
17. E. Kokiopoulou, J. Chen, and Y. Saad, "Trace optimization and eigenproblems in dimension reduction methods," Numer. Linear Algebra Appl., vol. 18, no. 3, pp. 565-602, 2011.
18. E. P. Xing, A. Y. Ng, M. I. Jordan, and S. Russell, "Distance metric learning, with application to clustering with side-information," in Advances in Neural Information Processing Systems, vol. 15. Cambridge, MA, USA: MIT Press, 2003, pp. 505-512.
19. K. Q. Weinberger and L. K. Saul, "Distance metric learning for large margin nearest neighbor classification," J. Mach. Learn. Res., vol. 10, pp. 207-244, Feb. 2009.
20. C. Shen, H. Li, and M. J. Brooks, "Supervised dimensionality reduction via sequential semidefinite programming," Pattern Recognit., vol. 41, no. 12, pp. 3644-3652, 2008.
21. P. Wang, C. Shen, and A. van den Hengel, "A fast semidefinite approach to solving binary quadratic problems," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2013, pp. 1312-1319.
22. S. Boyd and L. Xiao, "Least-squares covariance matrix adjustment," SIAM J. Matrix Anal. Appl., vol. 27, no. 2, pp. 532-546, 2005.
23. J. Malick, "A dual approach to semidefinite least-squares problems," SIAM J. Matrix Anal. Appl., vol. 26, no. 1, pp. 272-284, 2004.
24. X.-Y. Zhao, D. Sun, and K.-C. Toh, "A Newton-CG augmented Lagrangian method for semidefinite programming," SIAM J. Optim., vol. 20, no. 4, pp. 1737-1765, 2010.
25. Y. Gao and D. Sun, "Calibrating least squares covariance matrix problems with equality and inequality constraints," SIAM J. Matrix Anal. Appl., vol. 31, no. 3, pp. 1432-1457, 2009.
26. A. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro, "Robust stochastic approximation approach to stochastic programming," SIAM J. Optim., vol. 19, no. 4, pp. 1574-1609, 2009.
27. Z. Wen, D. Goldfarb, and W. Yin, "Alternating direction augmented Lagrangian methods for semidefinite programming," Math. Program. Comput., vol. 2, nos. 3-4, pp. 203-230, 2010.
28. H. Ouyang, N. He, L. Tran, and A. G. Gray, "Stochastic alternating direction method of multipliers," in Proc. 30th Int. Conf. Mach. Learn., 2013, pp. 80-88.
29. J. Chen, T. Yang, and S. Zhu, "Efficient low-rank stochastic gradient descent methods for solving semidefinite programs," in Proc. Int. Workshop Artif. Intell. Statist., 2014, pp. 122-130.
30. S. Shalev-Shwartz, Y. Singer, and A. Y. Ng, "Online and batch learning of pseudo-metrics," in Proc. Int. Conf. Mach. Learn., 2004, pp. 743-750.
31. J. V. Davis, B. Kulis, P. Jain, S. Sra, and I. S. Dhillon, "Information-theoretic metric learning," in Proc. 24th Int. Conf. Mach. Learn., 2007, pp. 209-216.
32. P. Jain, B. Kulis, I. S. Dhillon, and K. Grauman, "Online metric learning and fast similarity search," in Advances in Neural Information Processing Systems, vol. 8. Cambridge, MA, USA: MIT Press, 2008, pp. 761-768.
33. C. Shen, J. Kim, L. Wang, and A. Van Den Hengel, "Positive semidef-inite metric learning using boosting-like algorithms," J. Mach. Learn. Res., vol. 13, no. 1, pp. 1007-1036, 2012.
34. M. L. Overton and R. S. Womersley, "On the sum of the largest eigenvalues of a symmetric matrix," SIAM J. Matrix Anal. Appl., vol. 13, no. 1, pp. 41-45, 1992.
35. S. P. Boyd and L. Vandenberghe, Convex Optimization. Cambridge, U.K.: Cambridge Univ. Press, 2004.
36. H. Li, C. Shen, A. van den Hengel, and Q. Shi. (2014). "Worstcase linear discriminant analysis as scalable semidefinite feasibility problems." [Online]. Available: http://arxiv.org/abs/1411.7450
37. D. Henrion and J. Malick, "Projection methods for conic feasibility problems: Applications to polynomial sum-of-squares decompositions," Optim. Methods Softw., vol. 26, no. 1, pp. 23-46, 2011.
38. C. Zhu, R. H. Byrd, P. Lu, and J. Nocedal, "Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization," ACM Trans. Math. Softw., vol. 23, no. 4, pp. 550-560, 1997.
39. J. L. Morales and J. Nocedal, "Remark on 'Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound constrained optimization'," ACM Trans. Math. Softw., vol. 38, no. 1, p. 7, 2011.
40. K. C. Toh, M. J. Todd, and R. H. Tütüncü, "SDPT3-A MATLAB software package for semidefinite programming, version 1.3," Optim. Methods Softw., vol. 11, nos. 1-4, pp. 545-581, 1999.
41. J. F. Sturm, "Using SeDuMi 1.02, a MATLAB toolbox for optimization over symmetric cones," Optim. Methods Softw., vol. 11, nos. 1-4, pp. 625-653, 1999.
42. A. Frank and A. Asuncion. (2010). UCI Machine Learning Repository. School Inf. Comput. Sci., Univ. California, Irvine, CA, USA. [Online]. Available: http://archive.ics.uci.edu/ml
43. F. S. Samaria and A. C. Harter, "Parameterisation of a stochastic model for human face identification," in Proc. 2nd IEEE Workshop Appl. Comput. Vis., Dec. 1994, pp. 138-142.
44. P. N. Belhumeur, J. P. Hespanha, and D. Kriegman, "Eigenfaces vs. Fisherfaces: Recognition using class specific linear projection," IEEE Trans. Pattern Anal. Mach. Intell., vol. 19, no. 7, pp. 711-720, Jul. 1997.
45. T. Sim, S. Baker, and M. Bsat, "The CMU pose, illumination, and expression database," IEEE Trans. Pattern Anal. Mach. Intell., vol. 25, no. 12, pp. 1615-1618, 2003.
46. D. B. Graham and N. M. Allinson, "Characterizing virtual eigensigna-tures for general purpose face recognition," in Face Recognition: From Theory to Applications (NATO ASI Series F, Computer and Systems Sciences), vol. 163. New York, NY, USA: Springer-Verlag, 1998, pp. 446-456.
47. S. A. Nene, S. K. Nayar, and H. Murase, "Columbia object image library (COIL-20)," Dept. Comput. Sci., Columbia Univ., New York, NY, USA, Tech. Rep. CUCS-005-96, Feb. 1996.
48. S. A. Nene, S. K. Nayar, and H. Murase, "Columbia object image library (COIL-20)," Dept. Comput. Sci., Columbia Univ., New York, NY, USA, Tech. Rep. CUCS-006-96, Feb. 1996.
49. J.-M. Geusebroek, G. J. Burghouts, and A. W. M. Smeulders, "The Amsterdam library of object images," Int. J. Comput. Vis., vol. 61, no. 1, pp. 103-112, 2005.
50. S. Roweis. Data for MATLAB Hackers: Handwritten Digits. [Online]. Available: http://www.cs.nyu.edu/~roweis/data.html, accessed Jan. 2014.