A novel M-estimator for robust PCA

Journal of Machine Learning Research

Volumn 15, Issue , 2014, Pages 749-808

  Zhang, Teng ^a   Lerman, Gilad ^b  

^a UNIVERSITY OF MINNESOTA   (United States)

^b University of Minnesota   (United States)

Author keywords

Convex relaxation; Iteratively re weighted least squares; M estimator; Principal components analysis; Robust statistics

Indexed keywords

ALGORITHMS; COVARIANCE MATRIX; EIGENVALUES AND EIGENFUNCTIONS; ESTIMATION; PRINCIPAL COMPONENT ANALYSIS; RECOVERY; RELAXATION PROCESSES;

CONVEX RELAXATION; M-ESTIMATORS; PRINCIPAL COMPONENTS ANALYSIS; RE-WEIGHTED LEAST SQUARES; ROBUST STATISTICS;

ITERATIVE METHODS;

EID: 84897049186     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (78)

1. A. Agarwal, S. Negahban, and M. J. Wainwright. Noisy matrix decomposition via convex relaxation: optimal rates in high dimensions. Ann. Statist., 40(2):1171-1197, 2012a. ISSN 0090-5364. doi: 10.1214/12-AOS1000.
2. A. Agarwal, S. Negahban, and M. J. Wainwright. Fast global convergence of gradient methods for high-dimensional statistical recovery. The Annals of Statistics, 40(5):2452-2482, 2012b.
3. L. P. Ammann. Robust singular value decompositions: A new approach to projection pursuit. Journal of the American Statistical Association, 88(422):pp. 505-514, 1993. ISSN 01621459.
4. E. Arias-Castro, D. L. Donoho, X. Huo, and C. A. Tovey. Connect the dots: how many random points can a regular curve pass through? Adv. in Appl. Probab., 37(3):571-603, 2005. (Pubitemid 41488720)
5. E. Arias-Castro, G. Chen, and G. Lerman. Spectral clustering based on local linear approximations. Electron. J. Statist., 5:1537-1587, 2011.
6. O. Arslan. Convergence behavior of an iterative reweighting algorithm to compute multivariate M-estimates for location and scatter. Journal of Statistical Planning and Inference, 118(1-2):115-128, 2004. ISSN 0378-3758. doi: 10.1016/S0378-3758(02)00402-0. (Pubitemid 37272840)
7. A. Bargiela and J. K. Hartley. Orthogonal linear regression algorithm based on augmented matrix formulation. Comput. Oper. Res., 20:829-836, October 1993. ISSN 0305-0548.doi: 10.1016/0305-0548(93)90104-Q.
8. R. Basri and D. Jacobs. Lambertian reectance and linear subspaces. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(2):218-233, February 2003.
9. R. Basri, T. Hassner, and L. Zelnik-Manor. Approximate nearest subspace search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(2):266-278, 2011. ISSN 0162-8828. doi: http://doi.ieeecomputersociety.org/10. 1109/TPAMI.2010.110.
10. R. Bhatia and D. Drissi. Generalized Lyapunov equations and positive definite functions. SIAM J. Matrix Anal. Appl., 27(1):103-114, May 2005. ISSN 0895-4798. doi: 10.1137/040608970.
11. P. Bradley and O. Mangasarian. k-plane clustering. J. Global optim., 16(1):23-32, 2000.
12. S. C. Brubaker. Robust PCA and clustering in noisy mixtures. In Proceedings of the Twentieth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA '09, pages 1078-1087, Philadelphia, PA, USA, 2009. Society for Industrial and Applied Mathematics.
13. E. J. Candés, J. Romberg, and T. Tao. Stable signal recovery from incomplete and inaccurate measurements. Communications on Pure and Applied Mathematics, 59(8):1207-1223, 2006. doi: 10.1002/cpa.20124. (Pubitemid 43988295)
14. E. J. Candés, X. Li, Y. Ma, and J. Wright. Robust principal component analysis? Journal of the ACM (JACM), 58(3):11, 2011.
15. T. F. Chan and P. Mulet. On the convergence of the lagged difiusivity fixed point method in total variation image restoration. SIAM J. Numer. Anal., 36:354-367, 1999. ISSN 0036-1429. doi: 10.1137/S0036142997327075. (Pubitemid 129582569)
16. V. Chandrasekaran, S. Sanghavi, P. A. Parrilo, and A. S. Willsky. Rank-sparsity incoherence for matrix decomposition. SIAM J. Optim., 21(2):572-596, 2011. ISSN 1052-6234. doi: 10.1137/090761793.
17. T.-J. Chin, J. Yu, and D. Suter. Accelerated hypothesis generation for multistructure data via preference analysis. IEEE Trans. Pattern Anal. Mach. Intell., 34(4):625-638, April 2012. ISSN 0162-8828. doi: 10.1109/TPAMI.2011.169.
18. A. K. Cline. Rate of convergence of Lawson's algorithm. Mathematics of Computation, 26 (117):pp. 167-176, 1972. ISSN 00255718.
19. M. Coudron and G. Lerman. On the sample complexity of robust PCA. In NIPS, pages 3230-3238, 2012.
20. C. Croux and G. Haesbroeck. Principal component analysis based on robust estimators of the covariance or correlation matrix: Inuence functions and effciencies. Biometrika, 87: 603-618, 2000.
21. C. Croux, P. Filzmoser, and M. Oliveira. Algorithms for projectionc pursuit robust principal component analysis. Chemometrics and Intelligent Laboratory Systems, 87(2):218-225, 2007. (Pubitemid 46755420)
22. A. Daspremont, L. El Ghaoui, M. Jordan, and G. Lanckriet. A direct formulation for sparse PCA using semidefinite programming. SIAM Review, 49(3):434-448, 2007. doi: 10.1137/050645506. (Pubitemid 47377693)
23. I. Daubechies, R. DeVore, M. Fornasier, and C. S. Gunturk. Iteratively reweighted least squares minimization for sparse recovery. Communications on Pure and Applied Mathematics, 63:1-38, 2010. doi: 10.1002/cpa.20303.
24. G. David and S. Semmes. Singular integrals and rectiáble sets in Rn: au-delá des graphes Lipschitziens. Astfierisque, 193:1-145, 1991.
25. P. L. Davies. Asymptotic behaviour of s-estimates of multivariate location parameters and dispersion matrices. The Annals of Statistics, 15(3):pp. 1269-1292, 1987. ISSN 00905364.
26. C. Davis and W. M. Kahan. The rotation of eigenvectors by a perturbation. iii. SIAM J. on Numerical Analysis, 7:1-46, 1970.
27. S. J. Devlin, R. Gnandesikan, and J. R. Kettenring. Robust estimation of dispersion matrices and principal components. Journal of the American Statistical Association, 76(374):pp. 354-362, 1981. ISSN 01621459.
28. C. Ding, D. Zhou, X. He, and H. Zha. R1-PCA: rotational invariant L1-norm principal component analysis for robust subspace factorization. In ICML '06: Proceedings of the 23rd International Conference on Machine Learning, pages 281-288, New York, NY, USA, 2006. ACM. ISBN 1-59593-383-2. doi: 10.1145/1143844.1143880. (Pubitemid 46966855)
29. M. Fornasier, H. Rauhut, and R. Ward. Low-rank matrix recovery via iteratively reweighted least squares minimization. SIAM J. Optim., 21(4):1614-1640, 2011. ISSN 1052-6234. doi: 10.1137/100811404.
30. M. Hardt and A. Moitra. Algorithms and hardness for robust subspace recovery. In COLT, pages 354-375, 2013.
31. J. Ho, M. Yang, J. Lim, K. Lee, and D. Kriegman. Clustering appearances of objects under varying illumination conditions. In Proceedings of International Conference on Computer Vision and Pattern Recognition, volume 1, pages 11-18, 2003.
32. D. Hsu, S.M. Kakade, and Tong Zhang. Robust matrix decomposition with sparse corruptions. Information Theory, IEEE Transactions on, 57(11):7221 -7234, nov. 2011. ISSN 0018-9448. doi: 10.1109/TIT.2011.2158250.
33. P. J. Huber and E. M. Ronchetti. Robust Statistics. Wiley Series in Probability and Statistics. Wiley, Hoboken, NJ, 2nd edition, 2009. ISBN 978-0-470-12990-6. doi: 10. 1002/9780470434697.
34. Q. Ke and T. Kanade. Robust subspace computation using L1 norm. Technical report, Carnegie Mellon, 2003.
35. J. T. Kent and D. E. Tyler. Redescending M-estimates of multivariate location and scatter. The Annals of Statistics, 19(4):pp. 2102-2119, 1991. ISSN 00905364.
36. H. W. Kuhn. A note on Fermat's problem. Mathematical Programming, 4:98-107, 1973. ISSN 0025-5610. 10.1007/BF01584648.
37. N. Kwak. Principal component analysis based on L1-norm maximization. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 30(9):1672-1680, 2008. doi: 10.1109/TPAMI.2008.114.
38. C. L. Lawson. Contributions to the Theory of Linear Least Maximum Approximation. PhD thesis, University of California, Los Angeles, 1961.
39. K.-C. Lee, J. Ho, and D. Kriegman. Acquiring linear subspaces for face recognition under variable lighting. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 27 (5):684-698, 2005. ISSN 0162-8828. doi: 10.1109/TPAMI.2005.92. (Pubitemid 40608402)
40. G. Lerman and T. Zhang. lp-Recovery of the most significant subspace among multiple subspaces with outliers. ArXiv e-prints, December 2010. To Appear in Constructive Approximation.
41. G. Lerman and T. Zhang. Robust recovery of multiple subspaces by geometric lp minimization. Ann. Statist., 39(5):2686-2715, 2011. ISSN 0090-5364. doi: 10.1214/11-AOS914.
42. G. Lerman, M. McCoy, J. A. Tropp, and T. Zhang. Robust computation of linear models, or how to find a needle in a haystack. ArXiv e-prints, February 2012.
43. G. Li and Z. Chen. Projection-pursuit approach to robust dispersion matrices and principal components: Primary theory and monte carlo. Journal of the American Statistical Association, 80(391):759-766, 1985. ISSN 01621459. doi: 10.2307/2288497.
44. L. Li, W. Huang, I. Gu, and Q. Tian. Statistical modeling of complex backgrounds for foreground object detection. Image Processing, IEEE Transactions on, 13(11):1459-1472, nov. 2004. ISSN 1057-7149. doi: 10.1109/TIP.2004.836169.
45. Z. Lin, A. Ganesh, J. Wright, L. Wu, M. Chen, and Y. Ma. Fast convex optimization algorithms for exact recovery of a corrupted low-rank matrix. In In Intl. Workshop on Comp. Adv. in Multi-Sensor Adapt. Processing, Aruba, Dutch Antilles, 2009.
46. G. Liu, Z. Lin, and Y. Yu. Robust subspace segmentation by low-rank representation. In ICML, 2010.
47. G. Liu, Z. Lin, S. Yan, J. Sun, Y. Yu, and Y. Ma. Robust recovery of subspace structures by low-rank representation. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 35(1):171 -184, 2013. ISSN 0162-8828. doi: 10.1109/TPAMI.2012.88.
48. H. P. Lopuhaä and P. J. Rousseeuw. Breakdown points of afine equivariant estimators of multivariate location and covariance matrices. Ann. Statist., 19(1):229-248, 1991. ISSN 0090-5364.
49. R. A. Maronna. Robust M-estimators of multivariate location and scatter. The Annals of Statistics, 4(1):pp. 51-67, 1976. ISSN 00905364.
50. R. A. Maronna, R. D. Martin, and V. J. Yohai. Robust Statistics: Theory and methods. Wiley Series in Probability and Statistics. John Wiley & Sons Ltd., Chichester, 2006. ISBN 978-0-470-01092-1; 0-470-01092-4.
51. M. McCoy and J. Tropp. Two proposals for robust PCA using semidefinite programming. Elec. J. Stat., 5:1123-1160, 2011.
52. S. Mendelson. A few notes on statistical learning theory. In Lecture Notes in Computer Science, volume 2600, pages 1-40. Springer-Verlag, 2003.
53. H. Nyquist. Least orthogonal absolute deviations. Computational Statistics & Data Analysis, 6(4):361-367, 1988. ISSN 0167-9473. doi: 10.1016/0167-9473(88)90076-X.
54. M. R. Osborne and G. A. Watson. An analysis of the total approximation problem in separable norms, and an algorithm for the total l1 problem. SIAM Journal on Scientific and Statistical Computing, 6(2):410-424, 1985. doi: 10.1137/0906029.
55. A. M. Ostrowski. Solution of Equations and Systems of Equations. Academic Press, Second edition, September 1966. ISBN 0471889873.
56. M. Soltanolkotabi and E. J. Candés. A geometric analysis of subspace clustering with outliers. Ann. Stat., 40(4):2195-2238, 2012. doi: 10.1214/12-AOS1034.
57. H. Späth and G. A. Watson. On orthogonal linear approximation. Numer. Math., 51: 531-543, October 1987. ISSN 0029-599X. doi: 10.1007/BF01400354.
58. C. V. Stewart. Robust parameter estimation in computer vision. SIAM Reviews, 41:513-537, 1999. (Pubitemid 30538509)
59. T. Tao. Topics in Random Matrix Theory, volume 132 of Graduate Studies in Mathematics. American Mathematical Society, Providence, RI, 2012. ISBN 978-0-8218-7430-1.
60. M. Tipping and C. Bishop. Mixtures of probabilistic principal component analysers. Neural Computation, 11(2):443-482, 1999.
61. F. De La Torre and M. J. Black. Robust principal component analysis for computer vision. In Computer Vision, 2001. ICCV 2001. Proceedings. Eighth IEEE International Conference on, volume 1, pages 362 -369 vol.1, 2001. doi: 10.1109/ICCV.2001.937541. (Pubitemid 32794979)
62. F. De La Torre and M. J. Black. A framework for robust subspace learning. International Journal of Computer Vision, 54:117-142, 2003. ISSN 0920-5691. doi: 10.1023/A:1023709501986.
63. P. Tseng. Nearest q-at to m points. Journal of Optimization Theory and Applications, 105:249-252, 2000. ISSN 0022-3239. 10.1023/A:1004678431677.
64. D. E. Tyler. A distribution-free M-estimator of multivariate scatter. Ann. Statist., 15(1): 234-251, 1987. ISSN 0090-5364. doi: 10.1214/aos/ 1176350263.
65. R. Vershynin. Introduction to the non-asymptotic analysis of random matrices. In Compressed sensing, pages 210-268. Cambridge Univ. Press, Cambridge, 2012.
66. H. Voss and U. Eckhardt. Linear convergence of generalized weiszfeld's method. Computing, 25:243-251, 1980. ISSN 0010-485X. doi: 10.1007/BF02242002.
67. L. Wang and A. Singer. Exact and stable recovery of rotations for robust synchronization. Information and Inference, 2013. doi: 10.1093/imaiai/iat005.
68. G. A. Watson. Some Problems in Orthogonal Distance and Non-Orthogonal Distance Regression. Defense Technical Information Center, 2001. URL http://books.google.com/books?id=WKKWGwAACAAJ.
69. G. A. Watson. On the gauss-newton method for l1 orthogonal distance regression. IMA Journal of Numerical Analysis, 22(3):345-357, 2002. doi: 10.1093/imanum/22.3.345. (Pubitemid 37445901)
70. E. Weiszfeld. Sur le point pour lequel la somme des distances de n points donne's est minimum. Tohoku Math. J., 43:35-386, 1937.
71. H. Xu, C. Caramanis, and S. Mannor. Principal component analysis with contaminated data: The high dimensional case. In COLT, pages 490-502, 2010a.
72. H. Xu, C. Caramanis, and S. Sanghavi. Robust PCA via outlier pursuit. In NIPS, pages 2496-2504, 2010b.
73. H. Xu, C. Caramanis, and S. Sanghavi. Robust PCA via outlier pursuit. Information Theory, IEEE Transactions on, PP(99):1, 2012. ISSN 0018-9448. doi: 10.1109/TIT. 2011.2173156.
74. L. Xu and A.L. Yuille. Robust principal component analysis by self-organizing rules based on statistical physics approach. Neural Networks, IEEE Transactions on, 6(1):131-143, 1995. ISSN 1045-9227. doi: 10.1109/72.363442.
75. T. Zhang. Robust subspace recovery by geodesically convex optimization. ArXiv e-prints, 2012.
76. T. Zhang, A. Szlam, and G. Lerman. Median K-ats for hybrid linear modeling with many outliers. In Computer Vision Workshops (ICCV Workshops), 2009 IEEE 12th International Conference on Computer Vision, pages 234-241, Kyoto, Japan, 2009. doi: 10.1109/ICCVW.2009.5457695.
77. T. Zhang, A. Szlam, Y. Wang, and G. Lerman. Randomized hybrid linear modeling by local best-t ats. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, pages 1927 -1934, jun. 2010. doi: 10.1109/CVPR.2010. 5539866.
78. T. Zhang, A. Szlam, Y. Wang, and G. Lerman. Hybrid linear modeling via local best-t ats. International Journal of Computer Vision, 100:217-240, 2012. ISSN 0920-5691. doi: 10.1007/s11263-012-0535-6.