Dynamic anomalography: Tracking network anomalies via sparsity and low rank

IEEE Journal on Selected Topics in Signal Processing

Volumn 7, Issue 1, 2013, Pages 50-66

  Mardani, Morteza ^a   Mateos, Gonzalo ^a   Giannakis, Georgios B ^a  

^a University of Minnesota   (United States)

Author keywords

low rank; network cartography; online optimization; sparsity; Traffic volume anomalies

Indexed keywords

ACCELERATION TECHNIQUE; COMPUTATIONAL COMPLEXITY REDUCTIONS; DYNAMIC ENVIRONMENTS; END USERS; END-USER EXPERIENCE; LARGE-SCALE NETWORK; LEAST SQUARE; LOW RANK; NETWORK MONITORING; NUMERICAL TESTS; ON-LINE ALGORITHMS; ONLINE OPTIMIZATION; OPERATIONAL CONDITIONS; REAL NETWORKS; REAL TIME; REAL TIME ALGORITHMS; SERVICE REQUIREMENTS; SPARSITY; STATE-OF-THE-ART APPROACH; STOCHASTIC GRADIENT ALGORITHMS; TIME DIMENSION; TRACKING CAPABILITY; TRACKING NETWORKS; TRAFFIC ANOMALIES; TRAFFIC FLOW; TRAFFIC MATRICES; TRAFFIC MEASUREMENTS; TRAFFIC VOLUMES;

ALGORITHMS; MAPPING; MAPS; QUALITY OF SERVICE; TELECOMMUNICATION TRAFFIC;

TRAFFIC CONGESTION;

EID: 84873824486     PISSN: 19324553     EISSN: None     Source Type: Journal    
DOI: 10.1109/JSTSP.2012.2233193     Document Type: Article

References (44)

1. [Online]. Available: http://internet2.edu/observatory/archive/ datacollections. html[Online]. Available
2. A. Abdelkefi, Y. Jiang,W.Wang, A. Aslebo, and O. Kvittem, "Robust traffic anomaly detection with principal component pursuit," in Proc. ACM CoNEXT Student Workshop, Philadelphia, PA, Nov. 2010.
3. A. Agarwal, S. Negahban, and M. J. Wainright, "Noisy matrix decomposition via convex relaxation: Optimal rates in high dimensions," Ann. Statist., vol. 40, pp. 1171-1197, 2012.
4. T. Ahmed, M. Coates, and A. Lakhina, "Multivariate online anomaly detection using kernel recursive least squares," in Proc. IEEE/ACMInt. Conf. Comput. Commun., Anchorage, AK, May 2007.
5. L. Balzano, R. Nowak, and B. Recht, "Online identification and tracking of subspaces from highly incomplete information," in Proc. Allerton Conf. Communication, Control, and Computing, Monticello, IL, Jun. 2010.
6. A. Beck and M. Teboulle, "A fast iterative shrinkage-thresholding algorithm for linear inverse problems," SIAM J. Imag. Sci., vol. 2, pp. 183-202, Jan. 2009.
7. D. P. Bertsekas, Nonlinear Programming, 2nd ed. Nashua, NH: Athena-Scientific, 1999.
8. J. F. Cai, E. J. Candès, and Z. Shen, "A singular value thresholding algorithm for matrix completion," SIAM J. Optim., vol. 20, no. 4, pp. 1956-1982, 2008.
9. E. J.Candès,X.Li, Y.Ma, and J.Wright, "Robust principal component analysis?," J. ACM, vol. 58, no. 1, pp. 1-37, 2011.
10. E. J. Candès and Y. Plan, "Matrix completion with noise," Proc. IEEE, vol. 98, no. 6, pp. 925-936, Jun. 2009.
11. E. J. Candès and B. Recht, "Exact matrix completion via convex optimization," Found. Comput. Math., vol. 9, no. 6, pp. 717-722, 2009.
12. E. J. Candès and T. Tao, "Decoding by linear programming," IEEE Trans. Inf. Theory, vol. 51, no. 12, pp. 4203-4215, Dec. 2005.
13. E. J. Candès and M. Wakin, "An introduction to compressive sampling," IEEE Signal Process. Mag., vol. 25, no. 2, pp. 21-30, Mar. 2008.
14. V. Chandrasekaran, S. Sanghavi, P. R. Parrilo, and A. S. Willsky, "Rank-sparsity incoherence for matrix decomposition," SIAM J. Optim., vol. 21, no. 2, pp. 572-596, 2011.
15. Q. Chenlu and N. Vaswani, "Recursive sparse recovery in large but correlated noise," in Proc. 49th Allerton Conf. Commun., Control, Comput., Sep. 2011, pp. 752-759.
16. Y. Chi, Y. C. Eldar, and R. Calderbank, "PETRELS: Subspace estimation and tracking from partial observations," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., Kyoto, Japan, Mar. 2012, pp. 3301-3304.
17. A. Chistov and D. Grigorev, "Complexity of quantifier elimination in the theory of algebraically closed fields," in Math. Found. of Computer Science, ser. Lecture Notes in Computer Science. Berlin/Heidelberg, Germany: Springer, 1984, vol. 176, pp. 17-31.
18. T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning, 2nd ed. New York: Springer, 2009.
19. J. He, L. Balzano, and A. Szlam, "Incremental gradient on the Grassmannian for online foreground and background separation in subsampled video," in Proc. IEEE Conf. Comput. Visi. Pattern Recogn., Providence, RI, Jun. 2012.
20. D. Jakovetic, J. Xavier, and J.M. F.Moura, "FastDistributed Gradient Methods," arXiv:1112.2972v1 [cs.IT].
21. A. Lakhina, M. Crovella, and C. Diot, "Diagnosing network-wide traffic anomalies," in Proc. ACM SIGCOMM, Portland, OR, Aug. 2004.
22. A. Lakhina, K. Papagiannaki, M. Crovella, C. Diot, E. D. Kolaczyk, and N. Taft, "Structural analysis of network traffic flows," in Proc. ACM SIGMETRICS, New York, Jul. 2004.
23. L. Ljung and T. Söderström, Theory and Practice of Recursive Identification, 2nd ed. Cambridge, MA: MIT Press, 1983.
24. J.Mairal, J. Bach, J. Ponce, andG. Sapiro, "Online learning formatrix factorization and sparse coding," J. Mach. Learn. Res., vol. 11, pp. 19-60, Jan. 2010.
25. M. Mardani, G. Mateos, andG. B. Giannakis, "In-network sparsity regularized rank minimization: Applications and algorithms," IEEE Trans. Signal Process., 2013, see also arXiv:1203.1570v1 [cs.MA], submitted for publication.
26. M. Mardani, G. Mateos, and G. B. Giannakis, "Recovery of low-rank plus compressed sparse matrices with application to unveiling traffic anomalies," IEEE Trans. Inf. Theory., 2013, see also arXiv:1204.6537v1 [cs.IT], submitted for publication.
27. G. Mateos and G. B. Giannakis, "Robust PCA as bilinear decomposition with outlier-sparsity regularization," IEEE Trans. Signal Process., vol. 60, no. 10, pp. 5176-5190, Oct. 2012.
28. Z. Meng, A. Wiesel, and A. Hero, "Distributed principal component analysis on networks via directed graphical models," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., Kyoto, Japan, Mar. 2012, pp. 2877-2880.
29. B. K. Natarajan, "Sparse approximate solutions to linear systems," SIAM J. Comput., vol. 24, pp. 227-234, 1995.
30. Y. Nesterov, "A method of solving a convex programming problem with convergence rate ," Sov. Math. Doklady, vol. 27, pp. 372-376, 1983.
31. B. Recht, M. Fazel, and P.A. Parrilo, "Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization," SIAM Rev., vol. 52, no. 3, pp. 471-501, 2010.
32. B. Recht and C. Re, "Parallel stochastic gradient algorithms for large-scale matrix completion," Math. Programm. Comput., 2011. Available: http://www.optimization-online.org/DB-HTML/2011/04/3012.html, Preprint, submitted for publication
33. M. Roughan, "A case study of the accuracy of SNMP measurements," J. Electr. Comput. Eng., Dec. 2010, article ID 812979.
34. W. Rudin, Principles of Mathematical Analysis, 3rd ed. New York: McGraw-Hill, 1976.
35. A. H. Sayed, Fundamentals of Adaptive Filtering. New York:Wiley, 2003.
36. V. Solo and X. Kong, Adaptive Signal Processing Algorithms: Stability and Performance. Englewood Cliffs, NJ: Prentice-Hall, 1995.
37. M. Thottan and C. Ji, "Anomaly detection in IP networks," IEEE Trans. Signal Process., vol. 51, no. 8, pp. 2191-2204, Aug. 2003.
38. J. Tropp, "Just relax: Convex programming methods for identifying sparse signals," IEEE Trans. Inf. Theory, vol. 51, no. 3, pp. 1030-1051, Mar. 2006.
39. P. Tseng, "Convergence of a block coordinate descent method for nondifferentiable minimization," J. Optimiz. Theory Applicat., vol. 109, pp. 475-494, 2001.
40. A. W. Van Der Vaart, Asymptotic Statistics. Cambridge, U.K.: Cambridge Univ. Press, 2000.
41. B. Yang, "Projection approximation subspace tracking," IEEE Trans. Signal. Process., vol. 43, no. 1, pp. 95-107, Jan. 1995.
42. Y. Zhang, Z. Ge, A. Greenberg, and M. Roughan, "Network anomography," in Proc. ACM SIGCOM Conf. Internet Meas., Berekly, CA, Oct. 2005.
43. Y. Zhang, M. Roughan, W. Willinger, and L. Qiu, "Spatio-temporal compressive sensing and internet traffic matrices," in Proc. ACM SIGCOM Conf. Data Commun., New York, Oct. 2009.
44. Z. Zhou, X. Li, J. Wright, E. Candès, and Y. Ma, "Stable principal component pursuit," in Proc. Intl. Symp. Inf. Theory, Austin, TX, Jun. 2010, pp. 1518-1522.