Tracking cardinality distributions in network traffic

Proceedings - IEEE INFOCOM

Volumn , Issue , 2009, Pages 819-827

  Chen, Aiyou ^a   Li, Li Erran ^a   Cao, Jin ^a  

^a LUCENT TECHNOLOGIES   (United States)

Author keywords

Cardinality distribution; Data streaming; Expectation maximization; Mixture model; Network measurement

Indexed keywords

CARDINALITY DISTRIBUTION; DATA STREAMING; EXPECTATION-MAXIMIZATION; MIXTURE MODEL; NETWORK MEASUREMENT;

DATA REDUCTION; HIGHWAY ENGINEERING; INTERNET; MAXIMUM LIKELIHOOD ESTIMATION; MAXIMUM PRINCIPLE;

BLIND SOURCE SEPARATION;

EID: 70349658648     PISSN: 0743166X     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/INFCOM.2009.5061991     Document Type: Conference Paper

References (28)

1. N. Alon, Y. Matias, and M. Szegedy. The space complexity of approximating the frequency moments. In Proceedings of the twentyeighth annual ACM symposium on Theory of computing (STOC), pages 20-29, 1996.
2. Z. Bar-Yossef, T. S. Jayram, R. Kumar, D. Sivakumar, and L. Trevisan. Counting distinct elements in a data stream. In RANDOM, pages 1-10, 2002.
3. P. Bickel and K. Doksum. Mathematical Statistics: Basic Ideas and Selected Topics, Vol 1, (2nd Edition). Prentice Hall, 2000.
4. T. Bu, J. Cao, A. Chen, and P. P. C. Lee. A fast and compact method for unveiling significant patterns in high speed networks. In INFOCOM, pages 1893-1901, 2007.
5. J. Cao, Y. Jin, A. Chen, T. Bu, and Z. Zhang. Identifying high cardinality internet hosts. In Proceedings of IEEE INFOCOM (to appear), 2009.
6. A. Chakrabarti, G. Cormode, and A. McGregor. A near-optimal algorithm for computing the entropy of a stream. In SODA, pages 328-335, 2007.
7. A. Chen and J. Cao. Distinct counting with a self-learning bitmap. In Proceedings of the 25th International Conference on Data Engineering (ICDE), 2009.
8. A. Chen, J. Cao, and T. Bu. A simple and efficient estimation method for stream expression cardinalities. In VLDB, pages 171-182, 2007.
9. A. Chen, L. Li, and J. Cao. Estimating cardinality distributions in network traffic: extended abstract. In SIGMETRICS, pages 459-460, 2008.
10. G. Cormode, S. Muthukrishnan, and I. Rozenbaum. Summarizing and mining inverse distributions on data streams via dynamic inverse sampling. In VLDB, pages 25-36, 2005.
11. A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society. Series B (Methodological), pages 1-38, 1977.
12. N. Duffield, C. Lund, and M. Thorup. Estimating flow distributions from sampled flow statistics. In SIGCOMM, pages 325-336, 2003.
13. M. Durand and P. Flajolet. Loglog counting of large cardinalities. In Proceedings of European Symposium on Algorithms, pages 605-617, 2003.
14. B. Efron and R. Tibshirani. An Introduction to the Bootstrap. Chapman Hall, 1994.
15. C. Estan and G. Varghese. New directions in traffic measurement and accounting. In Proc. ACM SIGCOMM Internet Measurement Workshop, Nov. 2001.
16. C. Estan, G. Varghese, and M. Fisk. Bitmap algorithms for counting active flows on high speed links. IEEE/ACM Trans. on Networking, 14(5):925-937, 2006.
17. P. Flajolet and G. N. Martin. Probabilistic counting algorithms for data base applications. Journal of Computer and System Sciences, 31(2):182-209, 1985.
18. S. Ganguly. Counting distinct items over update streams. In ISAAC, pages 505-514, 2005.
19. F. Giroire. Order statistics and estimating cardinalities of massive data sets. In C. Martínez, editor, International Conference on Analysis of Algorithms, pages 157-166, 2005.
20. Y. He and C. Liu. The dynamic ECME algorithm. Technical report, 2009.
21. V. Karamcheti, D. Geiger, Z. Kedem, and S. Muthukrishnan. Detecting malicious network traffic using inverse distributions of packet contents. In MineNet: Proceeding of the ACM SIGCOMM workshop on Mining network data, pages 165-170, 2005.
22. A. Karasaridis, B. Rexroad, and D. Hoeflin. Wide-scale botnet detection and characterization. In First Workshop on Hot Topics in Understanding Botnets (HotBots), 2007.
23. A. Kumar, M. Sung, J. J. Xu, and J. Wang. Data streaming algorithms for efficient and accurate estimation of flow size distribution. In SIGMETRICS/Performance, pages 177-188, 2004.
24. A. Lakhina, M. Crovella, and C. Diot. Mining anomalies using traffic feature distributions. SIGCOMM Comput. Commun. Rev., 35(4):217-228, 2005.
25. A. Lall, V. Sekar, M. Ogihara, J. Xu, and H. Zhang. Data streaming algorithms for estimating entropy of network traffic. SIGMETRICS Perform. Eval. Rev., 34(1):145-156, 2006.
26. S. Sen and J. Wang. Analyzing peer-to-peer traffic across large networks. IEEE/ACM Trans. Netw., 12(2):219-232, 2004.
27. S. Venkataraman, D. Song, P. Gibbons, and A. Blum. New streaming algorithms for superspreader detection. In Proc. of Network and Distributed Systems Security Symposium, Feb. 2005.
28. L. Yang and G. Michailidis. Sampled based estimation of network traffic flow characteristics. In INFOCOM, pages 1775-1783, 2007.