A data streaming algorithm for estimating entropies of OD flows

Proceedings of the ACM SIGCOMM Internet Measurement Conference, IMC

Volumn , Issue , 2007, Pages 279-290

  Zhao, Haiquan ^a   Spatscheck, Oliver ^b   Lall, Ashwin ^c   Wang, Jia ^b   Ogihara, Mitsunori ^c   Xu, Jun ^a  

^a GEORGIA INSTITUTE OF TECHNOLOGY   (United States)

^b AT AND T LABS RESEARCH   (United States)

^c UNIVERSITY OF ROCHESTER   (United States)

Author keywords

Data streaming; Entropy estimation; Network measurement; Stable distributions; Traffic matrix

Indexed keywords

CHALLENGING PROBLEM; DATASTREAM (CO); INTERNET MEASUREMENTS; NETWORK MEASUREMENTS; O D FLOWS; OPEN PROBLEMS; ORIGIN-DESTINATION (OD); SINGLE POINT; TRAFFIC ANOMALIES;

ADMINISTRATIVE DATA PROCESSING; DATA REDUCTION; FINANCIAL DATA PROCESSING; INTERNET; SEMICONDUCTING INTERMETALLICS;

ENTROPY;

EID: 42149164252     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1298306.1298345     Document Type: Conference Paper

References (29)

1. A. Chakrabarti, K. Do Ba, and S. Muthukrishnan. Estimating entropy and entropy norm on data streams. In STACS, 2006.
2. L. Bhuvanagiri and S. Ganguly. Estimating entropy over data streams. In ESA, 2006.
3. D. Brauckhoff, B. Teilenbach, A. Wagner, M. May, and A. Lakhina. Impact of packet sampling on anomaly detection metrics. In IMC, 2006.
4. G. Casella and R. L. Berger. Statistical Inference. Duxbury, 2nd edition, 2002.
5. A. Chakrabarti and G. Cormode. A near-optimal algorithm for computing the entropy of a stream. In SODA, 2007.
6. J. M. Chambers, C. L. Mallows, and B. W. Stuck. A method for simulating stable random variables. Journal of the American Statistical Association, 71(354), 1976.
7. G. Cormode. Stable distributions for stream computations: It's as easy as 0,1,2. In Workshop on Management and Processing of Data Streams, 2003.
8. G. Cormode, P. Indyk, N. Koudas, and S. Muthukrishnan. Fast mining of massive tabular data via approximate distance computations. In ICDE, 2002.
9. M. Durand and P. Flajolet. Loglog counting of large cardinalities. In ESA, 2003.
10. C. Estan and G. Varghese. New Directions in Traffic Measurement and Accounting. In SIGCOMM, Aug. 2002.
11. L. Feinstein, D. Schnackenberg, R. Balupari, and D. Kindred. Statistical approaches to DDoS attack detection and response. In Proceedings of the DARPA Information Survivability Conference and Exposition, 2003.
12. P. Indyk. Stable distributions, pseudorandom generators, embeddings and data stream computation. In FOCS, 2000.
13. P. Indyk. Stable distributions, pseudorandom generators, embeddings, and data stream computation. J. A CM, 53(3):307-323, 2006.
14. A. Kuzmanovic and E. W. Knightly. Low-rate tcp targeted denial of service attacks (the shrew vs. the mice and elephants). In SIGCOMM, 2003.
15. A. Lakhina, M. Crovella, and C. Diot. Mining anomalies using traffic feature distributions. In SIGCOMM, 2005.
16. A. Lall, V. Sekar, M. Ogihara, J. Xu, and H. Zhang. Data streaming algorithms for estimating entropy of network traffic. In SIGMETRICS, 2006.
17. G. S. Manku and R. Motwani. Approximate frequency counts over data streams. In Proceedings of the 28th International Conference on Very Large Data Bases, 2002.
18. A. Medina, N. Taft, K. Salamatian, S. Bhattacharyya, and C. Diot. Traffic matrix estimation:existing techniques and new directions. In SIGCOMM, Aug. 2002.
19. S. Muthukrishnan. Data streams: algorithms and applications, available at http://athos.rutgers.edu/~muthu/.
20. J. Nolan. STABLE program, online at http://academic2.american.edu/ ~jpnolan/stable/stable.html.
21. A. Soule, A. Nucci, R. Cruz, E. Leonardi, and N. Taft. How to identify and estimate the largest traffic matrix elements in a dynamic environment. In SIGMETRICS, June 2004.
22. C. Tebaldi and M. West. Bayesian inference on network traffic using link count data. Journal of American Statistics Association, pages 557-576, 1998.
23. Y. Vardi. Internet tomography: estimating source-destination traffic intensities from link data. Journal of American Statistics Association, pages 365-377, 1996.
24. A. Wagner and B. Plattner. Entropy Based Worm and Anomaly Detection in Fast IP Networks. In Proceedings of IEEE International Workshop on Enabling Technologies, Infrastructures for Collaborative Enterprises, 2005.
25. K. Xu, Z.-L. Zhang, and S. Bhattacharya. Profiling internet backbone traffic: Behavior models and applications. In SIGCOMM, 2005.
26. Y. Zhang, Z. M. Mao, and J. Wang. Low-rate tcp-targeted dos attack disrupts internet routing. In Proc. 14th Annual Network & Distributed System Security Symposium, 2007.
27. Q. Zhao, Z. Ge, J. Wang, and J. Xu. Robust traffic matrix estimation with imperfect information: making use of multiple data sources. In SIGMETRICS, 2006.
28. Q. Zhao, A. Kumar, J. Wang, and J. Xu. Data streaming algorithms for accurate and efficient measurement of traffic and flow matrices. In SIGMETRICS, June 2005.
29. V. M. Zolotarev. One-Dimensional Stable Distributions, volume 65 of Translations of Mathematical Monographs. American Mathematical Society, Providence, RI, 1986.