A new algorithm for compressed counting with applications in Shannon entropy estimation in dynamic data

Journal of Machine Learning Research

Volumn 19, Issue , 2011, Pages 477-496

  Li, Ping ^a   Zhang, Cun Hui ^b  

^a Department of Statistical Science Cornell University ^*  (United States)

^b RUTGERS UNIVERSITY   (United States)

Author keywords

Data Streams; Entropy Estimation; Maximally Skewed Stable Random Projections

Indexed keywords

ALGORITHMS; DATA COMMUNICATION SYSTEMS; DATA MINING; FISHER INFORMATION MATRIX; LITHIUM;

CONSTANT FACTORS; DATA STREAM; EFFICIENT ESTIMATION; EMPIRICAL STUDIES; ENTROPY ESTIMATION; FISHER INFORMATION; RANDOM PROJECTIONS; SAMPLE COMPLEXITY;

ESTIMATION;

EID: 84898482261     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Conference Paper

References (36)

1. Charu C. Aggarwal, Jiawei Han, Jianyong Wang, and Philip S. Yu. On demand classification of data streams. In KDD, pages 503-508, Seattle, WA, 2004.
2. Noga Alon, Yossi Matias, and Mario Szegedy. The space complexity of approximating the frequency moments. In STOC, pages 20-29, Philadelphia, PA, 1996.
3. Lakshminath Bhuvanagiri and Sumit Ganguly. Estimating entropy over data streams. In ESA, pages 148-159, Zurich, Switzerland, 2006.
4. Daniela Brauckhoff, Bernhard Tellenbach, Arno Wagner, Martin May, and Anukool Lakhina. Impact of packet sampling on anomaly detection metrics. In IMC, pages 159-164, Rio de Janeriro, Brazil, 2006.
5. Amit Chakrabarti, Khanh Do Ba, and S. Muthukrishnan. Estimating entropy and entropy norm on data streams. Internet Mathematics, 3(1):63-78, 2006.
6. Amit Chakrabarti, Graham Cormode, and Andrew McGregor. A near-optimal algorithm for computing the entropy of a stream. In SODA, pages 328-335, New Orleans, Louisiana, 2007.
7. John M. Chambers, C. L. Mallows, and B. W. Stuck. A method for simulating stable random variables. Journal of the American Statistical Association, 71(354):340-344, 1976.
8. Carlotta Domeniconi and Dimitrios Gunopulos. Incremental support vector machine construction. In ICDM, pages 589-592, San Jose, CA, 2001.
9. Laura Feinstein, Dan Schnackenberg, Ravindra Balupari, and Darrell Kindred. Statistical approaches to DDoS attack detection and response. In DARPA Information Survivability Conference and Exposition, pages 303-314, 2003.
10. Izrail S. Gradshteyn and Iosif M. Ryzhik. Table of Integrals, Series, and Products. Academic Press, New York, fifth edition, 1994.
11. Sudipto Guha, Andrew McGregor, and Suresh Venkatasubramanian. Streaming and sublinear approximation of entropy and information distances. In SODA, pages 733 - 742, Miami, FL, 2006.
12. Anupam Gupta, John D. Lafferty, Han Liu, Larry A. Wasserman, and Min Xu. Forest density estimation. In COLT, pages 394-406, Haifa, Israel, 2010.
13. Nicholas J. A. Harvey, Jelani Nelson, and Krzysztof Onak. Sketching and streaming entropy via approximation theory. In FOCS, 2008a.
14. Nicholas J. A. Harvey, Jelani Nelson, and Krzysztof Onak. Streaming algorithms for estimating entropy. In ITW, 2008b.
15. M E. Havrda and F. Charvát. Quantification methods of classification processes: Concept of structural α-entropy. Kybernetika, 3:30-35, 1967.
16. Monika R. Henzinger, Prabhakar Raghavan, and Sridhar Rajagopalan. Computing on Data Streams. American Mathematical Society, Boston, MA, USA, 1999.
17. Piotr Indyk. Stable distributions, pseudorandom generators, embeddings, and data stream computation. Journal of ACM, 53(3):307-323, 2006.
18. Anukool Lakhina, Mark Crovella, and Christophe Diot. Mining anomalies using traffic feature distributions. In SIGCOMM, pages 217-228, Philadelphia, PA, 2005.
19. Ashwin Lall, Vyas Sekar, Mitsunori Ogihara, Jun Xu, and Hui Zhang. Data streaming algorithms for estimating entropy of network traffic. In SIGMETRICS, pages 145-156, Saint Malo, France, 2006.
20. α (0 < α ≤ 2) norm. In KDD, San Jose, CA, 2007.
21. α (0 < α ≤ 2) using stable random projections. In SODA, pages 10 - 19, San Francisco, CA, 2008.
22. Ping Li. Compressed counting. In SODA, New York, NY, 2009a.
23. Ping Li. Improving compressed counting. In UAI, Montreal, CA, 2009b.
24. Ping Li and Kenneth W. Church. A sketch algorithm for estimating two-way and multi-way associations. Computational Linguistics (Preliminary results appeared in HLT/EMNLP 2005), 33(3):305-354, 2007.
25. α (0 < α ≤ 2) using stable random projections. In NIPS, Vancouver, BC, Canada, 2007.
26. Ping Li, Kenneth W. Church, and Trevor J. Hastie. One sketch for all: Theory and applications of conditional random sampling. In NIPS (Preliminary results appeared in NIPS 2006), Vancouver, BC, Canada, 2008.
27. Ping Li, Michael Mahoney, and Yiyuan She. Approximating higher-order distances using random projections. In UAI, 2010.
28. Qiaozhu Mei and Kenneth Church. Entropy of search logs: How hard is search? with personalization? with backoff? In WSDM, pages 45 - 54, Palo Alto, CA, 2008.
29. S. Muthukrishnan. Data streams: Algorithms and applications. Foundations and Trends in Theoretical Computer Science, 1:117-236, 2 2005.
30. Liam Paninski. Estimation of entropy and mutual information. Neural Comput., 15(6): 1191-1253, 2003.
31. Alfred Rényi. On measures of information and entropy. In The 4th Berkeley Symposium on Mathematics, Statistics and Probability 1960, pages 547-561, 1961.
32. Constantino Tsallis. Possible generalization of boltzmann-gibbs statistics. Journal of Statistical Physics, 52:479-487, 1988.
33. Kuai Xu, Zhi-Li Zhang, and Supratik Bhattacharyya. Profiling internet backbone traffic: behavior models and applications. In SIGCOMM '05: Proceedings of the 2005 conference on Applications, technologies, architectures, and protocols for computer communications, pages 169-180, Philadelphia, Pennsylvania, USA, 2005.
34. Qiang Yang and Xindong Wu. 10 challeng problems in data mining research. International Journal of Information Technology and Decision Making, 5(4):597-604, 2006.
35. Haiquan Zhao, Ashwin Lall, Mitsunori Ogihara, Oliver Spatscheck, Jia Wang, and Jun Xu. A data streaming algorithm for estimating entropies of od flows. In IMC, San Diego, CA, 2007.
36. Haiquan Zhao, Nan Hua, Ashwin Lali, Ping Li, Jia Wang, and Jun Xu. Towards a universal sketch for origin-destination network measurements. Technical report, 2010.