A simple sketching algorithm for entropy estimation over streaming data

Journal of Machine Learning Research

Volumn 31, Issue , 2013, Pages 196-206

  Clifford, Peter ^a   Cosma, Ioana Ada ^b  

^a UNIVERSITY OF OXFORD   (United Kingdom)

^b UNIVERSITY OF OTTAWA   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; ARTIFICIAL INTELLIGENCE; ESTIMATION;

ASYMPTOTIC RELATIVE EFFICIENCY; ENTROPY ESTIMATION; ERROR PROBABILITIES; EXACT COMPUTATIONS; HIGH FREQUENCY DATA; PARAMETER CALIBRATION; PRACTICAL GUIDELINES; SINGLE-PASS ALGORITHM;

APPROXIMATION ALGORITHMS;

EID: 84954205022     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Conference Paper

References (27)

1. C. C. Aggarwal. Data streams: Models and Algo- rithms. Springer-Verlag, New York, 2007.
2. L. Bhuvanagiri and S. Ganguly. Estimating entropy over data streams. Lecture Notes in Computer Sci- ence, 4168: 148-159, 2006.
3. V. Braverman and R. Ostrovsky. Measuring independence of datasets. In Proceedings of the 42nd ACM symposium on Theory of computing (STOC), pages 271-280, New York, 2010. ACM.
4. A. Chakrabarti, K. Do Ba, and S. Muthukrishnan. Estimating entropy and entropy norm on data streams. Internet Mathematics, 3(1): 63-78, 2006.
5. A. Chakrabarti, G. Cormode, and A. McGregor. A near-optimal algorithm for estimating the entropy of a stream. ACM Transactions on Algorithms, 6 (3), 2010.
6. P. Clifford and I. A. Cosma. A statistical analysis of probabilistic counting algorithms. Scandinavian Journal of Statistics, 39(1): 1-14, 2012.
7. G. Grimmett and D. Stirzaker. Probability and random processes. Oxford University Press, USA, 3 edition, 2001.
8. S. Guha, P. Indyk, and A. McGregor. Sketching information divergences. Machine Learning, 72(1): 5-19, 2008.
9. N. J. A. Harvey, J. Nelson, and K. Onak. Streaming algorithms for estimating entropy. In IEEE Infor- mation Theory Workshop, 2008 (ITW), pages 227-231, 2008a.
10. N. J. A. Harvey, J. Nelson, and K. Onak. Sketching and streaming entropy via approximation theory. In 49th Annual IEEE Symposium on Foundations of Computer Science (FOCS), pages 489-498, 2008b.
11. W. Hoeffding. Inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58(301): 13-30, 1963.
12. P. Indyk. Stable distributions, pseudorandom generators, embeddings, and data stream computation. Journal of the Association for Computing Machin- ery (ACM), 53(3): 307-323, 2006.
13. P. Indyk and A. McGregor. Declaring independence via the sketching of sketches. In Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Dis- crete Algorithms (SODA), pages 737-745, 2008.
14. D. M. Kane, J. Nelson, E. Porat, and D. P. Woodruff. Fast moment estimation in data streams in optimal space. In Proceedings of the 43rd annual ACM sym- posium on Theory of computing (STOC), pages 745-754, New York, NY, 2011. ACM.
15. S. Kullback and R. A. Leibler. On information and sufficiency. The Annals of Mathematical Statistics, 22(1): 79-86, 1951.
16. A. Lall, V. Sekar, M. Ogihara, J. Xu, and H. Zhang. Data streaming algorithms for estimating entropy of network traffic. SIGMETRICS Performance Evalu- ation Review, 34(1): 145-156, 2006.
17. E. L. Lehmann. Theory of Point Estimation. Springer, New York, NY, 2 edition, 1998.
18. P. Li. Estimators and tail bounds for dimension reduction in la (0
19. P. Li. Improving compressed counting. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence (UAI 2009), pages 329-338. AUAI Press, 2009b.
20. P. Li and C.-H. Zhang. A new algorithm for compressed counting with applications in Shannon entropy estimation in dynamic data. Journal of Ma- chine Learning Research - Proceedings Track, 19: 477-496, 2011.
21. S. Muthukrishnan. Data streams: Algorithms and applications. Foundations and Trends in Theoretical Computer Science, 1: 117-236, 2005.
22. N. Nisan. Pseudorandom generators for spacebounded computation. Combinatorica, 12(4): 449-461, 1992.
23. A. Rényi. On Measures of Entropy and Information. Proc. Fourth Berkeley Symp. Math. Stat. and Prob- ability, 1: 547-561, 1961.
24. C. E. Shannon and W. Weaver. The Mathematical Theory of Communication. University of Illinois Press, Urbana, 1949.
25. C. Tsallis. Possible Generalization of Boltzmann- Gibbs Statistics. Journal of Statistical Physics, 52: 479-487, 1988.
26. H. C. Zhao, A. Lall, M. Ogihara, O. Spatscheck, J. Wang, and J. Xu. A data streaming algorithm for estimating entropies of OD flows. In Proceedings of the 7th ACM SIGCOMM conference on Internet measurement, pages 279-290, 2007.
27. V. M. Zolotarev. One-dimensional stable distribu- tions. American Mathematical Society, Providence, RI, 1986.