A Statistical Analysis of Probabilistic Counting Algorithms

Scandinavian Journal of Statistics

Volumn 39, Issue 1, 2012, Pages 1-14

  Clifford, Peter ^a   Cosma, Ioana A ^b  

^a UNIVERSITY OF OXFORD   (United Kingdom)

^b UNIVERSITY OF CAMBRIDGE   (United Kingdom)

Author keywords

Asymptotic relative efficiency; Cardinality; Data sketching; Data stream; Hash function; Maximum likelihood estimation; Space complexity; Stable distribution; Tail bounds

Indexed keywords

EID: 84857041420     PISSN: 03036898     EISSN: 14679469     Source Type: Journal    
DOI: 10.1111/j.1467-9469.2010.00727.x     Document Type: Article

References (35)

1. Aggarwal, C. C. (2007). Data streams: models and algorithms. Springer-Verlag, New York.
2. Akella, A., Bharambe, A., Reiter, M. and Seshan, S. (2003). Detecting DDoS attacks on ISP networks. In ACM SIGMOD/PODS workshop on Management and Processing of Data Streams (MPDS). ACM, New York.
3. Alon, N., Matias, Y. and Szegedy, M. (1999). The space complexity of approximating the frequency moments. J. Comput. System Sci. 58, 137-147.
4. Bar Youssef, Z., Jayram, T. S., Kumar, R., Sivakumar, D. and Trevisan, L. (2002). Counting distinct elements in a data stream. Lect. Notes Comput. Sci. 2483, 1-10.
5. Carter, J. L. and Wegman, M. N. (1979). Universal classes of hash functions. J. Comput. System Sci. 18, 143-154.
6. Chassaing, P. and Gerin, L. (2006). Efficient estimation of the cardinality of large data sets. In Proceedings of 4th Colloquium on Mathematics and Computer Science, vol. AG, Discrete Mathematics and Theoretical Computer Science Proceedings Nancy, France, 419-422.
7. Chen, A. and Cao, J. (2009). Distinct counting with a self-learning bitmap. In Proceedings of 25th International Conference on Data Engineering, IEEE Computer Society, Washington DC, 1171-1174.
8. Chernoff, H. (1952). A measure of asymptotic efficiency for tests of a hypothesis based on the sum of observations. Ann. Math. Statist. 23, 493-507.
9. Cormode, G., Datar, M., Indyk, P. and Muthukrishnan, S. (2003). Comparing data streams using Hamming norms (how to zero in). IEEE Trans. Knowl. Data Eng. 15, 529-540.
10. Cormode, G. and Muthukrishnan, S. (2005a). An improved data stream summary: the Count-Min sketch and its applications. J. Algorithms 55, 58-75.
11. Cormode, G. and Muthukrishnan, S. (2005b). What's new: finding significant differences in network data streams. IEEE/ACM Trans. Netw. (TON) 13, 1219-1232.
12. Durand, M. and Flajolet, P. (2003). Loglog counting of large cardinalities. Lect. Notes Comput. Sci. 2832, 605-617.
13. Feller, W. (1971). An introduction to probability theory and its applications, vol. 2, 2nd edn. John Wiley and Sons Inc. New York.
14. Fisher, R. A. (1925). Theory of statistical estimation. Math. Proc. Cambridge Philos. Soc. 22, 700-725.
15. Flajolet, P. and Martin, G. N. (1985). Probabilistic counting algorithms for database applications. J. Comput. System Sci. 31, 182-209.
16. Flajolet, P., Fusy, E., Gandouet, O. and Meunier, F. (2007). HyperLogLog: the analysis of a near-optimal cardinality estimation algorithm. In Proceedings of Conference on Analysis of Algorithms, vol. AH, Discrete Mathematics and Theoretical Computer Science Proceedings Nancy, France, 127-146.
17. Giroire, F. (2009). Order statistics and estimating cardinalities of massive data sets. Discrete Appl. Math. 157, 406-427.
18. Hammersley, J. M. (1950). On estimating restricted parameters. J. Roy. Statist. Soc. Ser. B 12, 192-240.
19. Harvey, N. J. A., Nelson, J. and Onak, K. (2008). Sketching and streaming entropy via approximation theory. In Proceedings of 49th Annual Symposium on Foundations of Computer Science, IEEE Computer Society, Washington DC, 489-498.
20. Indyk, P. (2006). Stable distributions, pseudorandom generators, embeddings, and data stream computation. J. ACM 53, 307-323.
21. Indyk, P. and Woodruff, D. P. (2003). Tight lower bounds for the distinct elements problem. In Proceedings of 44th Annual Symposium on Foundations of Computer Science, vol. 44, IEEE Computer Society, Washington DC, 283-289.
22. Kane, D. M., Nelson, J. and Woodruff, D. P. (2010). An optimal algorithm for the distinct elements problem. In Proceedings of the 29th Symposium on Principles of Database Systems (PODS), 41-52, Indiana.
23. Kirschenhofer, P. and Prodinger, H. (1993). A result in order statistics related to probabilistic counting. Computing 51, 15-27.
24. Knuth, D. E. (1998). The art of computer programming: sorting and searching, vol. 3, 2nd edn. Addison-Wesley, MA.
25. Lauritzen, S. L. (1988). Extremal families and systems of sufficient statistics. In Lecture Notes in Statistics, vol. 49 (eds J. Berger, S. Fienberg, J. Gani, K. Krickeberg and B. Singer ), 1-268. Springer, New York.
26. Lehmann, E. L. and Scheffé, H. (1950). Completeness, similar regions and unbiased estimation. Part I. Sankhy 10, 305-340.
27. Lévy, P. (1924). Théorie des erreurs. La loi de Gauss et les lois exceptionelles. Bull. Soc. Math. France 52, 49-85.
28. Muthukrishnan, S. (2005). Data streams: algorithms and applications, 1st edn. Now Publishers Inc, Cambridge, MA.
29. Nisan, N. (1992). Pseudorandom generators for space-bounded computation. Combinatorica 12, 449-461.
30. Nolan, J. P. (1997). Numerical calculation of stable densities and distribution functions. Stoch. Models 13, 759-774.
31. Nolan, J. P. (2001). Maximum likelihood estimation and diagnostics for stable distributions. In Lévy processes: theory and applications (eds O. E. Barndorff-Nielsen, T. Mikosch and S. I. Resnick ), 379-400. Birkhäuser, Boston, MA.
32. Press, W. H., Teukolsky, S. A., Vetterling, W. T. and Flannery, B. P. (2007). Numerical recipes: the art of scientific computing. Cambridge University Press, New York.
33. Rao, C. R. (1973). Linear statistical inference and its applications, 2nd edn. John Wiley and Sons, Inc., New York.
34. Whang, K.-Y., Vander-Zanden, B. T. and Taylor, H. M. (1990). A linear-time probabilistic counting algorithm for database applications. ACM Trans. Database Syst. 15, 208-229.
35. Zolotarev, V. M. (1986). One-dimensional stable distributions. American Mathematical Society, Providence RI.