Strong lower bounds for approximating distribution support size and the distinct elements problem

SIAM Journal on Computing

Volumn 39, Issue 3, 2009, Pages 813-842

  Raskhodnikova, Sofya ^a   Ron, Dana ^b   Shpilka, Amir ^c   Smith, Adam ^a  

^a The Pennsylvania State University   (United States)

^b TEL AVIV UNIVERSITY   (Israel)

^c TECHNION ISRAEL INSTITUTE OF TECHNOLOGY   (Israel)

Author keywords

Approximation algorithms; Distinct elements problem; Distribution support size; Lower bounds; Poissonization

Indexed keywords

ADDITIVE ERRORS; CHARIKAR; DISTINCT ELEMENTS; DISTINCT ELEMENTS PROBLEM; DISTRIBUTION SUPPORT SIZE; INPUT SEQUENCE; LOWER BOUND METHOD; LOWER BOUNDS; NEW YORK; NUMBER OF SAMPLES; POISSONIZATION; POSITIVE INTEGERS; PRINCIPLES OF DATABASE SYSTEMS; QUERY COMPLEXITY; SAMPLE COMPLEXITY;

COMPUTATION THEORY; CONTROL THEORY; DATABASE SYSTEMS; PROBABILITY DISTRIBUTIONS; RANDOM PROCESSES; RANDOM VARIABLES;

APPROXIMATION ALGORITHMS;

EID: 69049092204     PISSN: 00975397     EISSN: None     Source Type: Journal    
DOI: 10.1137/070701649     Document Type: Article

References (28)

1. [ABRS03] A. Akella, A. R. Bharambe, M. Reiter, and S. Seshan, Detecting DDoS attacks on ISP networks, in Proceedings of the ACM SIGMOD/PODS Workshop on Management and Processing of Data Streams, 2003.
2. [Akh65] N. I. Akhiezer, The Classical Moment Problem and Some Related Questions in Analysis, Hafner, New York, 1965.
3. [AMS99] N. Alon, Y. Matias, and M. Szegedy, The space complexity of approximating the frequency moments, J. Comput. System Sci., 58 (1999), pp. 137-147.
4. [And70] T. Ando, Truncated moment problems for operators, Acta Sci. Math. (Szeged), 31 (1970), pp. 319-334.
5. [Bar02] Z. Bar-Yossef, The Complexity of Massive Data Set Computations, Ph.D. thesis, Computer Science Division, University of California at Berkeley, Berkeley, CA, 2002.
6. [BJK+02] Z. Bar-Yossef, T. S. Jayram, R. Kumar, D. Sivakumar, and L. Trevisan, Counting distinct elements in a data stream, in Proceedings of the 6th International Workshop on Randomization and Approximation Techniques, Springer-Verlag, London, 2002, pp. 1-10.
7. [BKS01] Z. Bar-Yossef, R. Kumar, and D. Sivakumar, Sampling algorithms: Lower bounds and applications, in Proceedings of the Thirty-Third Annual ACM Symposium on Theory of Computing, ACM Press, New York, 2001, pp. 266-275; full version available online at http://www.ee.technion.ac.il/people/zivby/papers/ sampling/sampling-full.ps.
8. [BKS02] Z. Bar-Yossef, R. Kumar, and D. Sivakumar, Reductions in streaming algorithms, with an application to counting triangles in graphs, in Proceedings of the Thirteenth Annual ACM-SIAM Symposium on Discrete Algorithms, ACM, New York, SIAM, Philadelphia, 2002, pp. 623-632.
9. [BDKR02] T. Batu, S. Dasgupta, R. Kumar, and R. Rubinfeld, The complexity of approximating entropy, in Proceedings of the Thirty-Fourth Annual ACM Symposium on Theory of Computing, 2002, pp. 678-687.
10. [BDKR05] T. Batu, S. Dasgupta, R. Kumar, and R. Rubinfeld, The complexity of approximating the entropy, SIAM J. Comput., 35 (2005), pp. 132-150.
11. [BFF+01] T. Batu, E. Fischer, L. Fortnow, R. Kumar, R. Rubinfeld, and P. White, Testing random variables for independence and identity, in Proceedings of the 42nd Annual IEEE Symposium on Foundations of Computer Science, 2001, pp. 442-451.
12. [BFR+00] T. Batu, L. Fortnow, R. Rubinfeld, W. Smith, and P.White, Testing that distributions are close, in Proceedings of the 41st Annual IEEE Symposium on Foundations of Computer Science, 2000, pp. 259-269.
13. [BHR+07] K. Beyer, P. J. Haas, B. Reinwald, Y. Sismanis, and R. Gemulla, On synopses for distinct-value estimation under multiset operations, in Proceedings of the 2007 ACM SIGMOD International Conference on Management of Data, ACM, New York, 2007, pp. 199-210.
14. [Bun] J. Bunge, Bibliography on Estimating the Number of Classes in a Population, http://www.stat.cornell.edu/̃bunge/bibliography.htm.
15. [CCMN00] M. Charikar, S. Chaudhuri, R. Motwani, and V. R. Narasayya, Towards estimation error guarantees for distinct values, in Proceedings of the Nineteenth ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems, 2000, pp. 268-279.
16. [CF91] R. E. Curto and L. A. Fialkow, Recursiveness, positivity, and truncated moment problems, Houston J. Math., 17 (1991), pp. 603-635.
17. [FM85] P. Flajolet and G. N. Martin, Probabilistic counting algorithms for data base applications, J. Comput. System Sci., 31 (1985), pp. 182-209.
18. [GT02] P. B. Gibbons and S. Tirthapura, Distributed streams algorithms for sliding windows, in Proceedings of the Fourteenth Annual ACM Symposium on Parallel Algorithms and Architectures, 2002, pp. 63-72.
19. [IW03] P. Indyk and D. Woodruff, Tight lower bounds for the distinct elements problem, in Proceedings of the 44th Annual IEEE Symposium on Foundations of Computer Science, 2003, pp. 283-288.
20. [Pro53] Yu. V. Prohorov, Asymptotic behavior of the binomial distribution, Uspekhi Mat. Nauk, 8 (1953), pp. 135-142 (in Russian).
21. [RRRS07] S. Raskhodnikova, D. Ron, R. Rubinfeld, and A. Smith, Sublinear algorithms for approximating string compressibility, in Proceedings of the 11th RANDOM, Springer-Verlag, Berlin, Heidelberg, 2007, pp. 609-623.
22. [RRSS07] S. Raskhodnikova, D. Ron, A. Shpilka, and A. Smith, Strong lower bounds for approximating distribution support size and the distinct elements problem, in Proceedings of the 48th Annual IEEE Symposium on Foundations of Computer Science, IEEE Computer Society, Washington, DC, 2007, pp. 559-569.
23. [Szp01] W. Szpankowski, Average Case Analysis of Algorithms on Sequences, Wiley-Interscience, New York, 2001.
24. [Vad99] S. P. Vadhan, A Study of Statistical Zero-Knowledge Proofs, Ph.D. thesis, Massachusetts Institute of Technology, Cambridge, MA, 1999.
25. [Val08] P. Valiant, Testing symmetric properties of distributions, in Proceedings of the Fortieth Annual ACM Symposium on Theory of Computing, R. E. Ladner and C. Dwork, eds., ACM, New York, 2008, pp. 383-392.
26. [Was04] L. Wasserman, All of Statistics: A Concise Course in Statistical Inference, Springer-Verlag, New York, 2004.
27. [Web99] M. Weba, Bounds for the total variation distance between the binomial and the Poisson distribution in case of medium-sized success probabilities, J. Appl. Probab., 36 (1999), pp. 97-104.
28. [ZL77] J. Ziv and A. Lempel, A universal algorithm for sequential data compression, IEEETrans. Inform. Theory, 23 (1977), pp. 337-343.