Fast moment estimation in data streams in optimal space

Proceedings of the Annual ACM Symposium on Theory of Computing

Volumn , Issue , 2011, Pages 745-754

  Kane, Daniel M ^a   Nelson, Jelani ^b   Porat, Ely ^c   Woodruff, David P ^d  

^a HARVARD UNIVERSITY   (United States)

^b MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^c BAR ILAN UNIVERSITY   (Israel)

^d IBM ALMADEN RESEARCH CENTER   (United States)

Author keywords

data stream algorithms; frequency moments

Indexed keywords

COMPUTATION THEORY; VECTOR SPACES;

DATA STREAM ALGORITHMS; ENTROPY ESTIMATION; FREQUENCY MOMENTS; MOMENT ESTIMATION; NEAR-OPTIMAL; OPTIMAL ALGORITHM; OPTIMAL SPACE; STREAMING ALGORITHM;

DATA STREAMS;

EID: 79959741836     PISSN: 07378017     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1993636.1993735     Document Type: Conference Paper

References (54)

1. C. C. Aggarwal, A. Hinneburg, and D. A. Keim. On the surprising behavior of distance metrics in high dimensional spaces. In ICDT, pages 420-434, 2001.
2. N. Alon, P. B. Gibbons, Y. Matias, and M. Szegedy. Tracking join and self-join sizes in limited storage. J. Comput. Syst. Sci., 64(3):719-747, 2002.
3. N. Alon, Y. Matias, and M. Szegedy. The Space Complexity of Approximating the Frequency Moments. J. Comput. Syst. Sci., 58(1):137-147, 1999.
4. K. D. Ba, P. Indyk, E. Price, and D. P. Woodruff. Lower bounds for sparse recovery. In SODA, pages 1190-1197, 2010.
5. Z. Bar-Yossef. The Complexity of Massive Data Set Computations. PhD thesis, University of California at Berkeley, 2002.
6. Z. Bar-Yossef, T. S. Jayram, R. Kumar, and D. Sivakumar. An information statistics approach to data stream and communication complexity. J. Comput. Syst. Sci., 68(4):702-732, 2004.
7. L. Bhuvanagiri and S. Ganguly. Estimating entropy over data streams. In Proceedings of the 14th Annual European Symposium on Algorithms (ESA), pages 148-159, 2006.
8. L. Bhuvanagiri, S. Ganguly, D. Kesh, and C. Saha. Simpler algorithm for estimating frequency moments of data streams. In Proceedings of the 17th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 708-713, 2006.
9. A. Chakrabarti, G. Cormode, and A. McGregor. A near-optimal algorithm for estimating the entropy of a stream. ACM Transactions on Algorithms, 6(3), 2010.
10. A. Chakrabarti, K. Do Ba, and S. Muthukrishnan. Estimating Entropy and Entropy Norm on Data Streams. In Proceedings of the 23rd Annual Symposium on Theoretical Aspects of Computer Science (STACS), pages 196-205, 2006.
11. A. Chakrabarti, S. Khot, and X. Sun. Near-optimal lower bounds on the multi-party communication complexity of set disjointness. In Proceedings of the 18th Annual IEEE Conference on Computational Complexity (CCC), pages 107-117, 2003.
12. M. Charikar, K. Chen, and M. Farach-Colton. Finding frequent items in data streams. In Proceedings of the 29th International Colloquium on Automata, Languages and Programming (ICALP), pages 693-703, 2002.
13. K. L. Clarkson and D. Woodruff. Numerical linear algebra in the streaming model. In STOC, 2009.
14. G. Cormode, P. Indyk, N. Koudas, and S. Muthukrishnan. Fast mining of massive tabular data via approximate distance computations. In ICDE, pages 605-, 2002.
15. G. Cormode and S. Muthukrishnan. An improved data stream summary: the count-min sketch and its applications. J. Algorithms, 55(1):58-75, 2005. (Pubitemid 40357145)
16. G. Cormode, S. Muthukrishnan, and I. Rozenbaum. Summarizing and mining inverse distributions on data streams via dynamic inverse sampling. In VLDB, pages 25-36, 2005.
17. I. Diakonikolas, D. M. Kane, and J. Nelson. Bounded independence fools degree-2 threshold functions. In Proceedings of the 51st Annual IEEE Symposium on Foundations of Computer Science (FOCS), pages 11-20, 2010.
18. J. Feigenbaum, S. Kannan, M. Strauss, and M. Viswanathan. An approximate L1-difference algorithm for massive data streams. SIAM J. Comput., 32(1):131-151, 2002.
19. M. L. Fredman, J. Komlós, and E. Szemerédi. Storing a sparse table with 0(1) worst case access time. J. ACM, 31(3):538-544, 1984.
20. S. Ganguly. Estimating frequency moments of data streams using random linear combinations. In Proceedings of the 8th International Workshop on Randomization and Computation (RANDOM), pages 369-380, 2004.
21. S. Ganguly, A. N. Singh, and S. Shankar. Finding frequent items over general update streams. In Proceedings of the 20th International Conference on Scientific and Statistical Database Management (SSDBM), pages 204-221, 2008.
22. D. Geiger, T.-L. Liu, and M. J. Donahue. Sparse representations for image decompositions. International Journal of Computer Vision, 33(2):139-156, 1999. (Pubitemid 30532484)
23. A. Gronemeier. Asymptotically optimal lower bounds on the NIH-multi-party information complexity of the AND-function and disjointness. In Proceedings of the 26th International Symposium on Theoretical Aspects of Computer Science (STACS), pages 505-516, 2009.
24. Y. Gu, A. McCallum, and D. F. Towsley. Detecting anomalies in network traffic using maximum entropy estimation. In Proceedings of the Internet Measurement Conference (IMC), pages 345-350, 2005.
25. S. Guha, P. Indyk, and A. McGregor. Sketching information divergences. Machine Learning, 72(1-2):5-19, 2008.
26. S. Guha, A. McGregor, and S. Venkatasubramanian. Streaming and sublinear approximation of entropy and information distances. In Proceedings of the 17th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 733-742, 2006.
27. U. Haagerup. The best constants in the Khintchine inequality. Studia Math., 70(3):231-283, 1982.
28. N. J. A. Harvey, J. Nelson, and K. Onak. Sketching and streaming entropy via approximation theory. In Proceedings of the 49th Annual IEEE Symposium on Foundations of Computer Science (FOCS), pages 489-498, 2008.
29. P. Indyk. Stable distributions, pseudorandom generators, embeddings, and data stream computation. J. ACM, 53(3):307-323, 2006.
30. P. Indyk and A. McGregor. Declaring independence via the sketching of sketches. In SODA, pages 737-745, 2008.
31. P. Indyk and D. P. Woodruff. Optimal approximations of the frequency moments of data streams. In Proceedings of the 37th Annual ACM Symposium on Theory of Computing (STOC), pages 202-208, 2005.
32. T. S. Jayram. Hellinger strikes back: A note on the multi-party information complexity of AND. In APPROX-RANDOM, pages 562-573, 2009.
33. T. S. Jayram, R. Kumar, and D. Sivakumar. The one-way communication complexity of hamming distance. Theory of Computing, 4(1):129-135, 2008.
34. T. S. Jayram and D. P. Woodruff. Optimal bounds for Johnson-Lindenstrauss transforms and streaming problems with low error. In Proceedings of the 22nd Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 1-10, 2011.
35. D. M. Kane and J. Nelson. A derandomized sparse Johnson-Lindenstrauss transform. CoRR, abs/1006.3585v1, 2010.
36. D. M. Kane, J. Nelson, and D. P. Woodruff. On the exact space complexity of sketching and streaming small norms. In Proceedings of the 21st Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 1161-1178, 2010.
37. B. Krishnamurthy, S. Sen, Y. Zhang, and Y. Chen. Sketch-based change detection: methods, evaluation, and applications. In Proceedings of the 3rd ACM SIGCOMM Conference on Internet Measurement (IMC), pages 234-247, 2003.
38. p (0 < p ≤ 2) using stable random projections. In Proceedings of the 19th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 10-19, 2008.
39. P. Li. On practical algorithms for entropy estimation and the improved sample complexity of compressed counting. CoRR, abs/1004.3782, 2010.
40. S. Muthukrishnan. Data Streams: Algorithms and Applications. Foundations and Trends in Theoretical Computer Science, 1(2):117-236, 2005.
41. J. Nelson and D. Woodruff. Fast Manhattan sketches in data streams. In Proceedings of the 29th ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems (PODS), pages 99-110, 2010.
42. J. Nelson and D. P. Woodruff. A near-optimal algorithm for L1-difference. CoRR, abs/0904.2027, 2009.
43. N. Nisan. Pseudorandom generators for space-bounded computation. Combinatorica, 12(4):449-461, 1992.
44. Open Problems in Data Streams and Related Topics. IITK Workshop on Algorithms for Data Streams, 2006. http://www.cse.iitk.ac.in/users/sganguly/ data-stream-probs.pdf.
45. A. Pagh and R. Pagh. Uniform hashing in constant time and linear space. SIAM J. Comput., 38(1):85-96, 2008.
46. M. E. Saks and X. Sun. Space lower bounds for distance approximation in the data stream model. In Proceedings of the 34th Annual ACM Symposium on Theory of Computing (STOC), pages 360-369, 2002.
47. T. Sarlós. Improved approximation algorithms for large matrices via random projections. In In Proc. 47th Annu. IEEE Sympos. Found. Comput. Sci, pages 143-152. IEEE Computer Society, 2006.
48. D. Srivastava and S. Venkatasubramanian. Information theory for data management. In SIGMOD '10: Proceedings of the 2010 international conference on Management of data, pages 1255-1256, New York, NY, USA, 2010. ACM.
49. M. Thorup and Y. Zhang. Tabulation based 4-universal hashing with applications to second moment estimation. In Proceedings of the 15th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 615-624, 2004.
50. J. von zur Gathen and J. Gerhard. Modern Computer Algebra. Cambridge University Press, 1999.
51. M. Wong. Complex Analysis, volume 2. World Scientific, 2008.
52. D. P. Woodruff. Optimal space lower bounds for all frequency moments. In Proceedings of the 15th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 167-175, 2004.
53. D. P. Woodruff. Efficient and Private Distance Approximation in the Communication and Streaming Models. PhD thesis, Massachusetts Institute of Technology, 2007.
54. H. Zhao, A. Lall, M. Ogihara, O. Spatscheck, J. Wang, and J. Xu. A Data Streaming Algorithm for Estimating Entropies of OD Flows. In Proceedings of the Internet Measurement Conference (IMC), 2007.