Fast Manhattan sketches in data streams

Proceedings of the ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems

Volumn , Issue , 2010, Pages 99-109

  Nelson, Jelani ^a   Woodruff, David P ^b  

^a MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^b IBM ALMADEN RESEARCH CENTER   (United States)

Author keywords

clustering; data mining; sketching; streaming

Indexed keywords

ARBITRARY NUMBER; CLUSTERING DATA; DATA STREAM; MANHATTANS; NEAREST NEIGHBOR SEARCH; NETWORK MONITORING; POLY-LOGARITHMIC FACTORS; STREAMING ALGORITHM;

DATA MINING; TAXICABS;

DATABASE SYSTEMS;

EID: 77954738941     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1807085.1807101     Document Type: Conference Paper

References (57)

1. Charu C. Aggarwal, Alexander Hinneburg, and Daniel A. Keim. On the surprising behavior of distance metrics in high dimensional spaces. In ICDT, pages 420-434, 2001.
2. Rakesh Agrawal, King-Ip Lin, Harpreet S. Sawhney, and Kyuseok Shim. Fast similarity search in the presence of noise, scaling, and translation in time-series databases. In VLDB, pages 490-501, 1995.
3. Noga Alon, Yossi Matias, and Mario Szegedy. The Space Complexity of Approximating the Frequency Moments. J. Comput. Syst. Sci., 58(1):137-147, 1999.
4. Radu Berinde, Graham Cormode, Piotr Indyk, and Martin J. Strauss. Space-optimal heavy hitters with strong error bounds. In PODS, pages 157-166, 2009.
5. Kevin S. Beyer and Raghu Ramakrishnan. Bottom-up computation of sparse and iceberg cubes. In SIGMOD Conference, pages 359-370, 1999.
6. Emmanuel J. Candès, Justin Romberg, and Terence Tao. Stable signal recovery from incomplete and inaccurate measurements. Communications on Pure and Applied Mathematics, 59(8), 2006.
7. Moses Charikar, Kevin Chen, and Martin Farach-Colton. Finding frequent items in data streams. In Proceedings of the 29th International Colloquium on Automata, Languages and Programming (ICALP), pages 693-703, 2002.
8. Surajit Chaudhuri, Rajeev Motwani, and Vivek R. Narasayya. On random sampling over joins. In SIGMOD Conference, pages 263-274, 1999.
9. Cisco NetFlow. http://www.cisco.com/go/netflow.
10. Kenneth L. Clarkson. Subgradient and sampling algorithms for l1 regression. In Proceedings of the 16th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), 2005.
11. Edith Cohen, Nick G. Duffield, Haim Kaplan, Carsten Lund, and Mikkel Thorup. Algorithms and estimators for accurate summarization of internet traffic. In Internet Measurement Comference, pages 265-278, 2007.
12. Graham Cormode, Mayur Datar, Piotr Indyk, and S. Muthukrishnan. Comparing data streams using hamming norms (how to zero in). IEEE Trans. Knowl. Data Eng., 15(3):529-540, 2003.
13. Graham Cormode and Minos N. Garofalakis. Sketching streams through the net: Distributed approximate query tracking. In VLDB, pages 13-24, 2005.
14. Graham Cormode, Piotr Indyk, Nick Koudas, and S. Muthukrishnan. Fast mining of massive tabular data via approximate distance computations. In ICDE, pages 605-, 2002.
15. Graham Cormode, Flip Korn, S. Muthukrishnan, and Divesh Srivastava. Space- and time-efficient deterministic algorithms for biased quantiles over data streams. In PODS, pages 263-272, 2006.
16. Graham Cormode, Flip Korn, and Srikanta Tirthapura. Time-decaying aggregates in out-of-order streams. In PODS, pages 89-98, 2008.
17. Graham Cormode and S. Muthukrishnan. An improved data stream summary: the count-min sketch and its applications. J. Algorithms, 55(1):58-75, 2005.
18. Graham Cormode and S. Muthukrishnan. Space efficient mining of multigraph streams. In Proceedings of the 24th ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems (PODS), pages 271-282, 2005.
19. Graham Cormode and S. Muthukrishnan. What's hot and what's not: tracking most frequent items dynamically. ACM Trans. Database Syst., 30(1):249-278, 2005.
20. Graham Cormode, S. Muthukrishnan, and Irina Rozenbaum. Summarizing and mining inverse distributions on data streams via dynamic inverse sampling. In VLDB, pages 25-36, 2005.
21. Yadolah Dodge. L1-Statistical Procedures and Related Topics. Institute for Mathematical Statistics, 1997.
22. Min Fang, Narayanan Shivakumar, Hector Garcia-Molina, Rajeev Motwani, and Jeffrey D. Ullman. Computing iceberg queries efficiently. In VLDB, pages 299-310, 1998.
23. Joan Feigenbaum, Sampath Kannan, Martin Strauss, and Mahesh Viswanathan. An approximate L1-difference algorithm for massive data streams. SIAM J. Comput., 32(1):131-151, 2002.
24. Dan Feldman, Morteza Monemizadeh, Christian Sohler, and David P. Woodruff. Coresets and sketches for high dimensional subspace problems. In Proceedings of the 21st Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), to appear, 2010.
25. Sumit Ganguly. personal communication.
26. Sumit Ganguly and Graham Cormode. On estimating frequency moments of data streams. In Proceedings of the 11th International Workshop on Randomization and Computation (RANDOM), pages 479-493, 2007.
27. Sumit Ganguly, Abhayendra N. Singh, and Satyam Shankar. Finding frequent items over general update streams. In Proceedings of the 20th International Conference on Scientific and Statistical Database Management (SSDBM), pages 204-221, 2008.
28. Anna C. Gilbert, Martin J. Strauss, Joel A. Tropp, and Roman Vershynin. One sketch for all: fast algorithms for compressed sensing. In STOC, pages 237-246, 2007.
29. Jiawei Han, Jian Pei, Guozhu Dong, and Ke Wang. Efficient computation of iceberg cubes with complex measures. In SIGMOD Conference, pages 1-12, 2001.
30. John Hershberger, Nisheeth Shrivastava, Subhash Suri, and Csaba D. Tóth. Space complexity of hierarchical heavy hitters in multi-dimensional data streams. In Proceedings of the 24th ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems (PODS), pages 338-347, 2005.
31. Piotr Indyk. Stable distributions, pseudorandom generators, embeddings, and data stream computation. J. ACM, 53(3):307-323, 2006.
32. Piotr Indyk and Andrew McGregor. Declaring independence via the sketching of sketches. In SODA, pages 737-745, 2008.
33. Piotr Indyk and David P. Woodruff. Polylogarithmic private approximations and efficient matching. In TCC, pages 245-264, 2006.
34. T. S. Jayram and David P. Woodruff. The data stream space complexity of cascaded norms. In Proceedings of the 50th Annual IEEE Symposium on Foundations of Computer Science (FOCS), 2009.
35. Daniel M. Kane, Jelani Nelson, and David P. Woodruff. On the exact space complexity of sketching small norms. In Proceedings of the 21st Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), to appear, 2010.
36. Khaled Labib and V. Rao Vemuri. A hardware-based clustering approach for anomaly detection, 2006.
37. Wing Cheong Lau, Murali S. Kodialam, T. V. Lakshman, and H. Jonathan Chao. Datalite: a distributed architecture for traffic analysis via light-weight traffic digest. In BROADNETS, pages 622-630, 2007.
38. Kenneth D. Lawrence and Jeffrey L. Arthur. Robust Regression. Dekker, 1990.
39. p (0 < p ≤ 2) using stable random projections. In Proceedings of the 19th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 10-19, 2008.
40. Ping Li, Trevor Hastie, and Kenneth Ward Church. Very sparse random projections. In KDD, pages 287-296, 2006.
41. Hendrik P. Lopuhaä and Peter J. Rousseeuw. Breakdown points of affine equivalent estimators of multivarite location and covariance matrices. Annals of Statistics, 19(1):229-248, 1991.
42. Andre Madeira and S. Muthukrishnan. Functionally private approximation for negligibly-biased estimators. In Proceedings of the 29th International Conference on Foundations of Software Technology and Theoretical Computer Science (FSTTCS), 2009.
43. p sampling with applications. In Proceedings of the 21st Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), to appear, 2010.
44. Jelani Nelson and David P. Woodruff. A near-optimal algorithm for L1-difference. CoRR, abs/0904.2027, 2009.
45. Jiawang Nie, Pablo A. Parillo, and Bernd Sturmfels. Semidefinite representation of the k-ellipse. Algorithms in Algebraic Geometry, IMA Volumes in Mathematics and its Applications, 146:117-132, 2008.
46. Noam Nisan. Pseudorandom generators for space-bounded computation. Combinatorica, 12(4):449-461, 1992.
47. Open Problems in Data Streams and Related Topics. IITK Workshop on Algorithms for Data Streams, 2006. http://www.cse.iitk.ac.in/users/sganguly/ data-stream-probs.pdf.
48. Anna Pagh and Rasmus Pagh. Uniform hashing in constant time and linear space. SIAM J. Comput., 38(1):85-96, 2008.
49. Peter J. Rousseeuw and Annick M. Lerow. Robust Regression and Outlier Detection. John Wiley, 1987.
50. Robert T. Schweller, Zhichun Li, Yan Chen, Yan Gao, Ashish Gupta, Yin Zhang, Peter A. Dinda, Ming-Yang Kao, and Gokhan Memik. Reversible sketches: enabling monitoring and analysis over high-speed data streams. IEEE/ACM Trans. Netw., 15(5):1059-1072, 2007.
51. n. IEEE Transactions on Signal Processing, pages 771-782, 1999.
52. Mikkel Thorup and Yin Zhang. Tabulation based 4-universal hashing with applications to second moment estimation. In Proceedings of the 15th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 615-624, 2004.
53. Salil P. Vadhan. Pseudorandomness II. Manuscript. http://people.seas. harvard.edu/~salil/cs225/spring09/lecnotes/FnTTCS-vol2.pdf.
54. Endre V. Weiszfeld. Sur le point pour lequel la somme des distances de n points donnes est minimum. Tohoku Math, 43:355-386, 1937.
55. David P. Woodruff. Optimal space lower bounds for all frequency moments. In Proceedings of the 15th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 167-175, 2004.
56. p-distance for every p. Manuscript, 2009.
57. Byoung-Kee Yi and Christos Faloutsos. Fast time sequence indexing for arbitrary Lp norms. In VLDB, pages 385-394, 2000.