Efficient stream sampling for variance-optimal estimation of subset sums

SIAM Journal on Computing

Volumn 40, Issue 5, 2011, Pages 1402-1431

  Cohen, Edith ^a   Duffield, Nick ^a   Kaplan, Haim ^b   Lund, Carsten ^a   Thorup, Mikkel ^a  

^a AT AND T LABS RESEARCH   (United States)

^b TEL AVIV UNIVERSITY   (Israel)

Author keywords

Reservoir sampling; Sampling without replacement; Subset sum estimation; Weighted sampling

Indexed keywords

ARBITRARY SUBSETS; CONCRETE SET; ESTIMATION QUALITY; OPTIMALITY; SAMPLING SCHEMES; SAMPLING WITHOUT REPLACEMENT; STREAM SAMPLING; SUBSET SUM; UNBIASED ESTIMATION;

ESTIMATION; OPTIMIZATION;

SET THEORY;

EID: 81255191687     PISSN: 00975397     EISSN: None     Source Type: Journal    
DOI: 10.1137/10079817X     Document Type: Article

References (36)

1. R.J. Adler, R.E. Feldman, and M.S. Taqqu, A Practical Guide to Heavy Tails, Birkhäuser, Boston, 1998.
2. S. Alstrup, T. Husfeldt, and T. Rauhe, Marked ancestor problems, in Proceedings of the 39th Annual IEEE Symposium on Foundations of Computer Science (FOCS), 1998, pp. 534-544.
3. M.T. Chao, A general purpose unequal probability sampling plan, Biometrika, 69 (1982), pp. 653-656.
4. S. Chaudhuri, R. Motwani, and V.R. Narasayya, On random sampling over joins, in Proceedings of the ACM SIGMOD International Conference on Management of Data, 1999, pp. 263-274.
5. E. Cohen, N.G. Duffield, H. Kaplan, C. Lund, and M. Thorup, Composable, scalable, and accurate weight summarization of unaggregated data sets, in Proceedings of the 35th International Conference on Very Large Data Bases, Vol. 2, 2009, pp. 431-442.
6. E. Cohen and H. Kaplan, Bottom-k sketches: Better and more efficient estimation of aggregates (poster), in Proceedings of the 2007 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems, 2007, pp. 353-354. (Pubitemid 350158099)
7. E. Cohen and H. Kaplan, Summarizing data using bottom-k sketches, in Proceedings of the 26th ACM Symposium on Principles of Distributed Computing (PODC), 2007, pp. 225-234. (Pubitemid 350229319)
8. E. Cohen and H. Kaplan, Tighter estimation using bottom-k sketches, in Proceedings of the 34th International Conference on Very Large Data Bases, 2008, pp. 213-224.
9. T.H. Cormen, C.E. Leiserson, R.L. Rivest, and C. Stein, Introduction to Algorithms, 2nd ed., MIT Press, Cambridge, MA, McGraw-Hill, Boston, 2001.
10. C. Cranor, T. Johnson, V. Shkapenyuk, and O. Spatcheck, Gigascope: A stream database for network applications, in Proceedings of the 2003 ACM SIGMOD International Conference on Management of Data, 2003, pp. 647-651.
11. D. Dubhashi and D. Ranjan, Balls and bins: A study in negative dependence, Random Structures Algorithms, 13 (1998), pp. 99-124.
12. N.G. Duffield, C. Lund, and M. Thorup, Learn more, sample less: Control of volume and variance in network measurements, IEEE Trans. Inform. Theory, 51 (2005), pp. 1756-1775. (Pubitemid 40654447)
13. N.G. Duffield, C. Lund, and M. Thorup, Priority sampling for estimation of arbitrary subset sums, J. ACM, 54 (2007), 32.
14. P.S. Efraimidis and P.G. Spirakis, Weighted random sampling with a reservoir, Inform. Process. Lett., 97 (2006), pp. 181-185. (Pubitemid 43060080)
15. C. Estan and G. Varghese, New directions in traffic measurement and accounting, in Proceedings of the 2002 ACM SIGCOMM Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication, 2002, pp. 323-331.
16. C.T. Fan, M.E. Muller, and I. Rezucha, Development of sampling plans by using sequential (item by item) selection techniques and digital computers, J. Amer. Statist. Assoc., 57 (1962), pp. 387-402.
17. M.L. Fredman and M.E. Saks, The cell probe complexity of dynamic data structures, in Proceedings of the 21st Annual ACM Symposium on Theory of Computing (STOC), 1989, pp. 345-354.
18. M. Gibbons and Y. Matias, New sampling-based summary statistics for improving approximate query answers, in Proceedings of the 1998 ACM SIGMOD International Conference on Management of Data, 1998, pp. 331-342. (Pubitemid 128655980)
19. D.G. Horvitz and D.J. Thompson, A generalization of sampling without replacement from a finite universe, J. Amer. Statist. Assoc., 47 (1952), pp. 663-685.
20. T. Johnson, S. Muthukrishnan, and I. Rozenbaum, Sampling algorithms in a stream operator, in Proceedings of the 2005 ACM SIGMOD International Conference on Management of Data, 2005, pp. 1-12. (Pubitemid 43038912)
21. D.E. Knuth, The Art of Computer Programming, Vol. 2: Seminumerical Algorithms, Addison-Wesley, Reading, MA, 1969.
22. D. Moore, V. Paxson, S. Savage, C. Shannon, S. Staniford, and N. Weaver, Inside the slammer worm, IEEE Security and Privacy Magazine, 1 (2003), pp. 33-39.
23. S. Muthukrishnan, Data streams: Algorithms and applications, Found. Trends Theoret. Comput. Sci., 1 (2005), pp. 117-236. (Pubitemid 44111222)
24. The Netflix Prize, http://www.netflixprize.com/.
25. A. Panconesi and A. Srinivasan, Randomized distributed edge coloring via an extension of the Chernoff-Hoeffding bounds, SIAM J. Comput., 26 (1997), pp. 350-368. (Pubitemid 127652069)
26. K. Park, G.Kim, andM.Crovella, On the relationship between file sizes, transport protocols, and self-similar network traffic, in Proceedings of the 4th IEEE International Conference on Network Protocols (ICNP), 1996, pp. 171-180.
27. M. Pǎtraşcu, Private communication, 2009.
28. C.-E. Särndal, B. Swensson, and J. Wretman, Model Assisted Survey Sampling, Springer, New York, 1992.
29. A. Srinivasan, Distributions on level-sets with applications to approximation algorithms, in Proceedings of the 42nd Annual IEEE Symposium on Foundations of Computer Science (FOCS), 2001, pp. 588-597. (Pubitemid 33068420)
30. M. Szegedy, The DLT priority sampling is essentially optimal, in Proceedings of the 38th Annual ACM Symposium on Theory of Computing (STOC), 2006, pp. 150-158. (Pubitemid 44306548)
31. M. Szegedy and M. Thorup, On the variance of subset sum estimation, in Proceedings of the 15th Annual European Symposium on Algorithms, Lecture Notes in Comput. Sci. 4698, Springer, New York, 2007, pp. 75-86.
32. M. Thorup, Confidence intervals for priority sampling, in Proceedings of the Joint International Conference on Measurement and Modeling of Computer Systems, SIGMETRICS/ Performance, 2006, pp. 252-253.
33. M. Thorup, Equivalence between priority queues and sorting, J. ACM, 54 (2007), 28.
34. Y. Tillé, An elimination procedure for unequal probability sampling without replacement, Biometrika, 83 (1996), pp. 238-241.
35. Y. Tillé, Sampling Algorithms, Springer, New York, 2006.
36. J.S. Vitter, Random sampling with a reservoir, ACM Trans. Math. Software, 11 (1985), pp. 37-57.