Confident estimation for multistage measurement sampling and aggregation

SIGMETRICS'08: Proceedings of the 2008 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems

Volumn 36, Issue 1 SPECIAL ISSUE, 2008, Pages 109-120

  Cohen, Edith ^a   Duffield, Nick ^a   Lund, Carsten ^a   Thorup, Mikkel ^a  

^a AT AND T LABS RESEARCH   (United States)

Author keywords

Confidence intervals; Estimation; Network measurement; Sampling

Indexed keywords

AGGLOMERATION; COMPUTER NETWORKS; COMPUTER SYSTEMS;

AGGREGATION OPERATIONS; CHERNOFF BOUNDS; CONFIDENCE INTERVALS; ESTIMATION ERRORS; FLOW STATISTICS; MULTIPLE STAGES; NETWORK DATUMS; NETWORK MEASUREMENT; NETWORK USAGES; PACKET SAMPLINGS; PARAMETERIZATIONS; SAMPLE AND HOLDS; SAMPLING METHODS; SAMPLING SCHEMES; UNBIASED ESTIMATES;

SAMPLING;

EID: 57349186526     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1375457.1375471     Document Type: Conference Paper

References (31)

1. N. Alon, N. Duffield, C. Lund, and M. Thorup. Estimating arbitrary subset sums with few probes. In Proc. 24th ACM Symp. on Principles of Database Systems (PODS), pages 317-325, 2005.
2. D. Brauckhoff, B. Tellenbach, A. Wagner, and M. May. Impact of traffic sampling on anomaly detection metrics. In Proc. Internet Measuement Conference (IMC 2006), Rio de Janeiro, Brazil, October 25-27 2006.
3. Cisco. White paper-netflow services and applications, http://www.cisco.com/warp/public/cc/pd/iosw/ioft/neflct/tech/napps-wp.htm.
4. K. C. Claffy, G. C. Polyzos, and H.-W. Braun. Application of Sampling Methodologies to Network Traffic Characterization. Computer Communication Review, 23(4):194-203, October 1993. appeared in Proceedings ACM SIGCOMM'93, San Francisco, CA, September 13-17, 1993.
5. E. Cohen, N. Duffield, H. Kaplan, C. Lund, and M. Thorup. Sketching unaggregated data streams for subpopulation-size queries. In Proc. 26th ACM Symp. on Principles of Database Systems (PODS), 2007.
6. E, Cohen, N. Grossaug, and H. Kaplan. Processing top-k queries from samples. In Proceeding CoNEXT'06, Lisbon, Portugal, December 4-6 2006.
7. N. Duffield. (Ed.) A Framework for Packet Selection and Reporting. Internet Draft, June 2007. Work in Progress.
8. N. Duffield and M. Grossglauser. Trajectory Sampling with Unreliable Reporting. In INFOCOM, 2004.
9. N. Duffield and C. Lund. Predicting resource usage and estimation accuracy in an ip flow measurement collection infrastructure. In ACM SIGCOMM Internet Measurement Workshop, 2003. Miami Beach, Fl, October 27-29, 2003.
10. N. Duffield, C. Lund, and M. Thorup. Charging from sampled network usage. In ACM SIGCOMM Internet Measurement Workshop, 2001. San Francisco, CA, November 1-2, 2001.
11. N. Duffield, C. Lund, and M. Thorup. Flow sampling under hard resource constraints. In Proc. ACM IFIP Conference on Measurement and Modeling of Computer Systems (SIGMETRICS/Performance), pages 85-96, 2004.
12. N. Duffield, C. Lund, and M. Thorup. Estimating flow distributions from sampled flow statistics. IEEE/ACM Trans. Netw., 13(5):933-946, 2005.
13. N. Duffield, C. Lund, and M. Thorup. Optimal combination of sampled network measurements. In Proc. 5th ACM SIGCOMM Internet Measurement Workshop (IMC), page to appear, 2005.
14. N. Duffield, C. Lund, and M. Thorup. Sampling to estimate arbitrary subset sums. Technical Report cs.DS/0509026. Computing Research Repository (CoRR), 2005. http://arxiv.org/abs/cs.DS/0509026. Preliminary journal version of [11].
15. N. G. Duffield and M. Grossglauser. Trajectory sampling for direct traffic observation. IEEE/ACM Transactions on Networking, 9(3):280-292, June 2001.
16. C. Estan, K. Keys, D. Moore, and G. Varghese. Building a better netflow. In Proceedings of the ACM SIGCOMM 04, New York, NY, June 12-16 2004.
17. C. Estan and G. Varghese. New directions in traffic measurement and accounting. In Proc. ACM SIGCOMM '2002, Pittsburgh, PA, August 2002.
18. M. Gibbons and Y. Mafias. New sampling-based summary statistics for improving approximate query answers. In Proceedings ACM SIGMOD'98, Seattle, Washington, June 2-4 1998.
19. D. Horvitz and D. Thompson. A generalization of sampling without replacement from a finite universe. J. Arner. Statist. Assoc., 47, 663-685 1952.
20. J. Jedwab, P. Phaal, and B. Pinna. Traffic estimation for the largest sources in a network, using packet sampling with limited storage. Technical Report HPL-92-3, HP Laboratories, Bristol, March 1992. url: http://www.hpl.hp. com/ techreports /92/HPL-92-35.html.
21. T. Johnson, S. Muthukrishnan, and I. Rozenbaum. Sampling algorithms in a stream operator. In Proc. ACM SIGMOD, pages 1-12, 2005.
22. K. Keys, D. Moore, and C. Estan. A robust system for accurate real-time summaries of internet traffic. In Proceedings ACM SIGMETRICS'05, Banff, Alberta, Canada, June 6-10 2005.
23. R. R. Kompella and C. Estan. The power of slicing in internet flow measurement. In Proc. ACM Internet Measurement Conference, Berkeley, CA, October 2005.
24. J. Mai, C.-N. Chuah, A. Sridharan, T. Ye, and H. Zang. Is sampled data sufficient for anomaly detection? In Proc. Internet Measuement Conference (IMC 2006), Rio de Janeiro, Brazil, October 25-27 2006.
25. P. Phaal, S. Panchen, and N. McKee. Inmon corporation's snow: A method for monitoring traffic in switched and routed networks. RFC 3176, September 2001. url:http://www.ietf.org/rfc/rfc3176.txt.
26. J. Reeves and S. Panchen. Traffic monitoring with packet-based sampling for defense against security threats. In Proceedings of Passive and Active Measurement Workshop (PAM 2002), Fort Collins, CO, USA, March 25-26 2002.
27. M. Szegedy. The DLT priority sampling is essentially optimal. In Proc. 38th STOC, pages 150-158, 2006.
28. M. Szegedy and M. Thorup. On the variance of subset sum estimation. Technical Report cs.DS/0702029, Computing Research Repository (CoRR), 2007. http://arxiv.org/abs/cs.DS/0702029.
29. M. Thorup. Confidence intervals for priority sampling. In Proc. ACM SIGMETRICS/Performance 2006, pages 252-263, Saint-Malo, France, June 26-30 2006.
30. J. Turian and I. D. Melamed. Computational challenges in parsing by classification. In HLT-NAACL Workshop on Computationally Hard Problems and Joint Inference in Speech and Language Processing, New York, NY, 2006.
31. T. Zseby. Deployment of sampling methods for sla validation with non-intrusive measurements. In Proceedings of Passive and Active Measurement Workshop (PAM 2002), Fort Collins, CO, USA, March 25-26 2002.