Learn more, sample less: Control of volume and variance in network measurement

IEEE Transactions on Information Theory

Volumn 51, Issue 5, 2005, Pages 1756-1775

  Duffield, Nick ^a   Lund, Carsten ^a   Thorup, Mikkel ^a  

^a AT AND T LABS RESEARCH   (United States)

Author keywords

Estimation; Flows; Internet measurement; Sampling; Variance reduction

Indexed keywords

ALGORITHMS; CONVERGENCE OF NUMERICAL METHODS; NETWORK PROTOCOLS; PROBABILITY; ROUTERS; TELECOMMUNICATION TRAFFIC; THEOREM PROVING;

INTERNET MEASUREMENT; INTERNET PROTOCOL; VARIANCE REDUCTION;

INTERNET;

EID: 18544383616     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2005.846400     Document Type: Article

References (36)

1. J. Apisdorf, K. Claffy, K. Thompson, and R. Wilder. OC3MON: Flexible, Affordable, High Performance Statistics Collection. [Online]. Available: http://www.nlanr.net/NA/Oc3mon
2. B. Babcock, S. Babu, M. Datar, R. Motwani, and J. Widom, "Models and Issues in data stream systems," in Proc. 21st ACM SIGACT-SIGMOD-SIGART Symp. Principles of Database Systems, Madison, WI, Jun. 2002, pp. 1-16.
3. R. Cáceres, N. G. Duffield, A. Feldmann, J. Friedmann, A. Greenberg, R. Greer, T. Johnson, C. Kalmanek, B. Krishnamurthy, D. Lavelle, P. P. Mishra, K. K. Ramakrishnan, J. Rexford, F. Troe, and J. E. van der Merwe, "Measurement and analysis of IP network usage and behavior," IEEE Commun. Mag., vol. 38, no. 5, pp. 144-151, May 2000.
4. Cisco NetFlow. [Online]. Available: http://www.cisco.com/warp/public/ 732/netflow/index.html and http://www.cisco.com/univercd/cc/td/doc/ product/software/ios120/,120newft/1201imit/120s/120s11/12s_sanf.htm
5. K. C. Claffy, H.-W. Braun, and G. C. Polyzos, "Parameterizable methodology for internet traffic flow profiling," IEEE J. Sel. Areas Commun., vol. 13, no. 8, pp. 1481-1494, Oct. 1995.
6. K. C. Claffy, G. C. Polyzos, and H.-W. Braun, "Application of sampling methodologies to network traffic characterization," Comp. Commun. Rev., vol. 23, no. 4, pp. 194-203, Oct. 1993.
7. W. Cochran, Sampling Techniques. New York: Wiley, 1987.
8. T. H. Cormen, C. E. Leiserson, and R. L. Rivest, Introduction to Algorithms. Cambridge, MA: MIT Press/McGraw-Hill, 1990.
9. N. G. Duffield, "Sampling for passive internet measurement: A review," Statist. Sci., to be published.
10. N. G. Duffield and M. Grossglanser, "Trajectory sampling for direct traffic observation," IEEE/ACM Trans. Networking. To be published. Abridged version Proc. ACM Sigcomm 2000, Computer Commun. Rev., vol 30, no 4, Oct. 2000, pp. 271-282.
11. N. G. Duffield and C. Lund, "Predicting resource usage and estimation accuracy in ah IP flow measurement collection infrastructure," presented at the ACM SIGCOMM Internet Measurement Conf. 2003, Miami Beach, FL, Oct. 27-29, 2003.
12. N. G. Duffield. C. Lund, and M. Thomp, "Charging from sampled network usage," presented at the ACM SIGCOMM Internet Measurement Workshop 2001, San Francisco, CA, Nov. 1-2, 2001.
13. N. G. Duffield. C. Lund, and M. Thomp, "Estimating from distributions from sampled flow statistics," presented at the ACM Sigcomm 2003, Karlsruhe, Germany, Aug. 25-29, 2003,
14. N. G. Duffield. C. Lund, and M. Thomp, "Flow sampling under hard resource constraints," in Proc. ACM SIGMETRICS 2004, New York, NY, Jun. 12-16, 2004, pp. 85-96.
15. C. Estan and G. Varghese, "New directions in traffic measurement and accounting," in Proc. SIGCOMM 2002, Pittsburgh, PA, Aug. 19-23, 2002, pp. 323-336.
16. A. Feldmann, A. Greenberg, C. Lund, N. Reingold, J. Rexford, and F. True, "Deriving traffic demands for operational IP networks: Methodology and experience," in Proc. ACM Sigcomm 2000, Comp. Commun. Rev., vol. 30, Oct. 2000, pp. 257-270.
17. A. Feldmann, A. Greenberg, C. Lund, N. Reingold, and J. Rexford, "Netscope: Traffic engineering for IP networks," IEEE Network, vol. 14, no. 2, pp. 11-19, Mar./Apr. 2000.
18. A. Feldmann, A. C. Gilbert, and W. Willinger, "Data networks as cascades: Investigating the multifractal nature of internet WAN traffic," in Proc. ACM SIGCOMM '98, Vancouver, BC, Canada, Oct. 1998, pp. 42-55.
19. A. Feldmann, J. Rexford and R. Cáceres, "Efficient policies for carrying web traffic over flow-switched networks," IEEE/ACM Trans. Networking, vol. 6, no. 6, pp. 673-685, Dec. 1998.
20. B. Fortz and M. Thomp, "Internet traffic engineering by optimizing OSPF weights," in Proc. IEEE INFOCOM 2000, Tel Aviv, Israel, 2000, pp. 519-528.
21. B. Hajek, "Extremal splittings of point processes," Math. Oper. Res., vol. 10, pp. 543-556, 1985.
22. SFlow Accuracy and Billing. Inmon Corporation. [Online]. Available: http://www.inmon.com/PDF/sFlowBilling.pdf
23. Internet Protocol Flow Information Export (IPFIX). IETF Working Group. [Online]. Available: http://net.doit.wise.edu/ipfix/
24. P. L'Ecuyer, "Efficient and portable combined random number generators," Commun. ACM, vol. 31, pp. 742-749 and 774, 1988.
25. W. E. Leland, M. S. Taqqu, W. Willinger, and D. V. Wilson, "On the self-similar nature of ethernet traffic," in Proc. ACM SIGCOMM '93 , San Francisco, CA, Oct. 1993, pp. 183-193.
26. D. Moore, G. Voelker, and S. Savage, "Inferring internet denial of service activity," in Proc.2001 USENIX Security Symp., Washington, DC, Aug. 2001.
27. R. Motwani and P. Raghavan, Randomized Algorithms. Cambridge, U.K.: Cambridge Univ. Press, 1995.
28. S. Muthukrishnan, "Data streams: Algorithms and applications," presented at the 14th SODA, manuscript based on an invited talk, Baltimore, MD, 2003.
29. V. Paxson, "Automated packet trace analysis of TCP implementations," in Proc. ACM SIGCOMM '97, Cannes, France, Oct. 1997, pp. 167-179.
30. V. Paxson, G. Almes, J. Mahdavi, and M. Mathis. (1998) Framework for IP Performance Metrics. [Online]. Available: ftp://ftp.isi.edu/in-notes/ rfc2330.txt
31. J. Postel, "Transmission Control Protocol", RFC 793, 1981.
32. Argus. Qosient, LLC. [Online]. Available: http://www.qosient.com/argus/ index.htm
33. Real Time Flow Measurement. [Online]. Available: http://www.auckland.ac. nz/net/Internet/rtfm/
34. R. H. Riedi, M. S. Crouse, V. J. Ribeiro, and R. G. Baraniuk, "A multi-fractal wavelet model with application to network traffic," IEEE Trans. Inf. Theory, vol. 45, no. 3, pp. 992-1018, Apr. 1999.
35. [Online]. Available: http://www.riverstonenet.com/technology/
36. [Online]. Available: http://www.xacct.com