Cloud Resource Usage-Heavy Tailed Distributions Invalidating Traditional Capacity Planning Models

Journal of Grid Computing

Volumn 10, Issue 1, 2012, Pages 85-108

  Loboz, Charles ^a  

^a MICROSOFT   (United States)

Author keywords

Capacity planning; Power law; Probability distributions; Resource usage; Volatility

Indexed keywords

CAPACITY BUFFER; CAPACITY MODELS; CAPACITY PLANNING; CENTRAL LIMIT THEOREM; CONFIDENCE INTERVAL; HEAVY-TAILED; HEAVY-TAILED DISTRIBUTION; HIGH VOLATILITY; INTERNAL NETWORK; LOG-NORMAL; NON-NORMAL DISTRIBUTION; POWER LAW; RESOURCE USAGE; SAMPLE DISTRIBUTIONS; STOCK INDICES; STOCK MARKET ANALYSIS; UNDERLYING DISTRIBUTION; VOLATILITY; WORKLOAD MODELING;

FORECASTING; PROBABILITY DISTRIBUTIONS;

NORMAL DISTRIBUTION;

EID: 84859746610     PISSN: 15707873     EISSN: None     Source Type: Journal    
DOI: 10.1007/s10723-012-9211-x     Document Type: Article

References (31)

1. Clauset, A., Shalizi, C. R., Newman, M. E. J.: Power-law distributions in empirical data. SIAM Rev. 51, 661-703 (2009).
2. Clauset, A., Young, M., Gleditsch, K. S.: J. Conflict Resolut. 51, 58 (2007).
3. Goldstein, M. L., Morri, S. A., Yen, G. G.: Problems with fitting to the power-law distribution. Eur. Phys. J. B. 41(2), 255-258 (2004).
4. Gunther, N.: Guerilla capacity planning. iUniverse (October 31, 2000), ISBN-10: 3642065570.
5. James, A., Plank, M. J.: On fitting power laws to ecological data arxiv: 0712. 06131.
6. Leland, W., Taqqu, M., Willinger, W., Wilson, D.: On the self-similar nature of ethernet traffic, IEEE/ACM TON (1994).
7. Lilifoers, H. W.: J. Amer. Statist. Assoc. 64, 387-389 (1969).
8. Mantegna, R. N., Stanley, H. E.: An Introduction to Econophysics: Correlations and Complexity in Finance. Cambridge University Press, Cambridge (1999).
9. Marvasti, M. A.: How 'Normal' is your IT data. Proceedings of the Computer Measurement Group's 2009 International Conference, www. cmg. org.
10. Newman, M. E. J.: Power laws, Pareto distributions and Zipf's law. Contemp. Phys. 46, 323-351 (2006).
11. Shalizi, C.: Power law distributions, 1/f Noise, Long-Memory Time Series http://cscs. umich. edu/~crshalizi/notabene/power-laws. html.
12. van der Loo, M. P. J.: Distribution based outlier detection in univariate data, discussion paper 10003, Statistic Netherlands.
13. Agrawal, N., Bolosky, W. J., Douceur, J. R., Lorch, J. R.: A five-year study of file-system metadata. Trans. Storage 3, 3, Article 9 (October 2007). doi: 10. 1145/1288783. 1288788.
14. Li, H.: Workload dynamics on clusters and grids. J. Supercomput. 47(1), (2009).
15. Li, H., Muskulus, M., Wolters, L.: Modeling job arrivals in a data-intensive grid. In: Frachtenberg, E., Schwiegelshohn, U. (eds.) Int'l. Workshop on Job Scheduling Strategies for Parallel Processing (JSSPP), Revised Selected Papers, In: Lecture Notes in Computer Science, vol. 4376, pp. 210-231. Springer (2007).
16. Litzkow, M. J., Livny, M., Mutka, M. W.: Condor-a hunter of idle workstations, 8th International Conference on Distributed Computing Systems, pp. 104-111 (1988).
17. Iosup, A., Li, H., Jan, M., Anoep, S, Dumitrescu, C., Wolters, L., Dick, H., Epema, J.: The grid workloads archive. Future Gener. Comp. Sy. 24(7), 672-686 (2008).
18. Li, H., Heusdens, R., Muskulus, M. V., Wolters, L.: Analysis and synthesis of pseudo-periodic job arrivals in grids: a matching pursuit approach IEEE/ACM Intl. Symp. on Cluster Computing and the Grid (CCGrid) IEEE Computer Society, pp. 183-196 (2007).
19. Li, H., Muskulus, M., Wolters, L.: Modeling job arrivals in a data-intensive grid. In: Frachtenberg, E., Schwiegelshohn, U. (eds.) Int'l. Workshop on Job Scheduling Strategies for Parallel Processing (JSSPP), Revised Selected Papers, In: Lecture Notes in Computer Science, vol. 4376, pp. 210-231. Springer (2007).
20. Li, H., Wolters, L.: Towards a better understanding of workload dynamics on data-intensive clusters and grids. In: Int'l. Parallel & Distributed Processing Symposium (IPDPS), IEEE Computer Society, pp. 1-10 (2007).
21. Li, H.: Workload characterization, modeling, and prediction in grid computing. PhD thesis, https://openaccess. leidenuniv. nl/bitstream/1887/12574/1/Thesis. pdf.
22. Park, C., Hernandez-Campos, F., Marron, J. S., Donelson Smith, F.: Long-range dependence in a changing internet traffic mix. Comput. Netw. 48(3), 401-422 (2005).
23. Allspaw, J.: The art of capacity planning: scaling web resources, O'Reilly Media; 1 edn. (September 15, 2008), ISBN-10: 0596518579.
24. Albert, R., Barabási, A.-L.: Statistical mechanics of complex networks. Rev. Modern Phys. 74, 47-97 (2002).
25. Rasch, D., Guiard, V.: The robustness of parametric statistical methods. Psychol. Sci. 46(2), 175-208 (2004).
26. Peterson, D., Grossman, R.: Power laws in large shop DASD I/O Activity, CMG Proceedings, pp. 822-833 (Dec. 1995).
27. Peterson, D., Adams, D.: Fractal patterns in DASD I/O Traffic, CMG Proceedings, Dec, (1996).
28. Milligan, C., Peterson, D.: A practical approach for estimating true I/O skew, CMG Proceedings, pp. 970-981 (Dec. 1994).
29. Peterson, D.: Data center I/O patterns and power laws, CMG Proceedings (1996).
30. Adamic, L. A.: Zipf, Power-laws, and Pareto-a ranking tutorial. Xerox Palo Alto Research Center, Palo Alto, CA. Retrieved on 2011-07-26. http://www. hpl. hp. com/research/idl/papers/ranking/ranking. html.
31. Nicholls, P. T.: J. Am. Soc. Inform. Sci. 40, 379-385 (1989).