The datacenter as a computer: An introduction to the design of warehouse-scale machines

Synthesis Lectures on Computer Architecture

Volumn 6, Issue , 2009, Pages 1-107

  Barroso, Luiz André ^a   Hölzle, Urs ^a  

^a GOOGLE INC   (United States)

Author keywords

Cloud computing; Cluster computing; Computer organization and design; Data centers; Distributed systems; Energy efficiency; Fault tolerant computing; Internet services

Indexed keywords

CLOUD COMPUTING; COMPUTER ORGANIZATION AND DESIGN; DATA CENTERS; DISTRIBUTED SYSTEMS; FAULT-TOLERANT COMPUTING; INTERNET SERVICES;

COMPUTER SOFTWARE; ENERGY EFFICIENCY; FAULT TOLERANT COMPUTER SYSTEMS; INTERNET; MACHINE DESIGN; SOFTWARE ARCHITECTURE; WAREHOUSES;

CLUSTER COMPUTING;

EID: 67649170859     PISSN: 19353235     EISSN: 19353243     Source Type: Book Series    
DOI: 10.2200/S00193ED1V01Y200905CAC006     Document Type: Article

References (90)

1. M. Al-Fares, A. Loukissas, and A. Vahdat, "A scalable, commodity datacenter network architecture," in Proceedings of the ACM SIGCOMM 2008 Conference on Data Communication, Seattle, WA, August 17-22, 2008.
2. D. Atwood and J. G. Miner, "Reducing datacenter costs with an air economizer," IT@Intel Brief, August 2008.
3. L. N. Bairavasundaram, G. R. Goodson, S. Pasupathy, and J. Schindler, "An analysis of latent sector errors in disk drives," in Proceedings of the 2007 ACM SIGMETRICS international Conference on Measurement and Modeling of Computer Systems, San Diego, CA, June 12-16, 2007. SIGMETRICS '07. doi:10.1145/1254882.1254917
4. P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neugebauer, I. Pratt, and A. Warfeld, "Xen and the art of virtualization," in Proceedings of the Nineteenth ACM Symposium on Operating Systems Principles, Bolton Landing, NY, October 19-22, 2003. SOSP '03. doi:10.1145/945445.945462
5. L. Barroso, J. Dean, and U. Hölzle, "Web search for a planet: the architecture of the Google cluster," IEEE Micro, April 2003. doi:10.1109/MM.2003.1196112
6. L. A. Barroso and U. Hölzle, "The case for energy-proportional computing," IEEE Computer, vol. 40, no. 12 (Dec. 2007).
7. M. Burrows, "The chubby lock service for loosely - coupled distributed systems," in Proceedings of OSDI'06: Seventh Symposium on Operating System Design and Implementation, Seattle, WA, November 2006.
8. P. Bahl, R. Chandra, A. Greenberg, S. Kandula, D. A. Maltz, and M. Zhang, "Towards highly reliable enterprise network services via inference of multi - level dependencies," in Proceedings of SIGCOMM, 2007.
9. P. Barham, R. Isaacs, R. Mortier, and D. Narayanan, "Magpie: online modeling and performance-aware systems," in Proceedings of USENIX HotOS IX 2003.
10. E. A. Brewer, "Lessons from giant - scale services," IEEE Internet Computing, vol. 5, no. 4 ( July/Aug. 2001), pp. 46-55. doi:10.1109/4236.939450
11. E. V. Carrera, E. Pinheiro, and R. Bianchini, "Conserving disk energy in network servers," in Proceedings of the 17th Annual International Conference on Supercomputing, San Francisco, CA, June 23-26, 2003. ICS '03. doi:10.1145/782814.782829
12. B. Chandra, M. Dahlin, L. Gao, and A Nayate, "End-to-end WAN service availability," in Proceedings of the 3rd Conference on USENIX Symposium on Internet Technologies and Systems - Volume 3, San Francisco, CA, March 26-28, 2001.
13. F. Chang, J. Dean, S. Ghemawat, W. C. Hsieh, D. A. Wallach, M. Burrows, T. Chandra, A. Fikes, and R. E. Gruber, "Bigtable: a distributed storage system for structured data," in Proceedings of OSDI 2006, Seattle, WA, 2004.
14. J. Chase, D. Anderson, P. Thakar, A. Vahdat, and R. Doyle, "Managing energy and server resources in hosting centers," in Proceedings of the ACM Symposium on Operating Systems Principles (SOSP), Banff, Alberta, Canada, June 2001. doi:10.1145/502034.502045
15. G. Chen, W. He, J. Liu, S. Nath, L. Rigas, L. Xiao, and F. Zhao, "Energy aware server provisioning and load dispatching for connection-intensive Internet services," Microsoft Research Technical Report MSR-TR-2007-130, 2007.
16. Y. Chen, A. Das, W. Qin, A. Sivasubramaniam, Q. Wang, and N. Gautam, "Managing server energy and operational costs in hosting centers," in Proceedings of the ACM SIG-METRICS '05, Banff, Alberta, Canada, June 2005. doi:10.1145/1064212.1064253
17. Climate savers computing effciency specs. Available at http://www.climatesaverscomputing .org/about/tech-specs.
18. E. F. Coyle, "Improved muscular effciency displayed as tour de France champion matures," Journal of Applied Physiology, Mar. 2005. doi:10.1152/japplphysiol.00216.2005
19. J. Dean and S. Ghemawat, "MapReduce: simplifed data processing on large clusters," Communications of the ACM, vol. 51, no. 1 (2008), pp. 107-113.
20. G. DeCandia, D. Hastorun, M. Jampani, G. Kakulapati, A. Lakshman, A. Pilchin, S. Siv-asubramanian, P. Vosshall, and W. Vogels, "Dynamo: Amazon's highly available key-value store," in the Proceedings of the 21st ACM Symposium on Operating Systems Principles, Stevenson, WA, October 2007.
21. Dell Energy Calculator. Available at http://www.dell.com/calc.
22. T. J. Dell, "A white paper on the benefts of chipkill-correct ECC for PC server main memory," IBM Microelectronics Division, Rev 11/19/1997.
23. D. Dyer, "Current trends/challenges in datacenter thermal management-a facilities perspective," presentation at ITHERM, San Diego, CA, June 1, 2006.
24. J. Elerath and S. Shah, "Server class disk drives: how reliable are they?", IEEE Reliability and Maintainability, 2004 Annual Symposium - RAMS, January 2004. doi:10.1109/ RAMS.2004.1285439
25. Dupont Fabros Technology Inc. SEC Filing (S-11) 333-145294, August 9, 2007.
26. U.S. Environmental Protection Agency, "Report to Congress on server and datacenter energy effciency," Public Law 109-431, August 2, 2007.
27. X. Fan, W. Weber, and L. A. Barroso, "Power provisioning for a warehouse-sized computer," in Proceedings of the 34th Annual International Symposium on Computer Architecture, San Diego, CA, June 09-13, 2007. ISCA '07. doi:10.1145/1250662.1250665
28. M. E. Femal and V. W. Freeh, "Safe overprovisioning: using power limits to increase aggregate throughput," in 4th International Workshop on Power Aware Computer Systems (PACS 2004), December 2004, pp. 150-164.
29. M. E. Femal and V. W. Freeh, "Boosting datacenter performance through non-uniform power allocation," in Proceedings of the Second International Conference on Automatic Computing, June 13-16, 2005, pp. 250-261.
30. R. Fonseca, G. Porter, R. H. Katz, S. Shenker, and I. Stoica, "X-trace: a pervasive network tracing framework," in Proceedings of 4th USENIX Symposium on Networked Systems Design & Implementation, 2007, pp. 271-284.
31. S. Ghemawat, H. Gobioff, and S-T. Leung, "The Google fle system", in Proceedings of the 19th ACM Symposium on Operating Systems Principles, Lake George, NY, October 2003. doi:10.1145/945445.945450
32. Google Inc., "Effcient computing - step 2: effcient datacenters". Available at http://www .google.com/corporate/green/ datacenters/step2.html.
33. Google Inc., "Effcient Data Center Summit, April 2009". Available at http://www.google .com/corporate/green/datacenters/summit.html.
34. Google Inc., "Effcient Data Center, Part 1". Available at http://www.youtube.com/ watch?v=Ho1GEyftpmQ , starting at 0:5930.
35. J. Gray, "A census of tandem system availability between 1985 and 1990," Tandem Technical Report 90.1, January 1990.
36. The Green 500. Available at http://www.green500.org.
37. The Green Grid datacenter power effciency metrics: PUE and DCiE. Available at http:// www.thegreengrid.org/sitecore/content/Global/Content/white- papers/The-Green-Grid-Data-Center-Power-Effciency-Metrics-PUE-and-DCiE.aspx.
38. Green Grid, "Quantitative analysis of power distribution confgurations for datacenters". Available at http://www.thegreengrid.org/ gg-content/.
39. Green Grid, "Seven strategies to improve datacenter cooling effciency". Available at http:// www.thegreengrid.org/gg-content/.
40. SNIA Green Storage Initiative. Available at http://www.snia.org/forums/ green/.
41. S. Greenberg, E. Mills, and B. Tschudi, "Best practices for datacenters: lessons learned from benchmarking 22 datacenters," 2006 ACEEE Summer Study on Energy Effciency in Buildings. Available at http://eetd.lbl.gov/ EA/mills/emills/PUBS/PDF/ACEEE-datacenters.pdf.
42. The Hadoop Project. Available at http://hadoop.apache.org
43. J. Hamilton, "On designing and deploying internet-scale services," in Proceedings of USE-NIX LISA, 2007.
44. J. Hamilton, "Cooperative expendable micro-slice servers (CEMS): low cost, low power servers for internet-scale services," in Proceedings of the 4th Biennial Conference on Innovative Data Systems Research (CIDR), Asilomar, CA, January 4-7, 2009.
45. T. Heath, A. P. Centeno, P. George, L. Ramos, Y. Jaluria, and R. Bianchini, "Mercury and freon: temperature emulation and management for server systems," in Proceedings of the ACM International Conference on Architectural Support for Programming Languages and Operating Systems, San Jose, CA, October 2006.
46. HP Power Calculator. Available at http://www.hp.com/confgurator/ powercalcs.asp.
47. M. Isard, M. Budiu, Y. Yu, A. Birrell, and Fetterly, "Dryad: distributed data-parallel programs from sequential building blocks," in Proceedings of the 2nd ACM Sigops/Eurosys European Conference on Computer Systems 2007, Lisbon, Portugal, March 21-23, 2007.
48. M. Isard, "Autopilot: automatic datacenter management". SIGOPS Operating Systems Review, vol. 41, no. 2 (Apr. 2007), pp. 60-67. DOI: http://doi.acm.org/10.1145/1243418.1243426.
49. M. Kalyanakrishnam, Z. Kalbarczyk, and R. Iyer, "Failure data analysis of a LAN of Windows NT based computers," Reliable Distributed Systems, IEEE Symposium on, vol. 0, no. 0, pp. 178, 18th IEEE Symposium on Reliable Distributed Systems, 1999. doi:10.1109/ RELDIS.1999.805094
50. K. Lim, P. Ranganathan, J. Chang, C. Patel, T. Mudge, and S. Reinhardt, "Understanding and designing new server architectures for emerging warehouse-computing environments," Computer Architecture, International Symposium on, vol. 0, no. 0, pp. 315-326, 2008 International Symposium on Computer Architecture, 2008. doi:10.1145/1394608.1382148
51. C. Malone and C. Belady, "Metrics to characterize datacenter & IT equipment energy use," in Proceedings of the Digital Power Forum, Richardson, TX, September 2006.
52. W. D. McArdle, F. I. Katch, and V. L. Katch, Sports and Exercise Nutrition, second ed., LWW Publishers, 2004.
53. D. Meisner, B. Gold, and T. Wenisch, "PowerNap: eliminating server idle power," in Proceedings of the ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Washington, DC, March 2009.
54. J. Moore, J. Chase, P. Ranganathan, and R. Sharma, "Making scheduling 'cool': temperature-aware workload placement in datacenters," in Proceedings of the Annual Conference on USENIX Annual Technical Conference, Anaheim, CA, April 10-15, 2005.
55. D. Nelson, M. Ryan, S. DeVito, K. V. Ramesh, P. Vlasaty, B. Rucker, B. Da y Nelson, et al., "The role of modularity in datacenter design". Sun BluePrints Online, http://www.sun .com/storagetek/docs/EED.pdf.
56. D. Oppenheimer, A. Ganapathi, and D. A. Patterson, "Why do Internet services fail, and what can be done about it?," in Proceedings of the 4th Conference on USENIX Symposium on Internet Technologies and Systems-Volume 4, Seattle, WA, March 26-28, 2003.
57. L. Page and S. Brin, "The anatomy of a large-scale hypertextual search engine". Available at http://infolab.stanford.edu/∼backrub/ google.html.
58. C. Patel et al., "Thermal considerations in cooling large scale high compute density datacen-ters". Available at http://www.fomerics.com/ fotherm/technical-papers/t299.pdf.
59. C. Patel et al., "Cost model for planning, development and operation of a datacenter". Available at http://www.hpl.hp.com/techreports/2005/HPL- 2005-107R1.pdf.
60. D. A. Patterson, A. Brown, P. Broadwell, G. Candea, M. Chen, J. Cutler, P. Enriquez, A. Fox, E. Kiciman, M. Merzbacher, D. Oppenheimer, N. Sastry, W. Tetzlaff, J. Traupman, and N. Treuhaft. "Recovery-oriented computing (ROC): motivation, defnition, techniques, and case studies". UC Berkeley Computer Science Technical Report UCB//CSD-02-1175, March 15, 2002.
61. M. K. Patterson and D. Fenwick, "The state of datacenter cooling," Intel Corporation White Paper. Available at http://download. intel.com/technology/eep/data-center-effciency/state-of-date-center-cooling.pdf.
62. R. Pike, S. Dorward, R. Griesemer, and S. Quinlan, "Interpreting the data: parallel analysis with Sawzall", Scientifc Programming Journal, vol. 13, no. 4 (2005), pp. 227-298.
63. E. Pinheiro, W.-D. Weber, and L. A. Barroso, "Failure trends in a large disk drive population," in Proceedings of 5th USENIX Conference on File and Storage Technologies (FAST 2007), San Jose, CA, February 2007.
64. PG&E, "High performance datacenters". Available at http://hightech.lbl.gov/documents/ DATA-CENTERS/06-DataCenters-PGE.pdf.
65. W. Pitt Turner IV, J. H. Seader, and K. G. Brill, "Tier classifcations defne site infrastructure performance," Uptime Institute White Paper.
66. W. Pitt Turner IV and J. H. Seader, "Dollars per kW plus dollars per square foot are a better datacenter cost model than dollars per square foot alone," Uptime Institute White Paper, 2006.
67. R. Raghavendra, P. Ranganathan, V. Talwar, Z. Wang, and X. Zhu, "No 'power' struggles: coordinated multi-level power management for the datacenter', in Proceedings of the ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Seattle, WA, March 2008.
68. Reuters, "Dupont Fabros Technology, Inc. reports third quarter results," November 5, 2008.
69. P. Reynolds, C. Killian, J. L. Wiener, J. C. Mogul, M. A. Shah, and A. Vahdat, "Pip: detecting the unexpected in distributed systems," in Proceedings of USENIX NSDI, 2006.
70. P. Reynolds, J. L. Wiener, J. C. Mogul, M. K. Aguilera, and A. Vahdat, "WAP5: black box performance debugging for wide-area systems," in Proceedings of the 15th International World Wide Web Conference, 2006.
71. Robert Frances Group, "Total cost of ownership for Linux in the enterprise," July 2002. Available at http://www-03.ibm.com/linux/RFG- LinuxTCO-vFINAL-Jul2002.pdf.
72. SAVVIS press release, "SAVVIS sells assets related to two datacenters for $200 million," June 29, 2007. Available at http://www.savvis.net/corp/News/Press+Releases/Archive/SAVVIS+ Sells+Assets+Related+to+Two+Data+Centers+for+200+Million.htm.
73. B. Schroeder and G. A. Gibson, "Understanding failures in petascale computers," Journal of Physics: Conference Series 78 (2007). doi:10.1088/1742-6596/78/1/012022
74. B. Schroeder and G. A. Gibson, "Disk failures in the real world: what does an MTTF of 1,000,000 hours mean to you?," in Proceedings of the 5th USENIX Conference on File and Storage Technologies, February 2007.
75. B. Schroeder, E. Pinheiro, and W.-D. Weber, "DRAM errors in the wild: a large-scale feld study," to appear in the Proceedings of SIGMETRICS, 2009.
76. S. Siddha, V. Pallipadi, and A. Van De Ven, "Getting maximum mileage out of tickless," in Proceedings of the Linux Symposium, Ottawa, Ontario, Canada, June 2007.
77. SPEC Power. Available at http://www.spec.org/power-ssj2008/.
78. S. Rivoire, M. A. Shah, P. Ranganathan, C. Kozyrakis, and Justin Meza, "Models and metrics to enable energy-effciency optimizations," Computer, vol. 40, no. 12 (Dec. 2007), pp. 39-48.
79. S. Sankar, S. Gurumurthi, and M. R. Stan, "Intra-disk parallelism: an idea whose time has come," in Proceedings of the ACM International Symposium on Computer Architecture, June 2008, pp. 303-314.
80. Techarp.com, "Intel desktop CPU guide". Available at http://www.techarp.com/showarticle. aspx?artno=337&pgno=6.
81. Terrazon Semiconductor, "Soft errors in electronic memory - a white paper". Available at http://www.tezzaron.com/about/papers/soft-errors-1-1- secure.pdf.
82. M. Ton and B. Fortenbury, "High performance buildings: datacenters - server power supplies," Lawrence Berkeley National Laboratories and EPRI, December 2005.
83. Transaction Processing Performance Council. Available at http://www.tpc.org.
84. W. F. Tschudi, T. T. Xu, D. A. Sartor, and J. Stein, "High performance datacenters: a research roadmap," Lawrence Berkeley National Laboratory, Berkeley, CA, 2003.
85. TPC-C Executive summary for the Superdome-Itanium2, February 2007.
86. TPC-C Executive summary for the ProLiant ML350G5, September 2007.
87. VMware infrastructure architecture overview white paper. Available at http://www.vmware .com/pdf/vi-architecture-wp.pdf.
88. W. Vogels, "Eventually consistent," ACM Queue, October 2008. Available at http://queue .acm.org/detail.cfm?id=1466448.
89. B. Chandra, M. Dahlin, L. Gao, A.-A. Khoja, A .Nayate, A. Razzaq, and A. Sewani, "Resource management for scalable disconnected access to Web services," Proceedings of the 10th International Conference on World Wide Web, Hong Kong, Hong Kong, May 1-5, 2001, pp. 245-256.
90. "Microsoft's Top 10 Business Practices for Environmentally Sustainable Data Center." Available at http://www.microsoft.com/ environment/our-commitment/articles/datacenter-bp.aspx.