The datacenter as a computer designing warehouse-scale machines: Third edition

Synthesis Lectures on Communication Networks

Volumn 13, Issue 3, 2018, Pages 1-189

  Barroso, Luiz André ^a   Hölzle, Urs ^a   Ranganathan, Parthasarathy ^a  

^a GOOGLE INC   (United States)

Author keywords

Cloud computing; Cluster computing; Computer design; Computer organization; Data centers; Distributed systems; Energy efficiency; Fault tolerant computing; Hardware accelerators; Hyperscale systems; Internet services; Servers

Indexed keywords

CLOUD COMPUTING; CLUSTER COMPUTING; DIGITAL STORAGE; ENERGY EFFICIENCY; FAULT TOLERANT COMPUTER SYSTEMS; HARDWARE; LEARNING SYSTEMS; SERVERS; WAREHOUSES; WEB SERVICES;

COMPUTER DESIGNS; COMPUTER ORGANIZATION; DATA CENTERS; DISTRIBUTED SYSTEMS; HARDWARE ACCELERATORS; INTERNET SERVICES;

GREEN COMPUTING;

EID: 85055893281     PISSN: 19354185     EISSN: None     Source Type: Book Series    
DOI: 10.2200/S00874ED3V01Y201809CAC046     Document Type: Review

References (245)

1. 3D XPoint™: A breakthrough in non-volatile memory technology. https://www.intel.com/content/www/us/en/architecture-and-technology/intel-micron-3d-xpoint-webcast. html. 74, 159, 166
2. D. Abts, M. R. Marty, P. M. Wells, P. Klausler, and H. Liu. 2010. Energy proportional datacenter networks. SIGARCH Comput. Archit. News, 38(3) (June 2010), 338-347. DOI: 10.1145/1816038.1816004. 115
3. D. Abts and B. Felderman. A guided tour of data-center networking. Commun. ACM, 55(6) (2012), p. 44-51. DOI: 10.1145/2184319.2184335. 65
4. ANSI INCITS: T11: Fibre Channel Interfaces. https://standards.incits.org/apps/ group_public/workgroup.php?wg_abbrev=t11. 62
5. ANSI INCITS 462: Information technology, fibre channel: Backbone 5 (FC-BB-5). 62
6. D. Abts and J. Kim. High-performance Datacenter Networks: Architectures, Algorithms, and Opportunities. Morgan & Claypool, San Rafael, CA (2011). DOI: 10.2200/ S00341ED1V01Y201103CAC014. 65
7. M. Al-Fares, A. Loukissas, and A. Vahdat. A scalable, commodity datacenter network architecture. In Proceedings of the ACM SIGCOMM 2008 Conference on Data Communication, Seattle, WA, August 17-22 (2008).
8. D. Atwood and J. G. Miner. Reducing datacenter costs with an air economizer. IT@ Intel Brief (August 2008). 92
9. AMD, SeaMicro product pages, http://www.seamicro.com/SM10000 (2012).
10. G. Ananthanarayanan, A. Ghodsi, S. Shenker, and I. Stoica. Disk-locality in datacenter computing considered irrelevant. In Proceedings of the 13th Workshop on Hot Topics in Operating Systems (HotOS XIII), Napa, CA (May 2011). 70
11. R. Ananthanarayanan, V. Basker, S. Das, A. Gupta, H. Jiang, T. Qiu, A. Reznichenko, D. Ryabkov, M. Singh, and S. Venkataraman. Photon: Fault-tolerant and scalable joining of continuous data streams. SIGMOD (2013). https://static.googleusercontent.com/ media/research.google.com/en//pubs/archive/41318.pdf. 69
12. D. G. Andersen, J. Franklin, M. Kaminsky, A. Phanishayee, L. Tan, and V. Vasudevan. 2011. FAWN: a fast array of wimpy nodes. Commun. ACM, 54(7) (July 2011), 101-109. DOI: 10.1145/1965724.1965747. 53, 69
13. M. Armbrust, A. Fox, R. Griffith, A. D. Joseph, R. Katz, A. Konwinski, G. Lee, D. Patterson, A. Rabkin, I. Stoica, and M. Zaharia. A view of cloud computing. Commun. ACM, 53(4) (April 2010), 50-58. DOI: 10.1145/1721654.1721672. 6
14. P. Bahl, R. Chandra, A. Greenberg, S. Kandula, D. A. Maltz, and M. Zhang. Toward highly reliable enterprise network services via inference of multi-level dependencies. In Proceedings of SIGCOMM (2007). 31
15. L. N. Bairavasundaram, G. R. Goodson, S. Pasupathy, and J. Schindler. An analysis of latent sector errors in disk drives. In Proceedings of the 2007 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems, San Diego, CA (June 12-16, 2007). SIGMETRICS '07. 149
16. J. Baker, C. Bond, J. C. Corbett, J. J. Furman, A. Khorlin, J. S. Larson, J.-M. Leon, Y. Li, A. Lloyd, and V. Yushprakh. Megastore: Providing scalable, highly available storage for interactive services. In Proceedings of the Conference on Innovative Data System Research (CIDR) (2011), pp. 223-234. http://static.googleusercontent.com/external_content/ untrusted_dlcp/research.google.com/en/us/pubs/archive/36971.pdf. 69
17. L. A. Barroso, K. Gharachorloo, R. McNamara, A. Nowatzyk, S. Qadeer, B. Sano, S. Smith, R. Stets, and B. Verghese. 2000. Piranha: a scalable architecture based on single-chip multiprocessing. SIGARCH Comput. Archit. News, 28(2) (May 2000), pp. 282-293. DOI: 10.1145/342001.339696. 53
18. P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neugebauer, I. Pratt, and A. Warfield. Xen and the art of virtualization. In Proceedings of the Nineteenth ACM Symposium on Operating Systems Principles, (SOSP '03). Bolton Landing, NY (October 19-22, 2003).
19. P. Barham, R. Isaacs, R. Mortier, and D. Narayanan. Magpie: online modeling and performance-aware systems. In Proceedings of USENIX HotOS IX (2003). 31
20. L. A Barroso, M. Marty, D. Patterson, and P. Ranganathan. Attack of the killer microseconds. Comm. of the ACM, 60(4) (April 2017), pp. 48-54. DOI: 10.1145/3015146. 74, 162, 165
21. L. A. Barroso. Warehouse-scale computing: Entering the teenage decade. Plenary talk at the ACM Federated Computer Research Conference. In Proceedings of the 38th Annual International Symposium on Computer Architecture (ISCA '11). ACM, San Jose, CA (June 2011).
22. L. A Barroso, J. Dean, and U. Holzle. Web search for a planet: the architecture of the Google cluster. IEEE Micro (April 2003). 45, 53
23. L. A. Barroso and U. Holzle. The case for energy-proportional computing. IEEE Computer, 40(12) (December 2007). 110
24. C. Belady and C. Malone. Preliminary assessment from Uptime Institute: IDC data center of the future US server power spend for 2005 as a baseline($6bn); applied a cooling factor of 1; applied a .6 multiplier to US data for WW amount. Data Center Power Projection to 2014, 2006 ITHERM, San Diego, CA (June 2006). 103
25. E. Brewer, L. Ying, L. Greenfield, R. Cypher, and T. T'so. Disks for Data Centers, Google (2016) pp. 1-16. https://ai.google/research/pubs/pub44830. DOI: 10.1109/4236.939450. 66
26. E. A. Brewer. Lessons from giant-scale services. IEEE Internet Comput., 5(4) (July/ August 2001), pp. 46-55. 34, 142, 143
27. M. Burrows. The chubby lock service for loosely-coupled distributed systems. In Proceedings of OSDI'06: Seventh Symposium on Operating System Design and Implementation, Seattle, WA (November 2006). 17, 20
28. B. Chandra, M. Dahlin, L. Gao, and A. Nayate. End-to-end WAN service availability. In Proceedings of the 3rd Conference on USENIX Symposium on Internet Technologies and Systems, 3, San Francisco, CA (March 26-28, 2001).
29. B. Chandra, M. Dahlin, L. Gao, A.-A. Khoja, A. Nayate, A. Razzaq, and A. Sewani. Resource management for scalable disconnected access to Web services. Proceedings of the 10th International Conference on World Wide Web, Hong Kong (May 1-5, 2001), pp. 245-256. DOI: 10.1145/371920.372059. 143
30. J. Chase, D. Anderson, P. Thakar, A. Vahdat, and R. Doyle. Managing energy and server resources in hosting centers. In Proceedings of the ACM Symposium on Operating Systems Principles (SOSP), Banff, Alberta, Canada (June 2001). 123
31. F. Chang, J. Dean, S. Ghemawat, W. C. Hsieh, D. A. Wallach, M. Burrows, T. Chandra, A. Fikes, and R. E. Gruber. Bigtable: a distributed storage system for structured data. In Proceedings of OSDI (2006), Seattle, WA (2004). 9, 17, 20
32. C. Chambers, A. Raniwala, F. Perry, S. Adams, R. Henry, R. Bradshaw, and Nathan. FlumeJava: Easy, efficient data-parallel pipelines. ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), pp. 363-375 (2010). https:// ai.google/research/pubs/pub35650. 20, 68
33. Y. Chen, A. Das, W. Qin, A. Sivasubramaniam, Q. Wang, and N. Gautam. Managing server energy and operational costs in hosting centers. In Proceedings of the ACM SIGMETRICS '05, Banff, Alberta, Canada (June 2005). 123
34. G. Chen, W. He, J. Liu, S. Nath, L. Rigas, L. Xiao, and F. Zhao. Energy aware server provisioning and load dispatching for connection-intensive Internet services. Microsoft Research Technical Report MSR-TR-2007-130 (2007). 123
35. J. Chen, R. Monga, S. Bengio, and R. Jozefowicz. Revisiting distributed synchronous SGD. International Conference on Learning Representations Workshop Track (2016). https://arxiv.org/abs/1604.00981. 28
36. K. Christensen, P. Reviriego, B. Nordman, M. Bennett, M. Mostowfi, and J. A. Maestro. IEEE 802.3az: the road to energy efficient Ethernet. Comm. Mag., IEEE, 48(11), (November 2010) pp. 50-56. DOI: 10.1109/MCOM.2010.5621967. 115
37. Cisco Public White Paper. Cisco Visual Networking Index: Forecast and Methodology, 2016-2021 (June, 6, 2017).
38. Clark, J. Google turning its lucrative web search over to AI machines. Bloomberg Technology, (October 26, 2015) www.bloomberg.com. 28
39. Climate savers computing efficiency specs. http://www.climatesaverscomputing.org/ about/tech-specs. 106
40. L. Columbus. Cloud computing market projected to reach $411B by 2020. Forbes (October, 18, 2017). 1
41. J. C. Corbett et al. Spanner: Google's globally-distributed database. Proceedings of OSDI'12: Tenth Symposium on Operating System Design and Implementation, Hollywood, CA (October 2012). 9, 17, 20, 69
42. E. F. Coyle. Improved muscular efficiency displayed as tour de France champion matures. J. Appl. Physiolo. (March 2005). DOI: 10.1152/japplphysiol.00216.2005.
43. E. V. Carrera, E. Pinheiro, and R. Bianchini. Conserving disk energy in network servers. In Proceedings of the 17th Annual International Conference on Supercomputing, (ICS '03). San Francisco, CA (June 23-26, 2003). 114
44. Cavium ThunderX2R ARM Processors, High Performance ARMv8 Processors for Cloud and HPC Server Applications. https://www.cavium.com/product-thunderx2- arm-processors.html. 55
45. M. Dalton, D. Schultz, J. Adriaens, A. Arefin, A. Gupta, B. Fahs, D. Rubinstein, E. Cauich Zermeno, E. Rubow, J. A. Docauer, J. Alpert, J. Ai, J. Olson, K. DeCabooter, M. de Kruijf, N. Hua, N. Lewis, N. Kasinadhuni, R. Crepaldi, S. Krishnan, S. Venkata, Y. Richter, U. Naik, and A. Vahdat. Andromeda: Performance, isolation, and velocity at scale in cloud network virtualization. NSDI (2018). 42, 66
46. J. Dean and L. A. Barroso. The tail at scale. Commun. ACM, 56(2) (February 2013), pp. 80-86. DOI: 10.1145/2408776.2408794. 38, 162, 163
47. J. Dean. Machine learning for systems and systems for machine learning. In Proceedings of the NIPS. http://learningsys.org/nips17/assets/slides/dean-nips17.pdf. 58
48. J. Dean et al. Large scale distributed deep networks. NIPS (2012). https://ai.google/ research/pubs/pub40565. 28
49. J. Dean. Design lessons and advice from building large scale distributed systems, LADIS (2009). https://perspectives.mvdirona.com/2009/10/jeff-dean-design-lessonsand- advice-from-building-large-scale-distributed-systems/. 40
50. Dell Energy Calculator. http://www.dell.com/calc. 121
51. G. DeCandia, D. Hastorun, M. Jampani, G. Kakulapati, A. Lakshman, A. Pilchin, S. Sivasubramanian, P. Vosshall, and W. Vogels. Dynamo: Amazon's highly available keyvalue store. In Proceedings of the 21st ACM Symposium on Operating Systems Principles, Stevenson, WA (October 2007). DOI: 10.1145/1323293.1294281. 9, 17, 20, 68
52. T. J. Dell. A white paper on the benefits of chipkill-correct ECC for PC server main memory," IBM Microelectronics Division (Rev. 11/19/1997). 148
53. R. Dennard et al. Design of ion-implanted MOSFETs with very small physical dimensions. IEEE J. Solid State Circuits SC-9, 5 (October 1974), pp. 256-268. DOI:10.1109/ JSSC.1974.1050511. 163
54. J. Dean and S. Ghemawat. MapReduce: simplified data processing on large clusters. Commun. ACM, 51(1) (2008), pp. 107-113. DOI: 10.1145/1327452.1327492. 17, 20
55. C. Delimitrou and C. Kozyrakis. Quasar: resource-efficient and QoS-aware cluster management. In Proceedings of the 19th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS '14). ACM, New York, (2014) pp. 127-144. DOI: 10.1145/2541940.2541941. 117, 163, 165
56. Dell Modular Data Center: http://www.dell.com/learn/us/en/555/shared-content~data- sheets~en/documents~hyper-efficient-modular-data-center.pdf.
57. Dupont Fabros Technology Inc. SEC Filing (S-11) 333-145294, August 9, 2007.
58. DuPont Fabros Technology Annual Report 2011. http://www.dft.com/sites/default/ files/2011-10k.pdf. 130
59. D. DeWitt and J. Gray. Parallel database systems: The future of high performance database processing," CACM, 36(6) (June 1992). 54
60. D. Dyer. Current trends/challenges in datacenter thermal management - a facilities perspective. Presentation at ITHERM, San Diego, CA (June 1, 2006). 87
61. eBay Containers Push Frontiers of Efficiency: http://www.datacenterknowledge.com/ archives/2012/02/27/ebay-containers-push-frontiers-of-efficiency/. 96
62. R. Evans and J. Gao. DeepMind AI reduces Google data centre cooling bill by 40%. https://deepmind.com/blog/deepmind-ai-reduces-google-data-centre-cooling- bill-40/ (July 20, 2016). 103, 166
63. U.S. Environmental Protection Agency. Report to Congress on server and datacenter energy efficiency. Public Law 109-431 (August 2, 2007). 76
64. Beyond scaling: An electronics resurgence initiative. DARPA, (2017). https://www. darpa.mil/news-events/2017-06-01. 166
65. N. El-Sayed, I. Stefanovici, G. Amvrosiadis, A. A. Hwang, and B. Schroeder. Temperature management in data centers: Why some (might) like it hot. SIGMETRICS'12. http://www.cs.toronto.edu/~bianca/papers/temperature_cam.pdf. 104, 149
66. J. Elerath and S. Shah. Server class disk drives: how reliable are they? IEEE Reliability and Maintainability, 2004 Annual Symposium - RAMS (January 2004).
67. M. E. Femal and V. W. Freeh. Safe overprovisioning: using power limits to increase aggregate throughput. In 4th International Workshop on Power Aware Computer Systems (PACS 2004) (December 2004), pp. 150-164. 123
68. M. E. Femal and V. W. Freeh. Boosting datacenter performance through non-uniform power allocation. In Proceedings of the Second International Conference on Automatic Computing, (June 13-16, 2005), pp. 250-261. 20, 123
69. D. Firestone, A. Putnam, S. Mundkur, D. Chiou, A. Dabagh, M. Andrewartha, H. Angepat, V. Bhanu, A. M. Caulfield, E. S. Chung, H. Kumar Chandrappa, S. Chaturmohta, M. Humphrey, J. Lavier, N. Lam, F. Liu, K. Ovtcharov, J. Padhye, G. Popuri, S. Raindel, T. Sapre, M. Shaw, G. Silva, M. Sivakumar, N. Srivastava, A. Verma, Q. Zuhair, D. Bansal, D. Burger, K. Vaid, D. A. Maltz, and A. G. Greenberg. Azure Accelerated Networking: SmartNICs in the Public Cloud. NSDI 2018. 66
70. B. Fitzpatrick et al. (2003). Memcached. http://memcached.org/about. 69
71. S. H. Fuller and L. I. Millet (eds) The Future of Computing Performance: Game Over or Next Level. National Academies Press; Committee on Sustaning Growth in Computing Performance, National Research Council (2011). http://www.nap.edu/catalog. php?record_id=12980. 106
72. R. Fonseca, G. Porter, R. H. Katz, S. Shenker, and I. Stoica. X-trace: a pervasive network tracing framework. In Proceedings of 4th USENIX Symposium on Networked Systems Design & Implementation, (2007), pp. 271-284. 20
73. D. Ford, F. Labelle, F. I. Popovici, M. Stokely, V.-A. Truong, L. Barroso, C. Grimes, and S. Quinlan. Availability in globally distributed storage systems. In Proceedings of the 9th USENIX Sympsium on Operating Systems Design and Implementation (OSDI'10) (October 2010), Vancouver, BC, Canada. http://static.usenix.org/event/osdi10/tech/ full_papers/Ford.pdf. 68, 144
74. X. Fan, W. Weber, and L. A. Barroso. Power provisioning for a warehouse-sized computer. In Proceedings of the 34th Annual International Symposium on Computer Architecture, ISCA '07, San Diego, CA (June 9-13, 2007). DOI: /10.1145/1250662.1250665. 31, 110, 121, 122, 152
75. A. Gandhi, M. Harchol-Balter, R. Das, J. O. Kephart, and C. Lefurgy. Power capping via forced idleness. http://repository.cmu.edu/compsci/868/. 115
76. P. Gao, A. Narayan, S. Karandikar, J. Carreira, S. Han, R. Agarwal, S. Ratnasamy, and S. Shenker. Network requirements for resource disaggregation. OSDI (2016). https:// dl.acm.org/citation.cfm?id=3026897. 165
77. Google Cloud AutoML. https://cloud.google.com/automl/. 59
78. Google Compute Engine SLAs: https://cloud.google.com/compute/sla. 155
79. Google Cloud Platform. https://cloud.google.com/. 42, 155
80. Google Cloud: Regions and Zones. https://cloud.google.com/compute/docs/regions- zones/. 155
81. Google Data centers. http://www.google.com/about/datacenters/efficiency/internal/ index.html#measuring-efficiency. 101, 161
82. Google Data centers. Efficiency: How we do it. http://www.google.com/about/datacenters/ efficiency/internal/index.html#measuring-efficiency. 103
83. An Overview: Global Data Center Construction. https://wiredre.com/an-overview- global-data-center-construction/. 130
84. A. J. George and G. Ferrand. Cost study of AC vs DC data center power topologies based on system efficiency. Eltek. https://tinyurl.com/yaemk37v. 82
85. S. Ghemawat, H. Gobioff, and S-T. Leung. The Google file system, In Proceedings of the 19th ACM Symposium on Operating Systems Principles, Lake George, NY (October 2003). 20, 68, 109, 140
86. Google green, The Big Picture: http://www.google.com/green/bigpicture/#/. 101
87. C. Ghali, A. Stubblefield, E. Knapp, J. Li, B, Schmidt, J. Boeuf. Application Layer Transport Security, (2017), https://cloud.google.com/security/encryption-in-transit/ application-layer-transport-security/. 44
88. Google Inc.. Efficient Data Center Summit, April 2009. http://www.google. com/ about/datacenters/efficiency/external/2009-summit.html. 96, 105
89. Google Inc. Efficient computing - step 2: efficient datacenters. http://www. google. com/corporate/green/datacenters/step2.html.
90. Google Inc. Efficient Data Center, Part 1. http://www.youtube.com/ watch?v=Ho1GEyftpmQ starting at 0:5930.
91. S. Greenberg, E. Mills, and B. Tschudi. Best practices for data centers: lessons learned from benchmarking 22 datacenters. 2006 ACEEE Summer Study on Energy Efficiency in Buildings. http://evanmills.lbl.gov/pubs/pdf/aceee-datacenters.pdf. 104
92. S. Govindan, D. Wang, A. Sivasubramaniam, and B. Urgaonkar. Leveraging stored energy for handling power emergencies in aggressively provisioned datacenters. http:// csl.cse.psu.edu/publications/asplos12.pdf. DOI: 10.1145/2189750.2150985. 125
93. J. Gray. A census of tandem system availability between 1985 and 1990. Tandem Technical Report 90.1 (January 1990). 144
94. Google Stackdriver Debugger. https://cloud.google.com/debugger/. 31
95. Google Stackdriver. https://cloud.google.com/stackdriver/. 30, 31
96. Google Stackdriver Profiler. https://cloud.google.com/profiler/. 31
97. Google Stackdriver Trace. https://cloud.google.com/trace/.
98. The Hadoop Project. http://hadoop.apache.org. 17
99. J. Hamilton. On designing and deploying internet-scale services. In Proceedings of USENIX LISA (2007). 34, 139
100. J. Hamilton. Cooperative expendable micro-slice servers (CEMS): low cost, low power servers for internet-scale services. In Proceedings of the 4th Biennial Conference on Innovative Data Systems Research (CIDR), Asilomar, CA (January 4-7, 2009). 53
101. M. Hashemi, K. Swersky, J. A. Smith, G. Ayers, H. Litz, J. Chang, C. Kozyrakis, and P. Ranganathan. Learning memory access patterns. ICML (2018) 1924-1933. 166
102. K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition, 2015. https://arxiv.org/abs/1512.03385. 28
103. T. Heath, A. P. Centeno, P. George, L. Ramos, Y. Jaluria, and R. Bianchini. Mercury and freon: temperature emulation and management for server systems. In Proceedings of the ACM International Conference on Architectural Support for Programming Languages and Operating Systems, San Jose, CA (October 2006). DOI: 10.1145/1168918. 123
104. J. Hennessy, P. Turner, J. Masters, and M. Hill. Hot18: Spectre/Meltdown and the era of security. Keynote at Hotchips (2018). 42, 44, 166
105. K. Heslin. Uptime Institute: 2014 Data Center Industry Survey. November 2014. https://journal.uptimeinstitute.com/2014-data-center-industry-survey/. 100
106. Benjamin Hindman et al. Mesos: A platform for fine-grained resource sharing in the data center. In NSDI'11: Proceedings of the 8th USENIX Symposium on Networked Systems Design and Implementation, Boston, MA, USENIX Association (March, 2011). 117
107. U. Holzle. Brawny cores still beat wimpy cores, most of the time. IEEE MICRO (July/ August 2010). 54
108. U. Holzle. OpenFlow @ Google. Open Networking Summit, Santa Clara (April 2012). http://www.opennetsummit.org/archives/apr12/hoelzle-tue-openflow.pdf. 65
109. U. Holzle. Open compute project (OCP). Summit 2016 Keynote, San Jose CA.
110. HP Moonshot System: A new style of IT accelerating innovation at scale, Technical white paper (2013). https://goo.gl/3UGa4k. 54
111. HP Power Calculator. http://www.hp.com/configurator/powercalcs.asp. 121
112. HP Performance Optimized Datacenter (POD). http://h18004.www1.hp.com/products/ servers/solutions/datacentersolutions/pod/index.html.
113. A. A. Hwang, I. Stefanovici, and B. Schroeder. Cosmic rays don't strike twice: Understanding the nature of DRAM errors and the implications for system design, ASPLOS 2012 (March 2012). 149
114. C. H. Hsu, Y. Zhang, M. A. Laurenzano, D. Meisner, T. Wenisch, J. Mars, L. Tang, and R. G. Dreslinski. Adrenaline: Pinpointing and reining in tail queries with quick voltage boosting. IEEE 21st International Symposium on High Performance Computer Architecture (HPCA), Burlingame, CA (2015), pp. 271-282. 123
115. The 2015 International Building Code. https://codes.iccsafe.org/public/document/ IBC2015/chapter-6-types-of-construction. 77
116. Intel® 64 and IA-32 Architectures Software Developer's Manual Volume 3B: System Programming Guide, Part 2. May 2018. https://software.intel.com/en-us/articles/intel- sdm. 116
117. IntelR Turbo Boost Technology 2.0. https://www.intel.com/content/www/us/en/architecture- and-technology/turbo-boost/turbo-boost-technology.html. 118
118. IntelR XeonR Platinum 8164 Processor. https://ark.intel.com/products/120503/Intel- Xeon-Platinum-8164-Processor-35_75M-Cache-2_00-GHz. 118
119. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: distributed data-parallel programs from sequential building blocks. In Proceedings of the 2nd ACM Sigops/Eurosys European Conference on Computer Systems 2007, Lisbon, Portugal (March 21-23, 2007). 17
120. M. Isard. Autopilot: automatic datacenter management. SIGOPS Operating Systems Review, 41(2) (April 2007), pp. 60-67. DOI: 10.1145/1243418.1243426. 20
121. Internet Small Computer Systems Interface (iSCSI). The Internet Society: https:// tools.ietf.org/html/rfc3720. 62
122. S. Jain et al. B4: Experience with a globally-deployed software defined WAN. SIGCOMM (2013). 65, 165
123. W. Jiang, C. Hu, Y. Zhou, and A. Kanevsky. Are disks the dominant contributor for storage failures?: A comprehensive study of storage subsystem failure characteristics. Trans. Storage 4, 3, Article 7 (November 2008), 25 pages. DOI: 10.1145/1416944.141694. 144
124. N. P. Jouppi et al. In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th Annual International Symposium on Computer Architecture (ISCA), Toronto (2017) pp. 1-12. DOI: 10.1145/3079856.3080246. 28, 56, 58, 119, 120
125. N. Jouppi et al. Motivation for and evaluation of the first tensor processing unit. IEEE Micro (2018), pp 10-19. DOI: 10.1109/MM.2018.032271057. 27, 28
126. S. Kanev, J. Pablo Darago, K. Hazelwood, P. Ranganathan, T. Moseley, G. Y. Wei, and D. Brooks. Profiling a warehouse-scale computer. ACM SIGARCH Comput.Archit. News, 43, pp. 158-169 (2015). DOI: 10.1145/2872887.2750392. 21
127. S. Karandikar, H. Mao, D. Kim, D. Biancolin, A. Amid, D. Lee, N. Pemberton, E. Amaro, C. Schmidt, A. Chopra, Q. Huang, K. Kovacs, B. Nikolic, R. Katz, J. Bachrach, and K. Asanovic. FireSim: FPGA-accelerated cycle-exact scale-out system simulation in the public cloud. 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA), (2018), pp. 29-42. DOI: 10.1109/ISCA.2018.00014. 167
128. H. Kasture, D. B. Bartolini, N. Beckmann, and D. Sanchez. Rubik: Fast analytical power management for latency-critical systems. 48th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), Waikiki, HI (2015), pp. 598-610. 123
129. M. Kalyanakrishnam, Z. Kalbarczyk, and R. Iyer. Failure data analysis of a LAN of Windows NT based computers. IEEE Symposium on Reliable Distributed Systems, 0(0), p. 178, 18th IEEE Symposium on Reliable Distributed Systems (1999). DOI: 10.1109/ RELDIS.1999.805094. 145, 146
130. B. Koley. Software Defined Networking at Scale. 2014. https://ai.google/research/pubs/ pub42948. 165
131. V. Kontorinis, L. E. Zhang, B. Aksanli, J. Sampson, H. Homayoun, E. Pettis, D.M. Tullsen, and T. Simunic Rosing. Managing distributed ups energy for effective power capping in data centers. SIGARCH Comput. Archit. News 40(3) (June 2012), pp. 488-499. DOI: 10.1145/2366231.2337216. 125
132. J. G. Koomey, K. Brill, P. Turner, J. Stanley, and B. Taylor. A simple model for determining true total cost of ownership for data centers. Uptime Institute White Paper, Version 2 (October, 2007). 129
133. J.G. Koomey, S. Berard, M. Sanchez, and H. Wong. Implications of historical trends in the electrical efficiency of computing. Ann. Hist Comput., IEEE, 33(3), pp.46-54, (March 2011). DOI: 10.1109/MAHC.2010.28. 106
134. J. G. Koomey. Growth in Data Center Electricity Use 2005 to 2010. Analytics Press, 2011. 76
135. D. Kreutz, F. M. V. Ramos, P. . Verissimo, C. E. Rothenberg, S. Azodolmolky, and S. Uhlig. Software-Defined Networking: A Comprehensive Survey. OpenFlow Specification (2014). https://ieeexplore.ieee.org/abstract/document/6994333/. 165
136. A. Kumar, S. Jain, U. Naik, N. Kasinadhuni, E. C. Zermeno, C. S. Gunn, J. Ai, B. Carlin, M. Amarandei-Stavila, M. Robin, A. Siganporia, S. Stuart, and A. Vahdat. BwE: Flexible, hierarchical bandwidth allocation for WAN distributed computing, SIGCOMM, (2015) https://ai.google/research/pubs/pub43838. 65
137. J. Lee, C. Kim, K. Lin, L. Cheng, R. Govindaraju, and J. Kim. WSMeter: A performance evaluation methodology for Google's production warehouse-scale computers. USA (March 2018), https://dl.acm.org/citation.cfm?id=3173196. 167
138. K. Lim, P. Ranganathan, J. Chang, C. Patel, T. Mudge, and S. Reinhardt. Understanding and designing new server architectures for emerging warehouse-computing environments. International Symposium on Computer Architecture, 0(0), pp. 315-326, (2008). DOI: 10.1109/ISCA.2008.37. 53, 54
139. K. Lim, J. Chang, T. Mudge, P. Ranganathan, S. K. Reinhardt, and T. F. Wenisch. Disaggregated memory for expansion and sharing in blade servers. ISCA'09 (June 20-24, 2009), Austin, TX. https://web.eecs.umich.edu/~twenisch/papers/isca09-disaggregate. pdf. 165
140. K. Lim, D. Meisner, A. G. Saidi, P. Ranganathan, and T. Wenisch. Thin Servers with Smart Pipes: Designing SoC Accelerators for Memcached. In Proceedings of the 40th ACM International Symposium on Computer Architecture (ISCA), Tel Aviv, Israel (June 2013). 20, 55
141. D. Lo, L. Cheng, R. Govindaraju, L. A. Barroso, and C. Kozyrakis. Toward energy proportionality for large-scale latency-critical workloads. In Proceeding of the 41st Annual International Symposium on Computer Architecture (ISCA '14). IEEE Press, Piscataway, NJ, pp. 301-312. http://csl.stanford.edu/~christos/publications/2014.pegasus.isca.pdf. 116, 123
142. D. Lo, L. Cheng, R. Govindaraju, P. Ranganathan, and C. Kozyrakis. 2015. Heracles: improving resource efficiency at scale. In Proceedings of the 42nd Annual International Symposium on Computer Architecture (ISCA '15). ACM, New York, pp. 450-462. DOI: 10.1145/2749469.2749475. 117, 163, 165
143. A. Lottarini, A. Ramirez, J. Coburn, M. A. Kim, P. Ranganathan, D. Stodolsky, and M. Wachsler. vbench: Benchmarking video transcoding in the cloud. (2018). https:// dl.acm.org/citation.cfm?id=3173207. 25, 166
144. W. Lang, J. Patel, and S. Shankar. Wimpy Node Clusters: What about non-wimpy workloads? In Proceedings of the Sixth International Workshop on Data Management on New Hardware (June 2010). DOI: 10.1145/1869389.1869396.
145. M. Manos. The Capacity Problem. http://loosebolts.wordpress.com/2009/06/02/ chiller-side-chats-the-capacity-problem/. 121
146. J. Mars, L. Tang, R. Hundt, K. Skadron, and M. L. Soffa. 2011. Bubble-Up: increasing utilization in modern warehouse scale computers via sensible co-locations. In Proceedings of the 44th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO-44). ACM, New York, pp. 248-259. DOI: 10.1145/2155620.2155650. 117, 163, 165
147. C. Malone and C. Belady. Metrics to characterize datacenter & IT equipment energy use. In Proceedings of the Digital Power Forum, Richardson, TX (September 2006). 100
148. M. K. McKusick and S. Quinlan. GFS: Evolution on fast-forward. ACM Queue (August 1, 2009). DOI: 10.1145/1594204.1594206. 68
149. D. Meisner, C. M. Sadler, L. A. Barroso, W.-D. Weber, and T. F. Wenisch. 2011. Power management of online data-intensive services. SIGARCH Comput. Archit. News 39(3) ( June 2011), pp.319-330. DOI: 10.1145/2024723.2000103. 116
150. S. Melnik et al. Dremel: Interactive analysis of web-scale datasets. In Proceeding of the 36th International Conference on Very Large Data Bases (2010), pp. 330-339, 2010. 17
151. D. Meisner, B. T. Gold, and T. F. Wenisch. PowerNap: eliminating server idle power. SIGPLAN Not. 44(3) (March 2009) pp. 205-216. DOI: 10.1145/1508284.1508269. 115, 116
152. Microsoft Expands Dublin Cloud Computing Hub. Data Center Knowledge. February 23rd, 2012. http://www.datacenterknowledge.com/archives/2012/02/23/microsoft- expands-dublin-cloud-computing-hub/. 130
153. Microsoft's Top 10 Business Practices for Environmentally Sustainable Data Center. http://www.microsoft.com/environment/our_commitment/articles/datacenter_ bp.aspx. 96
154. R. Miller. Facebook has spent $210 million on Oregon data center. Data Center Knowledge. (January 30, 2012). http://www.datacenterknowledge.com/archives/2012/01/30/ facebookhas-spent-210-million-on-oregon-data-center/. 130
155. W. D. McArdle, F. I. Katch, and V. L. Katch. Sports and Exercise Nutrition, 2nd ed., LWW Publishers, 2004.
156. MLPerf. https://mlperf.org/assets/static/media/MLPerf-User-Guide.pdf. (2018). 28
157. J. Moore, J. Chase, P. Ranganathan, and R. Sharma. Making scheduling "cool": temperature- aware workload placement in datacenters. In Proceedings of the Annual Conference on USENIX Annual Technical Conference, Anaheim, CA (April 10-15, 2005). 123
158. Microsoft's Project Olympus. https://www.opencompute.org/wiki/Server/ProjectOlympus. 54
159. N. R. Murphy, B. Beyer, C. Jones, and J. Petoff. Site Reliability Engineering: How Google Runs Production Systems, 1st ed. (April 2016). 32
160. R. N. Mysore, A. Pamboris, N. Farrington, N. Huang, P. Miri, S. Radhakrishnan, V. Subramanya, and A. Vahdat. PortLand: A scalable fault-tolerant layer 2 data center network fabric. Department of Computer Science and Engineering, University of California San Diego. SIGCOMM'09 (August 17-21, 2009), Barcelona, Spain. 61
161. D. Nelson, M. Ryan, S. DeVito, K. V. Ramesh, P. Vlasaty, B. Rucker, B. Da y Nelson, et al. The role of modularity in datacenter design. Sun BluePrints Online. http://www.sun. com/storagetek/docs/EED.pdf. 104
162. NVM Express. http://nvmexpress.org/resources/specifications/. 62
163. OpenAI. AI and Compute. (May 2018). https://blog.openai.com/ai-and-compute/. 56, 57
164. Open Compute Project, Battery Cabinet Hardware v1.0. http://opencompute.org/wp/ wp-content/uploads/2011/07/DataCenter-Battery-Cabinet-Specifications.pdf. 80
165. D. Oppenheimer, A. Ganapathi, and D. A. Patterson. Why do Internet services fail, and what can be done about it? In Proceedings of the 4th Conference on USENIX Symposium on Internet Technologies and Systems, Volume 4, Seattle, WA (March 26-28, 2003). 143
166. Open Networking Foundation. Software-Defined Networking: The New Norm for Networks (April 2012). https://www.opennetworking.org/images/stories/downloads/ sdn-resources/white-papers/wp-sdn-newnorm.pdf. 65
167. J. Ousterhout, P. Agrawal, D. Erickson, C. Kozyrakis, J. Leverich, D. Mazieres, S. Mitra, A. Narayanan, G. Parulkar, M. Rosenblum, S. M. Rumble, E. Stratmann, and R. Stutsman. The case for RAMClouds: Scalable high-performance storage entirely in DRAM. SIGOPS Operating Systems Review, 43,(4) (December 2009), pp. 92-105. 69
168. J. Ousterhout, P. Agrawal, D. Erickson, C. Kozyrakis, J. Leverich, D. Mazieres, S. Mitra, A. Narayanan, D. Ongaro, G. Parulkar, M. Rosenblum, S. M. Rumble, E. Stratmann, and R. Stutsman. 2011. The case for RAMCloud. Commun. ACM 54(7) (July 2011), 121-130. DOI: 10.1145/1965724.1965751.
169. J. Ousterhout. A Philosophy of Software Design. Yaknyam Press (April 2018), 178 pages. 165
170. C. Patel et al. Thermal considerations in cooling large scale high compute density datacenters. http://www.flomerics.com/flotherm/technical_papers/t299.pdf.
171. D. A. Patterson, A. Brown, P. Broadwell, G. Candea, M. Chen, J. Cutler, P. Enriquez, A. Fox, E. Kiciman, M. Merzbacher, D. Oppenheimer, N. Sastry, W. Tetzlaff, J. Traupman, and N. Treuhaft. Recovery-oriented computing (ROC): motivation, definition, techniques, and case studies. UC Berkeley Computer Science Technical Report UCB//CSD- 02-1175 (March 15, 2002). 147
172. C. Patel et al. Cost model for planning, development and operation of a datacenter. http://www.hpl.hp.com/techreports/2005/HPL-2005-107R1.pdf. 129
173. L. Page and S. Brin. The anatomy of a large-scale hypertextual search engine. http:// infolab.stanford.edu/~backrub/google.html. 23
174. M. Pedram. Energy-efficient datacenters. Computer-aided design of integrated circuits and systems. IEEE Trans. Comput-Aided Des Integr Circ Sys. (October 2012). DOI: 10.1109/TCAD.2012.2212898. 123
175. M. K. Patterson and D. Fenwick. The state of datacenter cooling. Intel Corporation White Paper. http://download.intel.com/technology/eep/data-center-efficiency/stateof- date-center-cooling.pdf. 89, 94
176. PG&E. High performance datacenters. http://hightech.lbl.gov/documents/DATA_ CENTERS/06_DataCenters-PGE.pdf. 104
177. R. Pike, S. Dorward, R. Griesemer, and S. Quinlan. Interpreting the data: parallel analysis with Sawzall. Sci.Program. J., 13(4) (2005), pp. 227-298. 17
178. A. Pratt, P. Kumar, K. Bross, and T. Aldridge. Powering compute platforms in high efficiency data centers. IBM Technology Symposium, 2006. 82
179. Distribution of PUE Ratios. (January, 2010). http://www.energystar.gov/ia/partners/ prod_development/downloads/DataCenters_GreenGrid02042010.pdf. 100
180. A. Putnam, A. Caulfield, E. Chung, D. Chiou, K. Constantinides, J. Demme, H. Esmaeilzadeh, J. Fowers, J. Gray, . Haselman, S. Hauck, S. Heil, A. Hormati, J. Y. Kim, S. Lanka, E. Peterson, A. Smith, J. Thong, P. Yi Xiao, D. Burger, J. Larus, G. P. Gopal, and S. Pope. A Reconfigurable Fabric for Accelerating Large-Scale Datacenter Services. IEEE Press (June 1, 2014). 56
181. E. Pinheiro, W.-D. Weber, and L. A. Barroso. Failure trends in a large disk drive population. In Proceedings of 5th USENIX Conference on File and Storage Technologies (FAST 2007), San Jose, CA (February 2007). 14, 20, 104, 149, 150
182. Qualcomm Centriq 2400 Processor. https://www.qualcomm.com/products/qualcomm- centriq-2400-processor. 55
183. R. Raghavendra, P. Ranganathan, V. Talwar, Z. Wang, and X. Zhu. No "power" struggles: coordinated multi-level power management for the datacenter. In Proceedings of the ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Seattle, WA (March 2008). DOI: 10.1145/1353536.1346289. 117, 123, 165
184. P. Ranganathan, D. Irwin, P. Leech, and J. Chase. Ensemble-level power management for dense blade servers. Conference: Computer Architecture, February, 2006. ISCA '06. https://www.researchgate.net/publication/4244659_Ensemble-Level_Power_Management_ for_Dense_Blade_Servers. 83
185. P. Ranganathan, and J. Chang. (Re) Designing data-centric data centers. (2012). http:// www.hpl.hp.com/discover2012/pdf/2012_IEEEMicro_DCDC.pdf. 165
186. G. Ren, E. Tune, T. Moseley, Y. Shi, S. Rus, and R. Hundt. Google-wide profiling: A continuous profiling infrastructure for data centers. IEEE Micro 30(4) (July 2010), pp. 65-79. DOI: 10.1109/MM.2010.68. 31
187. Reuters. Dupont Fabros Technology, Inc. reports third quarter results. (November 5, 2008).
188. P. Reynolds, C. Killian, J. L. Wiener, J. C. Mogul, M. A. Shah, and A. Vahdat. Pip: detecting the unexpected in distributed systems. In Proceedings of USENIX NSDI (2006). 31
189. P. Reynolds, J. L. Wiener, J. C. Mogul, M. K. Aguilera, and A. Vahdat. WAP5: black box performance debugging for wide-area systems. In Proceedings of the 15th International World Wide Web Conference (2006). 31
190. Robert Frances Group. Total cost of ownership for Linux in the enterprise (July 2002). http://www-03.ibm.com/linux/RFG-LinuxTCO-vFINAL-Jul2002.pdf. 132
191. S. Rivoire, M. A. Shah, P. Ranganathan, C. Kozyrakis, and J. Meza. Models and metrics to enable energy-efficiency optimizations. Computer 40(12) (December 2007), pp. 39-48. 106
192. A. Saeed, N. Dukkipati, V. Valancius, V. The Lam, C. Contavalli, and A. Vahdat. Carousel: Scalable traffic shaping at end hosts. SIGCOMM (2017). 66
193. Samsung Z-NAND SSD technology brief. Ultra-low latency with Samsung Z-NAND SSD. 2017. https://www.samsung.com/us/labs/pdfs/collateral/Samsung_Z-NAND_ Technology_Brief_v5.pdf. 159, 166
194. U. Savagaonkar, N. Porter, N. Taha, B. Serebrin, and N. Mueller. Google Cloud Platform Blog, Titan in depth: Security in plaintext (August 2017). https://cloudplatform. googleblog.com/2017/08/Titan-in-depth-security-in-plaintext.html. 166
195. SAVVIS press release. SAVVIS sells assets related to two datacenters for $200 million. (June 29, 2007). http://news.centurylink.com/SAVVIS-Sells-Assets-Related-to-Two- Data-Centers-for-200-Million.
196. M. Schwarzkopf et al. Omega: flexible, scalable schedulers for large compute clusters. EuroSys (2013).
197. D. Serenyi. Keynote. 2nd Joint International Workshop on Parallel Data Storage & Data Intensive Scalable Intensive Computing Systems (November 2017). 9, 20, 152 , 154
198. B. Schroeder and G. A. Gibson. Understanding failures in petascale computers. J.Phys.: Conf. Ser. 78 (2007). DOI: 10.1088/1742-6596/78/1/012022. 145
199. B. Schroeder and G. A. Gibson. Disk failures in the real world: what does an MTTF of 1,000,000 hours mean to you?. In Proceedings of the 5th USENIX Conference on File and Storage Technologies (February 2007). 14, 149
200. S. Sankar, S. Gurumurthi, and M. R. Stan. Intra-disk parallelism: an idea whose time has come. In Proceedings of the ACM International Symposium on Computer Architecture, (June 2008), pp. 303-314. DOI: 10.1145/1394609.1382147. 114
201. A. Shehabi, S. Smith, D. Sartor, R. Brown, M. Herrlin, J. Koomey, E. Masanet, N. Horner, I. Azevedo, and W. Littner. U.S. Department of Energy: United States Data Center Usage Report (June 2016). 100
202. J. Shute, R. Vingralek, B. Samwel, B. Handy, C. Whipkey, E. Rollins, M. Oancea, K. Littlefield, D. Menestrina, S. Ellner, J. Cieslewicz, I. Rae, T. Stancescu, and H. Apte. F1: A distributed SQL database that scales. PVLDB 6(11): 1068-1079 (2013). 69
203. B. H. Sigelman et al. Dapper, a large-scale distributed systems tracing infrastructure. Google Research Report, http://research.google.com/pubs/pub36356.html. 2010. 31
204. D. Silver, A. Huang, C.J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, and S. Dieleman. Mastering the game of Go with deep neural networks and tree search. Nature 529 (7587) (2016). 28
205. A. Singh, J. Ong, A. Agarwal, G. Anderson, A. Armistead, R. Bannon, S. Boving, G. Desai, B. Felderman, P. Germano, A. Kanagala, J. Provost, J. Simmons, E. Tanda, J. Wanderer, U. Holzle, S. Stuart, and A. Vahdat. Jupiter rising: A decade of Clos topologies and centralized control in Google's datacenter network, SIGCOMM (2015). 62, 65, 152
206. Snappy: A fast compressor/decompressor. https://google.github.io/snappy/. 40
207. Storage Networking Industry Association (SNIA) Emerald Power Efficiency Measurement Specification V1.0 (August 2011). 106
208. SNIA Green Storage Initiative. http://www.snia.org/forums/green/.
209. W. Sounders. Server efficiency: Aligning energy use with workloads. Data Center Knowledge (June 12, 2012). 114
210. Storage Performance Council SPC Benchmark 2/Energy Extension V1.4 (November 2012). 106
211. SPEC Power. http://www.spec.org/power_ssj2008/. 106
212. S. Siddha, V. Pallipadi, and A. Van De Ven. Getting maximum mileage out of tickless In Proceedings of the Linux Symposium, Ottawa, Ontario, Canada (June 2007). 109
213. B. Schroeder, E. Pinheiro, and W.-D. Weber. DRAM errors in the wild: a large-scale field study. To appear in Proceedings of SIGMETRICS, 2009. 104, 149
214. Techarp.com, Intel desktop CPU guide. http://www.techarp.com/showarticle.aspx?artno= 337&pgno=6. 134
215. Terrazon Semiconductor. Soft errors in electronic memory - a white paper. http:// www.tezzaron.com/about/papers/soft_errors_1_1_secure.pdf. 148
216. M. Ton and B. Fortenbury. High performance buildings: datacenters - server power supplies. Lawrence Berkeley National Laboratories and EPRI (December 2005).
217. The Green 500. http://www.green500.org. 106
218. The Green Grid. A Framework for Data Center Energy Efficiency: https://tinyurl. com/yayave27.
219. The Green Grid. The Latest on The Green Grid's Productivity Research. https://tinyurl. com/y92umkyn. 99
220. The Green Grid datacenter power efficiency metrics: PUE and DCiE. https://tinyurl. com/d5jmst. 99
221. The Green Grid, WP#22-Usage and Public Reporting Guidelines for The Green Grid's Infrastructure Metrics PUE/DCiE. https://tinyurl.com/y9gwt4tk. 101, 102
222. The Green Grid, Quantitative analysis of power distribution configurations for datacenters. http://www.thegreengrid.org/gg_content/.
223. The Green Grid. Seven strategies to improve datacenter cooling efficiency. http://www. thegreengrid.org/gg_content/.
224. Telecommunications Industry Association standards. http://global.ihs.com/tia_telecom_ infrastructure.cfm?RID=Z56&MID=5280. 75, 76
225. TIA-942 Telecommunications Infrastructure Standard for Data Centers. http://www. adc.com/us/en/Library/Literature/102264AE.pdf Page 5.
226. Transaction Processing Performance Council. http://www.tpc.org. 45
227. TPC-C Executive Summary for the Superdome-Itanium2 (February 2007). 45
228. TPC-C Executive Summary for the ProLiant ML350G5 (September 2007). 45
229. W. Pitt Turner IV and J. H. Seader. Dollars per kW plus dollars per square foot are a better datacenter cost model than dollars per square foot alone. Uptime Institute White Paper (2006). 130
230. W. Pitt Turner IV, J. H. Seader, and K. G. Brill. Tier classifications define site infrastructure performance. Uptime Institute White Paper. 75
231. W. F. Tschudi, T. T. Xu, D. A. Sartor, and J. Stein. High performance datacenters: a research roadmap. Lawrence Berkeley National Laboratory, Berkeley, CA (2003).
232. Uptime Institute: Important to recognize the dramatic improvement in data center efficiency. http://blog.uptimeinstitute.com/2012/09/important-to-recognize-the-dramatic- improvement-in-data-center-efficiency/. 100
233. Uptime Institute: Data Center Site Infrastructure Tier Standard: Operational Sustainability. http://uptimeinstitute.com/publications. 75, 76
234. Uptime Institute Publications. http://uptimeinstitute.com/publications.
235. Uptime Institute: Data Center Site Infrastructure Tier Standard: Topology. http:// uptimeinstitute.com/publications. 76
236. A. Vahdat, M. Al-Fares, N. Farrington, R. N. Mysore, G. Porter, and S. Radhakrishnan. Scale-out networking in the data center. IEEE Micro 30(4):29-41 (2010). DOI: 10.1109/ MM.2010.72. 61, 65
237. A. Vahdat. ONS Keynote: Networking Challenges for the Next Decade. The Linux Foundation (April 2017). http://events17.linuxfoundation.org/sites/events/files/slides/ ONS%20Keynote%20Vahdat%202017.pdf. 165
238. A. Verma, L. Pedrosa, M. Korupolu, D. Oppenheimer, E. Tune, and J. Wilkes. Largescale cluster management at Google with Borg. In Proceedings of the Tenth European Conference on Computer Systems (EuroSys '15). ACM, New York, (2015). Article 18, 17 pages. DOI: 10.1145/2741948.2741964. 117, 152
239. VMware infrastructure architecture overview white paper. http://www.vmware.com/ pdf/vi_architecture_wp.pdf. 18
240. W. Vogels. Eventually consistent. ACM Queue (October 2008). http://queue.acm.org/ detail.cfm?id=1466448. 34, 37
241. D. Wang, C. Ren, A. Sivasubramaniam, B. Urgaonkar, and H. Fathy. Energy Storage in Datacenters: What, Where, and How much? DOI: 10.1145/2318857.2254780. 125
242. Wet bulb temperature. http://en.wikipedia.org/wiki/Wet-bulb_temperature. 92
243. W. Whitted, M. Sykora, K. Krieger, B. Jai, W. Hamburgen, J. Clidaras. D. L. Beaty, and G. Aigner. Data center uninterruptible power distribution architecture. http:// www. google.com/patents/US7560831. 80
244. Q. Wu, Q. Deng, L. Ganesh, C. H. Hsu, Y. Jin, S. Kumar, B. Li, J. Meza, and Y. J. Song. Dynamo: Facebook's Data Center-Wide Power Management System. 2016 ACM/ IEEE 43rd Annual International Symposium on Computer Architecture (ISCA), Seoul, (2016), pp. 469-480. 118, 165
245. Y. Wu, M. Schuster, Z. Chen., Q. V. Le, M. Norouzi., W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. Johnson, X. Liu, L. Kaiser, S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian, N. Patil, W. Wang, C. Young, J. Smith, J. Riesa, A. Rudnick, O. Vinyals, G. Corrado, M. Hughes, and J. Dean. Google's neural machine translation system: Bridging the gap between human and machine translation (September 26, 2016) http://arxiv.org/abs/1609.08144. 28