Trends in big data analytics

Journal of Parallel and Distributed Computing

Volumn 74, Issue 7, 2014, Pages 2561-2573

  Kambatla, Karthik ^a   Kollias, Giorgos ^b   Kumar, Vipin ^c   Grama, Ananth ^a  

^a PURDUE UNIVERSITY   (United States)

^b IBM T J WATSON RESEARCH CENTER   (United States)

^c UNIVERSITY OF MINNESOTA   (United States)

Author keywords

Analytics; Big data; Data centers; Distributed systems

Indexed keywords

APPLICATION PROGRAMS; COMPUTER SYSTEMS; FAULT TOLERANCE; HARDWARE; NETWORK SECURITY;

ANALYTICS; DATA CENTERS; DISTRIBUTED SYSTEMS; DISTRIBUTED TECHNIQUES; EMERGING APPLICATIONS; FUTURE GENERATIONS; METHOD AND SOFTWARE DEVELOPMENT; PARALLEL AND DISTRIBUTED SYSTEMS;

BIG DATA;

EID: 84901640093     PISSN: 07437315     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jpdc.2014.01.003     Document Type: Article

References (98)

1. Daniel J. Abadi, Don Carney, Ugur Çetintemel, Mitch Cherniack, Christian Convey, Sangdon Lee, Michael Stonebraker, Nesime Tatbul, and Stan Zdonik Aurora: a new model and architecture for data stream management VLDB J. 12 2 2003 120 139 (Pubitemid 37017924)
2. Azza Abouzeid, Kamil Bajda-Pawlikowski, Daniel Abadi, Alexander Rasin, Avi Silberschatz, HadoopDB: an architectural hybrid of MapReduce and DBMS technologies for analytical workloads, in: VLDB, 2009.
3. http://agbeat.com/tech-news/how-carriers-gather-track-and-sell-your- private-data/
4. Yanif Ahmad, Bradley Berg, Uur Cetintemel, Mark Humphrey, Jeong-Hyon Hwang, Anjali Jhingran, Anurag Maskey, Olga Papaemmanouil, Alexander Rasin, Nesime Tatbul, Wenjuan Xing, Ying Xing, and Stan Zdonik Distributed operation in the borealis stream processing engine Proceedings of the 2005 ACM SIGMOD International Conference on Management of Data SIGMOD'05 2005 ACM New York, NY, USA 882 884 (Pubitemid 43038997)
5. Mohammad Al-Fares, Alexander Loukissas, and Amin Vahdat A scalable, commodity data center network architecture Victor Bahl, David Wetherall, Stefan Savage, Ion Stoica, SIGCOMM 2008 ACM 63 74
6. Mohammad Alizadeh, Albert G. Greenberg, David A. Maltz, Jitendra Padhye, Parveen Patel, Balaji Prabhakar, Sudipta Sengupta, and Murari Sridharan Data center TCP (DCTCP) Shivkumar Kalyanaraman, Venkata N. Padmanabhan, K.K. Ramakrishnan, Rajeev Shorey, Geoffrey M. Voelker, SIGCOMM 2010 ACM 63 74
7. David G. Andersen, Jason Franklin, Michael Kaminsky, Amar Phanishayee, Lawrence Tan, and Vijay Vasudevan FAWN: a fast array of wimpy nodes Commun. ACM 54 7 2011 101 109
8. Rasmus Andersen, Brian Vinter, The scientific byte code virtual machine, in: GCA, 2008, pp. 175-181.
9. H. Andrade, B. Gedik, K.L. Wu, and P.S. Yu Processing high data rate streams in system S J. Parallel Distrib. Comput. 71 2 2011 145 156
10. Luiz André Barroso, and Urs Hölzle The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines Synthesis Lectures on Computer Architecture 2009 Morgan & Claypool Publishers
11. Krste Asanovic, Ras Bodik, Bryan Christopher Catanzaro, Joseph James Gebis, Parry Husbands, Kurt Keutzer, David A. Patterson, William Lester Plishker, John Shalf, Samuel Webb Williams, and Katherine A. Yelick The Landscape of Parallel Computing Research: A View from Berkeley. Technical Report UCB/EECS-2006-183, EECS Department, University of California, Berkeley 2006
12. M. Athanassoulis, A. Ailamaki, S. Chen, P. Gibbons, and R. Stoica Flash in a DBMS: where and how? Bull. IEEE Comput. Soc. Tech. Committee Data Eng. 2010
13. Jason Baker, Chris Bond, James C. Corbett, J.J. Furman, Andrey Khorlin, James Larson, Jean-Michel Leon, Yawei Li, Alexander Lloyd, Vadim Yushprakh, Megastore: Providing Scalable, Highly Available Storage for Interactive Services, in: CIDR'11, 2011.
14. J. Baliga, R.W.A. Ayre, K. Hinton, and R.S. Tucker Green cloud computing: balancing energy in processing, storage, and transport Proc. IEEE 99 1 2011 149 167
15. Costas Bekas, and Alessandro Curioni A new energy aware performance metric Comput. Sci.-Res. Dev. 25 3-4 2010 187 195
16. K. Birman, D. Freedman, Qi Huang, and P. Dowell Overcoming cap with consistent soft-state replication Computer 45 2 2012 50 58
17. E.A. Brewer, Towards robust distributed systems, in: Proc. 19th Annual ACM Symposium on Priniciples of Distributed Computing, PODC, 2000, pp. 7-10.
18. Mike Burrows The chubby lock service for loosely-coupled distributed systems Proceedings of the 7th Symposium on Operating Systems Design and Implementation OSDI'06 2006 USENIX Association Berkeley, CA, USA 335 350
19. Rajkumar Buyya, Chee Shin Yeo, Srikumar Venugopal, James Broberg, and Ivona Brandic Cloud computing and emerging IT platforms: vision, hype, and reality for delivering computing as the 5th utility Future Gener. Comput. Syst. 25 6 2009 599 616
20. Brad Calder, Ju Wang, Aaron Ogus, Niranjan Nilakantan, Arild Skjolsvold, Sam McKelvie, Yikang Xu, Shashwat Srivastav, Jiesheng Wu, Huseyin Simitci, Jaidev Haridas, Chakravarthy Uddaraju, Hemal Khatri, Andrew Edwards, Vaman Bedekar, Shane Mainali, Rafay Abbasi, Arpit Agarwal, Mian Fahim ul Haq, Muhammad Ikram ul Haq, Deepali Bhardwaj, Sowmya Dayanand, Anitha Adusumilli, Marvin McNett, Sriram Sankaran, Kavitha Manivannan, and Leonidas Rigas Windows azure storage: a highly available cloud storage service with strong consistency Proceedings of the Twenty-Third ACM Symposium on Operating Systems Principles SOSP'11 2011 ACM New York, NY, USA 143 157
21. Fay Chang, Jeffrey Dean, Sanjay Ghemawat, Wilson C. Hsieh, Deborah A. Wallach, Mike Burrows, Tushar Chandra, Andrew Fikes, Robert E. Gruber, Bigtable: a distributed storage system for structured data, in: OSDI, 2006.
22. Shimin Chen, Phillip B. Gibbons, and Suman Nath Rethinking database algorithms for phase change memory CIDR 2011 21 31 www.crdrdb.org
23. Yanpei Chen, Rean Griffith, Junda Liu, Randy H. Katz, and Anthony D. Joseph Understanding TCP incast throughput collapse in datacenter networks Jeffrey C. Mogul, WREN 2009 ACM 73 82
24. Joel Coburn, Adrian M. Caulfield, Ameen Akel, Laura M. Grupp, Rajesh K. Gupta, Ranjit Jhala, and Steven Swanson NV-heaps: making persistent objects fast and safe with next-generation, non-volatile memories Proceedings of the 16th International Conference on Architectural Support for Programming Languages and Operating Systems, 16th ASPLOS'11 2011 ACM Press Newport Beach, CA, USA 105 118
25. Tyson Condie, Neil Conway, Peter Alvaro, Joseph Hellerstein, Khaled Elmeleegy, Russell Sears, MapReduce online, in: NSDI, 2009.
26. Tyson Condie, Neil Conway, Peter Alvaro, Joseph M. Hellerstein, Khaled Elmeleegy, Russell Sears, MapReduce online, in: NSDI, 2010.
27. Jeremy Condit, Edmund B. Nightingale, Christopher Frost, Engin Ipek, Benjamin C. Lee, Doug Burger, and Derrick Coetzee Better I/O through byte-addressable, persistent memory Proceedings of the 22nd Symposium on Operating Systems Principles (22nd SOSP'09), Operating Systems Review (OSR) 2009 ACM SIGOPS Big Sky, MT 133 146
28. Brian F. Cooper, Raghu Ramakrishnan, Utkarsh Srivastava, Adam Silberstein, Philip Bohannon, Hans-Arno Jacobsen, Nick Puz, Daniel Weaver, and Ramana Yerneni Pnuts: Yahoo!'s hosted data serving platform Proc. VLDB Endow. 1 2 2008 1277 1288
29. Jeffrey Dean, Sanjay Ghemawat, MapReduce: simplified data processing on large clusters, in: OSDI, 2004.
30. Giuseppe DeCandia, Deniz Hastorun, Madan Jampani, Gunavardhan Kakulapati, Avinash Lakshman, Alex Pilchin, Swaminathan Sivasubramanian, Peter Vosshall, and Werner Vogels Dynamo: Amazon's highly available key-value store Proceedings of Twenty-First ACM SIGOPS Symposium on Operating Systems Principles SOSP'07 2007 ACM New York, NY, USA 205 220 (Pubitemid 351429377)
31. Patrick Th. Eugster, Pascal A. Felber, Rachid Guerraoui, and Anne-Marie Kermarrec The many faces of publish/subscribe ACM Comput. Surv. 35 2 2003 114 131
32. Nathan Farrington, George Porter, Sivasankar Radhakrishnan, Hamid Hajabdolali Bazzaz, Vikram Subramanya, Yeshaiahu Fainman, George Papen, and Amin Vahdat Helios: a hybrid electrical/optical switch architecture for modular data centers Shivkumar Kalyanaraman, Venkata N. Padmanabhan, K.K. Ramakrishnan, Rajeev Shorey, Geoffrey M. Voelker, SIGCOMM 2010 ACM 339 350
33. J. Fisher-Ogden, Hardware support for efficient virtualization.
34. Apache flume http://flume.apache.org
35. I. Foster, and C. Kesselman The Grid: Blueprint for a New Computing Infrastructure 1999 Morgan-Kaufmann
36. Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung The google file system Proceedings of the Nineteenth ACM Symposium on Operating Systems Principles SOSP'03 2003 ACM New York, NY, USA 29 43 (Pubitemid 40929686)
37. Apache giraph http://incubator.apache.org/giraph
38. Goldenorb http://www.raveldata.com/goldenorb
39. Chuanxiong Guo, Guohan Lu, Dan Li, Haitao Wu, Xuan Zhang, Yunfeng Shi, Chen Tian, Yongguang Zhang, and Songwu Lu BCube: a high performance, server-centric network architecture for modular data centers Pablo Rodriguez, Ernst W. Biersack, Konstantina Papagiannaki, Luigi Rizzo, SIGCOMM 2009 ACM 63 74
40. Apache hadoop http://hadoop.apache.org
41. Nathan Halko, Per-Gunnar Martinsson, and Joel A. Tropp Finding structure with randomness: probabilistic algorithms for constructing approximate matrix decompositions SIAM Rev. 53 2 2011 217 288
42. Nikos Hardavellas, Michael Ferdman, Babak Falsafi, and Anastasia Ailamaki Toward dark silicon in servers IEEE Micro 31 4 2011 6 15
43. Apache hbase http://hadoop.apache.org/hbase
44. Apache hadoop hdfs http://hadoop.apache.org/hdfs
45. Apache hedwig https://cwiki.apache.org/BOOKKEEPER/hedwig.html
46. Andrew Herdrich, Ramesh Illikkal, Ravi R. Iyer, Donald Newell, Vineet Chadha, and Jaideep Moses Rate-based QoS techniques for cache/memory in CMP platforms Proceedings of the 23rd International Conference on Supercomputing, 23rd ICS'09 2009 ACM Press. Intel Yorktown Heights, NY, USA 479 488
47. Improving MapReduce Performance in Heterogeneous Environments 2008 USENIX Association San Diego, CA 12/2008
48. Benjamin Hindman, Andy Konwinski, Matei Zaharia, Ali Ghodsi, Anthony D. Joseph, Randy Katz, Scott Shenker, and Ion Stoica Mesos: a platform for fine-grained resource sharing in the data center Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation NSDI'11 2011 USENIX Association Berkeley, CA, USA 22
49. U. Hölzle Brawny cores still beat wimpy cores, most of the time IEEE Micro 30 4 2010
50. Hortonworks blog http://hortonworks.com/blog/executive-video-series-the- hortonworks-vision-for-apache-hadoop
51. Patrick Hunt, Mahadev Konar, Flavio P. Junqueira, and Benjamin Reed Zookeeper: wait-free coordination for internet-scale systems Proceedings of the 2010 USENIX Conference on USENIX Annual Technical Conference USENIXATC'10 2010 USENIX Association Berkeley, CA, USA 11
52. Made in IBM labs: holey optochip first to transfer one trillion bits of information per second using the power of light, 2012. http://www-03.ibm.com/ press/us/en/pressrelease/37095.wss.
53. Michael Isard, Mihai Budiu, Yuan Yu, Andrew Birrell, Dennis Fetterly, Dryad: distributed data-parallel programs from sequential building blocks, in: EuroSys, 2007.
54. The International Technology Roadmap for Semiconductors 2010 http://www.itrs.net/
55. R. Iyer, R. Illikkal, L. Zhao, S. Makineni, D. Newell, J. Moses, and P. Apparao Datacenter-on-chip architectures: tera-scale opportunities and challenges Intel Tech. J. 11 3 2007 227 238
56. Karthik Kambatla, Naresh Rapolu, Suresh Jagannathan, Ananth Grama, Asynchronous algorithms in MapReduce, in: IEEE International Conference on Cluster Computing, CLUSTER, 2010.
57. S. Kandula, S. Sengupta, A. Greenberg, P. Patel, and R. Chaiken The nature of data center traffic: measurements & analysis Proceedings of the 9th ACM SIGCOMM Conference on Internet Measurement Conference 2009 ACM 202 208
58. Avinash Lakshman, Prashant Malik, Cassandra: a structured storage system on a p2p network, in: SPAA, 2009.
59. Benjamin C. Lee, Ping Zhou, Jun Yang, Youtao Zhang, Bo Zhao, Engin Ipek, Onur Mutlu, and Doug Burger Phase-change technology and the future of main memory IEEE Micro 30 1 2010 143
60. http://public.web.cern.ch/public/en/LHC/Computing-en.html
61. Kevin Lim, Jichuan Chang, Trevor Mudge, Parthasarathy Ranganathan, Steven K. Reinhardt, and Thomas F. Wenisch Disaggregated memory for expansion and sharing in blade servers Proc. 36th International Symposium on Computer Architecture, 36th ISCA'09 2009 ACM SIGARCH Austin, TX
62. Wyatt Lloyd, Michael J. Freedman, Michael Kaminsky, and David G. Andersen Don't settle for eventual: scalable causal consistency for wide-area storage with cops Proceedings of the Twenty-Third ACM Symposium on Operating Systems Principles PSOSP'11 2011 ACM New York, NY, USA 401 416
63. Grzegorz Malewicz, Matthew H. Austern, Aart J.C Bik, James C. Dehnert, Ilan Horn, Naty Leiser, Grzegorz Czajkowski, Pregel: a system for large-scale graph processing, in: SIGMOD, 2010.
64. http://www.information-management.com/issues/21-5/big-data-is-scaling-bi- and-analytics-10021093-1.html
65. Dejan S. Milojicic, Vana Kalogeraki, Rajan Lukose, Kiran Nagaraja, Jim Pruyne, Bruno Richard, Sami Rollins, and Zhichen Xu Peer-to-peer Computing. Technical Report HPL-2002-57R1, Hewlett Packard Laboratories, May 2 1900
66. Tomer Y. Morad, Uri C. Weiser, A. Kolodnyt, Mateo Valero, and Eduard Ayguadé Performance, power efficiency and scalability of asymmetric cluster chip multiprocessors Comput. Archit. Lett. 5 1 2006 14 17 (Pubitemid 46438057)
67. Christopher Olston, Benjamin Reed, Utkarsh Srivastava, Ravi Kumar, and Andrew Tomkins Pig Latin: a Not-So-Foreign language for data processing SIGMOD 2008 ACM ID: 1376726
68. Innovation at Google: The Physics of Data 2009 http://www.parc.com/event/ 936/innovation-at-google.html
69. Phoebus https://github.com/xslogic/phoebus
70. Keshav Pingali, Donald Nguyen, Milind Kulkarni, Martin Burtscher, Amber Hassaan, Rashid Kaleem, Tsung-Hsien Lee, Andrew Lenharth, Roman Manevich, Mario Méndez-Lojo, Dimitrios Prountzos, Xin Sui, The tao of parallelism in algorithms, in: PLDI, 2011.
71. Gerald J. Popek, and R.P. Goldberg Formal requirements for virtualizable third generation architectures Commun. ACM 17 7 1974
72. Russell Power, Jinyang Li, Piccolo: building fast, distributed programs with partitioned tables, in: OSDI, 2010.
73. Moinuddin K. Qureshi, Vijayalakshmi Srinivasan, and Jude A. Rivers Scalable high performance main memory system using phase-change memory technology Proc. 36th International Symposium on Computer Architecture 36th ISCA'09 June 2009 ACM SIGARCH Austin, TX IBM T.J. Watson RC
74. Parthasarathy Ranganathan From microprocessors to nanostores: rethinking data-centric systems IEEE Comput. 44 1 2011 39 48
75. Parthasarathy Ranganathan, and Jichuan Chang (Re)designing data-centric data centers IEEE Micro 32 1 2012 66 70
76. Colby Ranger, Ramanan Raghuraman, Arun Penmetsa, Gary Bradski, Christos Kozyrakis, Evaluating mapreduce for multi-core and multiprocessor system, in: Proceedings of the 13th Intl. Symposium on High-Performance Computer Architecture (HPCA), Phoenix, AZ, 2007.
77. Naresh Rapolu, Karthik Kambatla, Suresh Jagannathan, and Ananth Grama Transmr: data-centric programming beyond data parallelism Proceedings of the 3rd USENIX Conference on Hot Topics in Cloud Computing HotCloud'11 2011 USENIX Association Berkeley, CA, USA pp. 19-19
78. Benjamin Reed, and Flavio P. Junqueira A simple totally ordered broadcast protocol Proceedings of the 2nd Workshop on Large-Scale Distributed Systems and Middleware LADIS'08 2008 ACM New York, NY, USA 2:1 2:6
79. Erik Riedel, Garth Gibson, and Christos Faloutsos Active storage for large-scale data mining and multimedia Proceedings of the 24th International Conference on Very Large Data Bases VLDB'98 1998 Morgan Kaufmann East Sussex, San Francisco 62 73
80. Mendel Rosenblum, and Tal Garfinkel Virtual machine monitors: current technology and future trends IEEE Comput. 38 5 2005 39 47 (Pubitemid 40784341)
81. Apache s4 http://incubator.apache.org/s4
82. Mehul Shah, Parthasarathy Ranganathan, Jichuan Chang, Niraj Tolia, David Roberts, and Trevor Mudge Data dwarfs: Motivating a Coverage Set for Future Large Data Center Workloads, Technical Report HPL-2010-115, Hewlett Packard Laboratories, November 8 2010
83. J.E. Short, R.E. Bohn, and C. Baru How much information? 2010-report on enterprise server information. UCSD Global Information Industry Center 2011
84. Clinton Wills Smullen IV, Vidyabhushan Mohan, Anurag Nigam, Sudhanva Gurumurthi, and Mircea R. Stan Relaxing non-volatility for fast and energy-efficient STT-RAM caches HPCA 2011 IEEE Computer Society 50 61
85. Twitter storm https://github.com/nathanmarz/storm
86. Jie Tang, Shaoshan Liu, Zhimin Gu, Xiao-Feng Li, and Jean-Luc Gaudiot Achieving middleware execution efficiency: hardware-assisted garbage collection operations J. Supercomput. 59 3 2012 1101 1119
87. http://www.statisticbrain.com/twitter-statistics/
88. Scalable, Energy-Efficient Data Centers and Clouds 2012 http://iee.ucsb.edu/Data-Center-Report
89. Leslie G. Valiant A bridging model for parallel computation Commun. ACM 33 8 1990 103 111
90. Arun Venkataramani, Ravi Kokku, and Mike Dahlin Tcp nice: a mechanism for background transfers Proceedings of the 5th symposium on Operating Systems Design and Implementation OSDI'02 2002 ACM New York, NY, USA 329 343
91. Shivaram Venkataraman, Niraj Tolia, Parthasarathy Ranganathan, and Roy H. Campbell Consistent and durable data structures for non-volatile byte-addressable memory Gregory R. Ganger, John Wilkes, FAST 2011 USENIX 61 75
92. Ymir Vigfusson, Hussam Abu-Libdeh, Mahesh Balakrishnan, Ken Birman, Robert Burgess, Gregory Chockler, Haoyuan Li, and Yoav Tock Dr. multicast: Rx for data center communication scalability Proceedings of the 5th European conference on Computer systems EuroSys'10 2010 ACM New York, NY, USA 349 362
93. Haris Volos, Andres Jaan Tack, and Michael M. Swift Mnemosyne: lightweight persistent memory Proceedings of the 16th International Conference on Architectural Support for Programming Languages and Operating Systems, 16th ASPLOS'11 2011 ACM Press Newport Beach, CA, USA 91 104
94. March 2012. http://www.wallstreetdaily.com/2012/03/21/forever-growth- trends-and-five-stocks-set-to-profit/.
95. A. Weiss Computing in the clouds Computing 16 2007
96. Xiaoxia Wu, Jian Li, Lixin Zhang, Evan Speight, Ram Rajamony, and Yuan Xie Hybrid cache architecture with disparate memory technologies Proc. 36th International Symposium on Computer Architecture, 36th ISCA'09 2009 ACM SIGARCH Austin, TX
97. Apache yarn http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/ hadoop-yarn-site/YARN.html
98. http://www.youtube.com/yt/press/statistics.html