FaBRiQ: Leveraging Distributed Hash Tables towards Distributed Publish-Subscribe Message Queues

Proceedings - 2015 2nd IEEE/ACM International Symposium on Big Data Computing, BDC 2015

Volumn , Issue , 2016, Pages 11-20

  Sadooghi, Iman ^a   Wang, Ke ^a   Patel, Dharmit ^a   Zhao, Dongfang ^a   Li, Tonglin ^a   Srivastava, Shiva ^a   Raicu, Ioan ^a  

^a Illinois Institute of Technology   (United States)

Author keywords

Data Analytics; Distributed Hash Tables; Distributed Systems; Message Queue

Indexed keywords

DATA HANDLING; DATA STRUCTURES; DISTRIBUTED COMPUTER SYSTEMS; EFFICIENCY; FAULT TOLERANCE; NETWORK MANAGEMENT; QUEUEING THEORY; TELECOMMUNICATION NETWORKS; THROUGHPUT;

DATA ANALYTICS; DATA-INTENSIVE COMPUTING; DISTRIBUTED HASH TABLE; DISTRIBUTED SYSTEMS; HIGH PERFORMANCE COMPUTING (HPC); MESSAGE QUEUES; PUBLISH-SUBSCRIBE; STATE-OF-THE-ART SYSTEM;

BIG DATA;

EID: 84964350797     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/BDC.2015.42     Document Type: Conference Paper

References (39)

1. W. Vogels. " Improving the Cloud- More Efficient Queuing with SQS" [online] 2012, http://www.allthingsdistributed.com/2012/11/efficient-queueing-sqs.html
2. I. Sadooghi, S. Palur, et al. "Achieving Efficient Distributed Scheduling with Message Queues in the Cloud for Many-Task Computing and High-Performance Computing", Proceedings of the International Symposium on Cluster Computing and the Grid (CCGRID), 2014.
3. J. Kreps, N. Narkhede, and J. Rao. "Kafka: A distributed messaging system for log processing". NetDB, 2011.
4. A. Alten-Lorenz, Apache Flume, [online] 2013, https://cwiki.apache.org/FLUME/
5. A. Thusoo, Z. Shao, et al, "Data warehousing and analytics infrastructure at facebook, " in SIGMOD Conference, 2010, pp. 1013-1020.
6. J. Pearce, Scribe, [online] https://github.com/facebookarchive/scribe
7. T. J. Hacker and Z. Meglicki, "Using queue structures to improve job reliability, " in Proceedings of the 16th International Symposium on High-Performance Distributed Computing (HPDC), 2007, pp. 43-54.
8. I. Raicu, I. Foster, and Y. Zhao, " Many-task computing for grids and supercomputers", In MTAGS 2008, p. 1-11
9. A. Videla and J. J. Williams, "RabbitMQ in action". Manning, 2012.
10. B. Snyder, D. Bosanac, And R. Davies, "ActiveMQ in action" Manning, 2011.
11. S. Davies, and P. Broadhurst, "WebSphere MQ V6 Fundamentals", IBM Redbooks, 2005
12. Java Message Service Concepts, Oracle, [online] 2013, http://docs.oracle.com/javaee/6/tutorial/doc/bncdq.html
13. T. Li, X. Zhou, et. al. "ZHT: A light-weight reliable persistent dynamic scalable zero-hop distributed hash table, " in Proceedings of the IEEE IPDPS, 2013.
14. Amazon SQS, [online] 2014, http://aws.amazon.com/sqs/
15. D.Borthakur, HDFS architecture. Tech. rep., Apache Software Foundation, 2008.
16. Amazon Elastic Compute Cloud (Amazon EC2), Amazon Web Services, [online] 2013, http://aws.amazon.com/ec2/
17. P. Hunt, et. al., "ZooKeeper: wait-free coordination for internet-scale systems, " Proceedings of USENIXATC'10, 2010.
18. K. Ousterhout, P. Wendell, M. Zaharia, and I. Stoica. "Sparrow: distributed, low latency scheduling". Proceedings SOSP '13.
19. M. Wall, "Big Data: Are you ready for blast-off", BBC Business, March 2014
20. J. Lin and D. Ryaboy. Scaling big data mining infrastructure: The twitter experience. SIGKDD Explorations, 14 (2), 2012.
21. A. Reda, Y. Park, M. Tiwari, C. Posse, and S. Shah. Metaphor: a system for related search recommendations. In CIKM, 2012.
22. I. Raicu, P. Beckman, I. Foster. "Making a Case for Distributed File Systems at Exascale", Invited Paper, LSAP, 2011
23. R. Ramesh, L. Hu, and K. Schwan. "Project Hoover: auto-scaling streaming mapreduce applications". Proceedings of (MBDS '12). ACM, USA, 7-12. 2012
24. H. Liu, "Cutting mapreduce cost with spot market". 3rd USENIX Workshop on Hot Topics in Cloud Computing (2011).
25. T. White, "Hadoop: The Definitive Guide." O'Reilly Media, Inc., 2009
26. K. Wang, et. al. "Optimizing Load Balancing and Data-Locality with Data-aware Scheduling", IEEE Big Data 2014.
27. Amazon DynamoDB. [online] 2014 http://aws.amazon.com/dynamodb/
28. N. Liu, A. Haider, X.-H. Sun and D. Jin. "FatTreeSim: Modeling a Large-scale Fat-Tree Network for HPC Systems and Data Centers Using Parallel and Discrete Event Simulation, " ACM SIGSIM PADS, 2015.
29. N. Liu, et. al. "On the role of burst buffers in leadership-class storage systems, " IEEE MSST conference, 2012
30. I. Sadooghi, J. Martin, T. Li, I. Raicu, et. al. "Understanding the Performance and Potential of Cloud Computing for Scientific Applications", IEEE Transactions on Cloud Computing (TCC), 2015
31. T. Li, et. al. "A Convergence of Distributed Key-Value Storage in Cloud Computing and Supercomputing", Journal of Concurrency and Computation Practice and Experience (CCPE) 2015.
32. K. Wang, K. Qiao, I. Sadooghi, et. al. "Load-balanced and locality-aware scheduling for data-intensive workloads at extreme scales", Journal of Concurrency and Computation Practice and Experience (CCPE) 2015.
33. T. Li, I. Raicu, L. Ramakrishnan, "Scalable State Management for Scientific Applications in the Cloud", BigData 2014
34. T. Li, K. Keahey, K. Wang, D. Zhao, I. Raicu, "A Dynamically Scalable Cloud Data Infrastructure for Sensor Networks", ScienceCloud 2015
35. D. Patel, F. Khasib, I. Sadooghi, I. Raicu. "Towards In-Order and Exactly Once Delivery using Hierarchical Distributed Message Queues", (SCRAMBL'14) 2014
36. C. Dumitrescu, I. Raicu, I. Foster. "The Design, Usage, and Performance of GRUBER: A Grid uSLA-based Brokering Infrastructure", International Journal of Grid Computing, 2007
37. K. Wang, K. Brandstatter, I. Raicu. "SimMatrix: Simulator for MAny-Task computing execution fabRIc at eXascales", ACM HPC 2013
38. T. Li, R. Verma, X. Duan, H. Jin, I. Raicu. "Exploring Distributed Hash Tables in High-End Computing", ACM Performance Evaluation Review (PER), 2011
39. A. Rajendran, I. Raicu. "MATRIX: Many-Task Computing Execution Fabric for Extreme Scales", Illinois Institute of Technology, MS Thesis, 2013