Analysis of HDFS under HBase: A Facebook messages case study

Proceedings of the 12th USENIX Conference on File and Storage Technologies, FAST 2014

Volumn , Issue , 2014, Pages 199-212

  Harter, Tyler ^a   Borthakur, Dhruba ^b   Dong, Siying ^b   Aiyer, Amitanand ^b   Tang, Liyin ^b   Arpaci Dusseau, Andrea C ^a   Arpaci Dusseau, Remzi H ^a  

^a UNIVERSITY OF WISCONSIN MADISON   (United States)

^b FACEBOOK   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPACTION; SOCIAL NETWORKING (ONLINE);

COST SIMULATIONS; FACEBOOK; LARGE FILES; LAYERED DESIGN; NETWORK I/O;

RANDOM ACCESS STORAGE;

EID: 84988288142     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (30)

1. Remzi H. Arpaci-Dusseau and Andrea C. Arpaci-Dusseau. Operating Systems: Three Easy Pieces. Arpaci-Dusseau Books, 2014.
2. Mary Baker, John Hartman, Martin Kupfer, Ken Shirriff, and John Ousterhout. Measurements of a Distributed File System. In Proceedings of the 13th ACM Symposium on Operating Systems Principles (SOSP’91), pages 198–212, Pacific Grove, California, October 1991.
3. Fay Chang, Jeffrey Dean, Sanjay Ghemawat, Wilson C. Hsieh, Deborah A. Wallach, Michael Burrows, Tushar Chandra, Andrew Fikes, and Robert Gruber. Bigtable: A Distributed Storage System for Structured Data. In Proceedings of the 7th Symposium on Operating Systems Design and Implementation (OSDI’06), pages 205–218, Seattle, Washington, November 2006.
4. Chen, Yanpei and Alspaugh, Sara and Katz, Randy. Interactive Analytical Processing in Big Data Systems: A Cross-industry Study of MapReduce Workloads. Proc. VLDB Endow., August 2012.
5. Timothy E. Denehy, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. Journal-guided Resynchronization for Software RAID. In Proceedings of the 4th USENIX Symposium on File and Storage Technologies (FAST’05), pages 87–100, San Francisco, California, December 2005.
6. Dhruba Borthakur and Kannan Muthukkaruppan and Karthik Ranganathan and Samuel Rash and Joydeep Sen Sarma and Nicolas Spiegelberg and Dmytro Molkov and Rodrigo Schmidt and Jonathan Gray and Hairong Kuang and Aravind Menon and Amitanand Aiyer. Apache Hadoop Goes Realtime at Facebook. In Proceedings of the 2011 ACM SIGMOD International Conference on Management of Data (SIGMOD’11), Athens, Greece, June 2011.
7. E. W. Dijkstra. The Structure of the THE Multiprogramming System. Communications of the ACM, 11(5):341–346, May 1968.
8. IBM Product Documentation. Notes/domino best practices: Transaction logging. http://www-01.ibm.com/support/docview.wss?uid=swg27009309, 2013.
9. Ford, Daniel and Labelle, François and Popovici, Florentina I. and Stokely, Murray and Truong, Van-Anh and Barroso, Luiz and Grimes, Carrie and Quinlan, Sean. Availability in Globally Distributed Storage Systems. In Proceedings of the 9th Symposium on Operating Systems Design and Implementation (OSDI’10), Vancouver, Canada, December 2010.
10. Gregory R. Ganger. Blurring the Line Between Oses and Storage Devices. Technical Report CMU-CS-01-166, Carnegie Mellon University, December 2001.
11. Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung. The Google File System. In Proceedings of the 19th ACM Symposium on Operating Systems Principles (SOSP’03), pages 29–43, Bolton Landing, New York, October 2003.
12. Jim Gray. Tape is Dead. Disk is Tape. Flash is Disk, RAM Locality is King, 2006.
13. Tyler Harter, Chris Dragga, Michael Vaughn, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. A File is Not a File: Understanding the I/O Behavior of Apple Desktop Applications. In Proceedings of the 23rd ACM Symposium on Operating Systems Principles (SOSP’11), Cascais, Portugal, October 2011.
14. Joseph L. Hellerstein. Google cluster data. Google research blog, January 2010. Posted at http://googleresearch.blogspot.com/2010/01/google-cluster-data.html.
15. Qi Huang, Ken Birman, Robbert van Renesse, Wyatt Lloyd, Sanjeev Kumar, and Harry C. Li. An Analysis of Facebook Photo Caching. In Proceedings of the 24th ACM Symposium on Operating Systems Principles (SOSP’13), pages 167–181, Farmington, Pennsylvania, November 2013.
16. Rini T. Kaushik and Milind A Bhandarkar. GreenHDFS: Towards an Energy-Conserving, Storage-Efficient, Hybrid Hadoop Compute Cluster. In The 2010 Workshop on Power Aware Computing and Systems (HotPower’10), Vancouver, Canada, October 2010.
17. Niall Kennedy. Facebook’s Photo Storage Rewrite. http://www.niallkennedy.com/blog/2009/04/facebookhaystack.html, April 2009.
18. Butler W. Lampson. Hints for Computer System Design. In Proceedings of the 9th ACM Symposium on Operating System Principles (SOSP’83), pages 33–48, Bretton Woods, New Hampshire, October 1983.
19. Edward K. Lee and Chandramohan A. Thekkath. Petal: Distributed Virtual Disks. In Proceedings of the 7th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS VII), Cambridge, Massachusetts, October 1996.
20. Kannan Muthukkaruppan. Storage Infrastructure Behind Facebook Messages. In Proceedings of International Workshop on High Performance Transaction Systems (HPTS’11), Pacific Grove, California, October 2011.
21. John K. Ousterhout, Herve Da Costa, David Harrison, John A. Kunze, Mike Kupfer, and James G. Thompson. A Trace-Driven Analysis of the UNIX 4.2 BSD File System. In Proceedings of the 10th ACM Symposium on Operating System Principles (SOSP’85), pages 15–24, Orcas Island, Washington, December 1985.
22. Matt Perdeck. Speeding up database access. http://www.codeproject.com/Articles/296523/Speeding-updatabase-access-part-8-Fixing-memory-d, 2011.
23. Jerome H. Saltzer, David P. Reed, and David D. Clark. End-to-end arguments in system design. ACM Transactions on Computer Systems, 2(4):277–288, November 1984.
24. Konstantin Shvachko, Hairong Kuang, Sanjay Radia, and Robert Chansler. The Hadoop Distributed File System. In Proceedings of the 26th IEEE Symposium on Mass Storage Systems and Technologies (MSST’10), Incline Village, Nevada, May 2010.
25. Jason Sobel. Needle in a haystack: Efficient storage of billions of photos. http://www.flowgram.com/p/2qi3k8eicrfgkv, June 2008.
26. Nicolas Spiegelberg. Allow record compression for hlogs. https://issues.apache.org/jira/browse/HBASE-8155, 2013.
27. Chandramohan A. Thekkath, Timothy Mann, and Edward K. Lee. Frangipani: A Scalable Distributed File System. In Proceedings of the 16th ACM Symposium on Operating Systems Principles (SOSP’97), pages 224–237, Saint-Malo, France, October 1997.
28. Werner Vogels. File system usage in Windows NT 4.0. In Proceedings of the 17th ACM Symposium on Operating Systems Principles (SOSP’99), pages 93–109, Kiawah Island Resort, South Carolina, December 1999.
29. Yang Wang and Manos Kapritsos and Zuocheng Ren and Prince Mahajan and Jeevitha Kirubanandam and Lorenzo Alvisi and Mike Dahlin. Robustness in the Salus Scalable Block Store. In Presented as part of the 10th USENIX Symposium on Networked Systems Design and Implementation, Lombard, Illinois, April 2013.
30. Zaharia, Matei and Chowdhury, Mosharaf and Das, Tathagata and Dave, Ankur and Ma, Justin and McCauley, Murphy and Franklin, Michael J. and Shenker, Scott and Stoica, Ion. Resilient Distributed Datasets: A Fault-tolerant Abstraction for In-memory Cluster Computing. In Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation, San Jose, California, April 2010.