HARDFS: Hardening HDFS with selective and lightweight versioning

Proceedings of the 11th USENIX Conference on File and Storage Technologies, FAST 2013

Volumn , Issue , 2013, Pages 105-118

  Do, Thanh ^a   Harter, Tyler ^a   Liu, Yingchao ^a   Gunawi, Haryadi S ^a,b   Arpaci Dusseau, Andrea C ^a   Arpaci Dusseau, Remzi H ^a  

^a UNIVERSITY OF WISCONSIN MADISON   (United States)

^b UNIVERSITY OF CHICAGO   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CRIME; DIGITAL STORAGE; FILE ORGANIZATION; PROGRAM DEBUGGING;

HADOOP DISTRIBUTED FILE SYSTEM (HDFS); MEMORY CORRUPTION; NAMESPACES; NEW APPROACHES; ORDERS OF MAGNITUDE; REAL SOFTWARES; SOFTWARE BUG; SPACE OVERHEAD;

INFORMATION MANAGEMENT;

EID: 84959002967     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (60)

1. A block report processing may incorrectly cause the namenode to delete blocks. https://issues. apache.org/jira/browse/HADOOP-1135.
2. AspectJ. www.eclipse.org/aspectj.
3. Corrupt replicas are not tracked correctly through block report from DataNode. https://issues. apache.org/jira/browse/HDFS-900.
4. Decommissioning on NN restart can complete without blocks being replicated. https://issues.apache.org/jira/browse/ HDFS-3087.
5. HADOOP JIRA. http://issues.apache.org/ jira/browse/HADOOP.
6. Hadoop MapReduce. http://hadoop.apache. org/mapreduce.
7. HBase. http://hbase.apache.org.
8. HDFS JIRA. http://issues.apache.org/ jira/browse/HDFS.
9. HDFS should not remove blocks while in safemode. https://issues.apache.org/ jira/browse/HADOOP-3002.
10. Namenode accepts block report from dead datanodes. https://issues.apache.org/jira/ browse/HDFS-1250.
11. Amazon s3 availability event. http://status. aws.amazon.com/s3-20080720.html, 2008.
12. Observations on errors, corrections, and trust of dependent systems. http://perspectives. mvdirona.com/, 2012.
13. Peter Alvaro, Tyson Condie, Neil Conway, Khaled Elmeleegy, Joseph M. Hellerstein, and Russell C Sears. BOOM Analytics: Exploring Data-Centric, Declarative Programming for the Cloud. In EuroSys '10.
14. Algirdas A. Aviẑienis. The Methodology of NVersion Programming. In Michael R. Lyu, editor, Software Fault Tolerance, chapter 2. John Wiley & Sons Ltd., 1995.
15. L. Bairavasundaram, Garth R. Goodson, Bianca Schroeder, A. Arpaci-Dusseau, and R. Arpaci-Dusseau. An Analysis of Data Corruption in the Storage Stack. In FAST '08, pages 223-238.
16. L. Bairavasundaram, Swaminathan Sundararaman, A. Arpaci-Dusseau, and R. Arpaci-Dusseau. Tolerating File-System Mistakes with EnvyFS. In USENIX '09.
17. Wendy Bartlett and Lisa Spainhower. Commercial Fault Tolerance: A Tale of Two Systems. IEEE Transactions on Dependable and Secure Computing, 1(1):87-96, January 2004.
18. Ken Birman, Gregory Chockler, and Robbert van Renesse. Towards a Cloud Computing Research Agenda. ACM SIGACT News, 40(2):68-80, June 2009.
19. Dina Bitton and Jim Gray. Disk shadowing. In VLDB 14, pages 331-338, Los Angeles, CA, August 1988.
20. Burton H. Bloom. Space/time trade-offs in hash coding with allowable errors. Communications of the ACM, 13:422-426, 1970.
21. Flavio Bonomi, Michael Mitzenmacher, Rina Panigrahy, Sushil Singh, and George Varghese. An improved construction for counting bloom filters. In Proceedings of the 14th conference on Annual European Symposium-Volume 14, ESA'06, pages 684-695, London, UK, UK, 2006. Springer-Verlag.
22. Dhruba Borthakur. Hadoop avatarnode high availability. http:// hadoopblog.blogspot.com/2010/02/ hadoop-namenode-high-availability. html, 2010.
23. George Candea, Shinichi Kawamoto, Yuichi Fujiki, Greg Friedman, and Armando Fox. Microreboot-A Technique for Cheap Recovery. In OSDI '04, pages 31-44.
24. Miguel Castro and Barbara Liskov. Practical Byzantine Fault Tolerance. In OSDI '99, New Orleans, LA, February 1999.
25. Tushar Chandra, Robert Griesemer, and Joshua Redstone. Paxos Made Live-An Engineering Perspective. In PODC '07.
26. Fay Chang, Jeffrey Dean, Sanjay Ghemawat, Wilson C. Hsieh, Deborah A. Wallach, Michael Burrows, Tushar Chandra, Andrew Fikes, and Robert Gruber. Bigtable: A Distributed Storage System for Structured Data. In OSDI '06, pages 205-218.
27. Peter M. Chen, Edward K. Lee, Garth A. Gibson, Randy H. Katz, and David A. Patterson. RAID: High-performance, Reliable Secondary Storage. ACM Computing Surveys, 26(2):145-185, June 1994.
28. A. Clement, M. Kapritsos, S. Lee, Y. Wang, L. Alvisi, M. Dahlin, and T. Riche. Upright cluster services. In SOSP '09.
29. MiguelCorreia, DanielGomez Ferro, Flavio P. Junqueira, and Marco Serafini. Practical hardening of crash-tolerant systems. In USENIX '12.
30. Jeffrey Dean. Underneath the Covers at Google: Current Systems and Future Directions. In Google I/O '08.
31. Li Fan, Pei Cao, J. Almeida, and A.Z. Broder. Summary cache: a scalable wide-area web cache sharing protocol. IEEE/ACMTransactions on Networking, 8(3):281-293, jun 2000.
32. Daniel Fryer, Kuei Sun, Rahat Mahmood, TingHao Cheng, Shaun Benjamin, Ashvin Goel, and Angela Demke Brown. Recon: Verifying file system consistency at runtime. In FAST '12.
33. Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung. The Google File System. In SOSP '03, pages 29-43.
34. Jim Gray and Andreas Reuter. Transaction Processing: Concepts and Techniques. Morgan Kaufmann, 1993.
35. ZhongxianGu, Earl T. Barr, David J. Hamilton, and Zhendong Su. Has the bug really been fixed? In ICSE (1), pages 55-64, 2010.
36. H. Gunawi, V. Prabhakaran, S. Krishnan, A. Arpaci-Dusseau, and R. Arpaci-Dusseau. Improving File System Reliability with I/O Shepherding. In SOSP '07.
37. Haryadi S. Gunawi, Thanh Do, Pallavi Joshi, Peter Alvaro, Joseph M. Hellerstein, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau, Koushik Sen, and Dhruba Borthakur. FATE and DESTINI: A Framework for Cloud Recovery Testing. In NSDI '11.
38. James Hamilton. On Designing and Deploying Internet-Scale Services. In LISA '07.
39. Tyler Harter, Chris Dragga, Michael Vaughn, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. A file is not a file: understanding the i/o behavior of apple desktop applications. In SOSP '11.
40. Alyssa Henry. Cloud Storage FUD: Failure and Uncertainty and Durability. In FAST '09.
41. Gordon F. Hughes and Joseph F. Murray. Reliability and Security of RAID Storage Systems and D2D Archives Using SATA Disk Drives. ACM Transactions on Storage, 1(1):95-107, February 2005.
42. Avinash Lakshman and Prashant Malik. Cassandra-a decentralized structured storage system. In LADIS '09.
43. Leslie Lamport, Robert Shostak, and Marshall Pease. The Byzantine Generals Problem. ACM Transactions on Programming Languages and Systems, 4(3):382-401, July 1982.
44. Diego Ongaro, Stephen M. Rumble, Ryan Stutsman, John K. Ousterhout, Mendel Rosenblum, and Mendel Rosenblum. Fast crash recovery in ramcloud. In SOSP '11.
45. David Patterson, Garth Gibson, and Randy Katz. A Case for Redundant Arrays of Inexpensive Disks (RAID). In SIGMOD '88, pages 109-116, Chicago, IL, June 1988.
46. Patrick Reynolds, Janet L. Wiener, Jeffrey C. Mogul, Mehul A. Shah, Charles Killian, and Amin Vahdat. Pip: Detecting the unexpected in distributed systems. In NSDI '06.
47. Rodrigo Rodrigues, Miguel Castro, and Barbara Liskov. Base: using abstraction to improve fault tolerance. In SOSP '01.
48. Bianca Schroeder, Eduardo Pinheiro, and Wolf-Dietrich Weber. DRAM errors in the wild: A Large-Scale Field Study. In SIGMETRICS '09, Seattle, Washington, June 2007.
49. Konstantin Shvachko, Hairong Kuang, Sanjay Radia, and Robert Chansler. The Hadoop Distributed File System. In MSST '10, 2010.
50. Konstantin V. Shvachko. Hdfs scalability: The limits to growth. ;login:, 35, 2010.
51. Muthian Sivathanu, V. Prabhakaran, A. Arpaci-Dusseau, and R. Arpaci-Dusseau. Improving Storage System Availability with D-GRAID. In FAST '04, pages 15-30, San Francisco, CA, April 2004.
52. Muthian Sivathanu, V. Prabhakaran, Florentina I. Popovici, Timothy E. Denehy, A. Arpaci-Dusseau, and R. Arpaci-Dusseau. Semantically-Smart Disk Systems. In FAST '03, pages 73-88, San Francisco, CA, April 2003.
53. Jacek Sliwerski, Thomas Zimmermann, and Andreas Zeller. When do changes induce fixes? In ACM SIGSOFT Software Engineering Notes, pages 1-5, 2005.
54. Swaminathan Sundararaman, Sriram Subramanian, Abhishek Rajimwale, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau, Michael M. Swift. Membrane: Operating System Support for Restartable File Systems. In FAST '10.
55. Michael M. Swift, Brian N. Bershad, and Henry M. Levy. Improving the Reliability of Commodity Operating Systems. In SOSP '03.
56. Michael M. Swift, Brian N. Bershad, and Henry M. Levy. Recovering device drivers. In OSDI '04, pages 1-16.
57. John Wilkes, Richard Golding, Carl Staelin, and Tim Sullivan. The HP AutoRAID Hierarchical Storage System. ACM Transactions on Computer Systems, 14(1):108-136, February 1996.
58. Junfeng Yang, Tisheng Chen, Ming Wu, Zhilei Xu, Xuezheng Liu, Haoxiang Lin, Mao Yang, Fan Long, Lintao Zhang, and Lidong Zhou. MODIST: Transparent Model Checking of Unmodified Distributed Systems. In NSDI '09.
59. Junfeng Yang, Can Sar, and Dawson Engler. EXPLODE: A Lightweight, General System for Finding Serious Storage System Errors. In OSDI '06.
60. Zuoning Yin, Ding Yuan, Yuanyuan Zhou, Shankar Pasupathy, and Lakshmi Bairavasundaram. How do fixes become bugs?-a comprehensive characteristic study on incorrect fixes in commercial and open source operating systems. In FSE '11.