Comparing Hadoop and Fat-Btree based access method for small file I/O applications

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Volumn 6184 LNCS, Issue , 2010, Pages 182-193

  Luo, Min ^a   Yokota, Haruo ^a  

^a TOKYO INSTITUTE OF TECHNOLOGY   (Japan)

Author keywords

Fat Btree; File I O; Hadoop; Parallel Database

Indexed keywords

ACCESS METHODS; DATA ACCESS; DISTRIBUTED FILE SYSTEMS; FILE MANAGEMENT; I/O APPLICATIONS; PARALLEL DATABASE; SMALL FILES; WEB APPLICATION;

METADATA;

INFORMATION MANAGEMENT;

EID: 77955034064     PISSN: 03029743     EISSN: 16113349     Source Type: Book Series    
DOI: 10.1007/978-3-642-14246-8_20     Document Type: Conference Paper

References (26)

1. http://www.cloudera.com/blog/2009/02/02/the-small-files-problem
2. Dean, J., Ghemawat, S.: MapReduce: Simplified Data Processing on Large Clusters. In: USENIX Symposium on Operating Systems Design and Implementation, OSDI 2004 (2004)
3. Boral, H., Alexander, W., Clay, L., Copeland, G., Danforth, S., Franklin, M., Hart, B., Smith, M., Valduriez, P.: Prototyping Bubba, a highly parallel database system. IEEE TKDE 2(1), 4-24 (1990) (Pubitemid 20678496)
4. Dewitt, D.J., Ghandeharizadeh, S., Schneider, D.A., Bricker, A., Hsiao, H.I., Rasmussen, R.: The Gamma database machine project. IEEE TKDE 2(1), 44-62 (1990)
5. Yokota, H., Kanemasa, Y., Miyazaki, J.: Fat-Btree: An update conscious parallel directory structure. In: ICDE 1999, March 1999, pp. 448-457. IEEE Computer Society, Los Alamitos (1999)
6. Abouzeid, A., Bajda-Pawlikowski, K., Abadi, D., Rasin, A., Silberschatz, A.: Hadoopdb: An architectural hybrid of mapreduce and dbms technologies for analytical workloads. In: VLDB 2009: Proceedings of the 2009 VLDB Endowment (2009)
7. Pavlo, A., Paulson, E., Rasin, A., Abadi, D.J., Dewitt, D.J., Madden, S., Stonebraker, M.: A comparison of approaches to large-scale data analysis. In: Proceedings of the 2009 ACM SIGMOD International Conference, June 2009. ACM, New York (2009)
8. Hadoop, http://hadoop.apache.org/
9. Augusto, C., Baquero, S.: Performance test of Hadoop and iRODS Distributed Storage Systems. In: Seminario De Invesigation III, Mayo 18 De (2009)
10. Delmerico, J., Byrnes, N., Bruno, A., Jones, M., Gallo, S., Chaudhary, V.: Comparing the Performance of Clusters, Hadoop and Active Disks on Microarray Correlation Computations. In: The 16th IEEE International Conference on High Performance Computing, HiPC 2009, Cochin, India (2009)
11. http://dev.mysql.com/doc/refman/5.1/en/overview.html
12. Wu, S., Kemme, B.: Postgres-R(SI): Combining Replica Control with Concurrency Control Based on Snapshot Isolation. In: Proceedings of the 21st Int'l Conf. on Data Engineering, ICDE 2005, Washington, DC, April 05-08, pp. 422-433 (2005)
13. Pacitti, E., Ozsu, M.T., Coulon, C.: Preventive multi-master replication in a cluster of autonomous databases. In: Kosch, H., Böszörmé nyi, L., Hellwagner, H. (eds.) Euro-Par 2003. LNCS, vol. 2790, pp. 318-327. Springer, Heidelberg (2003)
14. http://slony.info/documentation/failover.html
15. Stonebraker, M., Abadi, D., DeWitt, D.J., Madden, S., Paulson, E., Pavlo, A., Rasin, A.: Mapreduce and parallel dbmss: friends or foes? Commun. ACM 53(1), 64-71 (2010)
16. Carns, P., Lang, S., Ross, R., Vilayannur, M., Kunkel, J., Ludwig, T.: Small-File Access in Parallel File Systems. In: Proceedings of the 23rd IEEE International Parallel and Distributed Processing Symposium (April 2009)
17. Liu, X., Han, J., et al.: Implementing WebGIS on Hadoop: A Case Study of Improving Small File IO Performance on HDFS. In: IEEE Cluster, New Orleans, LA (September 1, 2009)
18. Leo, S., Santoni, F., Zanetti, G.: Biodoop: Bioinformatics on Hadoop. In: International Conference on Parallel Processing Workshops, ICPPW 2009, September 22-25, pp. 415-422 (2009)
19. Seo, S., Jang, I., et al.: HPMR: Prefetching and pre-shuffling in shared MapReduce computation environment. In: Cluster Computing and Workshops (2009)
20. Namiki, Y., Kanbe, K., Kobayashi, D., Yokota, H.: An approach of using a parallel B-tree structure, Fat-Btree, in PostgreSQL for distributed retrieval. DBSJ Letters 6(2), 61-64 (2007)
21. Chaiken, R., Jenkins, B., Larson, P.-A., Ramsey, B., Shakib, D., Weaver, S., Zhou, J.: Scope: Easy and efficient parallel processing of massive data sets. In: Proc. of International Conference on Very Large Data Bases (2008)
22. Olston, C., Reed, B., Srivastava, U., Kumar, R., Tomkins, A.: Pig latin: a not-soforeign language for data processing. In: Proc. of SIGMOD (2008)
23. Facebook. Hive. Web page, http://issues.apache.org/jira/browse/HADOOP- 3601
24. Wang, F., Qiu, J., Yang, J., Dong, B., Li, X., Li, Y.: Hadoop high availability through metadata replication. In: CloudDB 2009: Proceeding of the First International Workshop on Cloud Data Management (2009)
25. Thusoo, A., et al.: Hive: A warehousing solution over a Map-Reduce framework. In: Proceedings of the Conference on Very Large Databases, pp. 1626-1629 (2009)
26. Amazon Web Service LLC. Amazon Elastic MapReduce (2009), http://aws.amazon.com/elasticmapreduce/