A storage-centric analysis of MapReduce workloads: File popularity, temporal locality and arrival patterns

Proceedings - 2012 IEEE International Symposium on Workload Characterization, IISWC 2012

Volumn , Issue , 2012, Pages 100-109

  Abad, Cristina L ^a,b,c   Roberts, Nathan ^b   Lu, Yi ^a   Campbell, Roy H ^a  

^a UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

^b YAHOO INC   (United States)

^c ESCUELA SUPERIOR POLITÉCNICA DEL LITORAL   (Ecuador)

Author keywords

Access patterns; Big Data; HDFS; MapReduce

Indexed keywords

DATA HANDLING; DIGITAL STORAGE;

ACCESS PATTERNS; ARRIVAL PATTERNS; FILE POPULARITIES; HDFS; MAP-REDUCE; STORAGE LAYERS; STORAGE SYSTEMS; TEMPORAL LOCALITY;

BIG DATA;

EID: 84873453654     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/IISWC.2012.6402909     Document Type: Conference Paper

References (19)

1. J. Dean and S. Ghemawat, "MapReduce: Simplified data processing on large clusters," in Proc. USENIX OSDI, 2004, pp. 137-150.
2. "Apache Hadoop," Jun. 2011, http://hadoop.apache.org.
3. K. Shvachko, H. Kuang, S. Radia, and R. Chansler, "The Hadoop Distributed File System," in MSST2010.
4. C. Abad, H. Luu, N. Roberts, K. Lee, Y. Lu, and R. Campbell, "Metadata traces and workload models for evaluating Big storage systems," in Proc. IEEE UCC, 2012.
5. A. Clauset, C. R. Shalizi, and M. Newman, "Power-law distributions in empirical data," SIAM Rev., vol. 51, no. 4, Nov. 2009.
6. L. Cherkasova and M. Gupta, "Analysis of enterprise media server workloads: Access patterns, locality, content evolution, and rates of change," IEEE/ACM Trans. Netw., vol. 12, no. 5, 2004.
7. G. Ananthanarayanan, S. Agarwal, S. Kandula, A. Greenberg, I. Stoica, D. Harlan, and E. Harris, "Scarlett: Coping with skewed popularity content in MapReduce clusters," in Proc. EuroSys, 2011.
8. B. Fan, W. Tantisiriroj, L. Xiao, and G. Gibson, "DiskReduce: RAID for data-intensive scalable computing," in Proc. PDSW, 2009, pp. 6-10.
9. C. Abad, Y. Lu, and R. Campbell, "DARE: Adaptive data replication for efficient cluster scheduling," in Proc. CLUSTER, 2011.
10. W. E. Leland, M. S. Taqqu, W. Willinger, and D. V. Wilson, "On the self-similar nature of Ethernet traffic," IEEE/ACM Trans. Netw., vol. 2, no. 1, 1994.
11. M. E. Crovella and A. Bestavros, "Self-similarity in World Wide Web traffic: Evidence and possible causes," IEEE/ACM Trans. on Netw., vol. 5, no. 6, 1997.
12. K. Park, G. Kim, and M. Crovella, "On the relationship between file sizes, transport protocols, and self-similar network traffic," in Proc. ICNP, 1996.
13. G. Casale, E. Z. Zhang, and E. Smirni, "Trace data characterization and fitting for Markov modeling," Perform. Eval., vol. 67, no. 2, 2010.
14. L. Breslau, P. Cao, L. Fan, G. Phillips, and S. Shenker, "Web caching and Zipf-like distributions: Evidence and implications," in Proc. INFOCOM, 1999.
15. Y. Chen, K. Srinivasan, G. Goodson, and R. Katz, "Design implications for enterprise storage systems via multi-dimensional trace analysis," in Proc. SOSP, 2011.
16. H. Li and L. Wolters, "Towards a better understanding of workload dynamics on data-intensive clusters and grids," in Proc. IPDPS, 2007.
17. Y. Chen, A. Ganapathi, R. Griffith, and R. Katz, "The case for evaluating MapReduce performance using workload suites," in Proc. MASCOTS, 2011.
18. Y. Chen, S. Alspaugh, and R. Katz, "Interactive query processing in Big Data systems: A cross-industry study of MapReduce workloads," in Proc. VLDB, 2012.
19. S. Patil, G. Gibson, G. Ganger, J. Lopez, M. Polte, W. Tantisiroj, and L. Xiao, "In search of an API for scalable file systems: Under the table or above it?" in Proc. USENIX HotCloud, 2009.