Data analytics in the cloud with flexible MapReduce workflows

CloudCom 2012 - Proceedings: 2012 4th IEEE International Conference on Cloud Computing Technology and Science

Volumn , Issue , 2012, Pages 427-434

  Goncalves, Carlos ^a,b   Assuncao, Luis ^a,b   Cunha, Jose C ^b  

^a INSTITUTO SUPERIOR DE ENGENHARIA DE LISBOA   (Portugal)

^b UNIVERSIDADE NOVA DE LISBOA   (Portugal)

Author keywords

Cloud; MapReduce; Text Mining; Workflow

Indexed keywords

AMAZON EC2; CLOUD ENVIRONMENTS; COMPLEX APPLICATIONS; COMPLEX WORKFLOWS; DATA ANALYTICS; ELASTIC COMPUTE CLOUDS; END USERS; FLEXIBLE SUPPORTS; LARGE DATASETS; MAP-REDUCE; MAP-REDUCE PROGRAMMING; MAPREDUCE MODELS; RECONFIGURABLE; SCIENTIFIC WORKFLOWS; TEXT MINING; WORK-FLOWS; WORKFLOW;

CLOUD COMPUTING; CLOUDS; COMPUTER SOFTWARE; DATA PROCESSING;

DATA MINING;

EID: 84874267470     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CloudCom.2012.6427527     Document Type: Conference Paper

References (51)

1. AlchemyAPI, [Accessed, (2012). [Online]. Available: http://www. alchemyapi.com
2. Calais, [Accessed, (2012). [Online]. Available: http://www.opencalais.com
3. L. Assuncao, C. Goncalves, and J. C. Cunha, "Autonomic Activities in the Execution of Scientific Workflows: Evaluation of the AWARD Framework," in to appear, Proceedings of the 9th IEEE International Conference on Autonomic and Trusted Computing (ATC 2012), 2012.
4. S. Melnik, A. Gubarev, J. J. Long, G. Romer, S. Shivakumar, M. Tolton, and T. Vassilakis, "Dremel: interactive analysis of webscale datasets," Commun. ACM, vol. 54, no. 6, pp. 114-123, Jun. 2011.
5. J. Dean and S. Ghemawat, "MapReduce: simplified data processing on large clusters," in Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation-Volume 6. Berkeley, CA, USA: USENIX Association, 2004, pp. 10-10.
6. Apache Hadoop, [Accessed, (2012, July)]. [Online]. Available: http://hadoop.apache.org/
7. R. Grossman and Y. Gu, "Data mining using high performance data clouds: experimental studies using sector and sphere," in Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, ser. KDD '08. New York, NY, USA: ACM, 2008, pp. 920-927.
8. Amazon EMR-Amazon Elastic MapReduce, [Accessed, (2012, July)]. [Online]. Available: http://aws.amazon.com/elasticmapreduce/
9. P. Riteau, K. Keahey, and C. Morin, "Bringing Elastic MapReduce to Scientific Clouds," in 3rd Annual Workshop on Cloud Computing and Its Applications: Poster Session, Argonne, Illinois, United States, 2011.
10. T. Gunarathne, T.-L. Wu, J. Qiu, and G. Fox, "MapReduce in the Clouds for Science," in Proceedings of the 2010 IEEE Second International Conference on Cloud Computing Technology and Science, ser. CLOUDCOM '10. Washington, DC, USA: IEEE Computer Society, 2010, pp. 565-572.
11. C. Ranger, R. Raghuraman, A. Penmetsa, G. Bradski, and C. Kozyrakis, "Evaluating MapReduce for multi-core and multiprocessor systems," in In HPCA '07: Proceedings of the 13th International Symposium on High-Performance Computer Architecture. IEEE Computer Society, 2007, pp. 13-24.
12. M. Rafique, B. Rose, A. Butt, and D. Nikolopoulos, "CellMR: A framework for supporting mapreduce on asymmetric cell-based clusters," in Parallel Distributed Processing, 2009. IPDPS 2009. IEEE International Symposium on, May 2009, pp. 1-12.
13. M. de Kruijf and K. Sankaralingam, "MapReduce for the Cell Broadband Engine Architecture," IBM Journal of Research and Development, vol. 53, no. 5, pp. 10:1-10:12, 2009.
14. B. He, W. Fang, Q. Luo, N. K. Govindaraju, and T. Wang, "Mars: a MapReduce framework on graphics processors," in Proceedings of the 17th international conference on Parallel architectures and compilation techniques, ser. PACT '08. New York, NY, USA: ACM, 2008, pp. 260-269.
15. J. Ekanayake, H. Li, B. Zhang, T. Gunarathne, S.-H. Bae, J. Qiu, and G. Fox, "Twister: a runtime for iterative MapReduce," in Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, ser. HPDC '10. New York, NY, USA: ACM, 2010, pp. 810-818.
16. D. Peng and F. Dabek, "Large-scale incremental processing using distributed transactions and notifications," in Proceedings of the 9th USENIX conference on Operating systems design and implementation, ser. OSDI'10. Berkeley, CA, USA: USENIX Association, 2010, pp. 1-15.
17. P. Bhatotia, A. Wieder, R. Rodrigues, U. A. Acar, and R. Pasquin, "Incoop: MapReduce for incremental computations," in Proceedings of the 2nd ACM Symposium on Cloud Computing, ser. SOCC '11. New York, NY, USA: ACM, 2011, pp. 7:1-7:14.
18. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly, "Dryad: distributed data-parallel programs from sequential building blocks," in Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007, ser. EuroSys '07. New York, NY, USA: ACM, 2007, pp. 59-72.
19. A. F. Gates, O. Natkovich, S. Chopra, P. Kamath, S. M. Narayanamurthy, C. Olston, B. Reed, S. Srinivasan, and U. Srivastava, "Building a high-level dataflow system on top of Map-Reduce: the Pig experience," Proc. VLDB Endow., vol. 2, pp. 1414-1425, August 2009.
20. A. Thusoo, J. S. Sarma, N. Jain, Z. Shao, P. Chakka, S. Anthony, H. Liu, P. Wyckoff, and R. Murthy, "Hive: a warehousing solution over a map-reduce framework," Proc. VLDB Endow., vol. 2, pp. 1626-1629, August 2009.
21. A. Abouzeid, K. Bajda-Pawlikowski, D. Abadi, A. Silberschatz, and A. Rasin, "HadoopDB: an architectural hybrid of MapReduce and DBMS technologies for analytical workloads," Proc. VLDB Endow., vol. 2, pp. 922-933, August 2009.
22. H.-c. Yang, A. Dasdan, R.-L. Hsiao, and D. S. Parker, "Map-reducemerge: simplified relational data processing on large clusters," in Proceedings of the 2007 ACM SIGMOD international conference on Management of data, ser. SIGMOD '07. New York, NY, USA: ACM, 2007, pp. 1029-1040.
23. I. Taylor, E. Deelman, D. Gannon, and M. Shields. Workflows for EScience: Scientific Workflows for Grids. London,: Springer, 2007.
24. A. Thusoo, Z. Shao, S. Anthony, D. Borthakur, N. Jain, J. Sen Sarma, R. Murthy, and H. Liu, "Data warehousing and analytics infrastructure at facebook," in Proceedings of the 2010 ACM SIGMOD International Conference on Management of data, ser. SIGMOD '10. New York, NY, USA: ACM, 2010, pp. 1013-1020.
25. Taverna Workflow Management System, [Accessed, (2012, July)]. [Online]. Available: http://www.taverna.org.uk/
26. E. Deelman, G. Singh, M.-H. Su, J. Blythe, Y. Gil, C. Kesselman, G. Mehta, K. Vahi, G. B. Berriman, J. Good, A. Laity, J. C. Jacob, and D. S. Katz, "Pegasus: A framework for mapping complex scientific workflows onto distributed systems," Sci. Program., vol. 13, no. 3, pp. 219-237, Jul. 2005.
27. Kepler project, Kepler User Manual, August 2011. [Online]. Available: https://kepler-project.org/
28. B. Linke, R. Giegerich, and A. Goesmann, "Conveyor," Bioinformatics, vol. 27, no. 7, pp. 903-911, Apr. 2011.
29. A. Thrasher, R. Carmichael, P. Bui, L. Yu, D. Thain, and S. Emrich, "Taming complex bioinformatics workflows with weaver, makeflow, and starch," in Workflows in Support of Large-Scale Science (WORKS), 2010 5th Workshop on, nov. 2010, pp. 1-6.
30. Y. Zhao, I. Raicu, I. T. Foster, M. Hategan, V. Nefedova, and M. Wilde, "Realizing Fast, Scalable and Reliable Scientific Computations in Grid Environments," CoRR, vol. abs/0808.3548, 2008.
31. A. Alexandrov, M. Heimel, V. Markl, D. Battré, F. Hueske, E. Nijkamp, S. Ewen, O. Kao, and D. Warneke, "Massively parallel data analysis with PACTs on Nephele," Proc. VLDB Endow., vol. 3, no. 1-2, pp. 1625-1628, Sep. 2010.
32. L. Hodgkinson, J. Rosa, and E. A. Brewer, "Parallel Software Architecture for Experimental Workflows in Computational Biology on Clouds," in PPAM (2), ser. Lecture Notes in Computer Science, R. Wyrzykowski, J. Dongarra, K. Karczewski, and J. Wasniewski, Eds., vol. 7204. Springer, 2011, pp. 281-291.
33. Hamake, [Accessed, (2012, September)]. [Online]. Available: http://code.google.com/p/hamake/
34. Apache Oozie Workflow Scheduler for Hadoop, [Accessed, (2012, July)]. [Online]. Available: http://incubator.apache.org/oozie/
35. Azkaban, [Accessed, (2012, September)]. [Online]. Available: http://data.linkedin.com/opensource/azkaban
36. Cascading, [Accessed, (2012, September)]. [Online]. Available: http://www.cascading.org
37. J. Wang, D. Crawl, and I. Altintas, "Kepler + Hadoop: a general architecture facilitating data-intensive applications in scientific workflow systems," in Proceedings of the 4th Workshop on Workflows in Support of Large-Scale Science, ser. WORKS '09. New York, NY, USA: ACM, 2009, pp. 12:1-12:8.
38. S. Valvag, D. Johansen, and A. Kvalnes, "Cogset vs. Hadoop: Measurements and Analysis," in Cloud Computing Technology and Science (CloudCom), 2010 IEEE Second International Conference on, 30 2010-dec. 3 2010, pp. 768-775.
39. Z. Vrba, P. Halvorsen, C. Griwodz, and P. Beskow, "Kahn Process Networks are a Flexible Alternative to MapReduce," in Proceedings of the 2009 11th IEEE International Conference on High Performance Computing and Communications, ser. HPCC '09. Washington, DC, USA: IEEE Computer Society, 2009, pp. 154-162.
40. Z. Vrba, P. Halvorsen, C. Griwodz, P. Beskow, and D. Johansen, "The Nornir Run-time System for Parallel Programs Using Kahn Process Networks," in Proceedings of the 2009 Sixth IFIP International Conference on Network and Parallel Computing, ser. NPC '09. Washington, DC, USA: IEEE Computer Society, 2009, pp. 1-8.
41. J. F. d. Silva, G. Dias, S. Guilloré, and J. G. P. Lopes, "Using LocalMaxs Algorithm for the Extraction of Contiguous and Noncontiguous Multiword Lexical Units," in Proceedings of the 9th Portuguese Conference on Artificial Intelligence: Progress in Artificial Intelligence, ser. EPIA '99. Springer-Verlag, 1999, pp. 113-132.
42. G. Kahn, "The semantics of a simple language for parallel programming," in Information processing, J. L. Rosenfeld, Ed., 1974, pp. 471-475.
43. T. M. Parks, "Bounded Scheduling of Process Networks," Ph.D. dissertation, University of California at Berkeley, 1995.
44. T. J. Lehman, A. Cozzi, Y. Xiong, J. Gottschalk, V. Vasudevan, S. Landis, P. Davis, B. Khavar, and P. Bowman, "Hitting the distributed computing sweet spot with TSpaces," Computer Networks, vol. 35, no. 4, pp. 457-472, 2001.
45. Hadoop Distributed File System, [Accessed, (2012, July)]. [Online]. Available: http://hadoop.apache.org/hdfs/
46. Amazon S3-Amazon Simple Storage Service, [Accessed, (2012, July)]. [Online]. Available: http://aws.amazon.com/s3/
47. N. Carriero and D. Gelernter, "Linda in context," Communication of the ACM, vol. 32, no. 4, pp. 444-458, 1989.
48. J. Schad, J. Dittrich, and J.-A. Quiané-Ruiz, "Runtime measurements in the cloud: observing, analyzing, and reducing variance," Proc. VLDB Endow., vol. 3, no. 1-2, pp. 460-471, Sep. 2010.
49. Amazon AWS-Simple Queue Service (SQS), [Accessed, (2012, July)]. [Online]. Available: http://aws.amazon.com/sqs/
50. Amazon AWS-SimpleDB, [Accessed, (2012, July)]. [Online]. Available: http://aws.amazon.com/simpledb/
51. Microsoft Azure-Table Storage service, [Accessed, (2012, July)]. [Online]. Available: http://www.windowsazure.com/enus/home/features/data- management/