Enabling big geoscience data analytics with a cloud-based, mapreduce-enabled and service-oriented workflow framework

PLoS ONE

Volumn 10, Issue 3, 2015, Pages

  Li, Zhenlong ^a   Yang, Chaowei ^a   Jin, Baoxuan ^a,b   Yu, Manzhu ^a   Liu, Kai ^a   Sun, Min ^a   Zhan, Matthew ^a,c  

^a GEORGE MASON UNIVERSITY   (United States)

^b Yunnan Bureau of Geological Survey   (China)

^c University of Texas at Austin   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ANALYTIC METHOD; ARTICLE; CLOUD COMPUTING; COMPUTER INTERFACE; CONCEPTUAL FRAMEWORK; DATA PROCESSING; HBASE; INFORMATION TECHNOLOGY; LEARNING ALGORITHM; LINEAR REGRESSION ANALYSIS; MAPREDUCE; METHODOLOGY; REMOTE SENSING; SERVICE ORIENTED ARCHITECTURE; SIMULATION; WORKFLOW; ALGORITHM; BIOLOGY; GEOLOGY; INTERNET; PROCEDURES;

ALGORITHMS; CLOUD COMPUTING; COMPUTATIONAL BIOLOGY; EARTH SCIENCES; INTERNET; WORKFLOW;

EID: 84924299962     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0116781     Document Type: Article

References (56)

1. Groot R, McLaughlin JD (2000) Geospatial data infrastructure: concepts, cases, and good practice: Oxford: Oxford university press. PMID: 25506959.
2. Schnase JL, Duffy DQ, Tamkin GS, Nadeau D, Thompson JH, et al. (2014) MERRA analytic services: Meeting the big data challenges of climate science through cloud-enabled climate analytics-as-a-service. Computers, Environment and Urban Systems doi:10.1016/j.compenvurbsys.2013.12.003.
3. Yang C, Li W, Xie J, Zhou B (2008) Distributed geospatial information processing: sharing distributed geospatial resources to support Digital Earth. International Journal of Digital Earth 1: 259-278.
4. Yang C, Goodchild M, Huang Q, Nebert D, Raskin R, et al. (2011) Spatial cloud computing: how can the geospatial sciences use and help shape cloud computing? International Journal of Digital Earth 4: 305-329.
5. Minchin S (2014) Big Data: Dealing with the Deluge. Available: http://www.atnf.csiro.au/research/workshops/2013/astroinformatics/talks/StuartMinchin-Astroinformatics.pdf. Accessed 2014 December 27.
6. Cugler DC, Oliver D, Evans MR, Shekhar S, Medeiros CB (2013) Spatial Big Data: Platforms, Analytics, and Science. GeoJournal. (in press).
7. Edwards PN (2010) A vast machine: Computer models, climate data, and the politics of global warming: MIT Press. PMID: 25506974.
8. Grassl H (2011) Climate change challenges. Surveys in Geophysics 32: 319-328.
9. Hodgson JA, Thomas CD, Wintle BA, Moilanen A (2009) Climate change, connectivity and conservation decision making: back to basics. Journal of Applied Ecology 46: 964-969.
10. Murphy JM, Sexton DM, Barnett DN, Jones GS, Webb MJ, et al. (2004) Quantification of modelling uncertainties in a large ensemble of climate change simulations. Nature 430: 768-772. PMID: 15306806.
11. Li Z, Yang C, Sun M, Li J, Xu C, et al. (2013) A high performance web-based system for analyzing and visualizing spatiotemporal data for climate studies. Web and Wireless Geographical Information Systems: Springer Berlin Heidelberg 190-198.
12. Cui D, Wu Y, Zhang Q (2010) Massive spatial data processing model based on cloud computing model. IEEE. pp. 347-350doi:10.1002/pst.472 PMID: 22328327.
13. Liu Y, Guo W, Jiang W, Gong J (2009) Research of remote sensing service based on cloud computing mode. Application Research of Computers 26: 3428-3431.
14. Yang C, Wu H, Huang Q, Li Z, Li J (2011) Using spatial principles to optimize distributed computing for enabling the physical science discoveries. Proceedings of the National Academy of Sciences 108: 5498-5503. doi:10.1073/pnas.0909315108 PMID: 21444779.
15. Li J, Wang FZ, Meng L, Zhang W, Cai Y (2014) A map-reduce-enabled SOLAP cube for large-scale remotely sensed data aggregation. Computers & Geosciences 70: 110-119. doi:10.4103/2231-0770.148499PMID: 25625082.
16. Juve G, Deelman E (2011) Scientific workflows in the cloud. In: Grids. Clouds and Virtualization: Springer 71-91.
17. Mell P, Grance T (2011) The NIST definition of cloud computing. Available: http://faculty.winthrop.edu/domanm/csci411/Handouts/NIST.pdf. Accessed 2014 December 27.
18. Porter JH (2000) Scientific databases. In: Michener W. K., & Brunt J. W. editors. Ecological data: Design, management and processing: 48-69.
19. Wright DJ, Wang S (2011) The emergence of spatial cyberinfrastructure. Proceedings of the National Academy of Sciences, 108(14), 5488-5491. doi:10.1073/pnas.1103051108 PMID: 21467227.
20. Dongarra J (2011) The international exascale software project roadmap. International Journal of High Performance Computing Applications: 1094342010391989.
21. Fiore S, Negro A, Aloisio G (2012) The Climate-G Portal: The context, key features and a multi-dimensional analysis. Future Generation Computer Systems 28: 1-8.
22. Cinquini L, Crichton D, Mattmann C, Bell GM, Drach B, et al. (2012) The Earth System Grid Federation: An open infrastructure for access to distributed geospatial data IEEE. pp. 1-10.
23. Stonebraker M (2010) SQL databases v. NoSQL databases. Communications of the ACM 53: 10-11.
24. Liu Y, Chen B, He W, Fang Y (2013) Massive image data management using HBase and MapReduce. IEEE. pp. 1-5. doi:10.1097/ACC.0b013e31829c6877 PMID: 25611599.
25. Khetrapal A, Ganesh V (2006) HBase and Hypertable for large scale distributed storage systems. Dept of Computer Science, Purdue University. Available: http://cloud.pubs.dbs.uni-leipzig.de/sites/cloud.pubs.dbs.uni-leipzig.de/files/Khetrapal2008HBaseandHypertableforlargescaledistributedstorage.pdf. Accessed 2014 December 27. PMID: 25590126.
26. Lakshman A, Malik P (2010) Cassandra: A decentralized structured storage system. ACM SIGOPS Operating Systems Review 44: 35-40.
27. Chang F, Dean J, Ghemawat S, Hsieh WC, Wallach DA, et al. (2008) Bigtable: A distributed storage system for structured data. ACM Transactions on Computer Systems (TOCS) 26: 4.
28. Chen J, Zheng G, Chen H(2013) ELM-MapReduce: MapReduce accelerated extreme learning machine for big spatial data analysis. IEEE. pp. 400-405. doi:10.1007/s13191-013-0341-z PMID: 25489164.
29. Zhang H, Liu M, Shi Y, Yuen DA, Yan Z, et al. (2007) Toward an automated parallel computing environment for geosciences. Physics of the Earth and Planetary Interiors 163: 2-22.
30. Geist A (1994) PVM: Parallel virtual machine: A users' guide and tutorial for networked parallel computing: MIT press. PMID: 25144107.
31. Gropp W, Lusk E, Skjellum A (1999) Using MPI: portable parallel programming with the messagepassing interface: MIT press. PMID: 25506965.
32. Foster I, Kesselman C, Tuecke S (2001) The anatomy of the grid: Enabling scalable virtual organizations. International journal of high performance computing applications 15: 200-222.
33. Dean J, Ghemawat S (2008) MapReduce: simplified data processing on large clusters. Communications of the ACM 51: 107-113.
34. Rizvandi NB, Boloori AJ, Kamyabpour N, Zomaya AY(2011) MapReduce implementation of prestack Kirchhoff time migration (PKTM) on seismic data. IEEE. pp. 86-91. doi:10.1007/s12262-011-0358-7 PMID: 24426523.
35. Zhao H, Ai S, Lv Z, Li B (2010) Parallel accessing massive NetCDF data based on mapreduce. Web Information Systems and Mining: Springer. pp. 425-431. doi:10.3390/s110100425 PMID: 22346584.
36. Duffy DQ, Schnase JL, Thompson JH, Freeman SM, Clune TL (2012) Preliminary Evaluation of MapReduce for High-Performance Climate Data Analysis. NASA new technology report white paper.
37. Ludäscher B, Altintas I, Berkley C, Higgins D, Jaeger E, et al. (2006) Scientific workflow management and the Kepler system. Concurrency and Computation: Practice and Experience 18: 1039-1065.
38. Wang S (2010) A CyberGIS framework for the synthesis of cyberinfrastructure, GIS, and spatial analysis. Annals of the Association of American Geographers 100: 535-557.
39. Yang C, Raskin R, Goodchild M, Gahegan M (2010) Geospatial cyberinfrastructure: past, present and future. Computers, Environment and Urban Systems 34: 264-277.
40. Deelman E, Gil Y, Zemankova M (2006) NSF workshop on the challenges of scientific workflows 1-2.
41. Gil Y (2008) From data to knowledge to discoveries: Scientific workflows and artificial intelligence. Scientific Programming 16: 4.
42. Taylor IJ, Deelman E, Gannon D, Shields M (2007) Workflows for e-Science: Springer-Verlag London Limited. doi:10.1093/jxb/erm028 PMID: 25506957.
43. Yue P, He L (2009) Geospatial data provenance in cyberinfrastructure. IEEE. pp. 1-4doi:10.1016/j.fas.2009.11.003 PMID: 21278824.
44. Oinn T, Addis M, Ferris J, Marvin D, Senger M, et al. (2004) Taverna: A tool for the composition and enactment of bioinformatics workflows. Bioinformatics 20: 3045-3054. PMID: 15201187.
45. Majithia S, Shields M, Taylor I, Wang I (2004) Triana: A graphical web service composition and execution toolkit IEEE. pp. 514-521.
46. Barga R, Jackson J, Araujo N, Guo D, Gautam N, et al.(2008) The trident scientific workflow workbench. IEEE. pp. 317-318. doi:10.1007/s12265-008-9065-6 PMID: 19956787.
47. Bavoil L, Callahan SP, Crossno PJ, Freire J, Scheidegger CE, et al. (2005) Vistrails: Enabling interactive multiple-view visualizations IEEE. pp. 135-142.
48. Hoffa C, Mehta G, Freeman T, Deelman E, Keahey K, et al. (2008) On the use of cloud computing for scientific workflows IEEE. pp. 640-645.
49. Juve G, Deelman E, Vahi K, Mehta G, Berriman B, et al. (2009) Scientific workflow applications on Amazon EC2 IEEE. pp. 59-66.
50. Simmhan Y, Barga R, van Ingen C, Lazowska E, Szalay A (2009) Building the trident scientific workflow workbench for data management in the cloud IEEE. pp. 41-50.
51. Mattmann CA, Crichton DJ, Hart AF, Goodale C, Hughes JS, et al. (2011) Architecting Data-Intensive Software Systems. Handbook of Data Intensive Computing: Springer. pp. 25-57.
52. Williams DN, Drach R, Ananthakrishnan R, Foster I, Fraser D, et al. (2009) The Earth System Grid: Enabling access to multimodel climate simulation data. Bulletin of the American Meteorological Society 90: 195-205.
53. Li Z, Yang C, Huang Q, Liu K, Sun M, et al. (2014) Building Model as a Service to support geosciences. Computers, Environment and Urban Systems. doi:10.1016/j.compenvurbsys.2014.06.004.
54. Huang Q, Yang C, Liu K, Xia J, Xu C, et al. (2013) Evaluating open-source cloud computing solutions for geosciences. Computers & Geosciences 59: 41-52. doi:10.4103/2231-0770.148499 PMID: 25625082.
55. Yue P, Di L, Yang W, Yu G, Zhao P (2007) Semantics-based automatic composition of geospatial Web service chains. Computers & Geosciences, 33(5), 649-665. doi:10.4103/2231-0770.148499 PMID: 25625082.
56. Li Z. Yang C, Wu H, Li W, Miao L (2011) An optimized framework for seamlessly integrating OGC Web Services to support geospatial sciences. International Journal of Geographical Information Science, 25(4), 595-613.