A spatiotemporal indexing approach for efficient processing of big array-based climate data with MapReduce

International Journal of Geographical Information Science

Volumn 31, Issue 1, 2017, Pages 17-35

  Li, Zhenlong ^a,b   Hu, Fei ^a   Schnase, John L ^c   Duffy, Daniel Q ^c   Lee, Tsengdar ^d   Bowen, Michael K ^c   Yang, Chaowei ^a  

^a GEORGE MASON UNIVERSITY   (United States)

^b UNIVERSITY OF SOUTH CAROLINA   (United States)

^c NASA GODDARD SPACE FLIGHT CENTER   (United States)

^d NASA HEADQUARTERS   (United States)

Author keywords

array based; big climate data; climate change; Hadoop MapReduce; HDFS; NASA MERRA; Spatiotemporal index

Indexed keywords

CLIMATE CHANGE; CLIMATE MODELING; DATA PROCESSING; PARALLEL COMPUTING; SPATIOTEMPORAL ANALYSIS;

EID: 84954206400     PISSN: 13658816     EISSN: 13623087     Source Type: Journal    
DOI: 10.1080/13658816.2015.1131830     Document Type: Article

References (27)

1. S.W.Berrick, S.S.Shen, and D.Ostrenga, 2008. Modern era retrospective-analysis for research and applications (MERRA) data and services at the GES DISC. In:AGU fall meeting abstracts, 15–19 December, San Francisco, CA. Vol. 1, 0225.
2. S.Bloom, et al., 2005. Documentation and validation of the Goddard Earth Observing System (GEOS) data assimilation system - version 4. Technical Report Series on Global Modeling and Data Assimilation 104606, v26.
3. J.B.Buck, et al., 2011. Scihadoop:array-based query processing in hadoop. In:Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis. ACM, 66.
4. V.Chandola, A.Banerjee, and V.Kumar, 2009. Anomaly detection:a survey. ACM Computing Surveys (CSUR), 41 (3), 15.
5. M.Das, and S.Parthasarathy, 2009. Anomaly detection and spatio-temporal analysis of global climate system. In:Proceedings of the third international workshop on knowledge discovery from sensor data, 28 June–1 July, Paris. New York:ACM, 142–150.
6. J.Dean, and S.Ghemawat, 2008. MapReduce:simplified data processing on large clusters. Communications of the ACM, 51 (1), 107–113. doi:10.1145/1327452
7. D.Q.Duffy, et al., 2012. Preliminary evaluation of MapReduce for high-performance climate data analysis [online]. NASA new technology report white paper. Available from:http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20120009187.pdf [Accessed 5 January 2016].
8. P.N.Edwards, 2010. A vast machine:Computer models, climate data, and the politics of global warming. Cambridge, MA:MIT Press, 518.
9. A.Eldawy, et al., 2015. SHAHED:a MapReduce-based system for querying and visualizing spatio-temporal satellite data. In:IEEE 31st international conference on data engineering, 13–17 April, Seoul. doi:10.1109/icde.2015.7113427
10. A.Eldawy, and M.F.Mokbel, 2013. A demonstration of spatialhadoop:an efficient mapreduce framework for spatial data. Proceedings of the VLDB Endowment, 6 (12), 1230–1233. doi:10.14778/2536274
11. R.A.Finkel, and J.L.Bentley, 1974. Quad trees a data structure for retrieval on composite keys. Acta Informatica, 4 (1), 1–9. doi:10.1007/BF00288933
12. Y.Geng, et al., 2013. SciHive:array-based query processing with HiveQL. In:12th IEEE international conference on Trust, security and privacy in computing and communications (TrustCom), 16–18 July, Melbourne. IEEE.
13. Y.Geng, X.Huang, and G.Yang, 2014. Adaptive indexing for distributed array processing. Adaptive indexing for distributed array processing. In:IEEE international congress on big data (BigData Congress), 27 June–2 July, Anchorage. IEEE.
14. A.Guttman, 1984. R-trees:a dynamic index structure for spatial searching. In:Proceedings of the 1984 ACM SIGMOD international conference on management of data. New York:ACM, 47–57.
15. Z.Li, et al., 2013. A high performance web-based system for analyzing and visualizing spatiotemporal data for climate studies. In:S. Liang, X. Wang, and C. Claramunt, eds. W2GIS. lecture notes in computer science, Vol. 7820. Berlin:Springer, 190–198.
16. Z.Li, et al., 2015. Enabling big geoscience data analytics with a cloud-based, MapReduce-enabled and service-oriented workflow framework. Plos One, 10 (3), e0116781. doi:10.1371/journal.pone.0116781
17. T.Malik, 2013. GeoBase:indexing NetCDF files for large-scale data analysis. In:W.C. Hu, ed. Big data management, technologies, and applications. H., PAHershey:IGI Global, 295–313.
18. V.Mayer-Schönberger, and K.Cukier, 2013. Big data:a revolution that will transform how we live, work, and think. Boston, MA:Houghton Mifflin Harcourt.
19. J.T.Overpeck, et al., 2011. Climate data challenges in the 21 st century. Science(Washington), 331 (6018), 700–702. doi:10.1126/science.1197869
20. M.M.Rienecker, et al., 2011. MERRA:NASA’s modern-era retrospective analysis for research and applications. Journal of Climate, 24 (14), 3624–3648. doi:10.1175/JCLI-D-11-00015.1
21. J.L.Schnase, et al., 2014. MERRA analytic services:meeting the big data challenges of climate science through cloud-enabled climate analytics-as-a-service. Computers, Environment and Urban Systems. doi:10.1016/j.compenvurbsys.2013.12.003
22. N.Skytland, 2012. Big data:what is NASA doing with big data today? Open.Gov open access article. Available from: http://open.nasa.gov/blog/2012/10/04/what-is-nasa-doing-with-big-data-today/ [Accessed 15March 2015].
23. M.Sun, et al., 2012. A web-based geovisual analytical system for climate studies. Future Internet, 4 (4), 1069–1085. doi:10.3390/fi4041069
24. T.White, 2009. Hadoop:the definitive guide:the definitive guide. Sebastopol, CA:O’Reilly Media, Inc.
25. K.Wu, et al., 2009. FastBit:interactively searching massive data. Journal of Physics:Conference Series, 180 (1), 012053.
26. C.Yang, et al., 2011. Using spatial principles to optimize distributed computing for enabling the physical science discoveries. Proceedings of the National Academy of Sciences, 108 (14), 5498–5503. doi:10.1073/pnas.0909315108
27. H.Zhao, et al., 2010. Parallel accessing massive NetCDF data based on mapreduce. In:F. Wang, ed. Web information systems and mining. Berlin:Springer, 425–431.