MERRA Analytic Services: Meeting the Big Data challenges of climate science through cloud-enabled Climate Analytics-as-a-Service

Computers, Environment and Urban Systems

Volumn 61, Issue , 2017, Pages 198-211

  Schnase, John L ^a   Duffy, Daniel Q ^a   Tamkin, Glenn S ^a   Nadeau, Denis ^a   Thompson, John H ^a   Grieg, Cristina M ^b   McInerney, Mark A ^a   Webster, William P ^a  

^a NASA GODDARD SPACE FLIGHT CENTER   (United States)

^b GEORGE MASON UNIVERSITY   (United States)

Author keywords

BAER; Cloud Computing; Data analytics; Data services; ESGF; Generativity; Hadoop; iRODS; MapReduce; MERRA

Indexed keywords

CLIMATE CHANGE; CLIMATE MODELS; CLOUD COMPUTING; DECISION SUPPORT SYSTEMS; INFORMATION MANAGEMENT; NASA; SELF ASSEMBLY; TECHNOLOGY TRANSFER;

BAER; DATA ANALYTICS; DATA SERVICES; ESGF; GENERATIVITY; HADOOP; IRODS; MAP-REDUCE; MERRA;

BIG DATA;

EID: 84892956198     PISSN: 01989715     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.compenvurbsys.2013.12.003     Document Type: Article

References (41)

1. Agrawal, D., Das, S., & Abbadi, A. E. (2011). Big data and cloud computing: current state and future opportunities. In Electronic proceedings of the 14th international conference on extending database technology (EBDT 2011) (4 pp.). < http://www.edbt.org/Proceedings/2011-Uppsala/papers/edbt/a50-agrawal.pdf>.
2. Baker, K.S., Bowker, G.C., Information ecology: Open system environment for data, memories, and knowing. Journal of Intelligent Information Systems: Integrating Artificial Intelligence and Database Technologies 29:1 (2007), 127–144.
3. Berman, F., Got data? A guide to data preservation in the information age. Communications of the ACM 51:12 (2008), 50–56.
4. Buck, J. B., Watkins, N., LeFevre, J., Ioannidou, K., Maltzahn, C., Polyzotis, N., et al. (2011). SciHadoop: Array-based query processing in Hadoop. In Proceedings of 2011 international conference for high performance computing, networking, storage and analysis (SC ‘11) (pp. 1–11). < http://users.soe.ucsc.edu/~alkis/papers/sc11_scihadoop.pdf>.
5. Burned Area Emergency Response (BAER) (2013). < http://www.nifc.gov/BAER>.
6. Christian, C., Lintott, C., Smith, A., Fortson, L., & Bamford, S. (2012). Citizen science: Contributions to astronomy research (12 pp.). Cornell University Library open access archive e-Print arXiv:1202.2577. < http://arxiv.org/abs/arXiv:1202.2577>.
7. Cloudera (2013). < http://cloudera.com>.
8. Dean, J., Ghemawat, S., MapReduce: Simplified data processing on large clusters. Communications of the ACM 51 (2008), 107–113.
9. DeGusta, M. (2012). Are smart phones spreading faster than any technology in human history? MIT Technology Review. < http://www.technologyreview.com/news/427787/are-smart-phones-spreading-faster-than-any-technology-in-human-history/>.
10. Duffy, D. Q., Schnase, J. L., Clune, T. L., Kim, E. J., Freeman, S. M., Thompson, J. H., et al. (2011). Preliminary Evaluation of MapReduce for high-performance climate data analysis. AGU fall meeting abstracts, 2011 AGU fall meeting, San Francisco, December. < http://adsabs.harvard.edu/abs/2011AGUFMIN44A.08>.
11. Duffy, D. Q., Schnase, J. L., Clune, T. L., Kim, E. J., Freeman, S. M., Thompson, J. H., et al. (2012). Preliminary evaluation of MapReduce for high-performance climate data analysis. NASA new technology report white paper (6 pp.). < http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20120009187_2012009164.pdf>.
12. Earth Observing System Distributed Information System (EOSDIS) (2013). < https://earthdata.nasa.gov>.
13. Edwards, P.N., A vast machine: Computer models, climate data, and the politics of global warming (518 pp.). 2010, MIT Press, Cambridge, Mass.
14. Ernst & Young (2011). Cloud computing issues and impacts. Publication in the Ernst & Young global technology industry discussion series. < http://www.ey.com/Publication/vwLUAssets/Cloud_computing_issues,_impacts_and_insights/$File/Cloud%20computing%20issues%20and%20impacts_14Apr11.pdf>.
15. Earth Systm Grid Federation (ESGF) (2013). < http://esgf.org>.
16. Hadoop Distributed File System (HDFS) (2013). < http://hadoop.apache.org>.
17. Intergovernmental Panel on Climate Change (IPCC) (2013). < http://www.ipcc.ch>.
18. Integrated Rule-Oriented Data System (iRODS) (2013). < http://www.irods.org>.
19. Kundra, V. (2011). Federal cloud computing strategy. White House publication of the U.S. Chief Information Officer (43 pp.). < http://www.whitehouse.gov/sites/default/files/omb/assets/egov_docs/federal-cloud-computing-strategy.pdf>.
20. Little, M. & Petraska, K. (2013). NASA science and cloud computing capabilities. In Presentation at the 2013 summer meeting of the federation of earth science information partners (ESIP 2013), July 9–12, 2013, Chapel Hill, NC. < http://esipfed.org/data-and-informatics>.
21. Mell, P. & Grace, T. (2011). The NIST definition of cloud computing. National Institutes of Standards (NIST) special publication 800-145. < http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdf>.
22. Modern-Era Retrospective Analysis for Research and Applications (MERRA) (2013). < http://gmao.gsfc.nasa.gov/merra/>.
23. NASA Center for Climate Simulation (NCCS) (2013). < http://www.nccs.nasa>.
24. Open Archive Information System (OAIS) Reference Model. < http://public.ccsds.org/sites/cwe/rids/Lists/CCSDS%206500P11/Attachments/650x0p11.pdf>.
25. Rienecker, M.M., Suarez, M.J., Gelaro, R., Todling, R., Bacmeister, J., Liu, E., et al. MERRA: NASA's modern-era retrospective analysis for research and applications, Journal of Climate, 2011, 24(14), 3624-3648. < http://dx.doi.org/10.1175/JCLI-D-11-00015.1>.
26. Saltzer, J.H., Reed, D.P., Clark, D.D., End-to-end arguments in system design. ACM Transactions on Computer Systems 2:4 (1984), 277–288.
27. Schnase, J. L., Webster, W. P., Parnell, L. A., & Duffy, D. Q. (2011). The NASA Center for Climate Simulation data management system: Toward an iRODS-based approach to scientific data services. In IEEE Xplore digital library proceedings of the 27th IEEE symposium on massive storage systems and technologies (MSST 2011), May 26–27, 2011, Denver, CO (6 pp.).
28. Schnase, J. L., Tamkin, G., Fladung, D., Sinno, S., & Gill, R. (2011). Federated observational and simulation data in the NASA Center for Climate Simulation data management system project. In Proceedings of the iRODS user group meeting 2011: Sustainable policy-based data management, sharing, and preservation, February 17–18, 2011, University of North Carolina, Chapel Hill, NC (14 pp.). < http://iren-web.renci.org/irods-meeting/nasa.pdf>.
29. Schnase, J. L., Duffy, D. Q., Tamkin, G. S., Strong, S., Ripley, D., Gill, R., et al. (2011). iRODS-based climate data services and virtualization-as-a-service in the NASA Center for Climate Simulation. American Geophysical Union (AGU) fall meeting poster, December 5–9, 2011, San Francisco, CA.
30. Schnase, J. L., Tamkin, G. S., Ripley, W. D., III, Strong, S., Gill, R., & Duffy, D. Q. (2012). The Virtual Climate Data Server (vCDS): An iRODS-based data management software appliance supporting climate data services and virtualization-as-a-service in the NASA Center for Climate Simulation. In Proceedings of the iRODS user group meeting 2012: Policy-based data management for managing community-driven collection life cycles, March 1–2, 2012, University of Arizona, Tempe, AZ (10 pp.).
31. Skytland, N. (2012). Big Data: What is NASA doing with Big Data today? Open.Gov open access article. < http://open.nasa.gov/blog/2012/10/04/what-is-nasa-doing-with-big-data-today/>.
32. Snijders, C., Matzat, U., Reips, U.D., Big data: Big gaps of knowledge in the field of Internet science. International Journal of Internet Science 7 (2012), 1–5.
33. Szalay, A.S., Kunszt, P.Z., Ani Thakar, A., Gray, J., Slutz, D., Brunner, R.J., Designing and mining multi-terabyte astronomy archives: The Sloan digital sky survey. Proceedings of the 2000 ACM SIGMOD international conference on Management of data (SIGMOD ‘00), 2000, ACM, New York, NY, USA, 451–462.
34. Tamkin, G. S. (2013). Hadoop for high-performance climate analytics. Hadoop summit, Palo Alto, July. < http://www.youtube.com/watch?v=1UMBiVlsPp0>.
35. Taylor, K. E., Balaji, V., Hankin, S., Juckes, M., Lawrence, B., & Pascoe, S. (2012). CMIP5 Data Reference Syntax (DRS) and controlled vocabularies (16 pp.). < http://cmip-pcmdi.llnl.gov/cmip5/docs/cmip5_data_reference_syntax.pdf>.
36. VanRoekel, S. (2013). Federal information technology FY 2013 budget priorities. Publication of the White House U.S. Chief Information Officer (16 pp.). < https://cio.gov/wp-content/uploads/2012/09/FY2013-IT1.pdf>.
37. Webster, P. (2013). Supercomputing the climate: NASA Big data mission. CSC World. Computer Sciences Corporation. < http://www.csc.com/cscworld/publications/81769/81773-supercomputing_the_climate_nasa_s_big_data_mission>.
38. Wei, J., Dirmeyer, P.A., Wisser, D., Bosilovich, M.G., Mocko, D.M., Where does irrigation water go? An estimate of the contribution of irrigation to precipitation using MERRA. Journal of Hydrometeorology 14:2 (2013), 271–289.
39. West, D. M. (2010). Saving money through cloud computing. A governance studies at Brookings Publication, Brookings Institution (14 pp.). < http://www.brookings.edu/research/papers/2010/04/07-cloud-computing-west>.
40. Young, J. R. (2010). Crowd science reaches new heights. Chronicle of Higher Education. < http://chronicle.com/article/The-Rise-of-Crowd-Science/65707/>.
41. Zittrain, J., The future of the Internet—And how to stop it (342 pp.). 2008, Yale University Press, New Haven.