Using a data grid to automate data preparation pipelines required for regional-scale hydrologic modeling

Environmental Modelling and Software

Volumn 78, Issue , 2016, Pages 31-39

  Billah, Mirza M ^a   Goodall, Jonathan L ^b,c   Narayan, Ujjwal ^d   Essawy, Bakinam T ^b   Lakshmi, Venkat ^e   Rajasekar, Arcot ^f   Moore, Reagan W ^f  

^a Virginia Polytechnic Institute and State University   (United States)

^b University of Virginia   (United States)

^c University of South Carolina   (United States)

^d UNIVERSITY OF MARYLAND   (United States)

^e UNIVERSITY OF SOUTH CAROLINA   (United States)

^f UNIVERSITY OF NORTH CAROLINA   (United States)

Author keywords

Data management; Hydrologic modeling; IRODS; Workflows

Indexed keywords

DATA HANDLING; GRID COMPUTING; METADATA; PIPELINES;

DATA MANAGEMENT SYSTEM; DATA PROCESSING PIPELINES; HETEROGENEOUS DATASETS; HYDROLOGIC CONDITIONS; HYDROLOGIC MODELING; IRODS; VARIABLE INFILTRATION CAPACITY MODELS; WORK-FLOWS;

INFORMATION MANAGEMENT;

EID: 84952682992     PISSN: 13648152     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.envsoft.2015.12.010     Document Type: Article

References (38)

1. Abdulla F.a., Lettenmaier D.P. Application of regional parameter estimation schemes to simulate the water balance of a large continental river. J. Hydrol. 1997, 197:258-285. 10.1016/S0022-1694(96)03263-5.
2. Abdulla F.a., Lettenmaier D.P. Development of regional parameter estimation equations for a macroscale hydrologic model. J. Hydrol. 1997, 197:230-257. 10.1016/S0022-1694(96)03262-3.
3. Abdulla F.a., Lettenmaier D.P., Wood E.F., Smith J.a. Application of a macroscale hydrologic model to estimate the water balance of the Arkansas-Red River Basin. J. Geophys. Res. 1996, 101:7449. 10.1029/95JD02416.
4. Billah M.M., Goodall J.L. Annual and interannual variations in terrestrial water storage during and following a period of drought in South Carolina, USA. J. Hydrol. 2011, 409:472-482. 10.1016/j.jhydrol.2011.08.045.
5. Billah M.M., Goodall J.L., Narayan U., Reager J.T., Lakshmi V., Famiglietti J.S. A methodology for evaluating evapotranspiration estimates at the watershed-scale using GRACE. J. Hydrol. 2015, 523:574-586. 10.1016/j.jhydrol.2015.01.066.
6. Chiang G.-T., Clapham P., Qi G., Sale K., Coates G. Implementing a genomic data management system using iRODS in the wellcome trust Sanger Institute. BMC Bioinforma. 2011, 12:361. 10.1186/1471-2105-12-361.
7. Crow W.T. Multiobjective calibration of land surface model evapotranspiration predictions using streamflow observations and spaceborne surface radiometric temperature retrievals. J. Geophys. Res. 2003, 10.1029/2002JD003292.
8. Fitch P., Perraud J.-M., Cuddy S., Seaton S., Bai Q., Hehir D., Sims J., Merrin L., Ackland R., Herron N. The hydrologists Workbench: more than a scientific workflow tool. Proceedings, Water Information Research and Development Alliance Science Symposium 2011.
9. Foster I. Globus online: accelerating and democratizing science through cloud-based services. IEEE Comput. Soc. 2011, 15:70-73. 10.1109/MIC.2011.64.
10. Furnans J., Olivera F. Watershed topology - the pfafstetter system. Esri User Conf 2001, Vol. 21.
11. Gil Y., Deelman E., Ellisman M., Fahringer T., Fox G., Gannon D., Goble C., Livny M., Moreau L., Myers J. Examining the challenges of scientific workflows. Comput. Long. Beach. Calif. 2007, 40:24-32. 10.1109/MC.2007.421.
12. Goff S.A., Vaughn M., McKay S., Lyons E., Stapleton A.E., Gessler D., Matasci N., Wang L., Hanlon M., Lenards A., Muir A., Merchant N., Lowry S., Mock S., Helmke M., Kubach A., Narro M., Hopkins N., Micklos D., Hilgert U., Gonzales M., Jordan C., Skidmore E., Dooley R., Cazes J., McLay R., Lu Z., Pasternak S., Koesterke L., Piel W.H., Grene R., Noutsos C., Gendler K., Feng X., Tang C., Lent M., Kim S.-J., Kvilekval K., Manjunath B.S., Tannen V., Stamatakis A., Sanderson M., Welch S.M., Cranston K.A., Soltis P., Soltis D., O'Meara B., Ane C., Brutnell T., Kleibenstein D.J., White J.W., Leebens-Mack J., Donoghue M.J., Spalding E.P., Vision T.J., Myers C.R., Lowenthal D., Enquist B.J., Boyle B., Akoglu A., Andrews G., Ram S., Ware D., Stein L., Stanzione D. The iPlant collaborative: cyberinfrastructure for plant biology. Front. Plant Sci. 2011, 2:34. 10.3389/fpls.2011.00034.
13. Goodall J.L., Maidment D.R. A spatiotemporal data model for river basin-scale hydrologic systems. Int. J. Geogr. Inf. Sci. 2009, 23:233-247. 10.1080/13658810802032193.
14. Guru S.M., Kearney M., Fitch P., Peters C. Challenges in using scientific workflow tools in the hydrology domain. 18th World IMACS Congress and MODSIM09 International Congress on Modelling and Simulation. Cairns, Qld 2009, 3514-3520.
15. Hedges M., Blanke T., Hasan A. Rule-based curation and preservation of data: a data grid approach using iRODS. Futur. Gener. Comput. Syst. 2009, 25:446-452. 10.1016/j.future.2008.10.003.
16. Hedges M., Hasan A., Blanke T. Management and preservation of research data with iRODS. Proc. ACM First Work. CyberInfrastructure Inf. Manag. EScience CIMS 07 2007, 17-22. 10.1145/1317353.1317358.
17. Horsburgh J.S., Tarboton D.G., Hooper R.P., Zaslavsky I. Managing a community shared vocabulary for hydrologic observations. Environ. Model. Softw. 2014, 52:62-73. 10.1016/j.envsoft.2013.10.012.
18. Lakshmi V., Piechota T., Narayan U., Tang C. Soil moisture as an indicator of weather extremes. Geophys. Res. Lett. 2004, 31:2-5. 10.1029/2004GL019930.
19. Leonard L., Duffy C.J. Essential terrestrial variable data workflows for distributed water resources modeling. Environ. Model. Softw. 2013, 50:85-96. 10.1016/j.envsoft.2013.09.003.
20. Leonard L., Duffy C.J. Automating data-model workflows at a level 12 HUC scale: watershed modeling in a distributed computing environment. Environ. Model. Softw. 2014, 61:174-190. 10.1016/j.envsoft.2014.07.015.
21. Liang X., Lettenmaier D.P. A simple hydrologically based model of land surface water and energy fluxes for general circulation models. J. Geophys. Res. 1994, 99:14,415-14,428. 10.1029/94JD00483.
22. Liang X., Lettenmaier D.P., Wood E.F. One-dimensional statistical dynamic representation of subgrid spatial variability of precipitation in the two-layer variable infiltration capacity model. J. Geophys. Res. 1996, 101:21403. 10.1029/96JD01448.
23. Liang X., Wood E.F., Lettenmaier D.P. Surface soil moisture parameterization of the VIC-2L model: evaluation and modification. Glob. Planet. Change 1996, 13:195-206. 10.1016/0921-8181(95)00046-1.
24. Lohmann D., Raschke E., Nijssen B., Lettenmaier D.P. Regional scale hydrology: I. Formulation of the VIC-2L model coupled to a routing model. Hydrol. Sci. J. 1998, 43:131-141. 10.1080/02626669809492107.
25. Ludäscher B., Altintas I., Berkley C., Higgins D., Jaeger E., Jones M., Lee E.A., Tao J., Zhao Y. Scientific workflow management and the Kepler system. Concurr. Comput. Pract. Exp. 2006, 18:1039-1065. 10.1002/cpe.994.
26. Maidment D.R. Arc Hydro: GIS for Water Resources 2002, ESRI Press.
27. Miles B., Band L. EcohydroWorkflowLib 2013, verified 05.29.2013. http://pythonhosted.org/ecohydroworkflowlib/index.html.
28. Moore R., Rajasekar A. Reproducible research within the DataNet federation consortium. International Environmental Modelling and Software Society 7th International Congress on Environmental Modelling and Software. San Diego, CA 2014.
29. Moriasi D.N., Arnold J.G., Liew M.W., Van Bingner R.L., Harmel R.D., Veith T.L. M e g s q a w s 2007, 50:885-900.
30. NASA NASA Climate Data Service (NCDS) 2015, [WWW Document]. URL, (accessed 10.16.15). http://cds.nccs.nasa.gov.
31. Oinn T., Greenwood M., Addis M., Alpdemir M.N., Ferris J., Glover K., Goble C., Goderis A., Hull D., Marvin D., Li P., Lord P., Pocock M.R., Senger M., Stevens R., Wipat A., Wroe C. Taverna: lessons in creating a workflow environment for the life sciences. Concurr. Comput. Pract. Exp. 2006, 18:1067-1100. 10.1002/cpe.993.
32. Perraud J., Fitch P.G., Bai Q. Challenges and solutions in implementing hydrological models within scientific workflow software. AGU Fall Meet. Abstr. -1 2010, 06.
33. Piasecki M., Lu B. Development of a hydrologic modeling platform using a workflow engine. AGU Fall Meeting Abstracts 2010, 1239.
34. Rajasekar A., Moore R., Hou C.-Y., Lee C.a., Marciano R., de Torcy A., Wan M., Schroeder W., Chen S.-Y., Gilbert L., Tooby P., Zhu B. iRODS Primer: Integrated Rule-oriented Data System, Synthesis Lectures on Information Concepts, Retrieval, and Services 2010, 10.2200/S00233ED1V01Y200912ICR012.
35. Rajasekar A., Moore R., Wan M., Schroeder W. Policy-based distributed data management systems. JODI J. Digit. Inf. 2010, 11:1-16.
36. Sheffield J., Goteti G., Wen F., Wood E.F. A simulated soil moisture based drought analysis for the United States. J. Geophys. Res. D. Atmos. 2004, 109:1-19. 10.1029/2004JD005182.
37. Sheffield J., Wood E.F. Characteristics of global and regional drought, 1950-2000: analysis of soil moisture data from off-line simulation of the terrestrial hydrologic cycle. J. Geophys. Res. Atmos. 2007, 112:1-21. 10.1029/2006JD008288.
38. Troy T.J., Wood E.F., Sheffield J. An efficient calibration method for continental-scale land surface modeling. Water Resour. Res. 2008, 44:1-13. 10.1029/2007WR006513.