CAMP: Community access MODIS pipeline

Future Generation Computer Systems

Volumn 36, Issue , 2014, Pages 418-429

  Hendrix, Valerie ^a   Ramakrishnan, Lavanya ^a   Ryu, Youngryel ^b   Van Ingen, Catharine ^c   Jackson, Keith R ^a   Agarwal, Deborah ^a  

^a LAWRENCE BERKELEY NATIONAL LABORATORY   (United States)

^b Seoul National University   (South Korea)

^c MICROSOFT RESEARCH   (United States)

Author keywords

Data intensive; High Performance computing; MODIS

Indexed keywords

CLOSED LOOP CONTROL SYSTEMS; DIGITAL STORAGE; PIPELINE PROCESSING SYSTEMS; PIPELINES; RADIOMETERS; SATELLITE IMAGERY;

DATA INTENSIVE; GROSS PHOTOSYNTHESIS; GROUND-BASED OBSERVATIONS; HIGH PERFORMANCE COMPUTING; HIGH PERFORMANCE COMPUTING (HPC); MODERATE RESOLUTION IMAGING SPECTRORADIOMETER; MODIS; TERRA AND AQUA MODIS;

DATA HANDLING;

EID: 84899658184     PISSN: 0167739X     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.future.2013.09.023     Document Type: Article

References (38)

1. NASA MODIS Web Site. URL http://modis.gsfc.nasa.gov.
2. R. Wolfe, D. Roy, and E. Vermote Modis land data storage, gridding, and compositing methodology: level 2 grid IEEE Transactions on Geoscience and Remote Sensing 36 4 1998 1324 1338 10.1109/36.701082 (Pubitemid 128748538)
3. T. Hey, S. Tansley, and K. Tolle The Fourth Paradigm: Data-Intensive Scientific Discovery 2009 Microsoft
4. P.E. Nugent, J.M. Simonson, M. Hagen, R. Miller, B. Sumpter, C. Yang, D. Miller, J. Belak, A. Boehnlein, and E. Dart Data and communications in basic energy sciences: creating a pathway for scientific discovery Report of a Department of Engergy Workshop, June 2011 URL http://science.energy.gov/~/media/ ascr/pdf/research/scidac/ASCR-BES-Data-Report.pdf
5. J. Ekanayake, H. Li, B. Zhang, T. Gunarathne, S.-H. Bae, J. Qiu, and G. Fox Twister: a runtime for iterative mapreduce Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing 2010 ACM 810 818
6. I. Raicu, Y. Zhao, C. Dumitrescu, I. Foster, and M. Wilde Falkon: a fast and light-weight task execution framework Proceedings of the 2007 ACM/IEEE Conference on Supercomputing, 2007. SC'07 2007 IEEE 1 12
7. Z. Fadika, E. Dede, M. Govindaraju, and L. Ramakrishnan Mariane: mapreduce implementation adapted for HPC environments 12th IEEE/ACM International Conference on Grid Computing (GRID), 2011 2011 IEEE 82 89
8. Y. Gu, R.L. Grossman, Y. Gu, and R.L. Grossman Sector and sphere: the design and implementation of a high-performance data cloud Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 367 1897 2009 2429 2445
9. R. Buyya, D. Abramson, and J. Giddy Nimrod/g: an architecture for a resource management and scheduling system in a global computational grid Proceedings. The Fourth International Conference/Exhibition on High Performance Computing in the Asia-Pacific Region, 2000. Vol. 1 2000 IEEE 283 289
10. K.E. Trenberth, L. Smith, T.T. Qian, A.G. Dai, and J. Fasullo Estimates of the global water budget and its annual cycle using observational and model data Journal of Hydrometeorology 8 4 2007 758 769 Go to ISI: //000249354400010 (Pubitemid 47491199)
11. Y. Ryu, D.D. Baldocchi, T.A. Black, M. Detto, B.E. Law, R. Leuning, A. Miyata, M. Reichstein, R. Vargas, C. Ammann, J. Beringer, L.B. Flanagan, L. Gu, L.B. Hutley, J. Kim, H. McCaughey, E.J. Moors, S. Rambal, and T. Vesala On the temporal upscaling of evapotranspiration from instantaneous remote sensing measurements to 8-day mean daily-sums Agricultural and Forest Meteorology 152 2012 212 222 http://www.sciencedirect.com/science/article/pii/S0168192311002863
12. Y. Ryu, D.D. Baldocchi, H. Kobayashi, C. van Ingen, J. Li, T.A. Black, J. Beringer, E. van Gorsel, A. Knohl, B.E. Law, and O. Roupsard Integration of MODIS land and atmosphere products with a coupled-process model to estimate gross primary productivity and evapotranspiration from 1 km to global scales Global Biogeochemical Cycles 25 4 2011 GB4017 10.1029/2011GB004053
13. R.K. Vinukollu, E.F. Wood, C.R. Ferguson, and J.B. Fisher Global estimates of evapotranspiration for climate studies using multi-sensor remote sensing data: evaluation of three process-based approaches Remote Sensing of Environment 115 3 2011 801 823 10.1016/j.rse.2010.11.006 URL http://www.sciencedirect.com/science/article/pii/S0034425710003251
14. NASA MODIS Land Web Site. URL http://modis-land.gsfc.nasa.gov.
15. NASA MODIS Atmosphere Web Site. URL http://modis-atmos.gsfc.nasa.gov.
16. NASA Ocean Color Web Site. URL http://oceancolor.gsfc.nasa.gov.
17. I. Raicu, I.T. Foster, and Y. Zhao Many-task computing for grids and supercomputers Workshop on Many-Task Computing on Grids and Supercomputers, 2008. MTAGS 2008 2008 IEEE 1 11
18. Q. Vanhellemont, B. Nechad, K. Ruddick, Grimas: gridding and archiving of satellite-derived ocean colour data for any region on earth, in: Proceedings of the CoastGIS 2011 conference, 2011.
19. J. Li, M. Humphrey, D.A. Agarwal, K.R. Jackson, C. van Ingen, and Y. Ryu eScience in the cloud: a MODIS satellite data reprojection and reduction pipeline in the Windows Azure platform IPDPS 2010 IEEE 1 10
20. B. Ludscher, I. Altintas, C. Berkley, D. Higgins, E. Jaeger, M. Jones, E. Lee, J. Tao, Y. Zhao, Scientific Workflow Management and the Kepler System, citeseer.ist.psu.edu/ludscher05scientific.html, 2005.
21. T. Oinn, M. Greenwood, M. Addis, M.N. Alpdemir, J. Ferris, K. Glover, C. Goble, A. Goderis, D. Hull, D. Marvin, P. Li, P. Lord, M.R. Pocock, M. Senger, R. Stevens, A. Wipat, and C. Wroe Taverna: lessons in creating a workflow environment for the life sciences: research articles Concurrency and Computation: Practice and Experience 18 10 2006 1067 1100 10.1002/cpe.v18:10 (Pubitemid 44258498)
22. E. Deelman, G. Singh, M. Hui Su, J. Blythe, Y. Gil, C. Kesselman, G. Mehta, K. Vahi, G.B. Berriman, J. Good, A. Laity, J.C. Jacob, and D.S. Katz Pegasus: a framework for mapping complex scientific workflows onto distributed systems Scientific Programming Journal 13 2005 219 237 (Pubitemid 43022937)
23. M. Wilde, M. Hategan, J.M. Wozniak, B. Clifford, D.S. Katz, and I. Foster Swift: a language for distributed parallel scripting Parallel Computing 37 9 2011 633 652 URL http://linkinghub.elsevier.com/retrieve/pii/S0167819111000524
24. J. O'Hara Toward a commodity enterprise middleware Queue 5 4 2007 48 55 (Pubitemid 46928865)
25. R.E. Bryant, Data-intensive supercomputing: the case for disc, 2007.
26. Rabbit MQ. URL http://www.amqp.org/.
27. RabbitMQ Performance Measurements, part 1. URL http://www.rabbitmq.com/ blog/2012/04/17/rabbitmq-performance-measurements-part-1.
28. RabbitMQ Performance Measurements, part 1. URL http://www.rabbitmq.com/ blog/2012/04/25/rabbitmq-performance-measurements-part-2.
29. J. Goecks, A. Nekrutenko, J. Taylor The galaxy team Galaxy: a comprehensive approach for supporting accessible, reproducible, and transparent computational research in the life sciences Genome Biology 11 8 2010 R86 10.1186/gb-2010-11-8-r86
30. I. Altintas, C. Berkley, E. Jaeger, M. Jones, B. Ludäscher, S. Mock, Kepler: An extensible system for design and execution of scientific workflows, in: Proceedings of the 16th International Conference on Scientific and Statistical Database Management, 2004, pp. 423-424.
31. E. Deelman, G. Singh, M. Su, J. Blythe, Y. Gil, C. Kesselman, G. Mehta, K. Vahi, G. Berriman, and J. Good et al. Pegasus: a framework for mapping complex scientific workflows onto distributed systems Scientific Programming 13 3 2005 219 237 (Pubitemid 43022937)
32. M. Wilde, M. Hategan, J.M. Wozniak, B. Clifford, D.S. Katz, and I. Foster Swift: a language for distributed parallel scripting Parallel Computing 37 9 2011 633 652 10.1016/j.parco.2011.05.005
33. T. Oinn et al. Taverna: lessons in creating a workflow environment for the life sciences Concurrency and Computation: Practice and Experience 18 10 2006 1067 1100 (Pubitemid 44258498)
34. I. Taylor, M. Shields, I. Wang, and A. Harrison Visual grid workflow in Triana Journal of Grid Computing 3 34 2005 153 169 (Pubitemid 43280300)
35. HIgh Performance Storage System. URL http://hpss-collaboration.org/.
36. J. Li, Y. Ryu, K. Jackson, D. Agarwal, C. van Ingen, Construction of a modis scientific data reprojection and reduction pipeline in windows azure platform, in: Microsoft Research eScience Workshop, Pittsburg, USA, 2009, microsoft Research eScience Workshop.
37. B. Allen, J. Bresnahan, L. Childers, I. Foster, G. Kandaswamy, R. Kettimuthu, J. Kordas, M. Link, S. Martin, K. Pickett, et al. Globus online: radical simplification of data movement via saas, Preprint CI-PP-5-0611, Computation Institute, The University of Chicago.
38. K. Yelick, S. Coghlan, B. Draney, and R.S. Canon Magellan Final Report, Tech. Rep., US Department of Energy Office of Science Office of Advanced Scientific Computing Research (ASCR) (Dec. 2011) 2011 http://science.energy.gov/ ~/media/ascr/pdf/program-documents/docs/Magellan-Final-Report.pdf