Efficient clustered server-side data analysis workflows using SWAMP

Earth Science Informatics

Volumn 2, Issue 3, 2009, Pages 141-155

  Wang, Daniel L ^a,b   Zender, Charles S ^b   Jenks, Stephen F ^b  

^a SLAC NATIONAL ACCELERATOR LABORATORY   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Data management; Geoscience; Parallel computing; Script compilation

Indexed keywords

DATA MANAGEMENT; RESERVOIR;

EID: 70349254612     PISSN: 18650473     EISSN: 18650481     Source Type: Journal    
DOI: 10.1007/s12145-009-0021-z     Document Type: Article

References (35)

1. Adams JM (2008) Ensemble handling in GrADS and GDS: working with TIGGE data. In: Proceedings of the 7th global organization for earth system science portal (GO-ESSP) workshop, Seattle, 17-19 September 2008
2. Alpern B, Wegman MN, Zadeck FK (1988) Detecting equality of variables in programs. In: POPL '88: proceedings of the 15th ACM SIGPLAN-SIGACT symposium on principles of programming languages, pp 1-11. ACM, New York. doi:10.1145/73560.73561
3. Amdahl GM (1967) Validity of the single processor approach to achieving large scale computing capabilities. AFIPS Conf Proc 30(8):483-485
4. Beynon M, Kurc T, Catalyurek U, Chang C, Sussman A, Saltz J (2001) Distributed processing of very large datasets with datacutter. Parallel Comput 27(11):1457-1478
5. Box D, Ehnebuske D, Kakivaya G, Layman A, Mendelsohn N, Nielsen HF, Thatte S, Winer D (2000) Simple object access protocol (SOAP) 1.1. Tech. rep., W3C. http://www.w3.org/TR/2000/NOTE-SOAP-20000508/
6. Bourne S (1978) An introduction to the UNIX shell. Bell Syst Tech J 57(6):1971-1990
7. Capps SB, Zender CS (2008) Observed and CAM3 GCM sea surface wind speed distributions: characterization, comparison, and bias reduction. J. Climate 21(24):6569
8. Cornillon P (2003) OPeNDAP: accessing data in a distributed, heterogeneous environment. Data Sci J 2:164-174
9. Cui Y, Moore R, Olsen K, Chourasia A, Maechling P, Minster B, Day S, Hu Y, Zhu J, Majumdar A (2007) Enabling very-large scale earthquake simulations on parallel machines. In: Proc intl conf computational science, part I, vol 4487. Springer, Berlin Heidelberg New York, pp 46-53
10. Dagum L, Menon R (1998) OpenMP: an industry standard API for shared-memory programming. IEEE Comput Sci Eng 5(1):46-55 (see also Comput Sci Eng)
11. Deelman E, Singh G, Su MH, Blythe J, Gil Y, Kesselman C, Mehta G, Vahi K, Berriman GB, Good J, Laity A, Jacob JC, Katz DS (2005) Pegasus: a framework for mapping complex scientific workflows onto distributed systems. Sci Prog 13(3):219-238
12. Domenico B, Caron J, Davis E, Kambic R, Nativi S (2002) Thematic real-time environmental distributed data services (THREDDS): incorporating interactive analysis tools into NSDL. J Digit Inf 2(4):2002-2005
13. Doty BE, Kinter III JL (1995) Geophysical data analysis and visualization using GrADS. Visualization techniques in space and atmospheric sciences. In: Szuszczewicz EP, Bredekamp JH (eds) NASA. Washington, D.C., pp 209-219
14. Fettig A (2006) Twisted network programming essentials. O'Reilly Media, Sebastopol
15. Foster I, Voeckler J, Wilde M, Zhao Y (2002) Chimera: a virtual data system for representing, querying, and automating data derivation. In: Proceedings of the 14th conference on scientific and statistical database management, Edinburgh, 24-26 July 2002, pp 37-46
16. Gropp W, Lusk E, Skjellum A (1999) Using MPI: portable parallel programming with the message-passing interface. MIT, Cambridge
17. Hankin S, Harrison DE, Osborne J, Davison J, O'Brien K (1996) A strategy and a tool, ferret, for closely integrated visualization and analysis. J Vis Comput Animat 7(3):149-157
18. Hanselman DC, Littlefield BL (2004) Mastering matlab 7. Prentice Hall, Englewood Cliffs
19. Hsu WW, Smith AJ, Young HC (2005) The automatic improvement of locality in storage systems. ACM Trans Comput Syst 23(4):424-473. doi:10.1145/1113574.1113577
20. Li J, Liao K-W, Choudhary AN, Ross RB, Thakur R, Gropp W, Latham R, Siegel A, Gallagher B, Zingale M (2003) Parallel netcdf: a high-performance scientific i/o interface. In: SC. ACM, New York, p 39
21. Mathis M, Heffner J, Reddy R (2003) Web100: extended tcp instrumentation for research, education and diagnosis. SIGCOMM Comput Commun Rev 33(3):69-79. doi:10.1145/956993.957002
22. Natrajan A, Crowley M, Wilkins-Diehr N, Humphrey MA, Fox AD, Grimshaw AS, Brooks III CL (2004) Studying protein folding on the grid: experiences using charmm on npaci resources under legion. Concurr Comput Pract Exper 16(4):385-397
23. Rew RK, Davis GP (1990) NetCDF: an interface for scientific data access. IEEE Comput Graph Appl 10(4):76-82
24. Rubio-Solar M, Vega-Rodríguez MA, Pérez JMS, Gómez-Iglesias A, Cárdenas-Montes M (2008) A FPGA optimization tool based on a multi-island genetic algorithm distributed over grid environments. In: Cluster computing and the grid, 2008. 8th IEEE international symposium on CCGRID '08, pp 65-72. doi:10.1109/CCGRID.2008.96
25. Schweitzer R, Manke A, Hankin S (2008) Server-side OPeNDAP analysis-concrete steps toward a generalized framework via a reference implementation using F-TDS. In: Proceedings of the 7th global organization for earth system science portal (GO-ESSP) workshop, Seattle, 17-19 September 2008
26. Seagate Technology (2006) Barracuda 7200.9 Serial ATA product manual, rev. d edn
27. Szalay A, Gray J, Thakar A, Kunszt P, Malik T, Raddick J, Stoughton C (2002) The SDSS skyserver: public access to the sloan digital sky server data. In: Proceedings of the 2002 ACM SIGMOD international conference on management of data, Madison, 3-6 June 2002, pp 570-581
28. Tejedor E, Badia RM (2008) COMP superscalar: bringing GRID superscalar and GCM together. In: Cluster computing and the grid, 2008. 8th IEEE International Symposium on CCGRID '08, pp 185-193. doi:10.1109/CCGRID.2008.104
29. Wall L, Christiansen T, Orwant J (2000) Programming perl, 3rd edn. O'Reilly Media, Sebastopol
30. Wang DL, Zender CS, Jenks SF (2007) Server-side parallel data reduction and analysis. In: Advances in grid and pervasive computing: 2nd international conference, GPC 2007, Paris, France, May 2-4, 2007. Proceedings, lecture notes in computer science, vol 4459. Springer, Heidelberg, pp 744-750
31. Wang DL, Zender CS, Jenks SF (2008) Cluster workflow execution of retargeted data analysis scripts. In: Cluster computing and the grid, 2008. 8th IEEE international symposium on CCGRID '08, pp 449-458. IEEE Computer Society, Lyon. doi:10.1109/CCGRID.2008.69
32. Winer D (1999) XML-RPC specification. http://www.xmlrpc.com/spec
33. Xue Y, Wan W, Li Y, Guang J, Bai L, Wang Y, Ai J (2008) Quantitative retrieval of geophysical parameters using satellite data. Computer 41(4):33-40. doi:10.1109/MC.2008.132
34. Zender CS (2008) Analysis of self-describing gridded geoscience data with netCDF operators (NCO). Environ Modell Softw 23(10):1338-1342. doi:10.1016/j.envsoft.2008.03.004
35. Zender CS, Mangalam HJ (2007) Scaling properties of common statistical operators for gridded datasets. Int J High Perform Comput Appl 21(4):458-498. doi:10.1177/1094342007083,802