Analyzing massive astrophysical datasets: Can Pig/Hadoop or a relational DBMS help?

Proceedings - IEEE International Conference on Cluster Computing, ICCC

Volumn , Issue , 2009, Pages

  Loebman, Sarah ^a   Nunley, Dylan ^a   Kwon, Yong Chul ^a   Howe, Bill ^a   Balazinska, Magdalena ^a   Gardner, Jeffrey P ^a  

^a UNIVERSITY OF WASHINGTON   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ASTROPHYSICAL SIMULATIONS; DATA PROCESSING SYSTEMS; DATA SETS; PARALLEL DATABASE; RELATIONAL DBMS; SCIENTIFIC APPLICATIONS; SCIENTIFIC DISCOVERY;

ASTROPHYSICS; CLUSTER COMPUTING; COMPUTER SCIENCE; DATA PROCESSING; HAND TOOLS; MANAGEMENT INFORMATION SYSTEMS; PARALLEL PROCESSING SYSTEMS; TECHNICAL PRESENTATIONS;

DATABASE SYSTEMS;

EID: 72049095583     PISSN: 15525244     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CLUSTR.2009.5289149     Document Type: Conference Paper

References (50)

1. "Oracle," http://www.oracle.com/database.
2. "Db2," http://www.ibm.com/db2/.
3. "Teradata," http://www.teradata.com/.
4. "Greenplum database," http://www.greenplum.com/.
5. J. Dean and S. Ghemawat, "MapReduce: simplified data processing on large clusters," in Proc. of the 6th USENIX Symposium on Operating Systems Design & Implementation (OSDI), 2004.
6. "Hadoop," http://hadoop.apache.org/.
7. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly, "Dryad: Distributed data-parallel programs from sequential building blocks," in Proc. of the European Conference on Computer Systems (EuroSys), 2007, pp. 59-72.
8. ISO/IEC 9075-*:2003, Database Languages - SQL. ISO, Geneva, Switzerland.
9. Y. Yu, M. Isard, D. Fetterly, M. Budiu, U. Erlingsson, P. K. Gunda, and J. Currey, "DryadLINQ: A system for general-purpose distributed data-parallel computing using a high-level language," 2008.
10. C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins, "Pig latin: a not-so-foreign language for data processing," in Proc. of the SIGMOD Conf., 2008, pp. 1099-1110.
11. R. Pike, S. Dorward, R. Griesemer, and S. Quinlan, "Interpreting the data: Parallel analysis with Sawzall," Scientific Programming, vol.13, no.4, 2005.
12. M. Stonebraker, J. Becla, D. DeWitt, K.-T. Lim, D. Maier, O. Ratzesberger, and S. Zdonik, "Requirements for science data bases and SciDB," in Fourth CIDR Conf. Perspectives, 2009.
13. "SDSS SkyServer DR7." [Online]. Available: http://skyserver. sdss.org
14. V. Singh, J. Gray, A. Thakar, A. S. Szalay, J. Raddick, B. Boroski, S. Lebedeva, and B. Yanny, "Skyserver traffic report - the first five years," 2007. [Online]. Available: http://www.citebase.org/abstract?id=oai: arXiv.org:cs/0701173
15. J. G. Stadel, "Cosmological N-body simulations and their analysis," Ph.D. dissertation, University of Washington, 2001.
16. "The NSF TeraGrid," http://www.teragrid.org.
17. "TIPSY: A Theoretical Image Processing SYstem," http://hpcc.astro.washington.edu/tools/tipsy/tipsy.html.
18. M. Davis, G. Efstathiou, C. S. Frenk, and S. D. M. White, "The evolution of large-scale structure in a universe dominated by cold dark matter," Astroph J, vol.292, pp. 371-394, May 1985.
19. D. H. Weinberg, L. Hernquist, and N. Katz, "Photoionization, Numerical Resolution, and Galaxy Formation," Astroph. J., vol.477, pp. 8-+, Mar. 1997.
20. "An Overview of SKID," http://www-hpcc.astro.washington.edu/ tools/skid.html.
21. S. R. Knollmann and A. Knebe, "AHF: Amiga's Halo Finder," Astroph. J. Suppl., vol.182, pp. 608-624, June 2009.
22. "IDL - Data Visualization Solutions," http://www.ittvis.com/ ProductServices/IDL.aspx.
23. J. P. Gardner, A. Connolly, and C. McBride, "Enabling rapid development of parallel tree search applications," in Proceedings of the 2007 Symposium on Challenges of Large Applications in Distri buted Environments (CLADE 2007). ACM Press, 2007.
24. --, "Enabling knowledge discovery in a virtual universe," in Proceedings of TeraGrid '07: Broadening Participation in the TeraGrid. ACM Press, 2007.
25. "The TPC-H benchmark," http://www.tpc.org.
26. H. Garcia-Molina, J. D. Ullman, and J. Widom, Database Systems: The complete book, 2nd ed. Prentice Hall, 2009.
27. C. Ballinger, "Born to be parallel: Why parallel origins give Teradata database an enduring performance edge," http://www.teradata.com/t/ page/87083/index.html.
28. "Vertica, inc." http://www.vertica.com/.
29. "IBM zSeries SYSPLEX," http://publib.boulder.ibm.com/ infocenter/\\dzichelp/v2r2/index.jsp?topic=/com.ibm.db2.doc.admin/xf6495.htm.
30. A. Pruscino, "Oracle RAC: Architecture and performance," in Proc. of the SIGMOD Conf., 2003, p. 635.
31. Pavlo A. et. al., "A comparison of approaches to large-scale data analysis," in Proc. of the SIGMOD Conf., 2009.
32. M. Ivanova, M. L. Kersten, N. Nes, and R. Goncalves, "An Architecture for Recycling Intermediates in a Column-store," in Proceedings of the ACM SIGMOD International Conference on Management of Data, Providence, RI, USA, June 2009, accepted for publication.
33. H. Boral, W. Alexander, L. Clay, G. Copeland, S. Danforth, M. Franklin, B. Hart, M. Smith, and P. Valduriez, "Prototyping Bubba, a highly parallel database system," IEEE TKDE, vol.2, no.1, pp. 4-24, 1990.
34. L. Chen, C. Olston, and R. Ramakrishnan, "Parallel evaluation of composite aggregate queries," in Proc. of the 24th International Conference on Data Engineering (ICDE), 2008.
35. A. Deshpande and L. Hellerstein, "Flow algorithms for parallel query optimization," in Proc. of the 24th International Conference on Data Engineering (ICDE), 2008.
36. D. DeWitt and J. Gray, "Parallel database systems: the future of high performance database systems," Communications of the ACM, vol.35, no.6, pp. 85-98, 1992.
37. D. J. Dewitt, S. Ghandeharizadeh, D. A. Schneider, A. Bricker, H. I. Hsiao, and R. Rasmussen, "The Gamma database machine project," IEEE TKDE, vol.2, no.1, pp. 44-62, 1990.
38. G. Graefe, "Encapsulation of parallelism in the Volcano query processing system," SIGMOD Record, vol.19, no.2, pp. 102-111, 1990.
39. Y. Xu, P. Kostamaa, X. Zhou, and L. Chen, "Handling data skew in parallel joins in shared-nothing systems," in Proc. of the SIGMOD Conf., 2008, pp. 1043-1052.
40. H. Yang, A. Dasdan, R.-L. Hsiao, and D. S. Parker, "Map-reduce- merge: simplified relational data processing on large clusters," in Proc. of the SIGMOD Conf., 2007, pp. 1029-1040.
41. D. J. DeWitt, E. Paulson, E. Robinson, J. Naughton, J. Royalty, S. Shankar, and A. Krioukov, "Clustera: an integrated computation and data management system," in Proc. of the 34th International Conference on Very Large DataBases (VLDB), 2008, pp. 28-41.
42. R. Chaiken, B. Jenkins, P.-A. Larson, B. Ramsey, D. Shakib, S. Weaver, and J. Zhou, "Scope: easy and efficient parallel processing of massive data sets," in Proc. of the 34th International Conference on Very Large DataBases (VLDB), 2008, pp. 1265-1276.
43. O. O'Malley and A. C. Murthy, "Winning a 60 second dash with a yellow elephant," http://developer.yahoo.net/blogs/hadoop/Yahoo2009.pdf, 2009.
44. "PigMix," http://wiki.apache.org/pig/PigMix.
45. A. Cary, Z. Sun, V. Hristidis, and N. Rishe, "Experiences on processing spatial data with mapreduce," in Proc. of the 21st SSDBM Conf., 2009.
46. "Sloan Digital Sky Survey," http://cas.sdss.org.
47. M. R. Palankar, A. Iamnitchi, M. Ripeanu, and S. Garfinkel, "Amazon S3 for science grids: a viable solution?" in DADC '08: Proceedings of the 2008 international workshop on Data-aware distributed computing, 2008, pp. 55-64.
48. "Amazon Simple Storage Service (Amazon S3)," http://www.amazon.com/gp/browse.html?node=16427261.
49. "Large Synoptic Survey Telescope," http://www.lsst.org/.
50. "Rasdaman: The intelligent RasterServer," http://www.rasdaman. com/.