A framework for building hypercubes using MapReduce

Computer Physics Communications

Volumn 185, Issue 5, 2014, Pages 1429-1438

  Tapiador, D ^a   O'Mullane, W ^a   Brown, A G A ^b   Luri, X ^c   Huedo, E ^d   Osuna, P ^a  

^a EUROPEAN SPACE AGENCY   (Spain)

^b LEIDEN UNIVERSITY   (Netherlands)

^c INSTITUT D ESTUDIS ESPACIALS DE CATALUNYA IEEC   (Spain)

^d UNIVERSIDAD COMPLUTENSE DE MADRID   (Spain)

Author keywords

Column oriented; Data mining; Framework; Gaia mission; Hadoop; Histogram; Hypercube; MapReduce

Indexed keywords

COLUMN-ORIENTED; FRAMEWORK; GAIA MISSION; HADOOP; HISTOGRAM; HYPERCUBE; MAP-REDUCE;

BENCHMARKING; COMPUTER SIMULATION; DATA MINING; GALAXIES; GEOMETRICAL OPTICS; GRAPHIC METHODS; STATISTICAL METHODS;

GEOMETRY;

EID: 84900587000     PISSN: 00104655     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.cpc.2014.02.010     Document Type: Article

References (18)

1. F. Mignard Overall science goals of the Gaia mission C. Turon, K.S. O'Flaherty, M.A.C. Perryman, ESA SP-576: The Three-Dimensional Universe with Gaia 2005 pp. 5-+
2. Euclid. Mapping the geometry of the dark Universe, Definition Study Report, Tech. rep., European Space Agency/SRE, July 2011
3. Z. Ivezic, J.A. Tyson, LSST: from science drivers to reference design and anticipated data products. ArXiv e-prints arXiv:0805.2366.
4. P. Dewdney, P. Hall, R. Schilizzi, and T. Lazio The square kilometre array Proc. IEEE 97 8 2009 1482 1496
5. J. Dean, and S. Ghemawat MapReduce: simplified data processing on large clusters Commun. ACM 51 1 2008 107 113
6. A. Pavlo, E. Paulson, A. Rasin, D.J. Abadi, D.J. DeWitt, S. Madden, and M. Stonebraker A comparison of approaches to large-scale data analysis Proceedings of the 2009 ACM SIGMOD International Conference on Management of Data SIGMOD'09 2009 ACM New York, NY, USA 165 178
7. J. Ekanayake, S. Pallickara, and G. Fox MapReduce for data intensive scientific analyses Proceedings of the 2008 Fourth IEEE International Conference on eScience ESCIENCE'08 2008 IEEE Computer Society Washington, DC, USA 277 284
8. T. Gunarathne, T.-L. Wu, J. Qiu, G. Fox, MapReduce in the Clouds for science, in: 2010 IEEE Second International Conference on Cloud Computing Technology and Science, CloudCom, 2010, pp. 565-572.
9. S.N. Srirama, O. Batrashev, P. Jakovits, and E. Vainikko Scalability of parallel scientific applications on the Cloud Sci. Program. 19 2-3 2011 91 105
10. A. Floratou, J.M. Patel, E.J. Shekita, and S. Tata Column-oriented storage techniques for MapReduce Proc. VLDB Endow. 4 7 2011 419 429
11. A.F. Gates, O. Natkovich, S. Chopra, P. Kamath, S.M. Narayanamurthy, C. Olston, B. Reed, S. Srinivasan, and U. Srivastava Building a high-level dataflow system on top of Map-Reduce: the Pig experience Proc. VLDB Endow. 2 2 2009 1414 1425 URL http://dl.acm.org/citation.cfm?id=1687553.1687568
12. A. Thusoo, J.S. Sarma, N. Jain, Z. Shao, P. Chakka, S. Anthony, H. Liu, P. Wyckoff, and R. Murthy Hive: a warehousing solution over a Map-Reduce framework Proc. VLDB Endow. 2 2 2009 1626 1629 URL http://dl.acm.org/citation. cfm?id=1687553.1687609
13. A.C. Robin, X. Luri, C. Reylé, Y. Isasi, E. Grux, S. Blanco-Cuaresma, F. Arenou, C. Babusiaux, M. Belcheva, R. Drimmel, C. Jordi, A. Krone-Martins, E. Masana, J.C. Mauduit, F. Mignard, N. Mowlavi, B. Rocca-Volmerange, P. Sartoretti, E. Slezak, and A. Sozzetti Gaia Universe model snapshot A&A 543 2012 A100
14. K.M. Grski, E. Hivon, A.J. Banday, B.D. Wandelt, F.K. Hansen, M. Reinecke, and M. Bartelmann HEALPix: a framework for high-resolution discretization and fast analysis of data distributed on the sphere Astrophys. J. 622 2 2005 759
15. Y. Kwon, M. Balazinska, B. Howe, and J. Rolia A study of skew in MapReduce applications 5th Open Cirrus Summit 2011
16. K. Shvachko, H. Kuang, S. Radia, R. Chansler, The Hadoop distributed file system, in: 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), 2010, pp. 1-10.
17. S. Melnik, A. Gubarev, J.J. Long, G. Romer, S. Shivakumar, M. Tolton, T. Vassilakis, Dremel: interactive analysis of web-scale datasets, in: Proc. of the 36th Int'l Conf on Very Large Data Bases, 2010, pp. 330-339. URL http://www.vldb2010.org/accept.htm.
18. J. Krueger, M. Grund, C. Tinnefeld, H. Plattner, A. Zeier, and F. Faerber Optimizing write performance for read optimized databases Proceedings of the 15th International Conference on Database Systems for Advanced Applications, Volume Part II DASFAA'10 2010 Springer-Verlag Berlin, Heidelberg 291 305