Sector and sphere: The design and of a high-performance data cloud

Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences

Volumn 367, Issue 1897, 2009, Pages 2429-2445

  Gu, Yunhong ^a   Grossman, Robert L ^a,b  

^a UNIVERSITY OF ILLINOIS AT CHICAGO   (United States)

^b Open Data Group   (United States)

Author keywords

Cloud computing; Data intensive computing; Distributed computing

Indexed keywords

SPHERES;

CLOUD COMPUTING; DATA CENTRES; DATA-INTENSIVE COMPUTING; DISTRIBUTED COMPUTING; EXPERIMENTAL STUDIES; LARGE DATASETS; OPEN SOURCES; PERFORMANCE DATA; PROGRAMMING MODELS;

COMPUTER SCIENCE;

EID: 67650302322     PISSN: 1364503X     EISSN: None     Source Type: Journal    
DOI: 10.1098/rsta.2009.0053     Document Type: Article

References (19)

1. Babcock, B., Babu, S., Datar, M., Motwani, R. & Widom, J. 2002 Models and issues in data stream systems. In Proc. 21st ACM SIGMOD-SIGACT-SIGART Symp. on Principles of Database Systems, PODS 2002, New York, pp. 1-16. (Pubitemid 35017047)
2. Beynon, M. D., Ferreira, R., Kurc, T., Sussman, A. & Saltz, J. 2000 DataCutter: middleware for filtering very large scientific datasets on archival storage systems. In Mass Storage Systems Conf., College Park, MD, March 2000.
3. Borthaku, D. 2007 The Hadoop distributed file system: architecture and design. See lucene.apache. org/hadoop.
4. Chang, F., Dean, J., Ghemawat, S., Hsieh, W. C.,Wallach, D. A., Burrows, M., Chandra, T., Fikes, A. & Gruber, R. E. 2006 BigTable: a distributed storage system for structured data. In OSDI.06, Seattle, WA, November 2006.
5. Chen, L., Reddy, K. & Agrawal, G. 2004 GATES: a grid-based middleware for processing distributed data streams. In 13th IEEE Int. Symp. on High Performance Distributed Computing (HPDC ) 2004, Honolulu, HI.
6. Dean, J. & Ghemawat, S. 2004 MapReduce: simplified data processing on large clusters. In OSDI.04: 6th Symp. on Operating System Design and Implementation, San Francisco, CA, December 2004.
7. Dilley, J., Maggs, B., Parikh, J., Prokop, H., Sitaraman, R. & Weihl, B. 2002 Globally distributed content delivery. IEEE Internet Comput. 6, 50-58. (doi:10.1109/MIC.2002.1036038)
8. Foster, I. 2005 Globus Toolkit version 4: software for service-oriented systems. In IFIP Int. Conf. on Network and Parallel Computing. Lecture Notes in Computer Science, no. 3779, pp. 2-13. Berlin, Germany: Springer. (Pubitemid 43948113)
9. Gedik, B., Bordawekar, R. & Yu, P. S. 2007 CellSort: high performance sorting on the cell processor. In VLDB, pp. 1286-1297.
10. Ghemawat, S., Gobioff, H. & Leung, S.-T. 2003 The Google file system. In 19th ACM Symp. on Operating Systems Principles, New York, NY.
11. Govindaraju, N. K., Gray, J., Kumar, R. & Manocha, D. 2006 GPUTeraSort: high performance graphics coprocessor sorting for large database management. In ACM SIGMOD, Chicago, IL.
12. Gropp, W., Lusk, E. & Skjellum, A. 1999 Using MPI: portable parallel programming with the message passing interface, 2nd edn. Cambridge, MA: MIT Press.
13. Grossman, R. L. & Gu, Y. 2008 Data mining using high performance clouds: experimental studies using Sector and Sphere. In Proc. 14th ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining (KDD 2008), Las Vegas, NV
14. Gu, Y. & Grossman, R. L. 2007 UDT: UDP-based data transfer for high-speed wide area networks. Comput. Netw. (Elsevier ) 51, 1777-1799. (doi:10.1016/j.comnet.2006.11.009)
15. Gu, Y., Grossman, R. L., Szalay, A. & Thakar, A. 2006 Distributing the Sloan Digital Sky Survey using UDT and Sector. In Proc. e-Science, Nottingham, UK.
16. Kramer,W. T. C., Shoshani, A., Agarwal, D. A., Draney, B. R., Jin, G., Butler, G. F. & Hules, J. A. 2004 Deep scientific computing requires deep data. IBM J. Res. Dev. 48, 209-232. (doi:10.1147/ rd.482.0209)
17. Owens, J. D., Luebke, D., Govindaraju, N., Harris, M., Kruger, J., Lefohn, A. E. & Purcell, T. J. 2005 A survey of general-purpose computation on graphics hardware. In Proc. of Eurographics 2005, Dublin, Ireland, pp. 21-51.
18. Thain, D., Tannenbaum, T. & Livny, M. 2005 Distributed computing in practice: the Condor experience. Concurrency Comput. Pract. Exp. 17, 323-356. (doi:10.1002/cpe.938)
19. Zhao, Y., Hategan, M., Clifford, B., Foster, I., von Laszewski, G., Raicu, I., Stef-Praun, T.&Wilde, M. 2007 Swift: fast, reliable, loosely coupled parallel computation. In IEEE Int.Workshop on Scientific Workflows, Salt Lake City, UT.