Design and implementation of task scheduling strategies for massive remote sensing data processing across multiple data centers

Software - Practice and Experience

Volumn 44, Issue 7, 2014, Pages 873-886

  Zhang, Wanfeng ^a,c   Wang, Lizhe ^a,b   Ma, Yan ^a   Liu, Dingsheng ^a  

^a INSTITUTE OF GEOLOGY AND GEOPHYSICS   (China)

^b CHINA UNIVERSITY OF GEOSCIENCES   (China)

^c UNIVERSITY OF CHINESE ACADEMY OF SCIENCES   (China)

Author keywords

data intensive computing; multi datacenter infrastructure; task scheduling; big data computing

Indexed keywords

BIG DATA; COMPUTER AIDED SOFTWARE ENGINEERING; DISTRIBUTED COMPUTER SYSTEMS; MULTITASKING; REMOTE SENSING; SCHEDULING; SCHEDULING ALGORITHMS;

DATA-INTENSIVE APPLICATION; DATA-INTENSIVE COMPUTING; DESIGN AND IMPLEMENTATIONS; DIRECTED ACYCLIC GRAPH (DAG); DISTRIBUTED COMPUTING ENVIRONMENT; MASSIVE REMOTE SENSING DATUM; MULTI-DATACENTER INFRASTRUCTURE; TASK-SCHEDULING;

TREES (MATHEMATICS);

EID: 84902532584     PISSN: 00380644     EISSN: 1097024X     Source Type: Journal    
DOI: 10.1002/spe.2229     Document Type: Article

References (37)

1. Ranjan R, Mitra K, Georgakopoulos D,. MediaWise cloud content orchestrator. Journal of Internet Services and Applications 2013; 4 (1): 1-14.
2. Ranjan R, Harwood A, Buyya R,. Coordinated load management in Peer-to-Peer coupled federated grid systems. The Journal of Supercomputing 2012; 61 (2): 292-316.
3. Wang L, Tao J, Ranjan R,. G-Hadoop: MapReduce across distributed data centers for data-intensive computing. Future Generation Computer Systems 2013; 29 (3): 739-750.
4. Wang L, Kunze M, Tao J,. Towards building a cloud for scientific applications. Advances in Engineering Software 2011; 42 (9): 714-722.
5. Wang L, Chen D, Huang F,. Virtual workflow system for distributed collaborative scientific applications on Grids. Computers & Electrical Engineering 2011; 37 (3): 300-310.
6. Ranjan R, Buyya R, Harwood A,. A case for cooperative and incentive based coupling of distributed clusters. Proceedings of the 7th IEEE International Conference on Cluster Computing (Cluster 2005), IEEE Computer Society Press: Boston, Massachusetts, USA, 2005; 1-11.
7. Rahman M, Ranjan R, Buyya R, Benattallah B,. A taxonomy and survey on autonomic management of applications in grid computing environments. Concurrency and Computation: Practice and Experience 2011; 23 (16): 1990-2019. DOI: 10.1002/cpe.1734.
8. Ranjan R, Harwood A, Buyya R,. Grid federation: An economy-based distributed resource management system for large-scale resource coupling. Grid Computing and Distributed Systems Laboratory, University of Melbourne, Australia, Technical Report, GRIDS-TR-2004-10, 2004.
9. Menzel M, Ranjan R,. CloudGenius: Decision support for web service cloud migration. Main Sceintific Track, International ACM Conference on World Wide Web (WWW 2012), Lyon, France, ACM Press, 2012.
10. Ranjan R, Harwood A, Buyya R,. A SLA-based coordinated super scheduling scheme and performance for computational grids. Proceedings of the 8th IEEE International Conference on Cluster Computing (Cluster 2006), Barcelona, Spain, IEEE Computer Society Press, 2006; 1-8.
11. Khanna G, Vydyanathan N, Kurc T, Catalyurek U, Wyckoff P, Saltz J, Sadayappan P,. A hypergraph partitioning based approach for scheduling of tasks with batch-shared I/O. Proceedings of the Fifth IEEE International Symposium on Cluster Computing and the Grid (CCGrid'05), 2005.
12. Catalyurek UV, Aykanat C,. Hypergraph-partitioning-based decomposition for parallel sparse-matrix vector multiplication. Parallel and Distributed Systems, IEEE Transactions on 1999; 10 (7): 673-693. (Pubitemid 30500688)
13. Sulistio A, Poduval G, Buyya R, Tham C-K,. Constructing a grid simulation with differentiated network service using GridSim. 6th International Conference on Internet Computing (ICOMP 2005), 2005.
14. Murshed M, Buyya R,. Using the GridSim toolkit for enabling grid computing education. International Conference on Communication Networks And Distributed Systems Modeling and Simulation (CNDS 2002). San Antonio, Texas, USA, 2002.
15. Sarkar V,. Determining average program execution times and their variance. ACM SIGPLAN Notices. ACM, 1989.
16. Aho AV, Lam M, Sethi R, Ullman J,. Compilers: principles, techniques, and tools. Vol. 1009. Pearson/Addison Wesley, 2007.
17. Ferrante J, Ottenstein KJ, Warren JD,. The program dependence graph and its use in optimization. ACM Transactions on Programming Languages and Systems (TOPLAS) 1987; 9 (3): 319-349. (Pubitemid 17641083)
18. Kosar T, Balman M,. A new paradigm: Data-aware scheduling in grid computing. Future Generation Computer Systems 2009; 25 (4): 406-413.
19. Balman M, Suslu I, Kosar T,. Distributed data management with PetaShare. Proceedings of the 15th ACM Mardi Gras conference: From lightweight mash-ups to lambda grids: Understanding the spectrum of distributed computing requirements, applications, tools, infrastructures, interoperability, and the incremental adoption of key capabilities. ACM, 2008.
20. Lee YC, Zomaya AY,. Data sharing pattern aware scheduling on grids. Proceedings of the International Conference on Parallel Processing (ICPP), Columbus, Ohio, 2006; 365-372 (Pubitemid 47159063)
21. Casanova H, Legrand A, Zagorodnov D, Berman F,. Heuristics for scheduling parameter sweep applications in grid environments. Proceedings of the 9th Heterogeneous Computing Workshop (HCW '00), 2000; 349-363. (Pubitemid 30592800)
22. Giersch A, Robert Y, Vivien F,. Scheduling tasks sharing files on heterogeneous master-slave platforms. Journal of Systems Architecture 2006; 52 (2): 88-104. (Pubitemid 43098940)
23. Hagerup T,. Allocating independent tasks to parallel processors: an experimental study. Journal of Parallel and Distributed Computing 1997; 47 (2): 185-197. (Pubitemid 127174400)
24. Heymann E, Senar MA, Luque E, Livny M,. Adaptive scheduling for master-worker applications on the computational grid. Grid Computing-GRID 2000, Springer-Verlag LNCS, 2000; 214-227.
25. da Silva F, Carvalho S, Hruschka E,. A scheduling algorithm for running bag-of-tasks data mining applications on the grid. Euro-Par 2004: Parallel Processing, Springer, 2004; 254-262.
26. Song W, Yue S, Wang L, Zhang W, Liu D,. Task scheduling of massive spatial data processing across distributed data centers: What's new? Proceedings of the 2011 IEEE 17th International Conference on Parallel and Distributed System (ICPADS '11), 2011; 976-981.
27. Wang L, Chen D, Zhao J,. Resource management of distributed virtual machines. International Journal of Ad Hoc and Ubiquitous Computing 2012; 10 (2): 96-111.
28. Kaya K, Aykanat C,. Iterative-improvement-based heuristics for adaptive scheduling of tasks sharing files on heterogeneous master-slave environments. Parallel and Distributed Systems, IEEE Transactions on 2006; 17 (8): 883-896. (Pubitemid 44070155)
29. Zhang W, Wang L, Liu D, Song W, Ma Y, Liu P, Chen D,. Towards building a multi-datacenter infrastructure for massive remote sensing image processing. Concurrency and Computation: Practice and Experience, 2013; 25: 1798-1812. DOI: 10.1002/cpe.2966.
30. Chen D, Wang L, Ouyang G,. Massively parallel neural signal processing on a many-core platform. Computing in Science and Engineering 2011; 13 (6): 42-51.
31. Dong F, Akl SG,. Scheduling algorithms for grid computing: State of the art and open problems. School of Computing, Queen's University, Kingston, Ontario, 2006.
32. Chaves CG, Batista DM, da Fonseca NL,. Scheduling grid applications on clouds. Global Telecommunications Conference (GLOBECOM 2010), IEEE, 2010.
33. Alhusaini AH, Prasanna VK, Raghavendra CS,. A unified resource scheduling framework for heterogeneous computing environments. 8th Heterogeneous Computing Workshop (HCW'99), IEEE, 1999.
34. Topcuoglu H, Hariri S, Wu M-Y,. Performance-effective and low-complexity task scheduling for heterogeneous computing. IEEE Transactions on Parallel and Distributed Systems 2002; 13 (3): 260-274. (Pubitemid 34448780)
35. Ali S, Siegel HJ, Maheswaran M,. Task execution time modeling for heterogeneous computing systems. 9th IEEE Heterogeneous Computing Workshop (HCW 2000), 2000; 185-199. (Pubitemid 30592814)
36. Silberstein M, Maruyama N,. An exact algorithm for energy-efficient acceleration of task trees on CPU/GPU architectures. Proceedings of the 4th Annual International Conference on Systems and Storage, 2011.
37. Buyya R, Murshed M,. Gridsim: A toolkit for the modeling and simulation of distributed resource management and scheduling for grid computing. Concurrency and Computation: Practice and Experience 2003; 14 (13-15): 1175-1220.