Dataflow detection and applications to workflow scheduling

Concurrency and Computation: Practice and Experience

Volumn 23, Issue 11, 2011, Pages 1261-1283

  Wang, Yang ^a,b   Lu, Paul ^a  

^a UNIVERSITY OF ALBERTA   (Canada)

^b NATIONAL UNIVERSITY OF SINGAPORE   (Singapore)

Author keywords

concurrency; dataflow; storage

Indexed keywords

COMMERCE; CONCURRENCY CONTROL; ECONOMIC AND SOCIAL EFFECTS; ENERGY STORAGE; FILE ORGANIZATION;

CONCURRENCY; DATA-FLOW INFORMATION; DATAFLOW; HIGH PERFORMANCE COMPUTING (HPC); SCHEDULING FLEXIBILITY; SCHEDULING POLICIES; SELECTIVE ISOLATION; WORKFLOW SCHEDULING;

SCHEDULING;

EID: 79960116799     PISSN: 15320626     EISSN: 15320634     Source Type: Journal    
DOI: 10.1002/cpe.1708     Document Type: Article

References (56)

1. Werner T,. Target gene identification from expression array data by promoter analysis. Biomolecular Engineering 2001; 17: 87-94. (Pubitemid 32140951)
2. Szafron D, Lu P, Greiner R, Wishart DS, Poulin B, Eisner R, Lu Z, Anvik J, Macdonell C, Fyshe A, Meeuwis D,. Proteome analyst: Custom predictions with explanations in a web-based tool for high-throughput proteome annotations. Nucleic Acids Research 2004; 32: W365-W371. Available at: [31 March 2011]. (Pubitemid 38997360)
3. GROMACS. Available at: http://www.gromacs.org [31 March 2011 ].
4. Schmidt M, Baldridge K, Boatz J, Elbert S, Gordon M, Jensen J, Koseki S, Matsunaga N, Montgomery J,. The general atomic and molecular electronic structure system. Journal of Computational Chemistry 1993; 14: 1347-1363. Available at: [31 March 2011].
5. Ludascher B, Altintas I, Berkley C, Higgins D, Jaeger E, Jones M, Lee EA, Tao J, Zhao Y,. Scientific workflow management and the Kepler system. Concurrency and Computation: Practice & Experience 2005; 18 (10): 1039-1065. Special Issue on Scientific Workflows.
6. Muchnick SS,. Advanced Compiler Design Implementation. Morgan Kaufmann: Los Altos, CA, 1997.
7. Lian CC, Tang F, Issac P, Krishnan A,. GEL: Grid execution language. Journal of Parallel and Distributed Computing 2005; 65: 857-869. (Pubitemid 40749341)
8. Taylor I, Shields M, Wang I, Rana O,. Triana applications within grid computing and peer to peer environments. Journal of Grid Computing 2004; 1: 199-217.
9. Condor Team, 2004. Available at: http://www.cs.wisc.edu/condor/dagman [31 March 2011].
10. Wang Y,. Transparent dataflow detection and use in workflow scheduling: Concurrency and deadlock avoidance. PhD Thesis, University of Alberta, Canada, 2008.
11. Sima D, Fountain T, Kacsuk P,. Advanced Computer Architectures, A Design Space Approach. Addison-Wesley: Reading, MA, 1997.
12. Hennessy JL, Patterson DA,. Computer Architecture: A Quantitative Approach (3rd edn). Morgan Kaufmann: Los Altos, CA, 2005.
13. Wang Y, Lu P,. On the benefits of a workflow-aware versioning filesystem in metacomputing systems. The Eighth International Conference on High Performance Computing in Asia Pacific Region, Beijing, China, 2005.
14. Wang Y, Lu P,. Using dataflow information to improve inter-workflow instance concurrency. Sixth International Conference on Parallel and Distributed Computing, Applications and Technologies (PDCAT), Dalian, China, 2005.
15. Abbott B et al., Search for gravitational waves from binary inspirals in S3 and S4 LIGO data. Available at:.
16. Wang J, Kuehl H, Sacchi MD,. Least-squares wave-equation avp imaging of 3d common azimuth data. Proceedings of the 73rd Annual International Meeting, Society of Exploration Geophysicists, Dallas, TX, U.S.A., 2003.
17. Blaha P, Schwarz K, Madsen G, Kvasnicka D, Luitz J,. WIEN2k: An augmented plane wave plus local orbitals program for calculating crystal properties. Technical Report, Institute of Physical and Theoretical Chemistry, Vienna University of Technology, 2001.
18. Lake R, Schaeffer J, Lu P,. Solving large retrograde analysis problems using a network of workstations. Advances in Computer Chess 1994; VII: 135-162.
19. Schaeffer J, Lake R,. Solving the game of checkers. In Games of No Chance, Nowakowski RJ, (ed.), vol. 20. Cambridge University Press: Cambridge, 1996.
20. Rosenberg AL,. On scheduling mesh-structured computations for internet-based computing. IEEE Transactions on Computers 2004; 53 (9): 1176-1186.
21. Glatard T, Montagnat J, Pennec X,. Grid-enabled workflows for data intensive medical applications. The 18th IEEE Symposium on Computer-based Medical Systems, Trinity College Dublin, Ireland, 2005; 537-542. (Pubitemid 41542951)
22. Crandall PE, Aydt RA, Chien AA, Reed DA,. Input/output characteristics of scalable parallel applications. Proceedings of the IEEE/ACM Conference on Supercomputing, San Diego, CA, U.S.A., 1995; 59-89.
23. Hulith P,. The amanda experiment. Proceedings of the XVII International Conference on Neutrino Physics and Astrophysics, Helsinki, Finland, June 1996.
24. Sum AK, de Pablo JJ,. Nautilus: Molecular simulation code. Technical Report, Dept. of Chemical Engineering, University of Wisconsin-Madison, 2002.
25. NASA Ames and the Courant Institute at NYU. Cart3D. Available at:. http://people.nas.nasa.gov/aftosmis/cart3d/cart3Dhome.html
26. Zhou S,. LSF: Load sharing in large-scale heterogeneous distributed systems. Proceedings of the Workshop on Cluster Computing, Florida State University, Tallahassee, U.S.A., December 1992.
27. Henderson R, Tweten D,. Portable batch system: External reference specification. NASA Ames Research Center, 1996.
28. Sulistio A, Buyya R,. A time optimization algorithm for scheduling bag-of-task applications in auction-based proportional share systems. Proceedings of the 17th International Symposium on Computer Architecture and High Performance Computing, October 2005; 235-242.
29. Yu Z, Shi W,. An adaptive rescheduling strategy for grid workflow applications. Proceedings of the IEEE International Parallel and Distributed Processing Symposium, Long Beach, CA, U.S.A., 2007; 214-220.
30. Blythe J, Gil Y, Deelman E,. Coordinating workflows in shared grid environments. Proceedings of the 14th International Conference on Automated Planning and Scheduling, Whistler, BC, Canada, 2004.
31. Zhang Y, Koelbel C, Kennedy K,. Relative performance of scheduling algorithms in grid environment. Proceedings of the Seventh IEEE International Symposium on Cluster Computing and the Grid, Rio de Janeiro, Brazil, 2007.
32. Gburzynski P,. SMURPH. Available at:. http://www.cs.ualberta.ca/pawel/ SMURPH/smurph.html
33. Thain D, Tannenbaum T, Livny M,. Condor and the grid. In Grid Computing: Making the Global Infrastructure a Reality, Berman F, Fox G, Hey T, (eds.). Wiley: New York, 2003.
34. Arvind K, Nikhil RS,. Executing a program on the MIT tagged-token dataflow architecture. IEEE Transactions on Computers 1990; 39 (3): 3.
35. Eggers SJ, Emer JS, Levy HM, Lo JL, Stamm RL, Tullsen DM,. Simultaneous multithreading: A platform for next-generation processors. IEEE Micro 1997; 17 (5): 12-19. (Pubitemid 127583978)
36. Ungerer T, Robic B, Silc J,. A survey of processors with explicit multithreading. ACM Computing Surveys 2003; 35 (1): 29-63. (Pubitemid 44159292)
37. Ramakrishnan A, Singh G, Zhao H, Deelman E, Sakellariou R, Vahi K, Blackburn K, Mayers D, Samidi M,. Scheduling data-intensive workflows onto storage-constrained distributed resources. Proceedings of the Seventh IEEE International Symposium on Cluster Computing and the Grid, Rio de Janeiro, Brazil, 2007; 401-409. (Pubitemid 47334578)
38. Edwards DA, Mckendry MS,. Exploiting read-mostly workloads in the filenet file system. Proceedings of the 12th ACM Symposium on Operating Systems Principles, Litchfield Park, AZ, U.S.A., 1989; 58-70.
39. Hartman JH, Ousterhout JK,. The Zebra striped network file system. In High Performance Mass Storage and Parallel I/O: Technologies and Applications, Jin H, Cortes T, Buyya R, IEEE Computer Society Press, Wiley: Silver Spring, MD, New York, NY, 2001; 309-329.
40. Santry D, Feeley M, Hutchinson N, Veitch A, Carton R, Ofir J,. Deciding when to forget in the elephant file system. The 17th ACM Symposium on Operating Systems Principles (SOSP), Kiawah Island Resort, near Charleston, SC, U.S.A., 1999; 110-123. (Pubitemid 129548351)
41. Santry DJ, Feeley MJ, Hutchinson NC, Veitch AC,. Elephant: The file system that never forgets. Workshop on Hot Topics in Operating Systems, Rio Rico, AZ, U.S.A., 1999; 2-7.
42. Ghemawat S, Gobioff H, Leung S-T,. The Google file system. Nineteenth ACM Symposium on Operating Systems Principles (SOSP), Bolton Landing, NY, U.S.A., 2003; 29-43. (Pubitemid 40929686)
43. Simmhan YL, Plale B, Gannon D,. A framework for collecting provenance in data-centric scientific workflows. Proceedings of the IEEE International Conference on Web Services, Chicago, IL, U.S.A., 2006.
44. Simmhan YL, Plale B, Gannon D,. Query capabilities of the karma provenance framework. Concurrency and Computation: Practice and Experience 2008; 20 (5): 441-451. (Pubitemid 351428358)
45. Holland DA, Seltzer MI, Braun U, Muniswamy-Reddy K-K,. Passing the provenance challenge. Concurrency and Computation: Practice and Experience 2008; 20 (5): 531-540. (Pubitemid 351428366)
46. Frew J, Metzger D, Slaughter P,. Automatic capture and reconstruction of computational provenance. Concurrency and Computation: Practice and Experience 2007; 20 (5): 485-496.
47. Bent J, Thain D, Arpaci-Dusseau AC, Arpaci-Dusseau RH, Livny M,. Explicit control in a batch-aware distributed file system. Proceedings of Networked Systems Design and Implementation (NSDI), Chichester, U.K., 2004; 365-378.
48. Muniswamy-Reddy K-K, Wright CP, Himmer A, Zodok E,. A versatile and user-oriented versioning file system. Proceedings of the Third USENIX Conference on File and Storage Technologies (FAST), San Francisco, CA, 2004; 115-128.
49. Yamamoto T, Matsushita M, Inoue K,. Accumulative versioning file system Moraine and its application to metrics environment mame. Proceedings of the Eighth ACM SIGSOFT International Symposium on Foundations of Software Engineering, San Diego, CA, U.S.A., 2000; 80-87. (Pubitemid 32404339)
50. Warneke D, Kao O,. Nephele: Efficient parallel data processing in the cloud. MTAG'09: Proceedings of the Second Workshop on Many-task Computing on Grids and Supercomputers, Portland, OR, November 2009.
51. Vecchiola C, Pandey S, Buyya R,. High-performance cloud computing: A view of scientific applications. Proceedings of the 10th International Symposium on Pervasive Systems, Algorithms and Networks, Kaohsiung, Taiwan, 2009.
52. Yazir YO, Matthews C, Farahbod R, Neville S, Guitouni A, Ganti S, Coady Y,. Dynamic resource allocation in computing clouds using distributed multiple criteria decision analysis. Proceedings of IEEE Third International Conference on Cloud Computing, Miami, Florida, July 2010; 91-98.
53. Armbrust M, Fox A, Griffith R, Joseph AD, Katz RH, Konwinski A, Lee G, Patterson DA, Rabkin A, Stoica I, Zaharia M,. Above the clouds: A berkeley view of cloud computing. Technical Report UCB/EECS-2009-28, EECS Department, University of California, Berkeley, February 2009.
54. Youseff L, Butrico M, Da Silva D,. Toward a unified ontology of cloud computing. Grid Computing Environments Workshop, Location: Austin, TX, November 2008.
55. Deelman E, Singh G, Su H-H, Blythe J, Gil Y, Kesselman C, Mehta G, Vahi K, Berriman GB, Good J, Laity A, Jacob JC, Katz DS,. Pegasus: A framework for mapping complex scientific workflows onto distributed systems. Scientific Programming Journal 2005; 13 (3): 219-237. (Pubitemid 43022937)
56. Isard M, Budiu M, Yu Y, Birrell A, Fetterly D,. Dryad: Distributed data-parallel programs from sequential building blocks. EuroSys'07: Proceedings of the Second ACM SIGOPS/EuroSys European Conference on Computer Systems, Lisboa, Portugal, 2007; 59-72. (Pubitemid 47281575)