Parallelizing XML data-streaming workflows via MapReduce

Journal of Computer and System Sciences

Volumn 76, Issue 6, 2010, Pages 447-463

  Zinn, Daniel ^a   Bowers, Shawn ^b,c   Köhler, Sven ^b   Ludäscher, Bertram ^a,b  

^a University of California   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c GONZAGA UNIVERSITY   (United States)

Author keywords

Collection Oriented Modeling and Design (COMAD); Data stream processing; Grouping; MapReduce; Parallelization; Static analysis; Virtual Data Assembly Line (VDAL); XML processing pipelines

Indexed keywords

DATA STREAMS; ECONOMIC AND SOCIAL EFFECTS; FAULT TOLERANCE; PIPELINES; STATIC ANALYSIS; XML;

ASSEMBLY LINE; DATA STREAM PROCESSING; GROUPING; MAP-REDUCE; PARALLELIZATIONS; XML PROCESSING;

PIPELINE PROCESSING SYSTEMS;

EID: 77955309365     PISSN: 00220000     EISSN: 10902724     Source Type: Journal    
DOI: 10.1016/j.jcss.2009.11.006     Document Type: Conference Paper

References (49)

1. G.C. Fox, D. Gannon, Special Issue: Workflow in Grid Systems Concurrency and Computation: Practice & Experience 18 10 2006 Wiley
2. I.J. Taylor, E. Deelman, D.B. Gannon, M. Shields, Workflows for e-Science: Scientific Workflows for Grids 2007 Springer
3. E. Deelman, D. Gannon, M. Shields, and I. Taylor Workflows and e-science: An overview of workflow system features and capabilities Future Gener. Comput. Syst. 25 2009 528 540
4. B. Ludäscher, I. Altintas, S. Bowers, J. Cummings, T. Critchlow, E. Deelman, D.D. Roure, J. Freire, C. Goble, M. Jones, S. Klasky, T. McPhillips, N. Podhorszki, C. Silva, I. Taylor, and M. Vouk Scientific process automation and workflow management A. Shoshani, D. Rotem, Scientific Data Management: Challenges, Existing Technology, and Deployment, Computational Science Series 2009 Chapman & Hall/CRC Ch. 13
5. P. Amnuaykanjanasin, and N. Nupairoj The BPEL orchestrating framework for secured grid services Information Technology: Coding and Computing (ITCC) 1 2005 348 353
6. J. Fagan Mashing up multiple Web feeds using Yahoo! Pipes Computers in Libraries 27 10 2007 10 18
7. Intl. Workshop on Web Service Choreography and Orchestration for BPM Nancy, France, September 2005 http://events.deri.at/bpm2005/
8. T. Oinn, M. Greenwood, M. Addis, M.N. Alpdemir, J. Ferris, K. Glover, C. Goble, A. Goderis, D. Hull, D. Marvin, P. Li, P. Lord, M.R. Pocock, M. Senger, R. Stevens, A. Wipat, and C. Wroe Taverna: Lessons in Creating a Workflow Environment for the Life Sciences G.C. Fox, D. Gannon, Special Issue: Workflow in Grid Systems Concurrency and Computation: Practice & Experience 18 10 2006 1067 1100 Wiley
9. B. Ludäscher, I. Altintas, C. Berkley, D. Higgins, E. Jaeger, M. Jones, E.A. Lee, J. Tao, and Y. Zhao Scientific Workflow Management and the Kepler System G.C. Fox, D. Gannon, Special Issue: Workflow in Grid Systems Concurrency and Computation: Practice & Experience 18 10 2006 1039 1065 Wiley
10. J. Hidders, N. Kwasnikowska, J. Sroka, J. Tyszkiewicz, and J.V. den Bussche A Formal Model of Dataflow Repositories Data Integration in the Life Sciences (DILS) 4544 2007 105 121
11. F. Tian, B. Reinwald, H. Pirahesh, T. Mayr, J. Myllymaki, Implementing a scalable XML publish/subscribe system using relational database systems, SIGMOD '04: Proceedings of the 2004 ACM SIGMOD International Conference on Management of Data (2004) 479-490
12. N. Walsh, and A. Milowski XProc: An XML Pipeline Language April 2007 W3C Working Draft
13. I. Avila-Campillo, T.J. Green, A. Gupta, M. Onizuka, D. Raven, D. Suciu, XMLTK: An XML toolkit for scalable XML stream processing, in: PLAN-X, Pittsburgh, PA, 2002
14. T. McPhillips, S. Bowers, D. Zinn, and B. Ludäscher Scientific Workflow Automation for Mere Mortals Future Gener. Comput. Syst. 25 5 2009 541 551
15. D. Zinn, S. Bowers, T. M.McPhillips, B. Ludäscher, X-CSR: Dataflow Optimization for Distributed XML Process Pipelines, in: ICDE, 2009, pp. 577-580, also see Technical Report CSE-2008-15, UC Davis
16. J. Dean, and S. Ghemawat MapReduce: Simplified data processing on large clusters Communications of the ACM 51 1 2008 107 113
17. Hadoop http://hadoop.apache.org/
18. V. Benzaken, G. Castagna, and A. Frisch CDuce: An XML-Centric General-Purpose Language ICFP '03: Proceedings of the Eighth ACM SIGPLAN International Conference on Functional Programming 2003 ACM New York, NY, USA 51 63
19. Y. Zhao, J. Dobson, I. Foster, L. Moreau, and M. Wilde A notation and system for expressing and executing cleanly typed workflows on messy scientific data SIGMOD Rec. 34 3 2005 37 43
20. N. Podhorszki, B. Ludäscher, and S. Klasky Workflow automation for processing plasma fusion simulation data WORKS '07: Proceedings of the 2nd Workshop on Workflows in Support of Large-Scale Science 2007 ACM New York, NY, USA 35 44
21. Image-magick http://www.imagemagick.org
22. C. Barton, P. Charles, D. Goyal, M. Raghavachari, M. Fontoura, and V. Josifovski Streaming XPath processing with forward and backward axes Proceedings of the International Conference on Data Engineering 1998 2003 IEEE Computer Society Press 455 466
23. A. Gupta, and D. Suciu Stream processing of XPath queries with predicates Proceedings of the 2003 ACM SIGMOD International Conference on Management of Data 2003 ACM New York, NY, USA 419 430
24. D. Borthakur The Hadoop Distributed File System: Architecture and Design 2007 Apache Software Foundation
25. T. Härder, M. Haustein, C. Mathis, and M. Wagner Node labeling schemes for dynamic XML documents reconsidered Data & Knowledge Engineering 60 1 2007 126 149
26. P. O'Neil, E. O'Neil, S. Pal, I. Cseri, G. Schaller, and N. Westbury ORDPATHs: Insert-friendly XML node labels Proceedings of the 2004 ACM SIGMOD International Conference on Management of Data 2004 ACM New York, NY, USA 903 908
27. Rocks clusters http://www.rocksclusters.org/
28. W. Gentzsch, Sun grid engine: Towards creating a compute power grid, in: First IEEE/ACM International Symposium on Cluster Computing and the Grid, 2001, Proceedings, 2001, pp. 35-36
29. I. Taylor, M. Shields, I. Wang, and O. Rana Triana applications within grid computing and peer to peer environments Journal of Grid Computing 1 2 2003 199 217
30. E. Deelman Pegasus: A framework for mapping complex scientific workflows onto distributed systems Scientific Programming 13 3 2005 219 237
31. Y. Zhao, M. Hategan, B. Clifford, I. Foster, G. von Laszewski, V. Nefedova, I. Raicu, T. Stef-Praun, M. Wilde, Swift: Fast, reliable, loosely coupled parallel computation, in: IEEE Congress on Services, 2007, pp. 199-206
32. T. Fahringer, R. Prodan, R. Duan, F. Nerieri, S. Podlipnig, J. Qin, M. Siddiqui, H. Truong, A. Villazon, and M. Wieczorek ASKALON: A grid application development and computing environment International Workshop on Grid Computing 2005 122 131
33. J. Qin, and T. Fahringer Advanced data flow support for scientific grid workflow applications Proceedings of the ACM/IEEE conference on Supercomputing (SC) 2007 ACM 1 12
34. X. Fei, S. Lu, C. Lin, A MapReduce-Enabled scientific workflow composition framework, in: IEEE International Conference on Web Services (ICWS), 2009, pp. 663-670
35. D.J. Goodman Introduction and evaluation of Martlet: A scientific workflow language for abstracted parallelisation, in: International World Wide Web Conference (WWW), 2007, pp. 983-992
36. C. Koch, S. Scherzinger, N. Schweikardt, and B. Stegmaier FluXQuery: An Optimizing XQuery Processor for Streaming XML Data VLDB 2004 1309 1312
37. S. Chandrasekaran, O. Cooper, A. Deshpande, M.J. Franklin, J.M. Hellerstein, W. Hong, S. Krishnamurthy, S.R. Madden, F. Reiss, and M.A. Shah TelegraphCQ: Continuous Dataflow Processing for an Uncertain World SIGMOD '03: Proceedings of the 2003 ACM SIGMOD International Conference on Management of Data 2003 ACM New York, NY, USA 668
38. J. Chen, D.J. DeWitt, F. Tian, Y. Wang, NiagaraCQ: A scalable continuous query system for internet databases, in: SIGMOD 2000: Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data, 2000, pp. 379-390
39. M. Balazinska, H. Balakrishnan, S.R. Madden, and M. Stonebraker Fault-tolerance in the borealis distributed stream processing system ACM Trans. Database Syst. 33 1 2008 1 44
40. C. Koch, S. Scherzinger, N. Schweikardt, and B. Stegmaier Schema-based scheduling of event processors and buffer minimization for queries on structured data streams VLDB '04: Proceedings of the Thirtieth International Conference on Very Large Data Bases 2004 VLDB Endowment 228 239
41. T.J. Green, A. Gupta, G. Miklau, M. Onizuka, and D. Suciu Processing XML streams with deterministic automata and stream indexes TODS 29 4 2004 752 788
42. Y. Chen, S.B. Davidson, and Y. Zheng An efficient XPath query processor for XML streams ICDE '06: Proceedings of the 22nd International Conference on Data Engineering 2006 IEEE Computer Society Washington, DC, USA 79
43. M.F. Fernández, T. Jim, K. Morton, N. Onose, J. Siméon, Highly distributed XQuery with DXQ, in: SIGMOD '07: Proceedings of the 2007 ACM SIGMOD International Conference on Management of Data, 2007, pp. 1159-1161
44. C. Re, J. Brinkley, K. Hinshaw, and D. Suciu Distributed XQuery Workshop on Information Integration on the Web 2004 116 121
45. G. Kahn The semantics of a simple language for parallel programming J.L. Rosenfeld, Proc. of the IFIP Congress vol. 74 1974 North-Holland 471 475
46. S. Abiteboul, A. Bonifati, G. Cobéna, I. Manolescu, T. Milo, Dynamic XML documents with distribution and replication, in: SIGMOD '03: Proceedings of the 2003 ACM SIGMOD international conference on Management of data, 2003, pp. 527-538
47. H.-C. Yang, A. Dasdan, R.-L. Hsiao, and D.S. Parker Map-reduce-merge: Simplified relational data processing on large clusters SIGMOD '07: Proceedings of the 2007 ACM SIGMOD International Conference on Management of Data 2007 ACM New York, NY, USA 1029 1040
48. T.M. McPhillips, and S. Bowers An approach for pipelining nested collections in scientific workflows SIGMOD Record 34 3 2005 12 17
49. T. McPhillips, S. Bowers, and B. Ludäscher Collection-oriented scientific workflows for integrating and analyzing biological data Data Integrating for Life Sciences (DILS) 2006 248 263