Dispel4py: A Python framework for data-intensive scientific computing

International Journal of High Performance Computing Applications

Volumn 31, Issue 4, 2017, Pages 316-334

  Filguiera, Rosa ^a   Krause, Amrey ^a   Atkinson, Malcolm ^a   Klampanos, Iraklis ^b   Moreno, Alexander ^c  

^a UNIVERSITY OF EDINBURGH   (United Kingdom)

^b INSTITUTE OF NANOSCIENCE AND NANOTECHNOLOGY   (Greece)

^c Georgia Institute of Technology   (United States)

Author keywords

data streaming; Data intensive computing; e infrastructures; programming frameworks; scientific workflows

Indexed keywords

DATA REDUCTION; DISTRIBUTED COMPUTER SYSTEMS; HIGH LEVEL LANGUAGES; MAPPING; MESSAGE PASSING; OPEN SOURCE SOFTWARE; SCALABILITY;

DATA STREAMING; DATA-INTENSIVE COMPUTING; E-INFRASTRUCTURES; PROGRAMMING FRAMEWORK; SCIENTIFIC WORKFLOWS;

COMPUTER ARCHITECTURE;

EID: 85021296575     PISSN: 10943420     EISSN: 17412846     Source Type: Journal    
DOI: 10.1177/1094342016649766     Document Type: Article

References (60)

1. Ács B, Llorà X, Auvil L, et al. (2010) A general approach to data-intensive computing using the Meandre component-based framework. In: Proceedings of the 1st international workshop on workflow approaches to new data-centric science, Indianapolis, IN, 6 June 2010, pp. 1-12. New York: ACM.
2. Agarwalla B, Ahmed N, Hilley D, et al. (2007) Streamline: Scheduling streaming applications in a wide area environment. In: Multimedia Systems, vol. 13, pp. 69-85.
3. Agrawal K, Anne Benoit FD, Rober Y, (2012) Mapping filtering streaming applications. Algorithmica 62 (1-2): 258-308.
4. Ahmad SG, Liew CS, Rafique MM, et al. (2014) Data-intensive workflow optimization based on application task graph partitioning in heterogeneous computing systems. In: 4th IEEE international conference on big data and cloud computing.
5. Aiche S, Sachsenberg T, Kenar E, et al. (2015) Workflows for automated downstream data analysis and visualization in large-scale computational mass spectrometry. Proteomics 15 (8): 1443-1447.
6. Atkinson M, (2013) Chapter 4: Data-intensive thinking with dispel. In: Atkinson M, Baxter R, Brezany P, et al.(eds) The DATA Bonanza: Improving Knowledge Discovery for Science, Engineering and Business. Wiley, pp. 61-122.
7. Atkinson M, Parsons M, (2013) Chapter 1: The digital-data challenge. In: Atkinson M, Baxter R, Brezany P, et al.(eds) The DATA Bonanza: Improving Knowledge Discovery for Science, Engineering and Business. John Wiley & Sons, pp. 5-13.
8. Atkinson M, Baxter R, Brezany P, et al. (eds) (2013) The DATA Bonanza: Improving Knowledge Discovery for Science, Engineering and Business. John Wiley & Sons.
9. Atkinson MP, Carpené M, Casarotti E, et al. (2015) VERCE delivers a productive e-Science environment for seismology research. In: Proceedings of the 11th IEEE e-Science conference, Munich.
10. Atkinson MP, Liew MG, Sun C, et al. (2012) Data-intensive architecture for scientific knowledge discovery. Distributed and Parallel Databases 30 (5-6): 307-324.
11. Baccianella S, Esuli A, Sebastiani F, (2010) Sentiwordnet 3.0: An enhanced lexical resource for sentiment analysis and opinion mining. In: Choukri K, Maegaard B, Mariani J, et al.(eds) Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10). Valletta, Malta: European Language Resources Association (ELRA).
12. Barker A, Walton CD, Robertson D, (2009) Choreographing web services. IEEE Transaction on Services Computing 2 (2): 152-166.
13. Barseghian D, Altintas I, Jones MB, et al. (2010) Workflows and extensions to the Kepler scientific workflow system to support environmental sensor data access and analysis. Ecological Informatics 5: 42-50.
14. Beisken S, Meinl T, Wiswedel B, et al. (2013) Knime-cdk: Workflow-driven cheminformatics. BMC Bioinformatics 14 (1): 257 (online). Available at: http://www.biomedcentral.com/1471-2105/14/257
15. Belhajjame K, Zhao J, Garijo D, et al. (2015) Using a suite of ontologies for preserving workflow-centric research objects. Journal of Web Semantics, 32: 16-42.
16. Bensen GD, Ritzwoller MH, Barmin MP, et al. (2007) Processing seismic ambient noise data to obtain reliable broad-band surface wave dispersion measurements. Geophysical Journal International 169 (3): 1239-1260.
17. Berriman GB, Deelman E, Good J, et al. (2007) Generating complex astronomy workflows. In: Workflows for e-Science. Springer.
18. Berriman GB, Deelman E, Groth PT, et al. (2010) The application of cloud computing to the creation of image mosaics and management of their provenance. In: Software and Cyberinfrastructure for Astronomy, vol. 7740. SPIE, p. 77401F.
19. Berthold MR, Cebron N, Dill F, et al. (2009) Knime-The konstanz information miner. SIGKDD Explorations 11: 26-31 (online). Available at: http://doi.acm.org/10.1145/1656274.1656280
20. Beyreuther M, Barsch R, Krischer L, et al. (2010) ObsPy: A python toolbox for seismology. Seismological Research Letters 81 (3): 530-533.
21. Blankenberg D, Kuster GV, Coraor N, et al. (2010) Galaxy: A Web-Based Genome Analysis Tool for Experimentalists. Wiley.
22. Bode MF, Cruz MJ, Molster FJ, (2008) The ASTRONET infrastructure roadmap. ASTRONET (online). Available at: http://books.google.co.uk/books?id=Y5tQPgAACAAJ
23. Buil-Aranda C, Arenas M, Corcho O, et al. (2013) Federating queries in SPARQL1.1: Syntax, semantics and evaluation. Web Semantics 18.
24. Carpene M, Klampanos I, Leong S, et al. (2013) Towards addressing CPU-intensive seismological applications in Europe. In: Kunkel J, Ludwig T, Meuer H, (eds) Supercomputing (vol. 7905, LNCS). Berlin/Heidelberg: Springer, pp. 55-66 (online). Available at: http://dx.doi.org/10.1007/978-3-642-38750-05
25. Cho H, Lee J, Kim S, (2013) Enhancing lexicon-based review classification by merging and revising sentiment dictionaries. In: Sixth International Joint Conference on Natural Language Processing (IJCNLP 2013), Nagoya, Japan, 14-18 October 2013, pp. 463-470. Asian Federation of Natural Language Processing/ACL (online). Available at: http://aclweb.org/anthology/I/I13/I13-1053.pdf
26. Churches D, Gombas G, Harrison A, et al. (2006) Programming scientific and distributed workflow with Triana services: Research articles. Concurrency and Computation: Practice and Experience 18 (10): 1021-1037.
27. De Roure D, Goble C, (2009) Software design for empowering scientists. IEEE Software 26 (1): 88-95.
28. De Roure D, Goble C, Stevens R, (2009) The design and realisation of the myexperiment virtual research environment for social sharing of workflows. Future Generation Computer Systems 25: 561-567 (online). DOI: 10.1016/j.future.2008.06.010.
29. Deelman E, Vahi K, Juve G, et al. (2015) Pegasus, a workflow management system for science automation. Future Generation Computer Systems 46C: 17-35.
30. Dutton WH, Jeffreys PW, (2010) World Wide Research: Reshaping the Sciences and Humanities. MIT Press.
31. Earle PS, Wald DJ, Jaiswal KS, et al. (2009) Prompt Assessment of Global Earthquakes for Response (PAGER): A System for Rapidly Determining the Impact of Earthquakes Worldwide (Technical Report). US Geological Survey.
32. Falt Z, Bednárek D, Kruliš M, et al. (2014) Bobolang: A language for parallel streaming applications. In: Proceedings of the 23rd international symposium on high-performance parallel and distributed computing (HPDC'14). New York: ACM, pp. 311-314 (online). Available at: http://doi.acm.org/10.1145/2600212.2600711
33. Filgueira R, Atkinson MP, Krause A, et al. (2015) dispel4py: An agile framework for data-intensive e-Science. In: Proceedings of the IEEE e-Science 2015.
34. Filgueira R, Atkinson MP, Tanimura Y, et al. (2014) Applying selectively parallel I/O compression to parallel storage systems. In: Euro-Par.
35. Gannon D, Plale B, Marru S, et al. (2007) Dynamic, adaptive workflows for mesoscale meteorology. In: Workflows for e-Science. Springer, pp. 126-142.
36. Garijo D, Corcho O, Gil Y, et al. (2014) Workflow reuse in practice: A study of neuroimaging pipeline users. In: e-Science. IEEE Computer Society, pp. 239-246 (online). Available at: http://dblp.uni-trier.de/db/conf/eScience/eScience2014.html#GarijoCGBHHJTT14
37. Gesing S, Atkinson M, Filgueira R, et al. (2014) Workflows in a dashboard: A new generation of usability. In: Proceedings of the WORKS'14. Piscataway, NJ: IEEE Press, 2014, pp. 82-93 (online). Available: http://dx.doi.org/10.1109/WORKS.2014.6
38. Guirado F, Roig C, Ripoll A, (2013) Enhancing throughput for streaming applications running on cluster systems. Journal of Parallel and Distributed Computing 73 (8): 1092-1105.
39. Hey AJG, Tansley S, Tolle K, (eds) (2009) The Fourth Paradigm: Data-Intensive Scientific Discovery. Redmond, WC: Microsoft Research.
40. Kelling S, Fink D, Hochachka W, et al. (2013) Chapter 22: Estimating species distributions-Across space, through time and with features of the environment. In: Atkinson MP, Baxter R, Besana P, et al.(eds) The DATA Bonanza-Improving Knowledge Discovery for Science, Engineering and Business. John Wiley & Sons, pp. 441-458.
41. Maechling P, Deelman E, Zhao L, et al. (2007) SCEC cybershake workflows-Automating probabilistic seismic hazard analysis calculations. In: Workflow for e-Sciences. Springer, pp. 143-163.
42. Medhat W, Hassan A, Korashy H, (2014) Sentiment analysis algorithms and applications: A survey. Ain Shams Engineering Journal 5 (4): 1093-1113 (online). Available at: http://www.sciencedirect.com/science/article/pii/S2090447914000550
43. MPI Forum (1994) MPI: A message-passing interface standard. International Journal of Supercomputer Applications 8: 165-414.
44. Nielsen FA, (2011) AFINN. Kgs. Lyngby: Technical University of Denmark.
45. Pak A, Paroubek P, (2010) Twitter as a corpus for sentiment analysis and opinion mining. In: Proceedings of the International Conference on Language Resources and Evaluation (LREC 2010), Valletta, Malta, 17-23 May 2010.
46. Ringler A, Hagerty M, Holland J, et al. (2015) The data quality analyzer: A quality control program for seismic data. Computers & Geosciences 76: 96-111.
47. Rogers D, Harvey I, Huu TT, et al. (2013) Bundle and pool architecture for multi-language, robust, scalable workflow executions. Journal of Grid Computing 11 (3): 457-480.
48. Rosner M, Tapias D, (eds) (2010) European Language Resources Association (online). Available at: http://www.lrec-conf.org/proceedings/lrec2010/summaries/385.html
49. Ruiz J, Garrido J, Santander-Vela J, et al. (2014) Astrotaverna-Building workflows with virtual observatory services (Special Issue on the Virtual Observatory). Astronomy and Computing 7-8: 3-11.
50. Rynge M, Juve G, Kinney J, et al. (2013) Producing an infrared multiwavelength galactic plane atlas using Montage, Pegasus and Amazon web services. In: ADASS Conference.
51. Segaran T, Hammerbacher J, (2009) Beautiful Data: The Stories behind Elegant Data Solutions. O'Reilly, 2009.
52. Shoshani A, Rotem D, (2010) Scientific Data Management: Challenges, Technology and Deployment. Chapman & Hall.
53. Simmhan Y, Barga R, van Ingen C, et al. (2009) Building the trident scientific workflow workbench for data management in the cloud. In: International Conference on Advanced Engineering Computing and Applications in Sciences (ADVCOMP), October 2009. IEEE.
54. Spinuso A, Cheney J, Atkinson M, (2013) Provenance for seismological processing pipelines in a distributed streaming workflow. In: Proceedings of the EDBT'13, pp. 307-312. New York: ACM.
55. Stonebraker M, Brown P, Zhang D, et al. (2013) SciDB: A DBMS for applications with complex analytics. Computing in Science & Engineering 15 (3): 54-62.
56. Vahi K, Rynge M, Juve G, et al. (2013) Rethinking data management for big data scientific workflows. In: Workshop on Big Data and Science: Infrastructure and Services.
57. Walter C, (2005) Kryder's law. Scientific American, pp. 32-33.
58. Wilde M, Hategan M, Wozniak JM, et al. (2011) Swift: A language for distributed parallel scripting. Parallel Computing 37 (9): 633-652.
59. Wolstencroft K, Haines R, Fellows D, et al. (2013) The Taverna workflow suite: designing and executing workflows of Web Services on the desktop, web or in the cloud. Nucleic Acids Research 41 (W1): W557-W561.
60. Wozniak JM, Armstrong TG, Maheshwari K, et al. (2013) Turbine: A distributed-memory dataflow engine for high performance many-task applications. Fundamenta Informaticae 128 (3): 337-366 (online). Available at: http://dx.doi.org/10.3233/FI-2013-949