A model-based framework for building extensible, high performance stream processing middleware and programming language for IBM InfoSphere Streams

Software - Practice and Experience

Volumn 42, Issue 11, 2012, Pages 1363-1391

  Gedik, Buǧra ^a   Andrade, Henrique ^a  

^a IBM RESEARCH   (United States)

Author keywords

data stream processing; model based code generation

Indexed keywords

BEHAVIORAL REASONING; BUILDING PROCESS; CODE GENERATION; COMMERCIAL GRADE; COMPILE TIME; DATA STREAM PROCESSING; DESIGN APPROACHES; INCREMENTAL COMPILATION; INFOSPHERE; INTERPROCESS COMMUNICATION; INTRINSIC PERFORMANCE; MODEL DESCRIPTION; MODEL-BASED DESIGN; RUNTIME ENVIRONMENTS; RUNTIME PERFORMANCE; RUNTIMES; STREAM PROCESSING;

COMPUTER HARDWARE DESCRIPTION LANGUAGES; MIDDLEWARE; NETWORK COMPONENTS; PROGRAM COMPILERS; SEMANTICS; SOFTWARE ENGINEERING;

STRUCTURAL DESIGN;

EID: 84867097696     PISSN: 00380644     EISSN: 1097024X     Source Type: Journal    
DOI: 10.1002/spe.1139     Document Type: Article

References (65)

1. Eick SG, Graves TL, Karr AF, Marron JS, Mockus A,. Does code decay? Assessing the evidence from change management data. Transactions on Software Engineering 2001; 27 (1): 1-12.
2. April A, Abran A,. A software maintenance maturity model (S3M): Measurement practices at maturity levels 3 and 4. Electronic Notes in Theoretical Computer Science 2009; 233: 73-87.
3. Bennett KH, Rajlich VT,. Software maintenance and evolution: A roadmap. Proceedings of the Conference on the Future of Software Engineering, Limerick, Ireland, 2000; 73-87.
4. IBM InfoSphere Streams. Available from: http://www-01.ibm.com/software/ data/infosphere/streams [accessed on 30 September 2010].
5. Amini L, Andrade H, Bhagwan R, Eskesen F, King R, Selo P, Park Y, Venkatramani C,. SPC: A distributed, scalable platform for data mining. Proceedings of the Workshop on Data Mining Standards, Services and Platforms (DM-SSP), Philadelphia, PA, 2006; 27-37.
6. De Pauw W, Andrade H,. Visualizing large-scale streaming applications. Information Visualization 2009; 8 (2): 87-106.
7. De Pauw W, Andrade H, Amini L,. StreamSight: A visualization tool for large-scale streaming applications. Proceedings of the Symposium on Software Visualization (SoftVis), Herrsching am Ammersee, Germany, 2008; 125-134.
8. Gedik B, Andrade H, Frenkiel A, De Pauw W, Pfeifer M, Allen P, Cohen N, Wu K-L,. Debugging tools and strategies for distributed stream processing applications. Software: Practice and Experience 2009; 39 (16): 1347-1376.
9. Jacques-Silva G, Challenger J, Degenaro L, Giles J, Wagle R,. Towards autonomic fault recovery in System-S. Proceedings of the International Conference on Autonomic Computing (ICAC), Jacksonville, FL, 2007; 31.
10. Jacques-Silva G, Gedik B, Andrade H, Wu K-L,. Language-level checkpointing support for stream processing applications. Proceedings of the International Conference on Dependable Systems and Networks (DSN), Estoril, Portugal, 2009; 145-154.
11. Park Y, King R, Nathan S, Most W, Andrade H,. Evaluation of a high-volume, low-latency market data processing sytem implemented with IBM middleware. Software: Practice and Experience, (to appear). DOI: 10.1002/spe.1047.
12. Turaga D, Andrade H, Gedik B, Venkatramani C, Verscheure O, Harris JD, Cox J, Szewczyk W, Jones P,. Design principles for developing stream processing applications. Software: Practice and Experience 2010; 40 (12): 1073-1104.
13. Wang H, Andrade H, Gedik B, Wu K-L,. A code generation approach for auto-vectorization in the SPADE compiler. Proceedings of the International Workshop on Languages and Compilers for Parallel Computing (LCPC), Newark, DE, 2009; 383-390.
14. Wolf J, Bansal N, Hildrum K, Parekh S, Rajan D, Wagle R, Wu K-L,. SODA: An optimizing scheduler for large-scale stream-based distributed computer systems. Proceedings of the International Middleware Conference (Middleware), Leuven, Belgium, 2008; 306-325.
15. Wolf J, Khandekar R, Hildrum K, Parekh S, Rajan D, Wu K-L, Andrade H, Gedik B,. COLA: Optimizing stream processing applications via graph partitioning. Proceedings of the International Middleware Conference (Middleware), Urbana, IL, 2009; 308-325.
16. Wu K-L, Yu PS, Gedik B, Hildrum KW, Aggarwal CC, Bouillet E, Fan W, George DA, Gu X, Luo G, Wang H,. Challenges and experience in prototyping a multi-modal stream analytic and monitoring application on System S. Proceedings of the International Conference on Very Large Data Bases (VLDB), Vienna, Austria, 2007; 1185-1196.
17. Zhang X, Andrade H, Gedik B, King R, Morar J, Nathan S, Park Y, Pavuluri R, Pring E, Schnier R, Selo P, Spicer M, Venkatramani C,. Implementing a high-volume, low-latency market data processing system on commodity hardware using IBM middleware. Proceedings of the Workshop on High Performance Computational Finance (WHPCF), Portland, OR, 2009.
18. Hirzel M, Andrade H, Gedik B, Kumar V, Losa G, Mendell M, Nasgaard H, Soulé R, Wu K-L,. SPLa-Stream Processing Language Specification. Technical Report RC24897, IBM Research. November 2009.
19. Andrade H, Gedik B, Wu K-L, Yu PS,. Scale-up strategies for processing high-rate data streams in System S. Proceedings of the International Conference on Data Engineering (ICDE), Shanghai, China, 2009; 1375-1378.
20. Gedik B, Andrade H, Wu K-L, Yu PS, Doo M,. SPADE: The System S declarative stream processing engine. Proceedings of the International Conference on Management of Data (SIGMOD), Vancouver, Canada, 2008; 1123-1134.
21. Codd E,. A relational model for large shared data banks. Communications of the ACM 1970; 13 (6): 377-387.
22. Gedik B, Andrade H, Wu K-L,. A code generation approach to optimizing high-performance distributed data stream processing. Proceedings of the International Conference on Information and Knowledge Management (CIKM), Hong Kong, China, 2009; 847-856.
23. De Pauw W, Letia M, Gedik B, Andrade H, Frenkiel A, Pfeifer M, Sow D,. Visual debugging for stream processing applications. Proceedings of the International Conference on Runtime Verification (RV), St Julians, Malta, 2010; 18-35.
24. Andrade H, Gedik B, Wu K-L, Yu PS,. Processing high data rate streams in System S. Journal of Parallel Distributed Computing 2011; 71 (2): 145-156.
25. Zou Q, Wang H, Soulé R, Hirzel M, Andrade H, Gedik B, Wu K-L,. From a stream of relational queries to distributed stream processing. Proceedings of the International Conference on Very Large Data Bases (VLDB), Singapore, 2009; 1394-1405.
26. Eclipse Modeling Framework Project. Available from:http://www.eclipse. org/modeling/emf [accessed on 30 September 2010].
27. Fallside DC, Walmsley P,. XML Schema Part 0: Primer Second Edition. Technical Report, W3C, September 2004.
28. Code Synthesis. C++/parser mappinga-getting started guide. Technical Report, Code Synthesis Tools CC, 2010.
29. Sun Microsystems. The Java architecture for XML bindinga-user's guide. Technical report, Sun Microsystems, Inc, October 2002.
30. The Object Management Group (OMG) Corba. Available from: http://www.corba.org [accessed on 30 September 2010].
31. OMG. C++ language mapping, version 1.2. Technical Report formal/2008-01-09, Object Management Group, January 2008.
32. OMG. IDL to Java language mapping, version 1.3. Technical Report formal/2008-01-11, Object Management Group, January 2008.
33. Grisby D, Lo S-L, Riddoch D,. The omniORB version 4.1a-user's guide. Technical Report formal/2008-01-11, Apasphere and AT&T Laboratories Cambridge, July 2009.
34. Schneider S, Andrade H, Gedik B, Biem A, Wu K-L,. Elastic scaling of data parallel operators in stream processing. Proceedings of the International Conference on Parallel and Distributed Processing Systems (IPDPS), Rome, Italy, 2009; 1-12.
35. Wall L, Christiansen T, Orwant J,. Programming Perl. O'Reilly Media, 2000.
36. Cazzola W,. SmartReflection: Efficient introspection in Java. Journal of Object Technology 2004; 3 (11): 117-132.
37. Andrade H, Gedik B, Mendell M, Nasgaard H, Wu K-L,. Method of selecting an expression evaluation technique for domain-specific language compilation. Patent filed as IBM Docket YOR920090642US1 in the United States, 2010.
38. IBM DB2. Available from: http://www.ibm.com/db2 [accessed on 30 September 2010].
39. IBM MQ. Available from: http://www.ibm.com/software/integration/wmq [accessed on 30 September 2010].
40. The GNU Foundation GCC. Available from: http://gcc.gnu.org [accessed on 30 September 2010].
41. Andrade H, Gedik B, Wu K-L,. On-demand marshalling and demarshalling of network messages. Patent filed as IBM Docket YOR920090029US1 in the United States, 2008.
42. Andrade H, Gedik B, Hou R, Wang H, Wu K-L,. Incrementally constructing executable code for component-based applications. Patent filed as IBM Docket YOR920080366US1 in the United States, 2009.
43. WebSphere MQ Low Latency Messaging. Available from: http://www-01.ibm. com/software/integration/wmq/llm [accessed on 30 September 2010].
44. Meng S, Kashyap SR, Venkatramani C, Liu L,. REMO: Resource-aware application state monitoring for large-scale distributed systems. Proceedings of the International Conference on Distributed Computing Systems (ICDCS), Montreal, Canada, 2009; 248-255.
45. Stallman RM, CygnusSupport. Debugging with GDBa-The GNU Source-Level Debugger. Free Software Foundation, 1996.
46. Valgrind Developers. Valgrind User Manual (Release 3.6.0 21). Apache Software Foundation, October 2010.
47. Microsoft. Distributed component object model (DCOM)a-remote protocol specification (revision 11.0.1). Technical Report, Microsoft Corporation, August 2010.
48. Booth D, Haas H, McCabe F, Newcomer E, Champion M, Ferris C, Orchard D,. Web services architecturea-W3C working group note 11 February 2004. Technical Report, W3C, February 2004.
49. Sun Microsystems, Inc. RPC: Remote procedure call protocol specification version 2. Technical Report RFC 1050, The Internet Engineering Task Force (IETF), June 1988.
50. Bray T, Paoli J, Sperberg-McQueen CM, Maler E, Yergeau F,. Extensible markup language (XML) 1.0a-fifth edition. Technical Report, W3C, November 2008.
51. Meyer B,. Object-Oriented Software Construction. Prentice Hall, 1997.
52. Austern M,. Generic Programming and the STL, Using and Extending the C++ Standard Template Library. Addison-Wesley, 1999.
53. Feldman S,. Makea-a program for maintaining computer programs. Software: Practice & Experience 1979; 9 (4): 255-265.
54. Stallman R, McGrath R, Smith P,. GNU Makea-A Program for Directing Recompilation (version 3.82). Free Software Foundation, July 2010.
55. MacKenzie D, Elliston B, Demaille A,. Autoconfa-Creating Automatic Configuration Scripts (version 2.68). Free Software Foundation, September 2010.
56. Habra A,. Apache Ant User Manual Guide (version 1.6.0). Apache Software Foundation, December 2003.
57. Challenges of large applications in distributed environments (CLADE 2010). Available from: http://sites.google.com/site/clade2010 [accessed on 30 September 2010].
58. StreamBase systems. Available from: http://www.streambase.com [accessed on 30 September 2010].
59. Progress apama event processing platform. Available from:http://web. progress.com/en/apama/event-processing-platform.html [accessed on 30 September 2010].
60. Balakrishnan H, Balazinska M, Carney D, Cetintemel U, Cherniack M, Convey C, Galvez E, Salz J, Stonebraker M, Tatbul N, Tibbetts R, Zdonik S,. Retrospective on Aurora. Very Large Data Bases Journal, Special Issue on Data Stream Processing 2004; 13 (4): 370-383.
61. Abadi DJ, Ahmad Y, Balazinska M, Cetintemel U, Cherniack M, Hwang J-H, Lindner W, Maskey AS, Rasin A, Ryvkina E, Tatbul N, Xing Y, Zdonik S,. The design of the Borealis stream processing engine. Proceedings of the Innovative Data Systems Research Conference (CIDR), Asilomar, CA, 2005; 277-289.
62. Arasu A, Babcock B, Babu S, Datar M, Ito K, Motwani R, Nishizawa I, Srivastava U, Thomas D, Varma R, Widom J,. STREAM: The Stanford stream data manager. IEEE Data Engineering Bulletin 2003; 26 (1): 19-26.
63. StreamSQL. Available from: http://www.streambase.com/products-streamsql. htm [accessed on 30 September 2010].
64. Arasu A, Babu S, Widom J,. The CQL continuous query language: Semantic foundations and query execution. Journal on Very Large Data Bases 2006; 15 (2): 121-142.
65. Thies W, Karczmarek M, Amarasinghe S,. StreamIt: A language for streaming applications. Proceedings of the International Conference on Compiler Construction (CC), Grenoble, France, 2002; 179-196.