Adding stream processing system flexibility to exploit low-overhead communication systems

Proceedings of the 3rd Workshop on High Performance Computational Finance, WHPCF 2010

Volumn , Issue , 2010, Pages

  Selo, Philippe ^a   Park, Yoonho ^a   Parekh, Sujay ^a   Venkatramani, Chitra ^a   Pyla, Hari K ^a   Zheng, Fang ^a  

^a IBM T J WATSON RESEARCH CENTER   (United States)

Author keywords

InfiniBand; Shared Memory; Stock Market Data; Stream Processing

Indexed keywords

COMMODITY HARDWARE; FINANCIAL APPLICATIONS; INFINIBAND; SHARED MEMORY; STOCK MARKET; STREAM PROCESSING; STREAM PROCESSING SYSTEMS; STREAMING APPLICATIONS;

COMMERCE; DATA PROCESSING; QUEUEING NETWORKS;

COMMUNICATION SYSTEMS;

EID: 84880191592     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/WHPCF.2010.5671828     Document Type: Conference Paper

References (30)

1. X. J. Zhang, H. Andrade, B. Gedik, R. King, J. Morar, S. Nathan, Y. Park, R. Pavuluri, E. Pring, R. Schnier, P. Selo, M. Spicer, V. Uhlig, and C. Venkatramani, "Implementing a high-volume, low-latency market data processing system on commodity hardware using IBM middleware," in Proceedings of the Workshop on High Performance Computational Finance (WHPCF), 2009.
2. "Options Price Reporting Authority (OPRA)," http://www. opradata.com.
3. J. Liu, W. Jiang, P. Wyckoff, D. K. Panda, D. Ashton, W. Gropp, D. Buntinas, and B. Toonen, "Design and implementation of MPICH2 over InfiniBand with RDMA support," in IEEE International Parallel and Distributed Processing Symposium (IPDPS), 2004.
4. H. Subramoni, P. Lai, M. Luo, and D. K. Panda, "RDMA over Ethernet-A preliminary study," in IEEE International Conference on Cluster Computing (Cluster), 2009.
5. S. Narravula, A. R. Mamidata, A. Vishnu, G. Santhanaraman, and D. K. Panda, "High performance MPI over iWARP," in International Conference on Parallel Processing (ICPP), 2007.
6. E. P. Mancini, G. Marsh, and D. K. Panda, "An MPI-stream hybrid programming model for computational clusters," in IEEE International Symposium on Cluster Computing and the Grid (CCGrid), 2010.
7. G. Jacques-Silva, J. Challenger, L. Degenaro, J. Giles, and R. Wagle, "Towards autonomic fault recovery in System S," in IEEE International Conference on Autonomic Computing (ICAC), 2007.
8. N. Jain, L. Amini, H. Andrade, R. King, Y. Park, P. Selo, and C. Venkatramani, "Design, implementation, and evaluation of the Linear Road Benchmark on the Stream Processing Core," in ACM International Conference on Management of Data (SIGMOD), 2006.
9. J. Wolf, N. Bansal, K. Hildrum, S. Parekh, D. Rajan, R. Wagle, and K.-L. Wu, "SODA: An optimizing scheduler for large-scale stream-based distributed computer systems," in ACM/IFIP/USENIX International Middleware Conference (Middleware), 2008.
10. A. Arasu, B. Babcock, S. Babu, M. Datar, K. Ito, R. Motwani, I. Nishizawa, U. Srivastava, D. Thomas, R. Varma, and J. Widom, "STREAM: The Stanford stream data manager," IEEE Data Engineering Bulletin, vol. 26, no. 1, 2003.
11. D. J. Abadi, Y. Ahmad, M. Balazinska, U. Cetintemel, M. Cherniack, J.-H. Hwang, W. Lindner, A. S. Maskey, A. Rasin, E. Ryvkina, N. Tatbul, Y. Xing, and S. Zdonik, "The design of the Borealis stream processing engine," in Conference on Innovative Data Systems Research Conference (CIDR), 2005.
12. "StreamBase Systems," http://www.streambase.com.
13. S. Chandrasekaran, O. Cooper, A. Deshpande, M. J. Franklin, J. M. Hellerstein, W. Hong, S. Krishnamurthy, S. R. Madden, V. Raman, F. Reiss, and M. A. Shah, "TelegraphCQ: Continuous dataflow processing for an uncertain world," in Conference on Innovative Data Systems Research (CIDR), 2003.
14. A. P. Black, J. Huang, R. Koster, J. Walpole, and C. Pu, "Infopipes: An abstraction for multimedia streaming," Springer-Verlag Multimedia Systems, vol. 8, no. 5, 2002.
15. M. Hirzel, H. Andrade, B. Gedik, V. Kumar, G. Losa, M. Mendell, H. Nasgaard, R. Soule, and K.-L. Wu, "SPADE language specification," IBM, Tech. Rep. RC24830, 2009.
16. B. Gedik, H. Andrade, and K.-L. Wu, "A code generation approach to optimizing high-performance distributed data stream processing," in ACM Conference on Information and Knowledge Management (CIKM), 2009.
17. M. Snir, S. W. Otto, D. W. Walker, J. Dongarra, and S. Huss-Lederman, MPI: The Complete Reference. Cambridge, MA, USA: MIT Press, 1995.
18. W. Gropp, S. Huss-Lederman, A. Lumsdaine, E. Lusk, B. Nitzberg, W. Saphir, and M. Snir, MPI-The Complete Reference: Volume 2, The MPI-2 Extensions. Cambridge, MA, USA: MIT Press, 1998.
19. N.-B. C. Laboratory, "MVAPICH/MVAPICH2: MPI-1/MPI-2 for InfiniBand and iWARP with OpenFabrics," http://mvapich. cse.ohio-state.edu, 2008.
20. E. Gabriel, G. E. Fagg, G. Bosilca, T. Angskun, J. J. Dongarra, J. M. Squyres, V. Sahay, P. Kambadur, B. Barrett, A. Lumsdaine, R. H. Castain, D. J. Daniel, R. L. Graham, and T. S. Woodall, "Open MPI: Goals, concept, and design of a next generation MPI implementation," in Proceedings, 11th European PVM/MPI Users' Group Meeting, Budapest, Hungary, September 2004, pp. 97-104.
21. S. Sur, H. W. Jin, L. Chai, and D. K. Panda, "RDMA read based rendezvous protocol for MPI over InfiniBand: Design alternatives and benefits," in ACM Symposium on Principles and Practice of Parallel Programming (PPoPP), 2006.
22. S. Froehlich, M. Hack, X. Meng, and L. Zhang, "Achieving precise coordinated cluster time in a cluster environment," in IEEE International Symposium on Precision Clock Synchronization for Measurement, Control, and Communication (ISPCS), 2008.
23. L. Zhang, Z. Liu, and C. H. Xia, "Clock synchronization algorithms for network measurements," in IEEE Conference on Computer Communications (Infocom), 2002.
24. "Red Hat Enterprise Linux 5," http://www.redhat.com/rhel.
25. "FAST Protocol," http://www.fixprotocol.org/fast.
26. "IBM Cell Broadband Engine," http://www-03.ibm.com/technology/ cell.
27. "InfiniBand Trade Association," http://www.infinibandta.org.
28. J. Liu, J. Wu, and D. K. Panda, "High performance RDMAbased MPI implementation over InfiniBand," in ACM International Conference on Supercomputing (ICS), 2003.
29. "IBM WebSphere MQ Low Latency Messaging," http://www-01.ibm. com/software/integration/wmq/llm.
30. M. Schlansker, Nagabhushan, E. Oertli, P. M. S. Jr., L. Rankin, D. Bradford, R. J. Carter, J. Mudigonda, N. Binkert, and N. P. Jouppi, "High-performance Ethernetbased communications for future multi-core processors," in ACM International Conference for High Performance Computing, Networking, Storage, and Analysis (SC), 2007.