Evaluation of a high-volume, low-latency market data processing system implemented with IBM middleware

Software - Practice and Experience

Volumn 42, Issue 1, 2012, Pages 37-56

  Park, Yoonho ^a   King, Richard ^a   Nathan, Senthil ^a   Most, Wesley ^a   Andrade, Henrique ^a  

^a IBM T J WATSON RESEARCH CENTER   (United States)

Author keywords

commodity hardware; IBM middleware; market data processing; performance optimizations

Indexed keywords

ALGORITHMIC TRADING; COMMODITY HARDWARE; CURRENT SYSTEM; DATA VOLUME; HARDWARE AND SOFTWARE; HARDWARE PLATFORM; LOW LATENCY; MARKET DATA; MARKET SURVEILLANCE; MARKET-MAKER; PERFORMANCE EVALUATION; PERFORMANCE OPTIMIZATIONS; SPECIALIZED SOFTWARE; STOCK MARKET; SYSTEM OPTIMIZATIONS; TRANSPORT TECHNOLOGY;

DATA HANDLING; HARDWARE; MIDDLEWARE; OPTIMIZATION; RISK ANALYSIS; WIRELESS SENSOR NETWORKS;

COMMERCE;

EID: 84855222517     PISSN: 00380644     EISSN: 1097024X     Source Type: Journal    
DOI: 10.1002/spe.1047     Document Type: Article

References (44)

1. Feldman M,. High-frequency traders get boost from FPGA acceleration. HPCWire, June 2007.
2. MacSweeny G,. Low-latency spending moves full speed ahead. Wall Street and Technology, December 2008.
3. Amini L, Jain N, Sehgal A, Silber J, Verscheure O,. Adaptive control of extreme-scale stream processing systems. Proceedings of the International Conference on Distributed Computing Systems (ICDCS 2006), Lisboa, Portugal, 2006.
4. Jacques-Silva G, Challenger J, Degenaro L, Giles J, Wagle R,. Towards autonomic fault recovery in System S. Proceedings of the International Conference on Autonomic Computing (ICAC 2007), Jacksonville, FL, U.S.A., 2007.
5. Jain N, Amini L, Andrade H, King R, Park Y, Selo P, Venkatramani C,. Design, implementation, and evaluation of the Linear Road Benchmark on the Stream Processing Core. Proceedings of the International Conference on Management of Data (SIGMOD 2006), Chicago, IL, U.S.A., 2006.
6. Wolf J, Bansal N, Hildrum K, Parekh S, Rajan D, Wagle R, Wu K-L,. SODA: An optimizing scheduler for large-scale stream-based distributed computer systems. Proceedings of the International Middleware Conference (Middleware 2008), Leuven, Belgium, 2008.
7. Wu K-L, Yu PS, Gedik B, Hildrum K, Aggarwal CC, Bouillet E, Fan W, George D, Gu X, Luo G, Wang H,. Challenges and experience in prototyping a multi-modal stream analytic and monitoring application on System S. Proceedings of the International Conference on Very Large Data Bases (VLDB 2007), Vienna, Austria, 2007.
8. Zhang XJ, Andrade H, Gedik B, King R, Morar J, Nathan S, Park Y, Pavuluri R, Pring E, Schnier R, Selo P, Spicer M, Uhlig V, Venkatramani C,. Implementing a high-volume, low-latency market data processing system on commodity hardware using IBM middleware. Proceedings of the Workshop on High Performance Computational Finance (WHPCF 2009), Portland, OR, U.S.A., 2009.
9. IBM WebSphere MQ Low Latency Messaging. Available at: [17, January 2011 ].
10. Andrade H, Gedik B, Wu K-L, Yu PS,. Scale-up strategies for processing high-rate data streams in System S. Proceedings of the IEEE International Conference on Data Engineering (ICDE 2009), Shanghai, China, 2009.
11. Gedik B, Andrade H, Wu K-L, Yu PS, Doo M,. SPADE: The System S declarative stream processing engine. Proceedings of the International Conference on Management of Data (SIGMOD 2008), Vancouver, BC, Canada, 2008.
12. Hirzel M, Andrade H, Gedik B, Kumar V, Losa G, Mendell M, Nasgaard H, R. Soulé, Wu K-L,. SPADE language specification. Technical Report RC24830, IBM, 2009.
13. Options Price Reporting Authority (OPRA). Available at: [17 January 2011 ].
14. Agarwal V, Bader DA, Dan L, Liu L-K, Pasetto D, Perrone M, Petrini F,. Faster FAST: Multicore acceleration of streaming financial data. Computer Science-Research and Development 2009; 23 (3-4): 249-257.
15. IBM System x3950 M2. Available at: [17 January 2011 ].
16. IBM BladeCenter Blade Servers. Available at: [17 January 2011 ].
17. Infiniband Trade Association. Available at: [17 January 2011 ].
18. IBM Blue Gene. Available at: [17 January 2011 ].
19. IBM Cell Broadband Engine. Available at: [17 January 2011 ].
20. IBM z/OS. Available at: [17 January 2011 ].
21. Security Industry Automation Corporation (SIAC). Available at: [17 January 2011 ].
22. FAST Protocol. Available at: [17 January 2011 ].
23. Froehlich S, Hack M, Meng X, Zhang L,. Achieving precise coordinated cluster time in a cluster environment. Proceedings of the Symposium on Precision Clock Synchronization for Measurement, Control and Communication (ISPCS 2008), Ann Arbor, MI, U.S.A., 2008.
24. Zhang L, Liu Z, Xia CH,. Clock synchronization algorithms for network measurements. Proceedings of the Conference on Computer Communications (INFOCOM 2002). IEEE, 2002.
25. Pantor Press Release. Available at: [17 January 2011 ].
26. Celoxica Market Data Line Handlers. Available at: [17 January 2011 ].
27. Redline Trading Solutions Products. Available at: [17 January 2011 ].
28. Klug T, Ott M, Weidendorfer J, Trinitis C,. autopin -Automated optimization of thread-to-core pinning on multicore systems. Transactions on High-Performance Embedded Architectures and Compilers 2008; 3 (4). Available at: [17 January 2011 ].
29. Brosky S, Rotolo S,. Shielded processors: Guaranteeing sub-millisecond response in standard Linux. Proceedings of the International Parallel and Distributed Processing Symposium (IPDPS 2003), Nice, France 2003.
30. The OpenFabrics Alliance. Available at: [17 January 2011 ].
31. Gedik B, Andrade H, Frenkiel A, Pauw WD, Pfeifer M, Allen P, Cohen N, Wu K-L,. Debugging tools and strategies for distributed stream processing applications. Software: Practice and Experience 2009; 39 (16): 1347-1376.
32. Petrini F, Kerbyson DJ, Pakin S,. The case of the missing supercomputer performance: Achieving optimal performance on the 8192 processors of ASCI Q. Proceedings of the Conference on High Performance Networking and Computing (SC 2003), Phoenix, AZ, U.S.A., 2003
33. Red Hat Enterprise Linux 5. Available at: [17 January 2011 ].
34. Gedik B, Andrade H, Wu K-L,. A code generation approach to optimizing high-performance distributed data stream processing. Proceedings of the 2009 Conference on Information and Knowledge Management (CIKM 2009), Hong Kong, China, 2009.
35. Wolf J, Khandekar R, Hildrum K, Parekh S, Rajan D, Wu K-L, Andrade H, Gedik B,. COLA: Optimizing stream processing applications via graph partitioning. Proceedings of the International Middleware Conference (Middleware 2009), Urbana-Champaign, IL, U.S.A., 2009.
36. Arasu A, Babcock B, Babu S, Datar M, Ito K, Motwani R, Nishizawa I, Srivastava U, Thomas D, Varma R, Widom J,. STREAM: The Stanford stream data manager. IEEE Data Engineering Bulletin 2003; 26 (1): 19-26.
37. Abadi DJ, Ahmad Y, Balazinska M, Cetintemel U, Cherniack M, Hwang J-H, Lindner W, Maskey AS, Rasin A, Ryvkina E, Tatbul N, Xing Y, Zdonik S,. The design of the Borealis stream processing engine. Proceedings of the Innovative Data Systems Research Conference (CIDR 2005), Asilomar, CA, U.S.A., 2005.
38. StreamBase Systems. Available at: [17 January 2011 ].
39. Chandrasekaran S, Cooper O, Deshpande A, Franklin MJ, Hellerstein JM, Hong W, Krishnamurthy S, Madden SR, Raman V, Reiss F, Shah MA,. TelegraphCQ: Continuous dataflow processing for an uncertain world. Proceedings of the Innovative Data Systems Research Conference (CIDR 2003), Asilomar, CA, U.S.A., 2003.
40. Jones T, Dawson S, Neely R, Tuel W, Brenner L, Fier J, Blackmore R, Caffrey P, Maskell B, Tomlinson P, Roberts M,. Improving the scalability of parallel jobs by adding parallel awareness to the operating system. Proceedings of the Conference on High Performance Networking and Computing (SC 2003), Phoenix, AZ, U.S.A., 2003.
41. Sottile MJ, Minnich R,. Analysis of microbenchmarks for performance tuning of clusters. Proceedings of the International Conference on Cluster Computing (CLUSTER 2004), San Diego, CA, U.S.A., 2004.
42. Nataraj A, Morris A, Malony AD, Sottile MJ, Beckman PH,. The ghost in the machine: Observing the effects of kernel operation on parallel application performance. Proceedings of the Conference on High Performance Networking and Computing (SC 2007), Reno, NV, U.S.A., 2007.
43. Ferreira KB, Bridges PG, Brightwell R,. Characterizing application sensitivity to OS interference using kernel-level noise injection. Proceedings of the Conference on High Performance Networking and Computing (SC 2008), Austin, TX, U.S.A., 2008.
44. Kim J-S, Andrade H, Sussman A,. Principles for designing data-/compute-intensive distributed applications and middleware systems for heterogeneous environments. Journal of Parallel Distributed Computing 2007; 67 (7): 755-771. (Pubitemid 46843490)