SODA: An optimizing scheduler for large-scale stream-based distributed computer systems

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Volumn 5346 LNCS, Issue , 2008, Pages 306-325

  Wolf, Joel ^a   Bansal, Nikhil ^a   Hildrum, Kirsten ^a   Parekh, Sujay ^a   Rajan, Deepak ^a   Wagle, Rohit ^a   Wu, Kun Lung ^a   Fleischer, Lisa ^b  

^a IBM T J WATSON RESEARCH CENTER   (United States)

^b DARTMOUTH COLLEGE   (United States)

Author keywords

Admission control; Flow balancing; Scheduling; Stream processing

Indexed keywords

COMPUTATIONAL METHODS; COMPUTER HARDWARE DESCRIPTION LANGUAGES; COMPUTER RESOURCE MANAGEMENT; COMPUTER SOFTWARE; DATA PROCESSING; DISTRIBUTED PARAMETER CONTROL SYSTEMS; HEURISTIC ALGORITHMS; MIDDLEWARE; PLANNING; RESOURCE ALLOCATION; SCHEDULING; WIRELESS LOCAL AREA NETWORKS (WLAN); ACCESS CONTROL; DATA HANDLING; OPTIMIZATION;

ADMISSION CONTROL; COMPLEX OPTIMIZATION PROBLEMS; DATA RATES; DISTRIBUTED STREAM PROCESSING; EXACT ALGORITHMS; FLOW BALANCING; PROBLEM DECOMPOSITIONS; RESOURCE AVAILABILITIES; STREAM PROCESSING; STREAMING APPLICATIONS;

DISTRIBUTED COMPUTER SYSTEMS;

COMPLEX OPTIMIZATION PROBLEMS; DISTRIBUTED STREAM PROCESSING; EXACT ALGORITHMS; FLOW BALANCING; PROBLEM DECOMPOSITION; RESOURCE AVAILABILITY; STREAM PROCESSING; STREAMING APPLICATIONS;

EID: 58049113792     PISSN: 03029743     EISSN: 16113349     Source Type: Book Series    
DOI: 10.1007/978-3-540-89856-6_16     Document Type: Conference Paper

References (23)

1. Amini, L., Andrade, H., Bhagwan, R., Eskesen, F., King, R., Selo, P., Park, Y., Venkatramani, C.: SPC: A distributed, scalable platform for data mining. In: International Workshop on Data Mining Standards, Services and Platforms (2006)
2. Douglis, F., Palmer, J., Richards, E., Tao, D., Tetzlaff, W., Tracey, J., Yin, J.: Position: Short object lifetimes require a delete-optimized storage system. In: ACM SIGOPS European Workshop (2004)
3. Hildrum, K., Douglis, F., Wolf, J., Yu, P.S., Fleischer, L., Katta, A.: Storage optimization for large-scale stream processing systems. In: ACM Transactions on Storage (2008)
4. Jain, N., Amini, L., Andrade, H., King, R., Park, Y., Selo, P., Venkatramani, C.: Design, implementation and evaluation of the linear road benchmark on the stream processing core. In: ACM SIGMOD International Conference on Management of Data (2006)
5. Jacques-Silva, G., Challenger, J., Degenaro, L., Giles, J., Wagle, R.: Towards autonomic fault recovery in System-S. In: International Conference on Autonomic Computing (2007)
6. Wu, K.-L., Yu, P.S., Gedik, B., Hildrum, K.W., Aggarwal, C.C., Bouillet, E., Fan, W., George, D.A., Gu, X., Luo, G., Wang, H.: Challenges and experience in prototyping a multi-modal stream analytic and monitoring application on System S. In: International Conference on Very Large Data Bases (2007)
7. Abadi, D.J., Ahmad, Y., Balazinska, M., Cetintemel, U., Cherniack, M., Hwang, J.H., Lindner, W., Maskey, A.S., Rasin, A., Ryvkina, E., Tatbul, N., Xing, Y., Zdonik, S.: The design of the Borealis stream processing engine. In: Conference on Innovative Data Systems Research (2005)
8. Chandrasekaran, S., Cooper, O., Deshpande, A., Franklin, M.J., Hellerstein, J.M., Hong, W., Krishnamurthy, S., Madden, S.R., Raman, V., Reiss, F., Shah, M.A.: TelegraphCQ: Continuous dataflow processing for an uncertain world. In: Conference on Innovative Data Systems Research (2003)
9. Arasu, A., Babcock, B., Babu, S., Datar, M., Ito, K., Motwani, R., Nishizawa, I., Srivastava, U., Thomas, D., Varma, R., Widom, J.: STREAM: The Stanford stream data manager. IEEE Data Engineering Bulletin 26 (2003)
10. Zdonik, S., Stonebraker, M., Cherniack, M., Cetintemel, U., Balazinska, M., Bal-akrishnan, H.: The Aurora and Medusa projects. IEEE Data Engineering Bulletin 26(1) (2003)
11. Wolf, J., Bansal, N., Hildrum, K., Parekh, S., Rajan, D., Wagle, R., Wu, K.L., Fleischer, L.: Scheduling optimizer for distributed applications: A reference paper. Technical Report 24453, IBM Research Report (2007)
12. Tatbul, N., Çetintemel, U., Zdonik, S.: Staying fit: Efficient load shedding techniques for distributed stream processing. In: International Conference on Very Large Data Bases, pp. 159-170 (2007)
13. Cormen, T., Leiserson, C., Rivest, R.: Introduction to Algorithms. McGraw-Hill, New York (1985)
14. Blazewicz, J., Ecker, K., Schmidt, G., Weglarz, J.: Scheduling in Computer and Manufacturing Systems. Springer, Heidelberg (1993)
15. Ibaraki, T., Katoh, N.: Resource Allocation Problems. MIT Press, Cambridge (1988)
16. Nemhauser, G.L., Wolsey, L.A.: Integer and Combinatorial Optimization. John Wiley and Sons, New York (1988)
17. ILOG: CPLEX, http://www.ilog.com/products/cplex
18. Xing, Y., Hwang, J.H., Çetintemel, U., Zdonik. S.: Providing resiliency to load variations in distributed stream processing. In: International Conference on Very Large Data Bases, VLDB Endowment, pp. 775-786 (2006)
19. Xing, Y., Zdonik, S., Hwang, J.H.: Dynamic load distribution in the Borealis stream processor. In: IEEE International Conference on Data Engineering, Washington, DC, USA, pp. 791-802. IEEE Computer Society, Los Alamitos (2005)
20. Pietzuch, P., Ledlie, J., Shneidman, J., Roussopoulos, M., Welsh, M., Seltzer, M.: Network-aware operator placement for stream-processing systems. In: IEEE International Conference on Data Engineering, Washington, DC, USA. IEEE Computer Society, Los Alamitos (2006)
21. Lakshmanan, G.T., Strom, R.E.: Biologically-Inspired Distributed Middleware Management for Stream Processing Systems. In: Issarny, V., Schantz, R. (eds.) Middleware 2008. LNCS, vol. 5346, pp. 223-242. Springer, Heidelberg (2008)
22. Motwani, R., Widom, J., Arasu, A., Babcokc, B., Babu, S., Datar, M., Manku, G., Olston, C., Rosenstein, J., Varma, R.: Query processing, approximation, and resource management in a data stream management system. In: Conference on Innovative Data Systems Research (2003)
23. Xia, C.H., Towsley, D., Zhang, C.: Distributed resource management and admission control of stream processing systems with max utility. In: ICDCS 2007: Proceedings of the 27th International Conference on Distributed Computing Systems (2007)