Storage optimization for large-scale distributed stream-processing systems

ACM Transactions on Storage

Volumn 3, Issue 4, 2008, Pages

  Hildrum, Kirsten ^a   Douglis, Fred ^a   Wolf, Joel L ^a   Yu, Philip S ^a   Fleischer, Lisa ^b   Katta, Akshay ^c  

^a IBM T J WATSON RESEARCH CENTER   (United States)

^b DARTMOUTH COLLEGE   (United States)

^c AMAZON   (United States)

Author keywords

File assignment problem; Load balancing; Optimization; Storage management; Streaming systems; Theory

Indexed keywords

FILE ASSIGNMENT PROBLEM; RETENTION VALUE FUNCTIONS; STORAGE MANAGEMENT; STREAMING SYSTEMS;

DISTRIBUTED COMPUTER SYSTEMS; FUNCTION EVALUATION; PROBLEM SOLVING; RESOURCE ALLOCATION;

DATA STORAGE EQUIPMENT;

EID: 40749098631     PISSN: 15533077     EISSN: 15533093     Source Type: Journal    
DOI: 10.1145/1326542.1326547     Document Type: Article

References (36)

1. ABADI, D. J., AHMAD, Y., BALAZINSKA, M., CENTINTEMEL, U., CHERNIACK, M., HWANG, J.-H., LINDNER, W., MASKEY, A. S., RASIN, A., RYVKINA, E., TATBUL, N., XING, Y., AND ZDONIK, S. 2005. The design of the Borealis stream processing engine. In Proceedings of the 2nd Biennial Conference on Innovative Data Systems Research (CIDR).
2. ABD-EL-MALEK, M., II, W. V. C., CRANOR, C., GANGER, G. R., HENDRICKS, J., KLOSTERMAN, A. J., MESNIER, M., PRASAD, M., SALMON, B., SAMBASIVAN, R. R., SINNAMOHIDEEN, S., STRUNK, J. D., THERESKA, E., WACHS, M., AND WYLIE, J. J. 2005. Ursa minor: Versatile cluster-based storage. In Proceedings of the USENIX Conference on File and Storage Technologies (FAST).
3. AHUJA, R., MAGNANTI, T., AND ORLIN, J. 1993. Network Flows. Prentice Hall.
4. ALVAREZ, G. A., BOROWSKY, E., GO, S., ROMER, T. H., BECKER-SZENDY, R., GOLDING, R., MERCHANT, A., SPASOJEVIC, M., VEITCH, A., AND WILKES, J. 2001. Minerva: An automated resource provisioning tool for large-scale storage systems. ACM Trans. Comput. Syst. 19, 4, 483-518.
5. AMINI, L., JAIN, N., SEHGAL, A., SILBER, J., AND VERSCHEURE, O. 2006. Adaptive control of extreme-scale stream processing systems. In Proceedings of IEEE International Conference on Distributed Computing Systems (ICDCS).
6. BENT, J., THAIN, D., ARPACI-DUSSEAU, A. C., ARPACI-DUSSEAU, R. H., AND LIVNY, M. 2004. Explicit control in the batch-aware distributed file system. In Proceedings of the ACMIUSENIX Symposium on Networked System Design and Implementation (NSDI). 365-378.
7. BERTSIMAS, D. AND TSITSIKLIS, J. 1997. Introduction to Linear Optimization. Athena Scientific.
8. BHAGWAN, R., DOUGLIS, F., HILDRUM, K., KEPHART, J. O., AND WALSH, W E. 2005. Time-Varying management of data storage. In 1st Workshop on Hot Topics in System Dependability.
9. BRANSON, M., DOUGLIS, F., FAWCETT, B., LIU, Z., RIABOV, A., AND YE, F. 2007. Autonomic operations in cooperative stream processing systems. In Proceedings of the 2nd Workshop on Hot Topics in Autonomic Computing. To appear.
10. CHANDRASEKARAN, S., COOPER, O., DESHPANDE, A., FRANKLIN, M. J., HELLERSTEIN, J. M., HONG, W., KRISHNAMURTHY, S., MADDEN, S. R., REISS, F., AND SHAH, M. A. 2003. TelegraphCQ: Continuous dataflow processing. In Proceedings of the ACM SIGMOD International Conference on Management of Data, 668-668.
11. DABEK, F., KAASHOEK, M. F., KARGER, D., MORRIS, R., AND STOICA, I. 2001. Wide-Area cooperative storage with CFS. In Proceedings of the 18th ACM Symposium on Operating Systems Principles (SOSP), Chateau Lake Louise, Banff, Canada.
12. DOUGLIS, F., BRANSON, M., HILDRUM, K., RONG, B., AND YE, F. 2006. Multi-Site cooperative data stream analysis. Oper. Syst. Rev. 40, 3, 31-37.
13. DOUGLIS, F., PALMER, J., RICHARDS, E. S., TAO, D., TETZLAFF, W. H., TRACEY, J. M., AND YIN, J. 2004. Position: Short object lifetimes require a delete-optimized storage system. In Proceedings of the 11th ACM SIGOPS European Workshop.
14. DOWDY, L. W. AND FOSTER, D. V. 1982. Comparative models of the file assignment problem. ACM Comput. Surv. 14, 2, 287-313.
15. FORREST, J. 2006. CLP- COIN-OR linear program solver. http://www.coin-or.org/Clp/index.html.
16. GRAHAM, R. 1969. Bounds on multiprocessor timing anomalies. SIAM J. Appl. Math. 17, 2.
17. HILDRUM, K., DOUGLIS, F., FLEISCHER, L., KATTA, A., WOLF, J. L., AND YU, P.S. 2006. Storage optimization for large-scale distributed stream processing systems. In Workshop on System Management Tools for Large-Scale Parallel Systems.
18. HUNTER, D. 1980. Modeling real DASD configurations. IBM Res. Rep. RC 8606.
19. JACQUES-SILVA, G., CHALLENGER, J., DEGENARO, L., GILES, J., AND WAGLE. 2007. Towards autonomic fault recovery in system-S. In Proceedings of the 4th IEEE International Conference an Autonomic Computing. To appear.
20. KNUTH, D. E. 1973. The Art of Computer Programming, Volume 3. Addison-Wesley.
21. LAVENBERG, S., Ed. 1983. Computer Performance Modeling Handbook. Academic Press.
22. LEE, L.-W., SCHEUERMANN, P., AND VINGRALEK, R. 2000. File assignment in parallel I/O systems with minimal variance of service time. IEEE Trans. Comput. 49, 2, 127-140.
23. LOUGEE-HEIMER, R., BARAHONA, F., DIETRICH, B., FASANO, J. P., FORREST, J., HARDER, R., LADANYI, L., PFENDER, T., RALPHS, SALTZMAN, M., AND SCHIENBERG, K. 2001. The COIN-OR initiative: Open-Source software accelerates operations research progress. ORMS Today 28, 5, 20-22.
24. MARCH, S. AND RHO, S. 1995. Allocating data and operations to nodes in distributed database design. IEEE Trans. Knowl. Data Eng. 7, 2.
25. PATTIPATI, K., WOLF, J., AND DEB, S. 1992. A calculus of variations approach to file allocation problems in computer systems. In Proceedings of the ACM SIGMETRICS Joint International Conference on Measurement and Modeling of Computer Systems.
26. PEREZ-DAVILA, A. AND DOWDY, L. 1984. Parameter interdependencies of file placement models in a Unix system. In Proceedings of the ACM SIGMETRICS Joint International Conference on Measurement and Modeling of Computer Systems.
27. PIETZUCH, P., LEDLIE, J., SHNEIDMAN, J., ROUSSOPOULOS, M., WELSH, M., AND SELTZER, M. 2006. Network-Aware operator placement for stream-processing systems. In Proceedings of the 22nd International Conference on Data Engineering (ICDE).
28. REPANTIS., T., GU., X., AND KALOGERAKI, V. 2006. Synergy: Sharing-Aware component composition for distributed stream processing systems. In. Proceedings of the ACM/IFIP/USENIX 7th International Middleware Conference, 322-341.
29. ROWSTRON, A. AND DRUSCHEL, P. 2001. Storage management and caching in PAST, a large-scale, persistent peer-to-peer storage utility. In Proceedings of the 18th ACM Symposium on Operating Systems Principles (SOSP). ACM Press, New York, 188-201.
30. STONEBRAKER, M., ÇETINTEMEL, U., AND ZDONIK, S. B. 2005. The 8 requirements of real-time stream processing. SIGMOD Rec. 34, 4, 42-47.
31. STREAMBASE SYSTEMS. 2007. Streambase. http://www.streambase.com/.
32. THE STREAM GROUP. 2003. STREAM: The Stanford stream data manager. IEEE Data Eng. Bull. 26, 1.
33. WAH, B. 1984. File placement on distributed computer systems. Comput. 17, 1, 23-32.
34. WOLF, J. 1989. The placement optimization problem. In Proceedings of the ACM SIGMETRICS Joint International Conference on Measurement and Modeling of Computer Systems and Performance.
35. ZDONIK, S., STONEBRAKER, M., CHERNIACK, M., CETINTEMEL, U., BALAZINSKA, M., AND BALAKRISHNAN, H. 2003. The Aurora and Medusa projects. IEEE Data Eng. Bull. 26, 1.
36. ZIPF, G. K. 1949. Human Behavior and the Principle of Least Effort. Addison-Wesley.