Fault-tolerant stream processing using a distributed, replicated file system

Proceedings of the VLDB Endowment

Volumn 1, Issue 1, 2008, Pages 574-585

  Kwon, YongChul ^a   Balazinska, Magdalena ^a   Greenberg, Albert ^b  

^a UNIVERSITY OF WASHINGTON   (United States)

^b MICROSOFT RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DISTRIBUTED PARAMETER CONTROL SYSTEMS; FILE ORGANIZATION; SCHEDULING;

DISTRIBUTED STREAM PROCESSING; FAULT TOLERANCE TECHNIQUES; MEMORY RESOURCES; PROTOTYPE IMPLEMENTATIONS; RESOURCE CONTENTION; RESOURCE UTILIZATIONS; ROLL-BACK RECOVERIES; STREAM PROCESSING;

FAULT TOLERANCE;

EID: 77957854659     PISSN: None     EISSN: 21508097     Source Type: Conference Proceeding    
DOI: 10.14778/1453856.1453920     Document Type: Article

References (54)

1. Abadi et. al. Aurora: A new model and architecture for data stream management. VLDB Journal, 12(2), Sept. 2003.
2. Abadi et. al. The design of the Borealis stream processing engine. In Proc. of the Second CIDR Conf., Jan. 2005.
3. A. Agbaria and J. S. Plank. Design, implementation, and performance of checkpointing in NetSolve. In Proc. of the Int. Conf. on Dependable Systems and Networks (FTCS-30 & DCCA-8), pages 49-54, June 2000.
4. R. K. Ahuja, T. L. Magnanti, and J. B. Orlin. Network flows: theory, algorithms, and applications. Prentice-Hall, Inc., 1993.
5. Aleri. http://www.aleri.com/index.html.
6. M. Balazinska, H. Balakrishnan, S. Madden, and M. Stonebraker. Fault-tolerance in the Borealis distributed stream processing system. In Proc. of the 2005 SIGMOD Conf., June 2005.
7. R. Barga, J. Goldstein, M. Ali, and M. Hong. Consistent streaming through time: A vision for event stream processing. In Proc. of the Third CIDR Conf., Jan. 2007.
8. A. Beguelin, E. Seligman, and P. Stephan. Application level fault tolerance in heterogeneous networks of workstations. Journal of Parallel and Distributed Computing, 43(2):147-155, 1997.
9. D. Borthakur. The Hadoop distributed file system: Architecture and design. http://lucene.apache.org/hadoop/hdfs_design.pdf, 2007.
10. G. Bronevetsky, D. Marques, K. Pingali, and P. Stodghill. Automated application-level checkpointing of mpi programs. In In Proc. of the 9th ACM SIGPLAN symposium on Principles and practice of parallel programming, pages 84-94, New York, NY, USA, 2003. ACM.
11. Chandrasekaran et al. TelegraphCQ: Continuous dataflow processing for an uncertain world. In Proc. of the First CIDR Conf., Jan. 2003.
12. Y. S. Chang, S. Y. Cho, and B. Y. Kim. Performance evaluation of the striped checkpointing algorithm on the distributed RAID for cluster computer. In Int. Conf. on Computational Science, pages 955-962, 2003.
13. Chen et. al. High availability and scalability guide for DB2 on linux, unix, and windows. IBM Redbooks http://www.redbooks.ibm.com/redbooks/pdfs/sg247363.pdf, Sept. 2007.
14. Cherniack et. al. Scalable distributed stream processing. In Proc. of the First CIDR Conf., Jan. 2003.
15. Coral8. http://coral8.com/.
16. C. Cranor, T. Johnson, O. Spatscheck, and V. Shkapenyuk. Gigascope: A stream database for network applications. In Proc. of the 2003 SIGMOD Conf., June 2003.
17. 2 system. Communications of the ACM, 34(10):34-48, Oct. 1991.
18. E. N. M. Elnozahy, L. Alvisi, Y.-M. Wang, and D. B. Johnson. A survey of rollback-recovery protocols in message-passing systems. ACM Computing Survey, 34(3):375-408, 2002.
19. M.J. Carey. et. al. Shoring up persistent applications. SIGMOD Rec., 23(2):383-394, June 1994.
20. G. DeCandia et. al. Dynamo: Amazon's highly available key-value store. In Proc. of the 21st ACM Symposium on Operating Systems Principles, Oct. 2007.
21. H. Garcia-Molina and K. Salem. Main memory database systems: An overview. IEEE TKDE, 04(6):509-516, 1992.
22. S. Ghemawat, H. Gobioff, and S.-T. Leung. The google file system. In Proc. of the 19th ACM Symposium on Operating Systems Principles, pages 29-43. ACM Press, 2003.
23. A. Goldberg, A. Gopal, K. Li, R. Strom, and D. F. Bacon. Transparent recovery of Mach applications. In Proc. of the Usenix Mach Workshop, Oct. 1990.
24. J. H. Howard, M. L. Kazar, S. G. Menees, D. A. Nichols, M. Satyanarayanan, R. N. Sidebotham, and M. J. West. Scale and performance in a distributed file system. ACM Trans. Comput. Syst., 6(1):51-81, 1988.
25. J.-H. Hwang, M. Balazinska, A. Rasin, U. Çetintemel, M. Stonebraker, and S. Zdonik. High-availability algorithms for distributed stream processing. In Proc. of the 21st ICDE Conf., Apr. 2005.
26. J.-H. Hwang, Y. Xing, U. Çetintemel, and S. B. Zdonik. A cooperative, self-configuring high-availability solution for stream processing. In Proc. of the 23rd ICDE Conf., pages 176-185, Apr. 2007.
27. Jain et. al. Design, implementation, and evaluation of the linear road benchmark on the stream processing core. In Proc. of the 2006 SIGMOD Conf., June 2006.
28. H. Jiang and V. Chaudhary. Migthread: Thread migration in dsm systems. In Proc. of the 2002 Int. Conf. on Parallel Processing Workshops, page 581, 2002.
29. Kosmix Corp. Kosmos distributed file system(kfs). http://kosmosfs.sourceforge.net, 2007.
30. C. Lamb, G. Landis, J. Orenstein, and D. Weinreb. The objectstore database system. Communications of the ACM, 34(10):50-63, Oct. 1991.
31. L. Lamport. Paxos made simple. SIGACT News, 32(4):51-58, Dec. 2001.
32. J. Lee, K. Kim, and S. K. Cha. Differential logging: A commutative and associative logging scheme for highly parallel main memory database. In Proc. of the 17th ICDE Conf., Apr. 2001.
33. T. J. Lehman and M. J. Carey. A study of index structures for main memory database management systems. In Proc. of the 12th VLDB Conf., pages 294-303, Aug. 1986.
34. E. Levy and A. Silberschatz. Incremental recovery in main memory database systems. IEEE TKDE, 04(6):529-540, 1992.
35. K. Li, J. Naughton, and J. Plank. Low-latency, concurrent checkpointing for parallel programs. IEEE Trans. on Parallel and Distributed Systems, 05(8), 1994.
36. W. Litwin. Linear hashing: A new tool for file and table addressing. In Proc. of the 6th VLDB Conf., pages 212-223, Oct. 1980.
37. C. Mohan, D. Haderle, B. Lindsay, H. Pirahesh, and P. Schwarz. ARIES: a transaction recovery method supporting fine-granularity locking and partial rollbacks using write-ahead logging. ACM TODS, 17(1):94-162, 1992.
38. Motwani et. al. Query processing, approximation, and resource management in a data stream management system. In Proc. of the First CIDR Conf., Jan. 2003.
39. Network Working Group, Sun Microsystems, Inc. RFC 1094 -NFS: Network file system protocol specification. http://www.faqs.org/rfcs/rfc1094.html, 1989.
40. Oracle Corp. Oracle berkeley db. http://www.oracle.com/technology/products/berkeley-db/db/index.html, 2008.
41. J. S. Plank. Efficient Checkpointing on MIMD Architectures. PhD thesis, Princeton University, June 1993.
42. J. S. Plank, M. Beck, G. Kingsley, and K. Li. Libckpt: Transparent checkpointing under Unix. In Usenix Winter Technical Conference, pages 213-223, Jan. 1995.
43. A. Ray. Oracle data guard: Ensuring disaster recovery for the enterprise. An Oracle white paper, Mar. 2002.
44. J. C. Sancho, F. Petrini, G. Johnson, J. Fernandez, and E. Frachtenberg. On the feasibility of incremental checkpointing for scientific computing. In Proc. of the Int. Parallel and Distributed Proc. Symp., Apr. 2004.
45. F. B. Schneider. Implementing fault-tolerant services using the state machine approach: a tutorial. ACM Computing Surveys, 22(4):299-319, 1990.
46. M. Shah, J. Hellerstein, and E. Brewer. Highly-available, fault-tolerant, parallel dataflows. In Proc. of the 2004 SIGMOD Conf., June 2004.
47. Silicon Graphics, Inc. Standard template library programmer's guide. http://www.sgi.com/tech/stl/.
48. Streambase. http://www.streambase.com/.
49. R. Talmage. Database mirroring in SQL Server 2005. http://www.microsoft.com/technet/prodtechnol/sql/2005/dbmirror.mspx, Apr. 2005.
50. S. S. Vadhiyar and J. Dongarra. Srs: A framework for developing malleable and migratable parallel applications for distributed systems. Parallel Processing Letters, 13(2):291-312, 2003.
51. N. H. Vaidya. Staggered consistent checkpointing. IEEE Trans. on Parallel and Distributed Systems, 10(7), 1999.
52. VMware Inc. Vmware. http://www.vmware.com, 2008.
53. J. Walters and V. Chaudhary. Application-level checkpointing techniques for parallel programs. In Proc. of the 3rd ICDCIT Conf., pages 221-234, Dec. 2006.
54. S. J. White and D. J. Dewitt. Quickstore: a high performance mapped object store. VLDB Journal, 4(4):629-673, Oct. 1995.