Improved file synchronization techniques for maintaining large replicated collections over slow networks

Proceedings - International Conference on Data Engineering

Volumn 20, Issue , 2004, Pages 153-164

  Suel, Torsten ^a   Noel, Patrick ^a   Trendafilov, Dimitre ^a  

^a POLYTECHNIC UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BANDWIDTH SAVINGS; FILE SYNCHRONIZATION; LARGE SCALE WEB SEARCHES; WEB SITE MONITORING;

BANDWIDTH; COMPACT DISKS; DISTRIBUTED COMPUTER SYSTEMS; INTERNET; PROBLEM SOLVING; SEARCH ENGINES; SERVERS;

FILE ORGANIZATION;

EID: 2442563450     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICDE.2004.1319992     Document Type: Conference Paper

References (49)

1. K. Abdel-Ghaffar and A. E. Abbadi. An optimal strategy for comparing file copies. IEEE Transactions on Parallel and Distributed Systems, 5(1):87-93, Jan. 1994.
2. S. Agarwal, D. Starobinski, and A. Trachtenberg. On the scalability of data synchronization protocols for PDAs and mobile devices. IEEE Network Magazine, special issue on Scalability in Communication Networks, July 2002.
3. S. Balasubramaniam and B. Pierce. What is a file synchronizer? In Proc. of the ACM/IEEE MOBICOM'98 Conference, pages 98-108, Oct. 1998.
4. D. Barbara and R. Lipton. A class of randomized strategies for low-cost comparison of file copies. IEEE Transactions on Parallel and Distributed Systems, 2(2):160-170, Apr. 1991.
5. O. Brandman, J. Cho, H. Garcia-Molina, and N. Shivakumar. Crawler-friendly web servers. In Proc. of the Workshop on Performance and Architecture of Web Servers (PAWS), June 2000.
6. B. Brewington and G. Cybenko. Keeping up with the changing web. IEEE Computer, 33(5), May 2000.
7. J. Cho and H. Garcia-Molina. The evolution of the web and implications for an incremental crawler. In Proc. of 26th Int. Conf. on Very Large Data Bases, pages 117-128, Sept. 2000.
8. J. Cho and H. Garcia-Molina. Synchronizing a database to improve freshness. In. Proc. of the ACM SIGMOD Int. Conf. on Management of Data, pages 117-128, May 2000.
9. J. Cho and A. Ntoulas. Effective change detection using sampling. In Proc. of the 28th Int. Conf. on Very Large Databases, pages 514-525, Sept. 2002.
10. G. Cormode, M. Paterson, S. Sahinalp, and U. Vishkin. Communication complexity of document exchange. In Proc. of the ACM-SIAM Symp. on Discrete Algorithms, Jan. 2000.
11. L. Cox, C. Murray, and B. Noble. Pastiche: Making backup cheap and easy. In Proc. of the 5th Symp. on Operating System Design and Implementation, December 2002.
12. R. Dorfman. The detection of defective members in large population. Annals of Mathematical Statistics, 14:436-440, 1943.
13. F. Douglis, A. Feldmann, B. Krishnamurthy, and J. Mogul. Rate of change and other metrics: a live study of the World Wide Web. In Proc. of the USENIX Symp. on Internet Technologies and Systems (ITS-97), pages 147-158, Berkeley, Dec. 8-11 1997. USENIX Association.
14. J. Edwards, K. McCurley, and J. Tomlin. An adaptive model for optimizing performance of an incremental web crawler. In Proc. of the 10th International World Wide Web Conference, pages 106-113, May 2001.
15. A. Evfimievski. A probabilistic algorithm for updating files over a communication link. In Proc. of the 9th Annual ACM-SIAM Symposium on Discrete Algorithms, pages 300-305, Jan. 1998.
16. J. Fiedler and J. Hammer. Using the web efficiently: Mobile crawlers. In Proceedings of thw 17th Int. Conf. on Computer Science, San Diego, CA, 1999.
17. H. Garcia-Molina. Webbase: Building a web warehouse. In Plenary Talk at the 2003 Federated Computing Research Conference, June 2003.
18. J. Gray, W. Chong, T. Barclay, A. Szalay, and J. Vandenberg. Terascale sneakernet: Using inexpensive disks for backup, archiving, and data exchange. Technical Report MSR-TR-2002-54, Microsoft Research, May 2002.
19. Grub. Distributed Internet Crawler, http://www.grub.org/.
20. J. Hirai, S. Raghavan, H. Garcia-Molina, and A. Paepcke. WebBase: A repository of web pages. In Proc. of the 9th Int. World Wide Web Conference, May 2000.
21. J. Hunt, K.-P. Vo, and W. Tichy. Delta algorithms: An empirical analysis. ACM Transactions on Software Engineering and Methodology, 7, 1998.
22. R. Karp and M. Rabin. Efficient randomized pattern-matching algorithms. IBM Journal of Research and Development, 31(2):249-260, 1987.
23. D. Korn and K.-P. Vo. Engineering a differencing and compression data format. In Proceedings of the Usenix Annual Technical Conference, pages 219-228, June 2002.
24. E. Kushilevitz and N. Nisan. Communication Complexity. Cambridge University Press, 1997.
25. J. Langford. Multiround rsync. January 2001. Unpublished manuscript.
26. J. MacDonald. File system support for delta compression. MS Thesis, UC Berkeley, May 2000.
27. T. Madej. An application of group testing to the file comparison problem. In Proc. of the 9th Int. Conf. on Distributed Computing Systems, pages 237-243, June 1989.
28. J. Metzner. A parity structure for large remotely located replicated data files. IEEE Transactions on Computers, 32(8):727-730, Aug. 1983.
29. J. Metzner. Efficient replicated remote file comparison. IEEE Transactions on Computers, 40(5):651-659, May 1991.
30. Y. Minsky, A. Trachtenberg, and R. Zippel. Set reconciliation with almost optimal communication complexity. Technical Report TR2000-1813, Cornell University, 2000.
31. A. Muthitacharoen, B. Chen, and D. Mazières. A low-bandwidth network file system. In Proc. of the 18th ACM Symp. on Operating Systems Principles, pages 174-187, October 2001.
32. A. Orlitsky. Worst-case interactive communication II: Two messages are not optimal. IEEE Transactions on Information Theory, 37(4):995-1005, July 1991.
33. A. Orlitsky. Interactive communication of balanced distributions and of correlated files. SIAM Journal of Discrete Math, 6(4):548-564, 1993.
34. A. Orlitsky and K. Viswanathan. Practical algorithms for interactive communication. In IEEE Int. Symp. on Information Theory, June 2001.
35. A. Orlitsky and K. Viswanathan. One-way communication and error-correcting codes. In Proc. of the 2002 IEEE Int. Symp. on Information Theory, page 394, June 2002.
36. C. Park and J. J. Metzner. Efficient location of discrepancies in multiple replicated large files. IEEE Transactions on Parallel and Distributed Systems, 13(6):597-610, June 2002.
37. A. Pelc. Searching games with errors - fifty years of coping with liars. Theoretical Computer Science, 270(1-2):71-109, Jan. 2002.
38. C. Pyne. Remote file transfer method and apparatus, 1995. US Patent Number 5446888.
39. N. Ramsey and E. Csirmaz. An algebraic approach to file synchronization. In Proc. of the 9th ACM Int. Symp. on Foundations of Software Engineering, pages 175-185, 2001.
40. D. Rasch and R. Burns. In-place rsync: File synchronization for mobile and wireless devices. In Proc. of the USENIX Annual Technical Conference, June 2003.
41. S. Rhea, K. Liang, and E. Brewer. Value-based web caching. In Proc. of the 12th Int. World Wide Web Conference, May 2003.
42. T. Schwarz, R. Bowdidge, and W. Burkhard. Low cost comparison of file copies. In Proc. of the 10th Int. Conf. on Distributed Computing Systems, pages 196-202, 1990.
43. N. Spring and D. Wetherall. A protocol independent technique for eliminating redundant network traffic. In Proc. of the ACM SIGCOMM Conference, 2000.
44. D. Starobinski, A. Trachtenberg, and S. Agarwal. Efficient PDA synchronization. IEEE Transactions on Mobile Computing, 2(1), 2003.
45. T. Suel and N. Memon. Algorithms for delta compression and remote file synchronization. In K. Sayood, editor, Lossless Compression Handbook. Academic Press, 2002.
46. D. Trendafilov, N. Memon, and T. Suel. zdelta: a simple delta compression tool. Technical Report TR-CIS-2002-02, Polytechnic University, CIS Department, June 2002.
47. A. Tridgell. Efficient Algorithms for Sorting and Synchronization. PhD thesis, Australian National University, April 2000.
48. A. Tridgell and P. MacKerras. The rsync algorithm. Technical Report TR-CS-96-05, Australian National University, June 1996.
49. S. Ulam. Adventures of a Mathematician. Scribner, 1976.