Principled dictionary pruning for low-memory corpus compression

SIGIR 2014 - Proceedings of the 37th International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn , Issue , 2014, Pages 283-292

  Tong, Jiancong ^a,b   Wirth, Anthony ^b   Zobel, Justin ^b  

^a NANKAI UNIVERSITY   (China)

^b UNIVERSITY OF MELBOURNE   (Australia)

Author keywords

Corpus compression; Optimization; Retrieval efficiency; String algorithms

Indexed keywords

COMPRESSION RATIO (MACHINERY); EFFICIENCY; INFORMATION RETRIEVAL; OPTIMIZATION;

MEMORY BOUNDS; MEMORY HIERARCHY; RANDOM ACCESS; RETRIEVAL EFFICIENCY; SPACE CONSUMPTION; STRING ALGORITHMS; TEXT COLLECTION; TEXT DATABASE;

ECONOMIC AND SOCIAL EFFECTS;

EID: 84904582651     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2600428.2609576     Document Type: Conference Paper

References (32)

1. R. A. Baeza-Yates and B. A. Ribeiro-Neto. Modern Information Retrieval - The Concepts and Technology Behind Search, Second edition. Addison-Wesley, 2011.
2. S. Büttcher, C. L. A. Clarke, and G. V. Cormack. Information Retrieval - Implementing and Evaluating Search Engines. MIT Press, 2010.
3. A. Cannane and H. E. Williams. General-purpose compression for efficient retrieval. JASIST, 52(5):430-437, 2001.
4. A. Cannane and H. E. Williams. A general-purpose compression scheme for large collections. ACM Transactions on Information Systems, 20(3):329-355, 2002.
5. G. Chen, S. J. Puglisi, and W. F. Smyth. Lempel-Ziv factorization using less time & space. Mathematics in Computer Science, 1(4):605-623, 2008.
6. C. Clarke, N. Craswell, and I. Soboroff. Overview of the trec 2004 terabyte track. In TREC, 2004.
7. W. B. Croft, D. Metzler, and T. Strohman. Search Engines - Information Retrieval in Practice. Addison-Wesley, 2009.
8. P. Ferragina and G. Manzini. On compressing the textual web. In WSDM, pages 391-400, 2010.
9. C. Hoobin, S. J. Puglisi, and J. Zobel. Relative Lempel-Ziv factorization for efficient storage and retrieval of web collections. PVLDB, 5(3):265-273, 2011.
10. C. Hoobin, S. J. Puglisi, and J. Zobel. Sample selection for dictionary-based corpus compression. In SIGIR, pages 1137-1138, 2011.
11. J. J. Hunt, K.-P. Vo, and W. F. Tichy. Delta algorithms: An empirical analysis. ACM Transactions on Software Engineering and Methodology, 7(2):192-214, 1998.
12. R. M. Karp and M. O. Rabin. Efficient randomized pattern-matching algorithms. IBM Journal of Research and Development, 31(2):249-260, 1987.
13. P. Kulkarni, F. Douglis, J. D. LaVoie, and J. M. Tracey. Redundancy elimination within large collections of files. In USENIX Annual Technical Conference, General Track, pages 59-72, 2004.
14. S. Kuruppu, B. Beresford-Smith, T. C. Conway, and J. Zobel. Iterative dictionary construction for compression of large dna data sets. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 9(1):137-149, 2012.
15. S. Kuruppu, S. J. Puglisi, and J. Zobel. Relative Lempel-Ziv compression of genomes for large-scale storage and retrieval. In SPIRE, pages 201-206, 2010.
16. S. Kuruppu, S. J. Puglisi, and J. Zobel. Optimized relative Lempel-Ziv compression of genomes. In ACSC, pages 91-98, 2011.
17. S. Kuruppu, S. J. Puglisi, and J. Zobel. Reference sequence construction for relative compression of genomes. In SPIRE, pages 420-425, 2011.
18. N. J. Larsson and A. Moffat. Offline dictionary-based compression. In DCC, pages 296-305, 1999.
19. C. D. Manning, P. Raghavan, and H. Schütze. Introduction to Information Retrieval. Cambridge University Press, 2008.
20. G. Navarro and V. Mä. Compressed full-text indexes. ACM Computing Surveys, 39(1), 2007.
21. Z. Ouyang, N. D. Memon, T. Suel, and D. Trendafilov. Cluster-based delta compression of a collection of files. In WISE, pages 257-268, 2002.
22. A. Peel, A. Wirth, and J. Zobel. Collection-based compression using discovered long matching strings. In CIKM, pages 2361-2364, 2011.
23. P. Shilane, M. Huang, G. Wallace, and W. Hsu. WAN-optimized replication of backup datasets using stream-informed delta compression. ACM Transations on Storage, 8(4):1-26, 2012.
24. P. Skibinski, S. Grabowski, and S. Deorowicz. Revisiting dictionary-based compression. Software: Practice and Experience, 35(15):1455-1476, 2005.
25. J. A. Storer and T. G. Szymanski. Data compression via textual substitution. Journal of the ACM, 29(4):928-951, 1982.
26. T. Suel, P. Noel, and D. Trendafilov. Improved file synchronization techniques for maintaining large replicated collections over slow networks. In ICDE, pages 153-164, 2004.
27. H. E. Williams and J. Zobel. Compressing integers for fast file access. The Computer Journal, 42(3):193-201, 1999.
28. I. H. Witten, A. Moffat, and T. C. Bell. Managing Gigabytes: Compressing and Indexing Documents and Images, Second Edition. Morgan Kaufmann, 1999.
29. J. Ziv and A. Lempel. A universal algorithm for sequential data compression. IEEE Transactions on Information Theory, 23(3):337-343, 1977.
30. J. Ziv and N. Merhav. A measure of relative entropy between individual sequences with application to universal classification. IEEE Transactions on Information Theory, 39(4):1270-1279, 1993.
31. N. Ziviani, E. Silva de Moura, G. Navarro, and R. A. Baeza-Yates. Compression: A key for next-generation text retrieval systems. IEEE Computer, 33(11):37-44, 2000.
32. J. Zobel and A. Moffat. Inverted files for text search engines. ACM Computing Surveys, 38(2), 2006.