Relative lempelziv factorization for efficient storage and retrieval of web collections

Proceedings of the VLDB Endowment

Volumn 5, Issue 3, 2011, Pages 265-273

  Hoobin, Christopher ^a   Puglisi, Simon J ^a,b   Zobel, Justin ^c  

^a RMIT UNIVERSITY   (Australia)

^b KING'S COLLEGE LONDON   (United Kingdom)

^c UNIVERSITY OF MELBOURNE   (Australia)

Author keywords

[No Author keywords available]

Indexed keywords

ADAPTIVE ALGORITHMS;

COMPRESSION TECHNIQUES; CORE COMPONENTS; DOCUMENT COLLECTION; DOCUMENT RETRIEVAL; REPRESENTATIVE SAMPLE; SCALABLE COMPRESSION; STATE-OF-THE-ART METHODS; STORAGE AND RETRIEVALS;

INFORMATION RETRIEVAL;

EID: 84863731168     PISSN: None     EISSN: 21508097     Source Type: Conference Proceeding    
DOI: 10.14778/2078331.2078341     Document Type: Article

References (36)

1. V. N. Anh and A. Moffat. Inverted index compression using word-aligned binary codes. Information Retrieval, 8(1):151-166, 2005.
2. J. Bentley and D. McIlroy. Data compression with long repeated strings. Journal of Information Sciences, 135(1-2):1-11, 2001.
3. F. Berman. Got data?: A guide to data preservation in the information age. Communications of the ACM, 51(12):50-56, 2008.
4. N. R. Brisaboa, A. Farina, G. Navarro, and J. R. Paramá. Improving semistatic compression via pair-based coding. In I. Virbitskaite and A. Voronkov, editors, Perspectives of Systems Informatics, volume 4378 of Lecture Notes in Computer Science, pages 124-134. Springer Berlin/Heidelberg, 2007.
5. N. R. Brisaboa, A. Farina, G. Navarro, and J. R. Paramá. Lightweight natural language text compression. Information Retrieval, 10:1-33, January 2007.
6. N. R. Brisaboa, A. Farina, G. Navarro, and J. R. Paramá. New adaptive compressors for natural language text. Software Practice and Experience, 38:1429-1450, November 2008.
7. N. R. Brisaboa, A. Farina, G. Navarro, and J. R. Paramá. Dynamic lightweight text compression. ACM Transactions on Information Systems, 28:10:1-10:32, July 2010.
8. S. Büttcher and C. L. A. Clarke. Index compression is good, especially for random access. In Proc. 16th ACM Conference on Information and Knowledge Management (CIKM'07), pages 761-770. ACM, 2007.
9. S. Büttcher, C. L. A. Clarke, and G. V. Cormack. Information Retrieval: Implementing and Evaluating Search Engines. MIT Press, 2010.
10. A. Cannane and H. E. Williams. A compression scheme for large databases. In Proc. 11th Australasian Database Conference (ADC'00), page 6. IEEE Computer Society, 2000.
11. A. Cannane and H. E. Williams. A general-purpose compression scheme for large collections. ACM Transactions on Information Systems, 20(3):329-355, 2002.
12. F. Chang, J. Dean, S. Ghemawat, W. C. Hsieh, D. A. Wallach, M. Burrows, T. Chandra, A. Fikes, and R. E. Gruber. Bigtable: A distributed storage system for structured data. ACM Transactions on Computer Systems, 26(2):1-26, June 2008.
13. G. Chen, S. J. Puglisi, and W. F. Smyth. Lempel-Ziv factorization using less time & space. Mathematics in Computer Science, 1(4):605-623, 2008.
14. B. Croft, D. Metzler, and T. Strohman. Search Engines: Information Retrieval in Practice. Addison-Wesley, 2010.
15. E. S. de Moura, G. Navarro, N. Ziviani, and R. Baeza-Yates. Fast and exible word searching on compressed text. ACM Transactions Information Systems, 18:113-139, April 2000.
16. P. Ferragina and G. Manzini. On compressing the textual web. In Proc. 3rd ACM International Conference on Web Search and Data Mining (WSDM'10), pages 391-400. ACM, 2010.
17. C. Hoobin, S. J. Puglisi, and J. Zobel. Sample selection for dictionary-based corpus compression. In Proc. 34th ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR'11). ACM, 2011.
18. D. Huffman. A method for the construction of minimum-redundancy codes. Proc. of the Institute of Radio Engineers, 40(9):1098-1101, 1952.
19. S. Kreft and G. Navarro. LZ77-like compression with fast random access. In Proc. 20th IEEE Data Compression Conference (DCC'10), pages 239-248. IEEE Computer Society, 2010.
20. S. Kuruppu, S. J. Puglisi, and J. Zobel. Relative Lempel-Ziv compression of genomes for large-scale storage and retrieval. In Proc. 17th Symposium on String Processing and Information Retrieval (SPIRE'10), pages 201-206. Springer, 2010.
21. N. J. Larsson and A. Moffat. Offline dictionary-based compression. In Proc. 9th Data Compression Conference (DCC'99), pages 296-305. IEEE Computer Society, March 1999.
22. U. Manber and E. W. Myers. Suffix arrays: A new method for on-line string searches. SIAM Journal on Computing, 22(5):935-948, 1993.
23. C. D. Manning, P. Raghavan, and H. Schütze. Introduction to Information Retrieval. Cambridge University Press, 2008.
24. A. Moffat. Word-based text compression. Software Practice and Experience, 19(2):185-198, 1989.
25. A. Moffat and A. Turpin. Compression and Coding Algorithms. Kluwer Academic Publishers, 2002.
26. F. Scholer, H. E. Williams, J. Yiannis, and J. Zobel. Compression of inverted indexes for fast query evaluation. In Proc. 25th ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR'02), pages 222-229. ACM, 2002.
27. A. Tombros and M. Sanderson. Advantages of query biased summaries in information retrieval. In Proc. 21st ACM SIGIR Internationl Conference on Research and Development in Information Retrieval (SIGIR'98), pages 2-10. ACM, 1998.
28. Y. Tsegay, S. J. Puglisi, A. Turpin, and J. Zobel. Document compaction for efficient query biased snippet generation. In Proc. 31st European Conference on IR Research on Advances in Information Retrieval (ECIR'09), pages 509-520. Springer-Verlag, 2009.
29. A. Turpin, Y. Tsegay, D. Hawking, and H. E. Williams. Fast generation of result snippets in web search. In Proc. 30th ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR'07), pages 127-134. ACM, 2007.
30. I. H. Witten, A. Moffat, and T. C. Bell. Managing Gigabytes: Compressing and Indexing Documents and Images. Morgan Kaufmann, 2. edition, 1999.
31. J. Ziv and A. Lempel. A universal algorithm for sequential data compression. IEEE Transactions on Information Theory, 23:337-343, 1977.
32. J. Ziv and A. Lempel. Compression of individual sequences via variable-rate coding. IEEE Transactions on Information Theory, 24(5):530-536, 1978.
33. J. Ziv and N. Merhav. A measure of relative entropy between individual sequences with application to universal classification. IEEE Transactions on Information Theory, 39(4):1270-1279, 1993.
34. N. Ziviani, E. S. de Moura, G. Navarro, and R. Baeza-Yates. Compression: A key for next-generation text retrieval systems. Computer, 33:37-44, 2000.
35. J. Zobel and A. Moffat. Adding compression to a full-text retrieval system. Software Practice and Experience, 25(8):891-903, 1995.
36. M. Zukowski, S. Héman, N. Nes, and P. Boncz. Super-scalar RAM-CPU cache compression. In Proc. 22nd International Conference of Data Engineering (ICDE'06), page 59. IEEE Computer Society, 2006.