Compressing term positions in web indexes

Proceedings - 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2009

Volumn , Issue , 2009, Pages 147-154

  Yan, Hao ^a   Ding, Shuai ^a   Suel, Torsten ^a  

^a POLYTECHNIC UNIVERSITY   (United States)

Author keywords

Index compression; Inverted index; Search engines

Indexed keywords

ACCESS PATTERNS; CLUSTERING EFFECT; EARLY TERMINATION; FREQUENCY DATA; INDEX COMPRESSION; INVERTED INDEX; INVERTED INDEX COMPRESSION; INVERTED INDICES; MASSIVE PARALLELISM; POSITION DATA; POSITION INFORMATION; RANKING FUNCTIONS; TERM OCCURRENCES; TEXT COLLECTION; WEB PAGE; WEB SEARCH ENGINES;

HUMAN COMPUTER INTERACTION; INFORMATION RETRIEVAL; INFORMATION SERVICES; NATURAL LANGUAGE PROCESSING SYSTEMS; OPERATING COSTS; RESEARCH; SEARCH ENGINES;

WORLD WIDE WEB;

EID: 72449208572     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1571941.1571969     Document Type: Conference Paper

References (31)

1. V. Anh. Impact-based document retrieval. PhD Thesis, The University of Melbourne, Amsterdam, Netherlands, April 2004.
2. V. Anh and A. Moffat. Index compression using fixed binary codewords. In Proc. of the 15th Int. Australasian Database Conference, pages 61-67, 2004.
3. V. Anh and A. Moffat. Inverted index compression using word-aligned binary codes. Information Retrieval, 8(1):151-166, Jan. 2005.
4. V. Anh and A. Moffat. Improved word-aligned binary compression for text indexing. IEEE Transactions on Knowledge and Data Engineering, 18(6):857-861, 2006.
5. D. Blandford and G. Blelloch. Index compression through document reordering. In Proc. of the Data Compression Conference, pages 342-351, 2002.
6. A. Bookstein, S. Klein, and T. Raita. Modeling word occurrences for the compression of concordances. ACM Transactions of Information Systems, 15(3):254-290, 1997.
7. A. Bookstein, S. Klein, and T. Raita. Markov models for clusters in concordance compression. In Proc. of the Data Compression Conference, March, 1994.
8. S. Buettcher, C. Clarke, and B. Lushman. Term proximity scoring for ad-hoc retrieval on very large text collections. In Proc. of the 29th Annual Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, 2006.
9. P. Elias. Universal codeword sets and representations of the integers. IEEE Transactions on Information Theory, 12(3):194-203, Mar. 1975.
10. A. Fraenkel and S. Klein. Novel compression of sparse bit-strings - preliminary report. Combinatorial Algorithms on Words, 12:169-183, 1985.
11. J. Gailly. zlib compression library. Available at http://www.gzip.org/ zlib/.
12. S. Golomb. Run-length encodingds. IEEE Transactions on Information Theory, 12(3):399-401, 1966.
13. S. Heman. Super-scalar database compression between ram and cpu-cache. MS Thesis, Centrum voor Wiskunde en Informatica (CWI), Amsterdam, Netherlands, July 2005.
14. J. Kleinberg. Bursty and hierarchical structure in streams. In Proc. of the 8th ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining, 2002.
15. D. Metzler and W. B. Croft. A markov random field model for term dependencies. In Proc. of the 28th Annual Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, 2005.
16. G. Mishne and M. Rijke. Boosting web retrieval through query operations. In Proc. of the 27th European Conference on IR Research, 2005.
17. A. Mo® at and V. Anh. Binary codes for locally homogeneous sequences. Information Processing Letters, 99(5):75-80, Sept. 2006.
18. A. Mo® at and L. Stuiver. Exploiting clustering in inverted file compression. Proc. of the Data Compression Conference, pages 82-91, 1996.
19. A. Moffat and L. Stuiver. Binary interpolative coding for e® ective index compression. Information Retrieval, 3(1):25-47, 2000.
20. A. Moffat and J. Zobel. Parameterised compression for sparse bitmaps. In Proc. of the 15th Annual Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, 1992.
21. R. Schenkel, A. Broschart, S. Hwang, M. Theobald, and G. Weikum. Efficient text proximity search. In 14th String Processing and Information Retrieval Symposium, 2007.
22. F. Scholer, H. Williams, J. Yiannis, and J. Zobel. Compression of inverted indexes for fast query evaluation. In Proc. of the 25th Annual SIGIR Conf. on Research and Development in Information Retrieval, pages 222-229, Aug. 2002.
23. W. Shieh, T. Chen, J. Shann, and C. Chung. Inverted file compression through document identifier reassignment. Information Processing and Management: an International Journal, 39(1):117-131, Oct. 2003.
24. F. Silvestri, S. Orlando, and R. Perego. Assigning identifiers to documents to enhance the clustering property of fulltext indexes. In Proc. of the 27th Annual Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 305-312, 2004.
25. F. Silvestri, R. Perego, and S. Orlando. Assigning document identifiers to enhance compressibility of web search engine indexes. In Proc. of the 19th ACM Symp. on Appliled Comp, pages 600-605, 2004.
26. T. Tao and C. Zhai. An exploration of proximity measures in information retrieval. In Proc. of the 30th Annual Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, 2007.
27. H. Williams and J. Zobel. Compressing integers for fast file access. Computer Journal, 42(3):193-201, 1999.
28. I. H. Witten, A. Moffat, and T. C. Bell. Managing Gigabytes: Compressing and Indexing Documents and Images. Morgan Kaufmann, second edition, 1999.
29. J. Zhang, X. Long, and T. Suel. Performance of compressed inverted list caching in search engines. In 17th International World Wide Web Conference (WWW), April 2008.
30. J. Zobel and A. Moffat. Inverted files for text search engines. ACM Computing Surveys, 38(2), 2006.
31. M. Zukowski, S. Heman, N. Nes, and P. Boncz. Super-scalar ram-cpu cache compression. In Proc. of the Int. Conf. on Data Engineering, 2006.