Inverted index compression and query processing with optimized document ordering

WWW'09 - Proceedings of the 18th International World Wide Web Conference

Volumn , Issue , 2009, Pages 401-410

  Yan, Hao ^a   Ding, Shuai ^a   Suel, Torsten ^a,b  

^a NEW YORK UNIVERSITY   (United States)

^b YAHOO RESEARCH   (United States)

Author keywords

Document ordering; Index compression; Inverted index; IR query processing; Search engines

Indexed keywords

COMPRESSION METHODS; COMPRESSION SCHEME; COMPRESSION TECHNIQUES; DOCUMENT ORDERING; INDEX COMPRESSION; INVERTED INDEX COMPRESSION; INVERTED INDICES; PROCESSING PERFORMANCE; QUERY PROCESSING TECHNIQUES;

OPTIMIZATION; SEARCH ENGINES; WEBSITES;

QUERY PROCESSING;

EID: 84865646680     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1526709.1526764     Document Type: Conference Paper

References (30)

1. V. Anh and A. Moffat. Index compression using fixed binary codewords. In Proc. of the 15th Int. Australasian Database Conference, pages 61-67, 2004.
2. V. Anh and A. Moffat. Inverted index compression using word-aligned binary codes. Inf. Retrieval, 8(1):151-166, 2005.
3. V. Anh and A. Moffat. Improved word-aligned binary compression for text indexing. IEEE Transactions on Knowledge and Data Engineering, 18(6):857-861, 2006.
4. J. Bentley, D. Sleator, R. Tarjan, and V. Wei. A locally adaptive data compression scheme. Comm. of the ACM, 29(4), 1986.
5. K. Berberich, S. Bedathur, T. Neumann, and G. Weikum. A time machine for text search. In Proc. of the 30th Annual Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 519-526, 2007.
6. R. Blanco and A. Barreiro. Document identifier reassignment through dimensionality reduction. In Proc. of the 27th European Conf. on Information Retrieval, pages 375-387, 2005.
7. D. Blandford and G. Blelloch. Index compression through document reordering. In Proc. of the Data Compression Conference, pages 342-351, 2002.
8. P. Boldi and S. Vigna. Compressed perfect embedded skip lists for quick inverted-index lookups. In Proc. of the 12th Int. Conf. on String Processing and Information Retrieval, 2005.
9. S. Brin and L. Page. The anatomy of a large-scale hypertextual web search engine. In Proc. of the Seventh World Wide Web Conference, 1998.
10. A. Broder, D. Carmel, M. Herscovici, A. Soffer, and J. Zien. Efficient query evaluation using a two-level retrieval process. In Proc. of the 12th Int. Conf. on Information and Knowledge Management, pages 426-434, November 2003.
11. A. Broder, N. Eiron, M. Fontoura, M. Herscovici, R. Lempel, J. McPherson, R. Qi, and E. Shekita. Indexing shared content in information retrieval systems. In Proc. of the 10th Int. Conf. on Extending Database Technology, pages 313-330, 2006.
12. F. Chierichetti, S. Lattanzi, F. Mari, and A. Panconesi. On placing skips optimally in expectation. In Proc. of the Int. Conf. on Web Search and Data Mining, pages 15-24, 2008.
13. R. Fagin. Combining fuzzy information: an overview. SIGMOD Record, 31(2):109-118, June 2002.
14. S. Heman. Super-scalar database compression between RAM and CPU-cache. MS Thesis, Centrum voor Wiskunde en Informatica, Amsterdam, Netherlands, July 2005.
15. M. Herscovici, R. Lempel, and S. Yogev. Efficient indexing of versioned document sequences. In Proc. of the 29th European Conf. on Information Retrieval, 2007.
16. X. Long and T. Suel. Optimized query execution in large search engines with global page ordering. In Proc. of the 29th Int. Conf. on Very Large Data Bases, pages 129-140, 2003.
17. A. Moffat and L. Stuiver. Binary interpolative coding for effective index compression. Inf. Retrieval, 3(1):25-47, 2000.
18. A. Moffat and J. Zobel. Parameterised compression for sparse bitmaps. In Proc. of the 15th Annual Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 274-285, 1992.
19. M. Persin, J. Zobel, and R. Sacks-Davis. Filtered document retrieval with frequency-sorted indexes. J. of the American Society for Information Science, 47(10):749-764, 1996.
20. M. Richardson, A. Prakash, and E. Brill. Beyond pagerank: machine learning for static ranking. In Proc. of the 15th Int. World Wide Web Conference, 2006.
21. K. Risvik, Y. Aasheim, and M. Lidal. Multi-tier architecture for web search engines. In First Latin American Web Congress, pages 132-143, 2003.
22. F. Scholer, H. Williams, J. Yiannis, and J. Zobel. Compression of inverted indexes for fast query evaluation. In Proc. of the 25th Annual SIGIR Conf. on Research and Development in Information Retrieval, pages 222-229, Aug. 2002.
23. W. Shieh, T. Chen, J. Shann, and C. Chung. Inverted file compression through document identifier reassignment. Inf. Processing and Management, 39(1):117-131, 2003.
24. F. Silvestri. Sorting out the document identifier assignment problem. In Proc. of 29th European Conf. on Information Retrieval, pages 101-112, 2007.
25. F. Silvestri, S. Orlando, and R. Perego. Assigning identifiers to documents to enhance the clustering property of fulltext indexes. In Proc. of the 27th Annual Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, 2004.
26. I. H. Witten, A. Moffat, and T. C. Bell. Managing Gigabytes: Compressing and Indexing Documents and Images. Morgan Kaufmann, second edition, 1999.
27. J. Zhang, X. Long, and T. Suel. Performance of compressed inverted list caching in search engines. In Proc. of the 17th Int. World Wide Web Conference, April 2008.
28. J. Zhang and T. Suel. Efficient search in large textual collection with redundancy. In Proc. of the 16th Int. World Wide Web Conference, 2007.
29. J. Zobel and A. Moffat. Inverted files for text search engines. ACM Computing Surveys, 38(2), 2006.
30. M. Zukowski, S. Heman, N. Nes, and P. Boncz. Super-scalar RAM-CPU cache compression. In Proc. of the Int. Conf. on Data Engineering, 2006.