Tinylex: Static n-gram index pruning with perfect recall

International Conference on Information and Knowledge Management, Proceedings

Volumn , Issue , 2008, Pages 409-418

  Coetzee, Derrick ^a  

^a MICROSOFT RESEARCH   (United States)

Author keywords

Algorithms; Experimentation

Indexed keywords

APPROXIMATE MATCHING; ERROR-RESILIENT; EXPERIMENTATION; FALSE POSITIVE; INCLUSION STRUCTURE; INVERTED INDICES; MAIN MEMORY; N-GRAMS; PRACTICAL PROBLEMS; QUERY STRING; QUERY TIME; SUB-STRINGS;

QUERY LANGUAGES;

KNOWLEDGE MANAGEMENT;

EID: 70349239392     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1458082.1458138     Document Type: Conference Paper

References (26)

1. E. S. Adams and A. C. Meltzer. Trigrams as index elements in full text retrieval: Observations and experimental results. In ACM Conference on Computer Science, pages 433-439, 1993.
2. V. N. Anh and A. Moffat. Inverted index compression using word-aligned binary codes. Inf. Retr., 8(1):151-166, 2005.
3. R. Arnold and T. Bell. A corpus for the evaluation of lossless compression algorithms. In DCC '97: Proceedings of the Conference on Data Compression, page 201, Washington, DC, USA, 1997. IEEE Computer Society.
4. D. Bahle, H. E. Williams, and J. Zobel. Efficient phrase querying with an auxiliary index. In SIGIR '02: Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, pages 215-221, New York, NY, USA, 2002. ACM.
5. B. H. Bloom. Space/time trade-offs in hash coding with allowable errors. Commun. ACM, 13(7):422-426, 1970.
6. S. Büttcher and C. L. A. Clarke. A document-centric approach to static index pruning in text retrieval systems. In CIKM '06: Proceedings of the 15th ACM international conference on Information and knowledge management, pages 182-189, New York, NY, USA, 2006. ACM.
7. D. Carmel, D. Cohen, R. Fagin, E. Farchi, M. Herscovici, Y. S. Maarek, and A. Soffer. Static index pruning for information retrieval systems. In SIGIR '01: Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, pages 43-50, New York, NY, USA, 2001. ACM.
8. R. J. D'Amore and C. P. Mah. One-time complete indexing of text: theory and practice. In SIGIR, pages 155-164, 1985.
9. R. Dementiev, J. Kärkkäinen, J. Mehnert, and P. Sanders. Better external memory suffix array construction. In ALENEX/ANALCO, pages 86-97, 2005.
10. L. Egghe. The distribution of n-grams. Scientometrics, 47(2):237-252, 2000.
11. International Human Genome Sequencing Consortium. Initial sequencing and analysis of the human genome. Nature, 409(6822):860-921, 2001.
12. W. Kent, C. W. Sugnet, T. S. Furey, K. Roskin, T. H. Pringle, A. M. Zahler, and D. Haussler. The Human Genome Browser at UCSC. Genome Res, 12(6):996-1006, 2002.
13. M.-S. Kim, K.-Y. Whang, J.-G. Lee, and M.-J. Lee. n-Gram/2L: A space and time efficient two-level n-gram inverted index structure. In VLDB, pages 325-336, 2005.
14. C. P. Mah and R. J. D'Amore. Complete statistical indexing of text by overlapping word fragments. SIGIR Forum, 17(3):6-16, 1982.
15. V. Mäkinen. Trade off between compression and search times in compact suffix array. In ALENEX '01: Revised Papers from the Third International Workshop on Algorithm Engineering and Experimentation, pages 189-201, London, UK, 2001. Springer-Verlag.
16. U. Manber and S. Wu. GLIMPSE: a tool to search through entire file systems. In WTEC'94: Proceedings of the USENIX Winter 1994 Technical Conference on USENIX Winter 1994 Technical Conference, pages 4-4, Berkeley, CA, USA, 1994. USENIX Association.
17. J. Mayfield and P. McNamee. Single n-gram stemming. In SIGIR, pages 415-416, 2003.
18. P. McNamee, J. Mayfield, and C. Piatko. Haircut: a system for multilingual text retrieval in java. J. Comput. Small Coll., 17(3):8-22, 2002.
19. A. Moffat and L. Stuiver. Binary interpolative coding for effective index compression. Inf. Retr., 3(1):25-47, 2000.
20. Y. Ogawa and T. Matsuda. An efficient document retrieval method using n-gram indexing. Systems and Computers in Japan, 33(2):54-63, 2002.
21. V. Siivola and B. Pellom. Growing an n-gram language model. 2005.
22. A. Stolcke. Entropy-based pruning of backoff language models. In Proceedings of the DRAPA News Transcription and Understanding Workshop, pages 270-274, 1998.
23. H. E. Williams, J. Zobel, and P. Anderson. What's next? index structures for efficient phrase querying. In Australasian Database Conference, pages 141-152, 1999.
24. S. Wu and U. Manber. Agrep: A fast approximate pattern-matching tool. In Proc. of the Winter 1992 USENIX Conference, pages 153-162, San Francisco, California, 1991.
25. M. Yamamoto and K. W. Church. Using suffix arrays to compute term frequency and document frequency for all substrings in a corpus. Computational Linguistics, 27(1):1-30, 2001.
26. J. Zobel, A. Moffat, and K. Ramamohanarao. Inverted files versus signature files for text indexing. ACM Trans. Database Syst., 23(4):453-490, 1998.