Efficient set intersection for inverted indexing

ACM Transactions on Information Systems

Volumn 29, Issue 1, 2010, Pages

  Culpepper, J Shane ^a   Moffat, Alistair ^b  

^a RMIT UNIVERSITY   (Australia)

^b UNIVERSITY OF MELBOURNE   (Australia)

Author keywords

Bitvector; Byte code; Compact data structures; Information retrieval; Set intersection; Set representation

Indexed keywords

BITVECTOR; BYTE-CODE; COMPACT DATA STRUCTURES; SET INTERSECTION; SET REPRESENTATION;

DATA STRUCTURES; INFORMATION RETRIEVAL SYSTEMS; SEARCH ENGINES;

INFORMATION RETRIEVAL;

EID: 80051486140     PISSN: 10468188     EISSN: 15582868     Source Type: Journal    
DOI: 10.1145/1877766.1877767     Document Type: Article

References (31)

1. BAEZA-YATES, R. A. 2004. A fast set intersection algorithm for sorted sequences. In Proceedings of the 15th Annual Symposium on Combinatorial Pattern Matching (CPM'04). S. C. Sahinalp, et al., Eds. Lecture Notes in Computer Science, vol. 3109. Springer, 400-408.
2. BARBAY, J. AND KENYON, C. 2002. Adaptive intersection and t-threshold problems. In Proceedings of the 13th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA'02). D. Eppstein, Ed., 390-399.
3. BARBAY, J., LÓ PEZ-ORTIZ, A., AND LU, T. 2006. Faster adaptive set intersections for text searching. In Proceedings of the 5th International Workshop on Experimental Algorithms (WEA'06). C. À lvarez and M. J. Serna, Eds. Lecture Notes in Computer Science, vol. 4007, Springer 146-157. (Pubitemid 44070934)
4. BENTLEY, J. AND YAO, A. C.-C. 1976. An almost optimal algorithm for unbounded searching. Inform. Process. Lett. 5, 3, 82-87.
5. BLANDFORD, D. K. AND BLELLOCH, G. E. 2004. Compact representations of ordered sets. In Proceedings of the 5th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA'04). J. I. Munro, Ed., 11-19.
6. BRODNIK, A. AND MUNRO, J. 1999. Membership in constant time and almost-minimum space. SIAM J. Comput. 28, 5, 1627-1640. (Pubitemid 129304145)
7. CLARK, D. 1996. Compact PAT trees. Ph. D. thesis, University of Waterloo.
8. CLAUDE, F. AND NAVARRO, G. 2008. Practical rank/select queries over arbitrary sequences. In Proceedings of the 15th International Symposium on String Processing and Information Retrieval (SPIRE'08). A. Amir et al., Eds. Lecture Notes in Computer Science, vol. 5280. Springer, 176-187.
9. DEMAINE, E. D., LÓPEZ-ORTIZ, A., AND MUNRO, J. I. 2000. Adaptive set intersections, unions, and differences. In Proceedings of the 11th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA'00). 743-752. (Pubitemid 30549325)
10. ELIAS, P. 1975. Universal codeword sets and representations of the integers. IEEE Trans. Inform. Theory IT-21, 2, 194-203.
11. GALLAGER, R. G. AND VAN VOORHIS, D. C. 1975. Optimal source codes for geometrically distributed integer alphabets. IEEE Trans. Inform. Theory IT-21, 2, 228-230.
12. GOLOMB, S. W. 1966. Run-length encodings. IEEE Trans. Inform. Theory IT-12, 3, 399-401.
13. GONNET, G. H., ROGERS, L. D., AND GEORGE, J. A. 1980. An algorithmic and complexity analysis of iterpolation search. Acta Inform. 13, 1, 39-52. (Pubitemid 10457202)
14. GUPTA, A., HON, W.-K., SHAH, R., AND VITTER, J. S. 2006. Compressed dictionaries: Space measures, data sets, and experiments. In Proceedings of the 5th International Workshop on Experimental Algorithms (WEA'06), C. À lvarez and M. J. Serna, Eds., Lecture Notes in Computer Science, vol. 4007. Springer, 158-169. (Pubitemid 44070935)
15. HENNESSY, J. L. AND PATTERSON, D. A. 2006. Computer Architecture: A Quantitative Approach 4th Ed. Morgan Kaufmann, San Francisco, CA.
16. HWANG, F. K. AND LIN, S. 1972. A simple algorithm for merging two disjoint linearly ordered list. SIAM J. Comput. 1, 31-39.
17. JACOBSON, G. 1989. Space-Efficient static trees and graphs. In Proceedings of the 30th Annual IEEE Symposium on Foundations of Computer Science (FOCS'89). 549-554. (Pubitemid 20726002)
18. MOFFAT, A. AND PORT, G. S. 1990. Splay tree melding: Theme and variations. In Proceedings of the 13th Australasian Computer Science Conference. 275-284.
19. MOFFAT, A. AND STUIVER, L. 2000. Binary interpolative coding for effective index compression. Inform. Retri. 3, 1, 25-47.
20. MOFFAT, A. AND TURPIN, A. 2002. Compression and Coding Algorithms 1st Ed. Kluwer Academic Publisher, Boston.
21. MOFFAT, A. AND ZOBEL, J. 1996. Self-indexing inverted files for fast text retrieval. ACM Trans. Inform. Syst. 14, 4, 349-379. (Pubitemid 126420100)
22. MUNRO, J. I. 1996. Tables. In Proceedings of the 16th Annual Conference on Foundations of Software Technology and Theoretical Computer Science, V. Chandru and V. Vinay, Eds. Lecture Notes in Computer Science, vol. 1180. Springer, 37-42.
23. OKANOHARA, D. AND SADAKANE, K. 2007. Practical entropy-compressed rank/select dictionary. In Proceedings of the 10th Workshop on Algorithm Engineering and Experiments (ALENEX'07). 60-70. (Pubitemid 47267203)
24. PAGH, R. 2001. Low redundancy in static dictionaries with constant time query. SIAM J. Comput. 31, 2, 353-363. (Pubitemid 33782954)
25. RAMAN, R., RAMAN, V., AND RAO, S. S. 2002. Succinct indexable dictionaries with applications to encoding k-ary trees and multisets. In Proceedings of the 13th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA'02), D. Eppstein, Ed. SIAM, 233-242.
26. SADAKANE, K. AND GROSSI, R. 2006. Squeezing succinct data structures into entropy bounds. In Proceedings of the 17th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA'06). ACM Press, 1230-1239. (Pubitemid 43275351)
27. SANDERS, P. AND TRANSIER, F. 2007. Intersection in integer inverted indices. In Proceedings of the 10th Workshop on Algorithm Engineering and Experiments (ALENEX'07). 71-83. (Pubitemid 47267204)
28. STROHMAN, T. AND CROFT, W. B. 2007. Efficient document retrieval in main memory. In Proceedings of the 30th Annual International Conference on Research and Development in Information Retrieval (SIGIR'07). ACM Press, 175-182. (Pubitemid 350164959)
29. TRANSIER, F. AND SANDERS, P. 2008. Compressed inverted indexes for in-memory search engines. In Proceedings of the 11th Workshop on Algorithm Engineering and Experiments (ALENEX'08). J. I. Munro and D. Wagner, Eds. 3-12.
30. WITTEN, I. H., MOFFAT, A., AND BELL, T. C. 1999. Managing Gigabytes: Compressing and Indexing Documents and Images 2nd Ed. Morgan Kaufmann, San Francisco.
31. ZOBEL, J. AND MOFFAT, A. 2006. Inverted files for text search engines. ACM Comput. Surv. 38, 2, 6-1-6-56.