Efficient in-memory extensible inverted file

Information Systems

Volumn 32, Issue 5, 2007, Pages 733-754

  Luk, Robert W P ^a   Lam, Wai ^b  

^a HONG KONG POLYTECHNIC UNIVERSITY   (Hong Kong)

^b CHINESE UNIVERSITY OF HONG KONG   (Hong Kong)

Author keywords

Indexing; Information retrieval; Optimization

Indexed keywords

AUTOMATIC INDEXING; DATA REDUCTION; OPTIMIZATION; PARALLEL PROCESSING SYSTEMS; PERSONAL COMPUTERS; RANDOM ACCESS STORAGE; STORAGE ALLOCATION (COMPUTER); WORD PROCESSING;

ARRIVAL RATE SCHEME; EXTENSIBLE INVERTED FILES; STORAGE UTILIZATION; STORAGE WASTES;

FILE ORGANIZATION;

EID: 34147114386     PISSN: 03064379     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.is.2006.06.001     Document Type: Article

References (51)

1. D. Hawking, N. Craswell, P.B. Thistlewaite, Overview of TREC-7 very large collection track, in: Proceedings of The Seventh TREC Conference, 1998, pp. 40-52.
2. C. Clarke, N. Craswell, I. Soboroff. Terabyte track, http://www-nlpir.nist.gov/projects/terabyte/, 2003.
3. J. Hirai, H. Garcia-Molina, A. Paepcke, S. Raghavan, WebBase: a repository of web pages, in: Proceedings of The Ninth International World Wide Web Conference, 2000, pp. 277-293.
4. NTCIR Patent Retrieval Task, http://www.slis.tsukuba.ac.jp/~fujii/ntcir5/cfp-en.html, 2005.
5. Barroso L.A., Dean J., and Hölzle U. Web search for a planet: the Google cluster architecture. IEEE Micro. 23 2 (2003) 22-28
6. Heinz S., and Zobel J. Efficient single-pass index construction for text databases. J. Am. Soc. Inform. Sci. Technol. 54 8 (2003) 713-729
7. R. Baeza-Yates, B. Ribeiro-Neto, Modern Information Retrieval, ACM Press, 1999.
8. Lester N., Zobel J., and Williams H. Efficient online index maintenance for contiguous inverted lists. Informat. Process. Manage. 42 (2006) 916-933
9. Navarro G., de Moura E.S., Neubert M., and Baeza-Yates R. Adding compression to block addressing inverted indexes. Inform. Retriev. 3 (2000) 49-77
10. A. MacFarlane, S.E. Robertson, J.A. McCann, On concurrency control of inverted files, in: F.C. Johnson (Ed.), Proceedings of the 18th MCS IRSG Annual Colloquium on Information Retrieval Research, 26-27 March 1996, pp. 67-79.
11. U. Manber, S. Wu, Glimpse: a tool to search through entire file systems, in: Proceedings of the USENIX Winter 1994 Technical Conference, 1994, pp. 23-32.
12. Ziviani N., de Moura E.S., Navarro G., and Baeza-Yates R. Compression: a key for next-generation text retrieval systems. IEEE Comput. 33 11 (2000) 37-44
13. Elias P. Universal codeword sets and the representation of the integers. IEEE Trans. Inform. Theory 21 2 (1975) 194-203
14. Trotman A. Compressing inverted files. Inform. Retriev. 6 1 (2003) 5-19
15. Williams H., and Zobel J. Compressing integers for fast file access. Comput. J. 42 3 (1999) 193-201
16. Golomb S.W. Run-length encodings. IEEE Trans. Inform. Theory 12 3 (1996) 399-401
17. Brewer E.A. Combining systems and databases: a search engine retrospective. In: Hellerstein J.M., and Stonebraker M. (Eds). Reading in Database Systems. Fourth Edition (2005), MIT Press, Cmabridge, MA
18. A. Fox, E.A. Brewer, Harvest, yield, and scalable tolerant systems, in: Proceedings of the 16th SOSP, St. Malo, France, October 1997.
19. Gilbert S., and Lynch N. Brewer's conjecture and the feasibility of consistent, available, partition-tolerant web services. Sigact News 33 2 (2002) 51-59
20. Heap H.S. Information Retrieval: Computational and Theoretical Aspects (1978), Academic Press, New York
21. C.L.A. Clarke, G. V. Cormack, Dynamic inverted indexes for a distributed full-text retrieval system, Technical Report MT-95-01, University of Waterloo, 1995.
22. B. A. Ribeiro-Nero, J. P. Kitajima, G. Navarro, C. Santana, N. Ziviani, Parallel generation of inverted files for distributed text collections, in: Proceedings of the 18th International Conference of the Chilean Society of Computer Science, Chile, 1998, pp. 149-157.
23. B. Ribeiro-Neto, E.S. Moura, M.S. Neubert, N. Ziviani, Efficient distributed algorithms to build inverted files, in: Proceedings of The 22nd Annual International ACM SIGIR conference on Research and development in information retrieval, Berkeley, 1999, pp. 105-112.
24. Heinz S., Zobel J., and Williams H.E. Burst tries: a fast, efficient data structure for string keys. ACM Trans. Inform. Syst. 20 2 (2002) 192-223
25. Kit C., Liu Y., and Liang N. On methods of Chinese automatic word segmentation. J. Chin. Inform. Process. 3 1 (1989) 13-20
26. Melnik S., Raghavan S., Yang B., and Garcia-Molina H. Building a distributed full-text index for the Web. ACM Trans. Inform. Syst. 19 3 (2001) 217-247
27. Zobel J., Moffat A., and Ramanohanarao K. Inverted files versus signature files for text indexing. ACM Trans. Database Syst. 23 4 (1998) 453-490
28. Witten I., Moffat A., and Bell T.C. Managing Gigabytes: Compressing and Indexing Documents and Images (1999), Morgan Kaufmann Publishers, Los Alios, CA
29. Faloutsos C., and Christodoulakis S. Description and performance analysis of signature file methods. ACM Trans. Office Inform. Syst. 5 3 (1987) 237-257
30. D. Cutting, J. Petersen, Optimizations for dynamic inverted index maintenance, in: Proceedings of the Thirteenth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 1990, pp. 405-411.
31. A. Tomasic, H. Garcia-Molina, K.A. Shoens, Incremental updates of inverted lists for text document retrieval, in: Proceedings of The ACM SIGMOD International Conference on Management of Data, 1994, pp. 289-300.
32. E.W. Brown, J.P. Callan, W.B. Croft, Fast incremental indexing for full-text information retrieval, in: Proceedings of The 20th VLDB Conference, 1994, pp. 192-202.
33. J. Zobel, A. Moffat, R. Sacks-Davis, Storage management for files of dynamic records, in: Proceedings of the Fourth Australian Database Conference, 1993, pp. 26-38.
34. Shieh W.-Y., and Chung C.-P. A statistics-based approach to incrementally update inverted files. Inform. Process. Manage. 41 2 (2005) 275-288
35. Tomasic A., and Garcia-Molina H. Issues in parallel information retrieval. Bull. Tech. Committee Data Eng. 17 3 (1994) 41-49
36. C. Badue, B.A. Ribeiro-Neto, R. Baeza-Yates, N. Ziviani, Distributed query processing using partitioned inverted files, in: Proceedings of the Eighth International Symposium on String Processing and Information Retrieval (SPIRE 2001), 2001, pp. 10-20.
37. A. MacFarlane, J.A. McCann, S.E. Robertson, Parallel search using partitioned inverted files, in: Proceedings of the Seventh International Symposium on String Processing and Information Retrieval (SPIRE 2000), 2000, pp. 209-220.
38. J.P. Callan, Z. Lu, W.B. Croft, Searching distributed collections with inference networks, in: Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Seattle, 1995, pp. 21-28.
39. C. Stanfill, Partitioned posting files: a parallel inverted file structure for information retrieval, in: Proceedings of the 13th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Brussels, 1990, pp. 413-428.
40. Jeong B.-S., and Omiecinski E. Inverted file partitioning schemes in multiple disk systems. IEEE Trans. Parallel Distribut. Syst. 6 2 (1995) 142-153
41. S.-H. Chung, S.-C. Oh, K.R. Ryu, S.-H. Park, Parallel information retrieval on a distributed memory multiprocessor system, in: Proceedings of the International Conference on Algorithms and Architectures for Parallel Processing (ICAPP 97), 1997, pp. 163-176.
42. P. Bailey, D. Hawking, A parallel architecture for query processing over a terabyte of text, Technical Report TR-CS-96-04, The Australia National University, June 1996.
43. N. Goharian, T. El-Ghazawi, D. Grossman, Enterprise text processing: a sparse matrix approach, in: Proceedings of International Conference on Information Technology: Coding and Computing,, 2001, pp. 71-75.
44. Stanfill C., and Kahle B. Parallel free-text search on the connection machine system. Commun. ACM 29 12 (1986) 1229-1239
45. Z. Lu, K.S. McKinley, B. Cahoon, The hardware/software balancing act for information retrieval on symmetric multiprocessors, in: Proceedings of Euro-Par 98, 1998, pp. 521-527.
46. S. Melnik, S. Raghavan, B. Yang, H. Garcia-Molina, Building a distributed full-text index for the Web, in: Proceedings of The 10th International Conference on World Wide Web, Hong Kong, 2001, pp. 396-406.
47. Cardellini V., Casalicchio E., Colajanni M., and Yu P.S. The state of the art in locally distributed web servers. ACM Comput. Surveys 34 2 (2002) 263-311
48. Brewer E.A. Lessons from giant-scale services. IEEE Internet Comput. 5 4 (2001) 46-55
49. Arasu A., Cho J., Garcia-Molina H., Paepcke A., and Raghavan S. Searching the web. ACM Trans. Internet Technol. 1 1 (2001) 2-43
50. Couvreur T.R., Benzel R.N., Miller S.F., Zeitler D.N., Lee D.L., Singhai M., Shivaratri N., and Wong W.Y.P. An analysis of performance and cost factors in searching large text databases using parallel search systems. J. Am. Soc. Inform. Sci. 45 7 (1994) 443-464
51. Cahoon B., McKinley K.S., and Lu Z. Evaluating the performance of distributed architectures for information retrieval using a variety of workloads. ACM Trans. Inform. Syst. 18 1 (2000) 1-43