A general-purpose compression scheme for large collections

ACM Transactions on Information Systems

Volumn 20, Issue 3, 2002, Pages 329-355

  Cannane, Adam ^a   Williams, Hugh E ^a  

^a RMIT UNIVERSITY   (Australia)

Author keywords

Phrase based compression; Random access; Sampling

Indexed keywords

COMPRESS SCHEMES; PHRASE BASED COMPRESSIONS; QUERY EVALUATION; SINGLE PASS COMPRESSION;

DATA ACQUISITION; DATA STRUCTURES; GENERAL PURPOSE COMPUTERS; MATHEMATICAL MODELS; NONBIBLIOGRAPHIC RETRIEVAL SYSTEMS; QUERY LANGUAGES; SEARCH ENGINES;

DATA COMPRESSION;

EID: 33645851469     PISSN: 10468188     EISSN: 10468188     Source Type: Journal    
DOI: 10.1145/568727.568730     Document Type: Article

References (50)

1. ADAMS, M., CELNIKER, S., ET AL. 2000. The genome sequence of drosophila melanogaster. Science 287, 5461 (Mar.), 2185-2195. (See http://www.ncbi.nlm.nih. gov:80/entrez/query.fcgi?cmd=Retrieve&db=PubMed&list_uids= 10731132&dopt=Abstract for complete list of authors.)
2. AMIR, A. AND BENSON, G. 1992. Efficient two-dimensional compressed matching. In Proceedings of the IEEE Data Compression Conference (Snowbird, Utah), J. Storer and M. Cohn, Eds., IEEE Computer Society Press, Los Alamitos, Calif., 279-288.
3. APOSTOLICO, A. AND LONARDI, S. 1998. Some theory and practice of greedy off-line textual substitution. In Proceedings of the IEEE Data Compression Conference (Snowbird, Utah), J. Storer and M. Cohn, Eds., IEEE Computer Society Press, Los Alamitos, Calif., 119-128.
4. APOSTOLICO, A. AND LONARDI, S. 2000. Off-line compression by greedy textual substitution. Proc. IEEE 88, 11, 1733-1744.
5. BELL, T., CLEARY, J., AND WITTEN, I. 1990. Text Compression. Prentice-Hall, Englewood Cliffs, N.J.
6. BENSON, D., BOGUSKI, M., LIPMAN, D., OSTELL, J., AND OUELLETTE, B. 1998. GenBank. Nucleic Acids Res. 26, 1, 1-7.
7. BENTLEY, J. AND MCILROY, D. 1999. Data compression using long common strings. In Proceedings of the IEEE Data Compression Conference (Snowbird, Utah), J. Storer and M. Cohn, Eds. IEEE Computer Society Press, Los Alamitos, Calif., 287-295.
8. BENTLEY, J. AND MCILROY, D. 2001. Data compression with long repeated strings. Inf. Sci. 135, 1-2 (June), 1-11.
9. BENTLEY, J. L. AND SEDGEWICK, R. 1997. Fast algorithms for sorting and searching strings. In Proceedings of the Eighth Annual ACM-SIAM Symposium on Discrete Algorithms (SODA'97) (New Orleans), 360-369.
10. CANNANE, A. AND WILLIAMS, H. 2000. A compression scheme for large databases. In Proceedings of the Australasian Database Conference (Canberra), M. Orlowska, Ed., Vol. 22., IEEE Computer Society Press, Los Alamitos, Calif., 6-11.
11. CANNANE, A. AND WILLIAMS, H. 2001. General-purpose compression for efficient retrieval. J. Am. Soc. Inf. Sci. Tech. 52, 5 (Apr.), 430-437.
12. CANNANE, A., WILLIAMS, H., AND ZOBEL, J. 1999. A general-purpose compression scheme for databases. In Proceedings of the IEEE Data Compression Conference (Snowbird, Utah), J. Storer and M. Cohn, Eds., IEEE Computer Society Press, Los Alamitos, Calif., 519 (refereed poster).
13. COREL CORPORATION. 1994. Corel professional photos CD-ROM.
14. GAILLY, J. 1993. Gzip program and documentation. Available by anonymous ftp from prep.ai.mit.edu:/pub/gnu/gzip-*.tar.
15. GEORGE, D., BARKER, W., MEWES, H., PFEIFFER, F., AND TSUGITA, A. 1996. The PIR-international protein sequence database. Nucleic Acids Res. 24, 17-20.
16. HARMAN, D. 1995. Overview of the second text retrieval conference (TREC-2). Inf. Process. Manage. 31, 3, 271-289.
17. HAWKING, D., CRESWELL, N., AND THISTLEWAITE, P. 1999. Overview of TREC-7 very large collection track. In Proceedings of the Text Retrieval Conference (TREC), E. Voorhees and D. Harman., Eds., National Institute of Standards and Technology Special Publication 500-242, Washington, D.C., 91-104.
18. HORSPOOL, R. 1991. Improving LZW. In Proceedings of the IEEE Data Compression Conference (Snowbird, Utah), J. Storer and J. Reif, Eds., IEEE Computer Society Press, Los Alamitos, Calif., 332-341.
19. LARSSON, N. J. AND MOFFAT, A. 2000. Offline dictionary-based compression. Proc. IEEE 88, 11 (Nov.), 1722-1732.
20. LELEWER, D. A. AND HIRSCHBERG, D. S. 1987. Data compression. Comput. Surv. 19, 3 (Sept.), 261-296.
21. LYNCH, M. 1973. Compression of bibliographic files using an adaption of run-length coding. Inf. Storage Retrieval 9, 207-214.
22. MAYNE, A. AND JAMES, E. B. 1975. Information compression by factoring common strings. Comput. J. 18, 157-160.
23. MOFFAT, A. 1989. Word based text compression. Softw. Pract. Exper. 19, 2 (Feb.), 185-198.
24. MOFFAT, A., NEAL, R., AND WITTEN, I. 1995. Arithmetic coding revisited. In Proceedings of the IEEE Data Compression Conference (Snowbird, Utah), J. Storer and M. Cohn, Eds., IEEE Computer Society Press, Los Alamitos, Calif., 202-211.
25. MOFFAT, A., NEAL, R. M., AND WITTEN, I. H. 1998. Arithmetic coding revisited. ACM Trans. Inf. Syst. 16, 3 (July), 256-294.
26. MOFFAT, A., ZOBEL, J., AND SHARMAN, N. 1997. Text compression for dynamic document databases. IEEE Trans. Knowl. Data Eng. 9, 2, 302-313.
27. NAKAMURA, H. AND MURASHIMA, S. 1996. Data compression by concatenations of symbol pairs. In Proceedings of the IEEE International Symposium on Information Theory and Its Applications (Victoria, BC), 496-499.
28. NEVILL-MANNING, C. AND WITTEN, I. 1997. Compression and explanation using hierarchical grammars. Comput. J. 40, 2/3, 103-116.
29. NEVILL-MANNING, C. AND WITTEN, I. 2000. Online and offline heuristics for inferring hierarchies of repetitions in sequences. Proc. IEEE 88, 11, 1745-1755.
30. RUBIN, F. 1976. Experiments in text file compression. Commun. ACM 19, 11 (Nov.), 617-623.
31. SCHUEGRAF, E. AND HEAPS, H. 1974. A comparison of algorithms for data base compression by use of fragments as language elements. Inf. Storage Retrieval 10, 309-319.
32. SEWARD, J. 2000. The bzip2 and Iibbzip2 home page. Available by anonymous ftp from sourceware.cygnus.com:/pub/bzip2/v100/bzip2-*.tar.gz.
33. SILVA DE MOURA, E., NAVARRO, G., ZIVIANI, N., AND BAEZA-YATES, R. 2000. Fast and flexible word searching on compressed text. ACM Trans. Inf. Syst. 18, 2 (Apr.), 113-139.
34. SPINK, A., WOLFRAM, D., JANSEN, B. J., AND SARACEVIC, T. 2001. Searching the Web: The public and their queries. J. Am. Soc. Inf. Sci. 52, 3, 226-234.
35. STORER, J. AND SZYMANSKI, T. 1982. Data compression via textual substitution. J. ACM 29, 928-951.
36. TEAHAN, W. AND CLEARY, J. 1996. The entropy of English using PPM-based models. In Proceedings of the IEEE Data Compression Conference (Snowbird, Utah), J. Storer and M. Cohn, Eds., IEEE Computer Society Press, Los Alamitos, Calif., 53-62.
37. THOMAS, S. AND OROST, J. 1985. Compress (version 4.0) program and documentation.
38. TURPIN, A. 1999. Efficient prefix coding. PhD thesis, The University of Melbourne.
39. TURPIN, A. AND MOFFAT, A. 1997. Fast file search using text compression. In Proceedings of the Australasian Computer Science Conference, M. Patel, Ed., Vol. 19, Australian Computer Science Communications, Sydney, 1-8.
40. WAGNER, R. A. 1973. Common phrases and minimum-space text storage. Commun. ACM 16, 3 (Mar.), 148-152.
41. WELCH, T. 1984. A technique for high performance data compression. IEEE Comput. 17, 8-20.
42. WILLIAMS, H. AND ZOBEL, J. 1999. Compressing integers for fast file access. Comput. J. 42, 3, 193-201.
43. WILLIAMS, H. AND ZOBEL, J. 2001. Searchable words on the Web (submitted).
44. WITTEN, I., MOFFAT, A., AND BELL, T. 1999. Managing Gigabytes: Compressing and Indexing Documents and Images, second ed., Morgan Kaufmann Publishers, Los Altos, Calif.
45. WOLFF, J. 1978. Receding of natural language for economy of transmission or storage. Comput. J. 21, 1, 42-44.
46. ZIV, J. AND LEMPEL, A. 1977. A universal algorithm for sequential data compression. IEEE Trans. Inf. Theor. IT-23, 3, 337-343.
47. ZIV J. AND LEMPEL, A. 1978. Compression of individual sequences via variable rate coding. IEEE Trans. Inf. Theor. IT-24, 5, 530-536.
48. ZOBEL, J. AND MOFFAT, A. 1995. Adding compression to a full-text retrieval system. Softw. Pract. Exper. 25, 8 (Aug.), 891-903.
49. ZOBEL, J. AND WILLIAMS, H. 1999. Combined models for high-performance compression of large text collections. In Proceedings of the Sixth String Processing and Information Retrieval Conference (SPIRE'99) (Cancun), R. Baeza-Yates, E. Chàvez, and J. Favela, Eds., IEEE Computer Society Press, Los Alamitos, Calif., 224-231.
50. ZOBEL, J., HELNZ, S., AND WILLIAMS, H. 2001. In-memory hash tables for accumulating text vocabularies. Inf. Proc. Lett. 80, 271-277.