On compressing and indexing repetitive sequences

Theoretical Computer Science

Volumn 483, Issue , 2013, Pages 115-133

  Kreft, Sebastian ^a   Navarro, Gonzalo ^a  

^a UNIVERSITY OF CHILE   (Chile)

Author keywords

Compression; Lempel Ziv; Repetitive texts; Self indexing; Succinct data structures

Indexed keywords

BIOLOGICAL SEQUENCES; LEMPEL-ZIV; NEW MEMBERS; REPETITIVE TEXTS; SELF-INDEXING; SOFTWARE REPOSITORIES; SUCCINCT DATA STRUCTURE; TEXT EXTRACTION;

COMPACTION; DATA STRUCTURES;

INDEXING (OF INFORMATION);

EID: 84876408746     PISSN: 03043975     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.tcs.2012.02.006     Document Type: Conference Paper

References (70)

1. S. Kreft, G. Navarro, LZ77-like compression with fast random access, in: Proc. 20th Data Compression Conference, DCC'10, pp. 239-248.
2. S. Kreft, G. Navarro, Self-indexing based on LZ77, in: Proc. 22nd Annual Symposium on Combinatorial Pattern Matching, CPM'11, in: LNCS, vol. 6661, pp. 41-54.
3. N. Ziviani, E.S. de Moura, G. Navarro, and R. Baeza-Yates Compression: a key for next-generation text retrieval systems IEEE Computer 33 2000 37 44
4. G. Navarro, and V. Mäkinen Compressed full-text indexes ACM Computing Surveys 39 2007 (article 2)
5. P. Ferragina, G. Manzini, Opportunistic data structures with applications, in: Proc. 41st Annual Symposium on Foundations of Computer Science, FOCS'00, pp. 390-398.
6. R. Grossi, J.S. Vitter, Compressed suffix arrays and suffix trees with applications to text indexing and string matching, in: Proc. 32nd Annual ACM Symposium on Theory of Computing, STOC'00, pp. 397-406.
7. A. Apostolico The myriad virtues of subword trees Combinatorial Algorithms on Words NATO ISI Series 1985 Springer-Verlag 85 96
8. U. Manber, and G. Myers Suffix arrays: a new method for on-line string searches SIAM Journal on Computing 22 1993 935 948
9. G. Manzini An analysis of the Burrows-Wheeler transform Journal of the ACM 48 2001 407 430
10. R. Grossi, A. Gupta, J.S. Vitter, High-order entropy-compressed text indexes, in: Proc. 14th Annual ACM-SIAM Symposium on Discrete Algorithms, SODA'03, pp. 841-850.
11. P. Ferragina, G. Manzini, V. Mäkinen, and G. Navarro Compressed representations of sequences and full-text indexes ACM Transactions on Algorithms 3 2007 article 20
12. T. Gagie Large alphabets and incompressibility Information Processing Letters 99 2006 246 251
13. C. Nevill-Manning, I. Witten, D. Maulsby, Compression by induction of hierarchical grammars, in: Proc. 4th Data Compression Conference, DCC'94, pp. 244-253.
14. N.J. Larsson, and A. Moffat Off-line dictionary-based compression Proceedings of the IEEE 88 2000 1722 1732
15. A. Lempel, and J. Ziv On the complexity of finite sequences IEEE Transactions on Information Theory 22 1976 75 81
16. J. Ziv, and A. Lempel A universal algorithm for sequential data compression IEEE Transactions on Information Theory 23 1977 337 343
17. J. Ziv, and A. Lempel Compression of individual sequences via variable-rate coding IEEE Transactions on Information Theory 24 1978 530 536
18. F. Claude, G. Navarro, Self-indexed text compression using straight-line programs, in: Proc. 34th International Symposium on Mathematical Foundations of Computer Science, MFCS'09, in: LNCS, vol. 5734, pp. 235-246.
19. P. Bille, G.M. Landau, R. Raman, K. Sadakane, S.R. Satti, O. Weimann, Random access to grammar-compressed strings, in: Proc. 22nd Annual ACM-SIAM Symposium on Discrete Algorithms, SODA'11, pp. 373-389.
20. S. Kuruppu, B. Beresford-Smith, T. Conway, J. Zobel, Repetition-based compression of large DNA datasets, in: Proc. 13th Annual International Conference on Computational Molecular Biology, RECOMB'09, Poster.
21. F. Claude, A. Fariña, M. Martínez-Prieto, G. Navarro, Compressed q-gram indexing for highly repetitive biological sequences, in: Proc. 10th IEEE Conference on Bioinformatics and Bioengineering, BIBE'10, pp. 86-91.
22. F. Claude, A. Fariña, M. Martínez-Prieto, G. Navarro, Indexes for highly repetitive document collections, in: Proc. 20th ACM International Conference on Information and Knowledge Management, CIKM'11, pp. 463-468.
23. K. Sadakane, R. Grossi, Squeezing succinct data structures into entropy bounds, in: Proc. 17th Annual ACM-SIAM Symposium on Discrete Algorithms, SODA'06, pp. 1230-1239.
24. G. Navarro Indexing text using the Ziv-Lempel trie Journal of Discrete Algorithms 2 2004 87 114
25. P. Ferragina, and G. Manzini Indexing compressed text Journal of the ACM 52 2005 552 581
26. D. Arroyuelo, G. Navarro, K. Sadakane, Reducing the space requirement of LZ-index, Proc. 17th Annual Symposium on Combinatorial Pattern Matching, CPM, in: LNCS, vol. 4009, pp. 319-330.
27. L. Russo, and A. Oliveira A compressed self-index using a Ziv-Lempel dictionary Information Retrieval 5 2008 501 513
28. S. Kuruppu, S. Puglisi, J. Zobel, Relative Lempel-Ziv compression of genomes for large-scale storage and retrieval, in: Proc. 17th International Symposium on String Processing and Information Retrieval, SPIRE'10, pp. 201-206.
29. J. Sirén, N. Välimäki, V. Mäkinen, G. Navarro, Run-length compressed indexes are superior for highly repetitive sequence collections, in: Proc. 15th International Symposium on String Processing and Information Retrieval, SPIRE'08, in: LNCS, vol. 5280, pp. 164-175.
30. V. Mäkinen, G. Navarro, J. Sirén, and N. Välimäki Storage and retrieval of highly repetitive sequence collections Journal of Computational Biology 17 2010 281 308
31. M. Burrows, D. Wheeler, A block sorting lossless data compression algorithm, Technical Report 124, Digital Equipment Corporation, 1994.
32. R. Grossi, and J.S. Vitter Compressed suffix arrays and suffix trees with applications to text indexing and string matching SIAM Journal of Computing 35 2005 378 407
33. V. Mäkinen, and G. Navarro Succinct suffix arrays based on run-length encoding Nordic Journal of Computing 12 2005 40 66
34. K. Sadakane New text indexing functionalities of the compressed suffix arrays Journal of Algorithms 48 2003 294 313
35. W. Rytter Application of Lempel-Ziv factorization to the approximation of grammar-based compression Theoretical Computer Science 302 2003 211 222
36. J. Kärkkäinen, E. Ukkonen, Lempel-Ziv parsing and sublinear-size index structures for string matching, in: Proc. 3rd South American Workshop on String Processing, WSP'96, pp. 141-155.
37. J. Kärkkäinen, Repetition-based text indexes, Ph.D. Thesis, Department of Computer Science, University of Helsinki, Finland, 1999.
38. S. Kreft, Self-Index based on LZ77, MSc Thesis, University of Chile, 2010. Available as Tech. Report TR/DCC-2011-13, Dept. of Computer Science, University of Chile. http://www.dcc.uchile.cl/TR/2011/DCC-20111220-013.pdf.
39. M. Farach, M. Thorup, String matching in Lempel-Ziv compressed strings, in: Proc. 27th ACM Annual Symposium on the Theory of Computing, STOC, pp. 703-712.
40. S.R. Kosaraju, and G. Manzini Compression of low entropy strings with Lempel-Ziv algorithms SIAM Journal on Computing 29 1999 893 911
41. E. Plotnik, M. Weinberger, and J. Ziv Upper bounds on the probability of sequences emitted by finite-state sources and on the redundancy of the Lempel-Ziv algorithm IEEE Transactions on Information Theory 38 1992 66 72
42. R. Hamming Coding and Information Theory 1986 Prentice-Hall
43. E.R. Fiala, and D.H. Greene Data compression with finite windows Communications of the ACM 32 1989 490 505
44. R. Raman, V. Raman, S.S. Rao, Succinct indexable dictionaries with applications to encoding k-ary trees and multisets, in: Proc. 13th Annual ACM-SIAM Symposium on Discrete Algorithms, SODA'02, pp. 233-242.
45. J.I. Munro, R. Raman, V. Raman, S.S. Rao, Succinct representations of permutations, in: Proc. 30th International Colloquium on Automata, Languages and Computation, ICALP'03, in: LNCS, vol. 2719, pp. 345-356.
46. D. Morrison PATRICIA - Practical algorithm to retrieve information coded in alphanumeric Journal of the ACM 15 1968 514 534
47. D. Benoit, E. Demaine, I. Munro, R. Raman, V. Raman, and S. Rao Representing trees of higher degree Algorithmica 43 2005 275 292
48. V. Mäkinen, and G. Navarro Rank and select revisited and extended Theoretical Computer Science 387 2007 332 347
49. M. Pǎtraşcu, Succincter, in: Proc. 49th IEEE Annual Symposium on Foundations of Computer Science, FOCS'08, pp. 305-313.
50. N. Brisaboa, S. Ladra, G. Navarro, Directly addressable variable-length codes, in: Proc. 16th International Symposium on String Processing and Information Retrieval, SPIRE'09, pp. 122-130.
51. D. Arroyuelo, R. Cánovas, G. Navarro, K. Sadakane, Succinct trees in practice, in: Proc. 11th Workshop on Algorithm Engineering and Experiments, ALENEX'10, pp. 84-97.
52. J. Fischer, Optimal succinctness for range minimum queries, in: Proceedings of the Latin American Symposium on Theoretical Informatics, LATIN'10, in: LNCS, vol. 6034, pp. 158-169.
53. S. Muthukrishnan, Efficient algorithms for document retrieval problems, in: Proc. 13th Annual ACM-SIAM Symposium on Discrete Algorithms, SODA'02, pp. 657-666.
54. S. Gog, J. Fischer, Advantages of shared data structures for sequences of balanced parentheses, in: Proc. 20th Data Compression Conference, DCC'10, pp. 406-415.
55. J. Fischer, V. Mäkinen, and G. Navarro Faster entropy-bounded compressed suffix trees Theoretical Computer Science 410 2009 5354 5364
56. M. Crochemore, C.S. Iliopoulos, M. Kubica, M.S. Rahman, T. Walen, Improved algorithms for the range next value problem and applications, in: Proc. 25th International Symposium on Theoretical Aspects of Computer Science, STACS'08, pp. 205-216.
57. D. Okanohara, K. Sadakane, Practical entropy-compressed rank/select dictionary, in: Proc. 9th Workshop on Algorithm Engineering and Experiments, ALENEX'07.
58. G. Chen, S.J. Puglisi, and W.F. Smyth Lempel-Ziv factorization using less time & space Mathematics in Computer Science 1 2008 605 623
59. J. Kärkkäinen, P. Sanders, Simple linear work suffix array construction, in: Proc. 30th International Colloquium on Automata, Languages and Programming, ICALP'03, in: LNCS, vol. 2719, pp. 943-955.
60. D. Willard Log-logarithmic worst-case range queries are possible in space Θ (n) Information Processing Letters 17 1983 81 84
61. G. Navarro, K. Sadakane, Fully-functional static and dynamic succinct trees, CoRR 0905.0768v5 (2010).
62. J.I. Munro An implicit data structure supporting insertion, deletion, and search in O (log n) time Journal of Computer System Sciences 33 1986 66 74
63. D. Arroyuelo, G. Navarro, and K. Sadakane Stronger Lempel-Ziv based compressed text indexing Algorithmica 62 1 2012 54 101
64. D. Okanohara, K. Sadakane, An online algorithm for finding the longest previous factors, in: Proc. 16th Annual European Symposium on Algorithms, ESA'08, pp. 696-707.
65. G. Navarro Implementing the LZ-index: Theory versus practice ACM Journal of Experimental Algorithmics 13 2009 (article 2)
66. J. Fischer, V. Heun, A new succinct representation of RMQ-information and improvements in the enhanced suffix array, in: Proc. 1st International Symposium on Combinatorics, Algorithms, Probabilistic and Experimental Methodologies, ESCAPE'07, in: LNCS, vol. 4614, pp. 459-470.
67. F. Claude, and G. Navarro Self-indexed grammar-based compression Fundamenta Informaticae 111 2010 313 337
68. W.-K. Hon, K. Sadakane, and W.-K. Sung Breaking a time-and-space barrier in constructing full-text indices SIAM Journal on Computing 38 2009 2162 2178
69. L.M.S. Russo, G. Navarro, A.L. Oliveira, Fully-compressed suffix trees, in: 8th Latin American Symposium on Theoretical Informatics, LATIN'08, in: LNCS, vol. 4957, pp. 362-373.
70. D. Arroyuelo, G. Navarro, Smaller and faster Lempel-Ziv indices, in: Proc. 18th International Workshop on Combinatorial Algorithms, IWOCA'07, pp. 11-20.