High-throughput DNA sequence data compression

Briefings in Bioinformatics

Volumn 16, Issue 1, 2013, Pages 1-15

  Zhu, Zexuan ^a   Zhang, Yongpeng ^a   Ji, Zhen ^a   He, Shan ^b   Yang, Xiao ^c  

^a SHENZHEN UNIVERSITY   (China)

^b UNIVERSITY OF BIRMINGHAM   (United Kingdom)

^c BROAD INSTITUTE OF MIT AND HARVARD   (United States)

Author keywords

Compression; Next generation sequencing; Reference based compression; Reference free compression

Indexed keywords

HIGH THROUGHPUT SEQUENCING; INFORMATION PROCESSING; MOLECULAR GENETICS; PROCEDURES;

DATA COMPRESSION; HIGH-THROUGHPUT NUCLEOTIDE SEQUENCING; MOLECULAR SEQUENCE DATA;

EID: 84908236785     PISSN: 14675463     EISSN: 14774054     Source Type: Journal    
DOI: 10.1093/bib/bbt087     Document Type: Article

References (77)

1. Sanger F, Nicklen S, Coulson AR. DNA sequencing with chain-terminating inhibitors. ProcNatlAcadSci 1977;74:5463-7.
2. Margulies M, Egholm M, Altman WE, et al. Genome sequencing in microfabricated high-density picolitre reactors. Nature 2005;437:376-80.
3. Shendure J, Ji H. Next-generation DNA sequencing. Nat Biotechnol 2008;26:1135-45.
4. Metzker ML. Sequencing technologies - the next generation. Nat Rev Genet 2010;11:31-46.
5. Branton D, Deamer DW, Marziali A, et al. The potential and challenges of nanopore sequencing. NatBiotechnol 2008;26:1146-53.
6. Pennisi E. Will computers crash genomics? Science 2011; 331:666-8.
7. Giancarlo R, Scaturro D, Utro F. Textual data compression in computational biology: a synopsis. Bioinformatics 2009;25:1575-86.
8. Grumbach S, Tahi F. Compression of DNA sequences. In:Proceedings of the 1993 IEEEData Compression Conference (DCC '93), Snowbird,Utah 1993;340-50.
9. Grumbach S, Tahi F. A new challenge for compression algorithms. Genet Seq Inform ProcessManag 1994;30:875-86.
10. Chen X, Kwong S, Li M. A compression algorithm for DNA sequences and its applications in genome comparison. Genome Informat Ser 1999;10:51-61.
11. Chen X, Li M, Ma B, et al. DNACompress: fast and effective DNA sequence compression. Bioinformatics 2002;18:1696-8.
12. Korodi G, Tabus I, Rissanen J, et al. DNA sequence compression - based on the normalized maximum likelihood model. IEEE Sign ProcessMag 2007;24:47-53.
13. Cao MD, Dix TI, Allison L, et al. A simple statistical algorithm for biological sequence compression. In: Proceedings of the 2007 IEEE Data Compression Conference (DCC '07). Snowbird,Utah 2007;43-52.
14. Matsumoto T, Sadakane K, Imai H. Biological sequence compression algorithms. Genome Inform Ser 2000;11:43-52.
15. Kahn SD. On the future of genomic data. Science 2011;331:728-9.
16. Kuruppu S, Beresford-Smith B, Conway T, et al. Iterative dictionary construction for compression of large DNA data sets. IEEE-ACM Trans Computat Biol Bioinformatics 2012;9:137-49.
17. Bose T, Mohammed MH, Dutta A, et al. BIND - An algorithm for loss-less compression of nucleotide sequence data. J Biosci 2012;37:785-9.
18. Brandon MC, Wallace DC, Baldi P. Data structures and compression algorithms for genomic sequence data. Bioinformatics 2009;25:1731-8.
19. Christley S, Lu Y, Li C, et al. Human genomes as email attachments. Bioinformatics 2009;25:274-5.
20. Wang C, Zhang D. A novel compression tool for efficient storage of genome resequencing data. Nucleic Acids Res 2011;39:E45-U74.
21. Pinho AJ, Pratas D, Garcia SP. GReEn: a tool for efficient compression of genome resequencing data. NucleicAcidsRes 2012;40:e27.
22. Kuruppu S, Puglisi SJ, Zobel J. Relative Lempel-Ziv compression of genomes for large-scale storage and retrieval. String Process Informat Retrieval, Lecture Notes in Computer Science (LNCS) 2010;6393:201-6.
23. Kuruppu S, Puglisi SJ, Zobel J. Optimized relative Lempel-Ziv compression of genomes. In: Proceedings of Australasian Computer Science Conference,Darlinghurst, Australia 2011;91-8.
24. Horspool RN. The effect of non-greedy parsing in Ziv-Lempel compression methods. In: Proceedings of the1995 IEEE Data Compression Conference (DCC '95), Snowbird, Utah 1995; 302-11.
25. Deorowicz S, Grabowski S. Robust relative compression of genomes with random access. Bioinformatics 2011;27:2979-86.
26. Mohammed MH, Dutta A, Bose T, et al. DELIMINATE-a fast and efficient method for loss-less compression of genomic sequences. Bioinformatics 2012;28:2527-9.
27. Kozanitis C, Saunders C, Kruglyak S, et al. Compressing genomic sequence fragments using SlimGene, Journal of Computational Biology 2011;18:401-13.
28. Jones DC, Ruzzo WL, Peng X, et al. Compression of nextgeneration sequencing reads aided by highly efficient de novo assembly. Nucleic Acids Res 2012;40:e171.
29. Popitsch N, von Haeseler A. NGC: lossless and lossy compression of aligned high-throughput sequencing data. Nucleic Acids Res 2013;41:e27.
30. Tembe W, Lowey J, Suh E. G-SQZ: compact encoding of genomic sequence and quality data. Bioinformatics 2010;26:2192-4.
31. Deorowicz S, Grabowski S. Compression of DNA sequence reads in FASTQ format. Bioinformatics 2011;27:860-2.
32. Bhola V, Bopardikar AS, Narayanan R, et al. No-reference compression of genomic data stored in FASTQ format. In:2011 IEEE International Conference on Bioinformatics and Biomedicine (BIBM'2011), Atlanta,Georgia 2011;147-50.
33. Bonfield JK, Mahoney MV. Compression of FASTQ and SAM format sequencing data. PLoS One 2013;8:e59190.
34. Yanovsky V. ReCoil - an algorithm for compression of extremely large datasets of DNA data. Algorithms Mol Biol 2011;6:23.
35. Cox AJ, Bauer MJ, Jakobi T, et al. Large-scale compression of genomic sequence databases with the Burrows-Wheeler transform. Bioinformatics 2012;28:1415-9.
36. Hach F, Numanagic I, Alkan C, et al. SCALCE: boosting sequence compression algorithms using locally consistent encoding. Bioinformatics 2012;28:3051-7.
37. Ziv J, Lempel A. A universal algorithm for sequential data compression. IEEETrans InformTheory 1977;23:337-43.
38. Rissanen J, Langdon GG. Arithmetic coding. IBMJResDev 1979;23:149-62.
39. Willems FM, Shtarkov YM, Tjalkens TJ. The context-tree weighting method: basic properties. IEEETrans InformTheory 1995;41:653-64.
40. Peterson J, Garges S, Giovanni M, et al. The NIH human microbiome project. GenomeRes 2009;19:2317-23.
41. Altshuler DM, Durbin RM, Abecasis GR, et al. An integrated map of genetic variation from 1,092 human genomes. Nature 2012;491:56-65.
42. Didelot X, Bowden R, Wilson DJ, et al. Transforming clinical microbiology with bacterial genome sequencing. Nat Rev Genet 2012;13:601-12.
43. Golomb S. Run-length encodings. IEEETrans InformTheory 1965;12:399-401.
44. Elias P. Universal codeword sets and representations of the integers. IEEETrans InformTheory 1975;21:194-203.
45. Huffman DA. A method for the construction of minimumredundancy codes. Proc IRE 1952;40:1098-101.
46. Miller W, Myers EW. A file comparison program. Software Pract Exp 1985;15:1025-40.
47. Hunt JJ, Vo K-P, Tichy WF. Delta algorithms: an empirical analysis. ACMTrans Software EngMethodol (TOSEM) 1998;7:192-214.
48. Pinho AJ, Ferreira PJSG, Neves AJR, et al. On the representability of complete genomes by multiple competing finite-context (Markov) models. PLoS One 2011;6:e21588.
49. Ahn S-M, Kim T-H, Lee S, et al. The first Korean genome sequence and analysis: full genome sequencing for a socioethnic group. GenomeRes 2009;19:1622-29.
50. Huala E, Dickerman AW, Garcia-Hernandez M, et al. The Arabidopsis Information Resource (TAIR): a comprehensive database and web-based information retrieval, analysis, and visualization system for a model plant. NucleicAcidsRes 2001;29:102-5.
51. Cock PJA, Fields CJ, Goto N, et al. The Sanger FASTQ file format for sequences with quality scores, and the Solexa/Illumina FASTQ variants. Nucleic Acids Res 2010;38:1767-71.
52. Li H, Handsaker B, Wysoker A, et al. The sequence alignment/map format and SAMtools. Bioinformatics 2009;25:2078-9.
53. Fritz MH-Y, Leinonen R, Cochrane G, et al. Efficient storage of high throughput DNA sequencing data using reference- based compression. GenomeRes 2011;21:734-40.
54. Langmead B, Trapnell C, Pop M, et al. Ultrafast and memory-efficient alignment of short DNA sequences to the human genome. Genome Biol 2009;10:R25.
55. Li H, Durbin R. Fast and accurate short read alignment with Burrows-Wheeler transform. Bioinformatics 2009;25:1754-60.
56. Li H, Homer N. A survey of sequence alignment algorithms for next-generation sequencing. Brief Bioinformatics 2010;11:473-83.
57. Daily K, Rigor P, Christley S, et al. Data structures and compression algorithms for high-throughput sequencing technologies. BMC Bioinformatics 2010;11:514.
58. Baldi P, Benz RW, Hirschberg DS, et al. Lossless compression of chemical fingerprints using integer entropy codes improves storage and retrieval. J Chem Inform Model 2007;47:2098-109.
59. Pevzner PA, Tang HX, Waterman MS. An eulerian path approach to DNA fragment assembly. Proc Natl Acad Sci USA 2001;98:9748-53.
60. Storer JA. Data Compression:Methods andTheory. New York, NY, USA: Computer Science Press, Inc., 1988.
61. Nelson M. Data compression book. Foster City, CA, USA:IDG Books Worldwide, Inc., 1991.
62. Dementiev R, Sanders P, Schultes D, et al. Engineering an external memory minimum spanning tree algorithm. Explor New Front Theoretical Informat, International Federation for Information Processing (IFIP) 2004;155:195-208.
63. Burrows M, Wheeler DJ. A block-sorting lossless data compression algorithm. SRCResearch Report 1994.
64. Sahinalp SC, Vishkin U. Efficient approximate and dynamic matching of patterns using a labeling paradigm. In:Proceedings of the 37th Annual Symposium on Foundations of Computer Science, Burlington,Vermont 1996;320-8.
65. Wan R, Anh VN, Asai K. Transformations for the compression of FASTQ quality scores of next-generation sequencing data. Bioinformatics 2012;28:628-35.
66. Ochoa I, Asnani H, Bharadia D, et al. QualComp: a new lossy compressor for quality scores based on rate distortion theory. BMC Bioinformatics 2013;14:187.
67. Cover TM, Thomas JA. Elements of information theory. New York, NY, USA: John Wiley & Sons, Inc, 1991.
68. Janin L, Rosone G, Cox AJ. Adaptive reference-free compression of sequence quality scores. Bioinformatics; doi:10.1093/bioinformatics/btt257 (Advance Access publication 9 May 2013).
69. Bauer MJ, Cox AJ, Rosone G, et al. Lightweight LCP construction for next-generation sequencing datasets. Algorithm Bioinformatics, LectureNotes in Computer Science (LNCS) 2012;7534:326-37.
70. Loh P-R, Baym M, Berger B. Compressive genomics. Nat Biotechnol 2012;30:627-30.
71. Ferragina P, Manzini G. Indexing compressed text. JACM 2005;52:552-81.
72. Makinen V, Navarro G, Siren J, et al. Storage and retrieval of highly repetitive sequence collections. J Computatl Biol 2010;17:281-308.
73. Makinen V, Navarro G, Siren J, et al. Storage and retrieval of individual genomes. Res ComputatMol Biol, Lecture Notes in Computer Science (LNCS) 2009;5541:121-37.
74. Arroyuelo D, Navarro G, Sadakane K. Stronger Lempel-Ziv based compressed text indexing. Algorithmica 2012;62:54-101.
75. Afgan E, Baker D, Coraor N, et al. Harnessing cloud computing with Galaxy Cloud. Nat Biotechnol 2011;29:972-4.
76. Stein LD. The case for cloud computing in genome informatics. Genome Biol 2010;11:207.
77. Cochrane G, Cook CE, Birney E. The future of DNA sequence archiving. GigaScience 2012;1:2.