Mash: Fast genome and metagenome distance estimation using MinHash

Genome Biology

Volumn 17, Issue 1, 2016, Pages

  Ondov, Brian D ^a   Treangen, Todd J ^a   Melsted, Páll ^b   Mallonee, Adam B ^a   Bergman, Nicholas H ^a   Koren, Sergey ^c   Phillippy, Adam M ^c  

^a National Biodefense Analysis and Countermeasures Center   (United States)

^b UNIVERSITY OF ICELAND   (Iceland)

^c NATIONAL HUMAN GENOME RESEARCH INSTITUTE   (United States)

Author keywords

Alignment; Comparative genomics; Genomic distance; Metagenomics; Nanopore; Sequencing

Indexed keywords

ARTICLE; BIOINFORMATICS; CLUSTER ANALYSIS; GENE CLUSTER; GENE SEQUENCE; GENE STRUCTURE; GENETIC ANALYSIS; GENETIC DATABASE; GENETIC ENGINEERING; GENETIC PROCEDURES; HUMAN; METAGENOME; METAGENOMICS; MINHASH SKETCH; NONHUMAN; PHYLOGENETIC TREE; PHYLOGENY; SEQUENCE ALIGNMENT; GENOME; GENOMICS; MOLECULAR EVOLUTION; NUCLEIC ACID DATABASE; PROCEDURES; SOFTWARE;

CLUSTER ANALYSIS; DATABASES, NUCLEIC ACID; EVOLUTION, MOLECULAR; GENOME; GENOMICS; METAGENOME; METAGENOMICS; PHYLOGENY; SOFTWARE;

EID: 84978998700     PISSN: 14747596     EISSN: 1474760X     Source Type: Journal    
DOI: 10.1186/s13059-016-0997-x     Document Type: Article

References (63)

1. Altschul SF, Gish W, Miller W, Myers EW, Lipman DJ. Basic local alignment search tool. J Mol Biol. 1990;215:403-10.
2. GenBank and WGS Statistics. http://www.ncbi.nlm.nih.gov/genbank/statistics. Accessed 31 May 2016.
3. Stephens ZD, Lee SY, Faghri F, Campbell RH, Zhai C, Efron MJ, et al. Big data: astronomical or genomical? PLoS Biol. 2015;13:e1002195.
4. Broder AZ. On the resemblance and containment of documents. Compression and Complexity of Sequences 1997 - Proceedings 1998:21-29.
5. Indyk P, Motwani R. Approximate nearest neighbors: towards removing the curse of dimensionality. In: Proceedings of the Thirtieth Annual ACM Symposium on Theory of Computing. Dallas, TX: ACM; 1998.
6. Broder AZ. Identifying and filtering near-duplicate documents. In: COM '00 Proceedings of the 11th Annual Symposium on Combinatorial Pattern Matching. London: Springer; 2000. p. 1-10.
7. Chum O, Philbin J, Zisserman A. Near Duplicate Image Detection: min-Hash and tf-idf Weighting. In: Proceedings of the British Machine Vision Conference 2008. Durham, UK: British Machine Vision Association and Society for Pattern Recognition; 2008.
8. Narayanan M, Karp RM. Gapped local similarity search with provable guarantees. Algorithms in Bioinformatics, Proceedings. 2004;3240:74-86.
9. Berlin K, Koren S, Chin CS, Drake JP, Landolin JM, Phillippy AM. Assembling large genomes with single-molecule sequencing and locality-sensitive hashing. Nat Biotechnol. 2015;33:623-30.
10. Yang X, Zola J, Aluru S. Parallel metagenomic sequence clustering via sketching and maximal quasi-clique enumeration on map-reduce clouds. In: Parallel & Distributed Processing Symposium (IPDPS), 2011 IEEE International. IEEE. 2011. p. 1223-33.
11. Drew J, Hahsler M. Strand: fast sequence comparison using mapreduce and locality sensitive hashing. In: Proceedings of the 5th ACM Conference on Bioinformatics, Computational Biology, and Health Informatics. Newport Beach, CA: ACM; 2014.
12. Rasheed Z, Rangwala H. A Map-Reduce Framework for Clustering Metagenomes. In: 2013 IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum: IEEE. 2013.
13. Vinga S, Almeida J. Alignment-free sequence comparison-a review. Bioinformatics. 2003;19:513-23.
14. Haubold B. Alignment-free phylogenetics and population genetics. Brief Bioinform. 2014;15:407-18.
15. Blaisdell BE. A measure of the similarity of sets of sequences not requiring sequence alignment. Proc Natl Acad Sci U S A. 1986;83:5155-9.
16. Torney DC, Burks C, Davison D, Sirotkin KM. Computation of d2: a measure of sequence dissimilarity. In: Bell GI, Marr TG, editors. Computers and DNA: the proceedings of the Interface between Computation Science and Nucleic Acid Sequencing Workshop, held December 12 to 16, 1988 in Santa Fe, New Mexico. Redwood City: Addison-Wesley Pub. Co; 1990.
17. Lippert RA, Huang H, Waterman MS. Distributional regimes for the number of k-word matches between two random sequences. Proc Natl Acad Sci U S A. 2002;99:13980-9.
18. Yang K, Zhang L. Performance comparison between k-tuple distance and four model-based distances in phylogenetic tree reconstruction. Nucleic Acids Res. 2008;36:e33.
19. Deloger M, El Karoui M, Petit MA. A genomic distance based on MUM indicates discontinuity between most bacterial species and genera. J Bacteriol. 2009;191:91-9.
20. Yi H, Jin L. Co-phylog: an assembly-free phylogenomic approach for closely related organisms. Nucleic Acids Res. 2013;41:e75.
21. Haubold B, Klotzl F, Pfaffelhuber P. andi: fast and accurate estimation of evolutionary distances between closely related genomes. Bioinformatics. 2015;31:1169-75.
22. Fan H, Ives AR, Surget-Groba Y, Cannon CH. An assembly and alignment-free method of phylogeny reconstruction from next-generation sequencing data. BMC Genomics. 2015;16:522.
23. Konstantinidis KT, Tiedje JM. Genomic insights that advance the species definition for prokaryotes. Proc Natl Acad Sci U S A. 2005;102:2567-72.
24. Schatz MC, Phillippy AM. The rise of a digital immune system. Gigascience. 2012;1:4.
25. Pruitt KD, Tatusova T, Brown GR, Maglott DR. NCBI Reference Sequences (RefSeq): current status, new features and genome annotation policy. Nucleic Acids Res. 2012;40:D130-5.
26. Saitou N, Nei M. The neighbor-joining method: a new method for reconstructing phylogenetic trees. Mol Biol Evol. 1987;4:406-25.
27. Miller W, Rosenbloom K, Hardison RC, Hou M, Taylor J, Raney B, et al. 28-way vertebrate alignment and conservation track in the UCSC Genome Browser. Genome Res. 2007;17:1797-808.
28. Perelman P, Johnson WE, Roos C, Seuanez HN, Horvath JE, Moreira MA, et al. A molecular phylogeny of living primates. PLoS Genet. 2011;7:e1001342.
29. Kuhner MK, Felsenstein J. A simulation comparison of phylogeny algorithms under equal and unequal evolutionary rates. Mol Biol Evol. 1994;11:459-68.
30. Loman NJ, Quick J, Simpson JT. A complete bacterial genome assembled de novo using only nanopore sequencing data. Nat Methods. 2015;12:733-5.
31. Song L, Florea L, Langmead B. Lighter: fast and memory-efficient sequencing error correction without counting. Genome Biol. 2014;15:509.
32. Seth S, Valimaki N, Kaski S, Honkela A. Exploration and retrieval of whole-metagenome sequencing samples. Bioinformatics. 2014;30:2471-9.
33. Maillet N, Lemaitre C, Chikhi R, Lavenier D, Peterlongo P. Compareads: comparing huge metagenomic experiments. BMC Bioinformatics. 2012;13 Suppl 19:S10.
34. Maillet N, Collet G, Vannier T, Lavenier D, Peterlongo P. COMMET: comparing and combining multiple metagenomic datasets. In: 2014 IEEE International Conference on Bioinformatics and Biomedicine (BIBM): IEEE. 2014.
35. Rusch DB, Halpern AL, Sutton G, Heidelberg KB, Williamson S, Yooseph S, et al. The Sorcerer II Global Ocean Sampling expedition: northwest Atlantic through eastern tropical Pacific. PLoS Biol. 2007;5:e77.
36. Human Microbiome Project C. Structure, function and diversity of the healthy human microbiome. Nature. 2012;486:207-14.
37. Qin J, Li R, Raes J, Arumugam M, Burgdorf KS, Manichanh C, et al. A human gut microbial gene catalogue established by metagenomic sequencing. Nature. 2010;464:59-65.
38. Freedman MJ, Nissim K, Pinkas B. Efficient private matching and set intersection. Advances in Cryptology - Eurocrypt 2004, Proceedings. 2004;3027:1-19.
39. De Cristofaro E, Faber S, Gasti P, Tsudik G. Genodroid: are privacy-preserving genomic tests ready for prime time? In: Proceedings of the 2012 ACM workshop on Privacy in the electronic society. Raleigh, NC: ACM; 2012.
40. Solomon B, Kingsford C. Large-scale search of transcriptomic read sets with sequence bloom trees. bioRxiv. 2015. doi: 10.1101/017087.
41. Fofanov Y, Luo Y, Katili C, Wang J, Belosludtsev Y, Powdrill T, et al. How independent are the appearances of n-mers in different genomes? Bioinformatics. 2004;20:2421-8.
42. Roberts M, Hayes W, Hunt BR, Mount SM, Yorke JA. Reducing storage requirements for biological sequence comparison. Bioinformatics. 2004;20:3363-9.
43. Roberts M, Hunt BR, Yorke JA, Bolanos RA, Delcher AL. A preprocessor for shotgun assembly of large genomes. J Comput Biol. 2004;11:734-52.
44. Deorowicz S, Kokot M, Grabowski S, Debudaj-Grabysz A. KMC 2: fast and resource-frugal k-mer counting. Bioinformatics. 2015;31:1569-76.
45. Wood DE, Salzberg SL. Kraken: ultrafast metagenomic sequence classification using exact alignments. Genome Biol. 2014;15:R46.
46. Patrascu M, Thorup M. The power of simple tabulation hashing. J ACM. 2012;59:14.
47. Ukkonen E. Approximate string-matching with Q-grams and maximal matches. Theor Comput Sci. 1992;92:191-211.
48. Bar-Yossef Z, Jayram TS, Kumar R, Sivakumar D, Trevisan L. Counting distinct elements in a data stream. In: Proceedings of the 6th International Workshop on Randomization and Approximation Techniques. Springer-Verlag; 2002. p. 1-10.
49. Phillippy AM, Schatz MC, Pop M. Genome assembly forensics: finding the elusive mis-assembly. Genome Biol. 2008;9:R55.
50. Felsenstein J. PHYLIP - Phylogeny Inference Package (Version 3.2). Cladistics. 1989;5:164-6.
51. UCSC multiz20way. http://hgdownload.cse.ucsc.edu/goldenPath/hg38/multiz20way/. Accessed 31 May 2016.
52. HMP Illumina WGS Reads. http://hmpdacc.org/HMIWGS/all/. Accessed 31 May 2016.
53. HMP Illumina WGS Assemblies. http://hmpdacc.org/HMASM/all/. Accessed 31 May 2016.
54. MetaHIT assemblies. http://www.bork.embl.de/~arumugam/Qin_et_al_2010/. Accessed 31 May 2016.
55. Li H, Durbin R. Fast and accurate short read alignment with Burrows-Wheeler transform. Bioinformatics. 2009;25:1754-60.
56. Cap'n Proto. https://capnproto.org. Accessed 31 May 2016.
57. MurmurHash3. https://code.google.com/p/smhasher. Accessed 31 May 2016.
58. Gough B. GNU scientific library reference manual. Godalming: Network Theory Ltd.; 2009.
59. Open Bloom Filter Library. https://code.google.com/p/bloom. Accessed 31 May 2016.
60. Siek JG, Lee L-Q, Lumsdaine A. The Boost Graph Library: User Guide and Reference Manual. New York, NY: Pearson Education; 2001.
61. Shannon P, Markiel A, Ozier O, Baliga NS, Wang JT, Ramage D, et al. Cytoscape: a software environment for integrated models of biomolecular interaction networks. Genome Res. 2003;13:2498-504.
62. Kamada T, Kawai S. An algorithm for drawing general undirected graphs. Inform Process Lett. 1989;31:7-15.
63. Bankevich A, Nurk S, Antipov D, Gurevich AA, Dvorkin M, Kulikov AS, et al. SPAdes: a new genome assembly algorithm and its applications to single-cell sequencing. J Comput Biol. 2012;19:455-77.