Dataset size and composition impact the reliability of performance benchmarks for peptide-MHC binding predictions

BMC Bioinformatics

Volumn 15, Issue 1, 2014, Pages

  Kim, Yohan ^a   Sidney, John ^a   Buus, Søren ^b   Sette, Alessandro ^a   Nielsen, Morten ^c,d   Peters, Bjoern ^a  

^a LA JOLLA INSTITUTE FOR ALLERGY AND IMMUNOLOGY   (United States)

^b UNIVERSITY OF COPENHAGEN   (Denmark)

^c TECHNICAL UNIVERSITY OF DENMARK   (Denmark)

^d INSTITUTO DE INVESTIGACIONES BIOTECNOLÓGICAS   (Argentina)

Author keywords

Benchmarking of MHC class I predictors; Cross validation; Epitope prediction; Sequence similarity

Indexed keywords

BINDING ENERGY; EPITOPES; FORECASTING; ITERATIVE METHODS; PEPTIDES;

CLASS I; CROSS VALIDATION; EPITOPE PREDICTIONS; EXPECTED ERROR RATES; PERFORMANCE METRICS; PREDICTION PERFORMANCE; SEQUENCE SIMILARITY; TRAINING AND TESTING;

BENCHMARKING;

EPITOPE; HLA ANTIGEN; MHC BINDING PEPTIDE; OLIGOPEPTIDE;

ALLELE; ANIMAL; ARTICLE; BIOLOGY; GENETICS; HUMAN; IMMUNOLOGY; METABOLISM; METHODOLOGY; MOUSE; PROTEIN BINDING; QUALITY CONTROL; REPRODUCIBILITY;

ALLELES; ANIMALS; BENCHMARKING; COMPUTATIONAL BIOLOGY; EPITOPES; HLA ANTIGENS; HUMANS; MICE; OLIGOPEPTIDES; PROTEIN BINDING; REPRODUCIBILITY OF RESULTS;

EID: 84904076057     PISSN: None     EISSN: 14712105     Source Type: Journal    
DOI: 10.1186/1471-2105-15-241     Document Type: Article

References (28)

1. Murphy KM. Janeway's Immunobiology 2011, 7. NY, USA: Garland Science New York.
2. Sette A, Vitiello A, Reherman B, Fowler P, Nayersina R, Kast WM, Melief CJ, Oseroff C, Yuan L, Ruppert J. The relationship between class I binding affinity and immunogenicity of potential cytotoxic T cell epitopes. J Immunol 1994, 153:5586-5592.
3. Assarsson E, Sidney J, Oseroff C, Pasquetto V, Bui H-H, Frahm N, Brander C, Peters B, Grey H, Sette A. A quantitative analysis of the variables affecting the repertoire of T cell specificities recognized after vaccinia virus infection. J Immunol 2007, 178:7890-7901.
4. Brusic V, Rudy G, Harrison LC. MHCPEP, a database of MHC-binding peptides: update 1997. Nucleic Acids Res 1998, 26:368-371.
5. Schönbach C, Koh JLY, Sheng X, Wong L, Brusic V. FIMM, a database of functional molecular immunology. Nucleic Acids Res 2000, 28:222-224.
6. Bhasin M, Singh H, Raghava GPS. MHCBN: a comprehensive database of MHC binding and non-binding peptides. Bioinformatics 2003, 19:665-666.
7. Toseland CP, Clayton DJ, McSparron H, Hemsley SL, Blythe MJ, Paine K, Doytchinova IA, Guan P, Hattotuwagama CK, Flower DR. AntiJen: a quantitative immunology database integrating functional, thermodynamic, kinetic, biophysical, and cellular data. Immunome Res 2005, 1:4.
8. Vita R, Zarebski L, Greenbaum JA, Emami H, Hoof I, Salimi N, Damle R, Sette A, Peters B. The immune epitope database 2.0. Nucleic Acids Res 2010, 38:D854-D862.
9. Peters B, Bui H-H, Frankild S, Nielsen M, Lundegaard C, Kostem E, Basch D, Lamberth K, Harndahl M, Fleri W, Wilson SS, Sidney J, Lund O, Buus S, Sette A. A community resource benchmarking predictions of peptide binding to MHC-I molecules. PLoS Comput Biol 2006, 2:e65.
10. Lin HH, Ray S, Tongchusak S, Reinherz EL, Brusic V. Evaluation of MHC class I peptide binding prediction servers: applications for vaccine research. BMC Immunol 2008, 9:8.
11. Kim Y, Ponomarenko J, Zhu Z, Tamang D, Wang P, Greenbaum J, Lundegaard C, Sette A, Lund O, Bourne PE, Nielsen M, Peters B. Immune epitope database analysis resource. Nucleic Acids Res 2012, 40:W525-W530.
12. Kim Y, Sidney J, Buus S, Sette A, Nielsen M, Peters B. IEDB Analysis Resource Benchmark Datasets. [http://tools.iedb.org/benchmark].
13. Wang P, Sidney J, Kim Y, Sette A, Lund O, Nielsen M, Peters B. Peptide binding predictions for HLA DR, DP and DQ molecules. BMC Bioinformatics 2010, 11:568.
14. Kim Y, Sidney J, Pinilla C, Sette A, Peters B. Derivation of an amino acid similarity matrix for peptide: MHC binding and its application as a Bayesian prior. BMC Bioinformatics 2009, 10:394.
15. Zhang H, Lundegaard C, Nielsen M. Pan-specific MHC class I predictors: a benchmark of HLA class I pan-specific prediction methods. Bioinformatics 2009, 25:83-89.
16. El-Manzalawy Y, Dobbs D, Honavar V. On evaluating MHC-II binding peptide prediction methods. PLoS One 2008, 3:e3268.
17. Hosmer DW, Lemeshow S. Applied Logistic Regression 2004, 354. Hoboken, New Jersey: John Wiley & Sons.
18. Briesemeister S, Rahnenführer J, Kohlbacher O. No longer confidential: estimating the confidence of individual regression predictions. PLoS One 2012, 7:e48723.
19. Nielsen M, Lundegaard C, Worning P, Lauemøller SL, Lamberth K, Buus S, Brunak S, Lund O. Reliable prediction of T-cell epitopes using neural networks with novel sequence representations. Protein Sci 2003, 12:1007-1017.
20. Lundegaard C, Lamberth K, Harndahl M, Buus S, Lund O, Nielsen M. NetMHC-3.0: accurate web accessible predictions of human, mouse and monkey MHC class I affinities for peptides of length 8-11. Nucleic Acids Res 2008, 36:W509-W512.
21. Hoof I, Peters B, Sidney J, Pedersen LE, Sette A, Lund O, Buus S, Nielsen M. NetMHCpan, a method for MHC class I binding prediction beyond humans. Immunogenetics 2009, 61:1-13.
22. Nielsen M, Lundegaard C, Blicher T, Lamberth K, Harndahl M, Justesen S, Røder G, Peters B, Sette A, Lund O, Buus S. NetMHCpan, a method for quantitative predictions of peptide binding to any HLA-A and-B locus protein of known sequence. PLoS One 2007, 2:e796.
23. Zhang GL, Ansari HR, Bradley P, Cawley GC, Hertz T, Hu X, Jojic N, Kim Y, Kohlbacher O, Lund O, Lundegaard C, Magaret CA, Nielsen M, Papadopoulos H, Raghava GPS, Tal V-S, Xue LC, Yanover C, Zhu S, Rock MT, Crowe JE, Panayiotou C, Polycarpou MM, Duch W, Brusic V. Machine learning competition in immunology - prediction of HLA class I binding peptides. J Immunol Methods 2011, 374:1-4.
24. Zhang L, Udaka K, Mamitsuka H, Zhu S. Toward more accurate pan-specific MHC-peptide binding prediction: a review of current methods and tools. Briefings Bioinforma 2012, 13:350-364.
25. Swets JA. Measuring the accuracy of diagnostic systems. Science 1988, 240:1285-1293.
26. Paul S, Weiskopf D, Angelo MA, Sidney J, Peters B, Sette A. HLA class I alleles are associated with peptide-binding repertoires of different size, affinity, and immunogenicity. J Immunol 2013, 191:5831-5839.
27. Nielsen M, Lundegaard C, Blicher T, Peters B, Sette A, Justesen S, Buus S, Lund O. Quantitative predictions of peptide binding to Any HLA-DR molecule of known sequence: Netmhciipan. PLoS Comput Biol 2008, 4:e1000107.
28. MacKay DJC. Information theory, inference and learning algorithms 2003, New York, New York: Cambridge University Press.