sbv IMPROVER diagnostic signature challenge: Scoring strategies

Systems Biomedicine

Volumn 1, Issue 4, 2014, Pages 208-216

  Norel, Raquel ^a   Bilal, Erhan ^a   Conrad Chemineau, Nathalie ^b   Bonneau, Richard ^c   de la Fuente, Alberto ^d   Jurisica, Igor ^e   Marbach, Daniel ^f   Meyer, Pablo ^a   Rice, J Jeremy ^a   Tuller, Tamir ^g   Stolovitzky, Gustavo ^a  

^a IBM T J WATSON RESEARCH CENTER   (United States)

^b IBM   (Switzerland)

^c NEW YORK UNIVERSITY   (United States)

^d LEIBNIZ INSTITUTE FOR FARM ANIMAL BIOLOGY FBN   (Germany)

^e PRINCESS MARGARET HOSPITAL   (Canada)

^f MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^g TEL AVIV UNIVERSITY   (Israel)

Author keywords

Crowdsourcing; Diagnostic signature; Gene expression; Improver; Molecular classification; Peer review

Indexed keywords

ADENOCARCINOMA; AREA UNDER THE CURVE; ARTICLE; AVERAGE AREA UNDER THE PRECISION RECALL CURVE; BELIEF CONFUSION METRIC; CANCER STAGING; CHRONIC OBSTRUCTIVE LUNG DISEASE; CLINICAL ASSESSMENT; CLINICAL EVALUATION; CORRECT CLASS ENRICHMENT METRIC; DIAGNOSTIC IMAGING; DIAGNOSTIC TEST; DISEASE CLASSIFICATION; GENE EXPRESSION; GOLD STANDARD; HUMAN; LUNG CANCER; MAJOR CLINICAL STUDY; MULTIPLE SCLEROSIS; OUTCOME ASSESSMENT; PHENOTYPE; PSORIASIS; RECURRENT DISEASE; SCORING SYSTEM; SQUAMOUS CELL CARCINOMA; TRANSCRIPTOMICS;

EID: 84984671808     PISSN: 21628130     EISSN: 21628149     Source Type: Journal    
DOI: 10.4161/sysb.26326     Document Type: Article

References (23)

1. Meyer P, Hoeng J, Rice JJ, Norel R, Sprengel J, Stolle K, Bonk T, Corthesy S, Royyuru A, Peitsch MC, et al. Industrial methodology for process verification in research (IMPROVER): toward systems biology verification. Bioinformatics 2012; 28:1193-201; PMID:22423044; http://dx.doi.org/10.1093/bioinformatics/bts116
2. Diamandis EP. Cancer biomarkers: can we turn recent failures into success? J Natl Cancer Inst 2010; 102:1462-7; PMID:20705936; http://dx.doi. org/10.1093/jnci/djq306
3. Norel R, Rice JJ, Stolovitzky G. The self-assessment trap: can we all be better than average? Mol Syst Biol 2011; 7:537; PMID:21988833; http://dx.doi. org/10.1038/msb.2011.70
4. Van Noorden R. Science publishing: The trouble with retractions. Nature 2011; 478:26-8; PMID:21979026; http://dx.doi.org/10.1038/478026a
5. Wager E. Who is responsible for investigating suspected research misconduct? Anaesthesia 2012; 67:462-6; PMID:22404337; http://dx.doi. org/10.1111/j.1365-2044.2012.07132.x
6. Costello JC, Stolovitzky G. Seeking the wisdom of crowds through challenge-based competitions in biomedical research. Clin Pharmacol Ther 2013; 93:396-8; PMID:23549146; http://dx.doi. org/10.1038/clpt.2013.36
7. Meyer P, Alexopoulos LG, Bonk T, Califano A, Cho CR, de la Fuente A, de Graaf D, Hartemink AJ, Hoeng J, Ivanov NV, et al. Verification of systems biology research in the age of collaborative competition. Nat Biotechnol 2011; 29:811-5; PMID:21904331; http://dx.doi.org/10.1038/nbt.1968
8. Koren Y. Factorization Meets the Neighborhood: a Multifaceted Collaborative Filtering Model. In: Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining. New York, NY, USA: ACM, KDD '08, 426-34
9. Mariani V, Kiefer F, Schmidt T, Haas J, Schwede T. Assessment of template based protein structure predictions in CASP9. Proteins 2011; 79(Suppl 10):37-58; PMID:22002823; http://dx.doi.org/10.1002/prot.23177
10. Rhrissorrakrai K, Rice JJ, Boue S, Talikka M, Bilal E, Martin F, Meyer P, Norel R, Xiang Y, Stolovitzky G, et al. sbv IMPROVER Diagnostic Signature Challenge: Design and results. Systems Biomedicine 2013; e26325
11. Tarca AL, Lauria M, Unger M, Bilal E, Boue S, Dey KK, Hoeng J, Koeppl H, Martin F, Meyer P, et al.; IMPROVER DSC Collaborators. Strengths and limitations of microarray-based phenotype prediction: Lessons learned from the IMPROVER Diagnostic Signature Challenge. Bioinformatics 2013; In press; PMID:23966112; http://dx.doi.org/10.1093/bioinformatics/btt492
12. Moult J, Fidelis K, Kryshtafovych A, Tramontano A. Critical assessment of methods of protein structure prediction (CASP)--round IX. Proteins 2011; 79(Suppl 10):1-5; PMID:21997831; http://dx.doi. org/10.1002/prot.23200
13. Janin J. Assessing predictions of protein-protein interaction: the CAPRI experiment. Protein Sci 2005; 14:278-83; PMID:15659362; http://dx.doi. org/10.1110/ps.041081905
14. Leitner F, Mardis SA, Krallinger M, Cesareni G, Hirschman LA, Valencia A. An Overview of BioCreative II.5. IEEE/ACM Trans Comput Biol Bioinform 2010; 7:385-99; PMID:20704011; http://dx.doi.org/10.1109/TCBB.2010.61
15. Marbach D, Costello JC, Küffner R, Vega NM, Prill RJ, Camacho DM, Allison KR, Kellis M, Collins JJ, Stolovitzky G; DREAM5 Consortium. Wisdom of crowds for robust gene network inference. Nat Methods 2012; 9:796-804; PMID:22796662; http://dx.doi.org/10.1038/nmeth.2016
16. Shi L, Campbell G, Jones WD, Campagne F, Wen Z, Walker SJ, Su Z, Chu TM, Goodsaid FM, Pusztai L, et al.; MAQC Consortium. The MicroArray Quality Control (MAQC)-II study of common practices for the development and validation of microarray-based predictive models. Nat Biotechnol 2010; 28:827-38; PMID:20676074; http://dx.doi.org/10.1038/nbt.1665
17. Ransohoff DF. Proteomics research to discover markers: what can we learn from Netflix? Clin Chem 2010; 56:172-6; PMID:20040622; http://dx.doi. org/10.1373/clinchem.2009.126698
18. Dror G. The 2009 Knowledge Discovery in Data Competition (KDD Cup 2009). Challenges in Machine Learning 2011; 3
19. Cozzetto D, Kryshtafovych A, Fidelis K, Moult J, Rost B, Tramontano A. Evaluation of template-based models in CASP8 with standard measures. Proteins 2009; 77(Suppl 9):18-28; PMID:19731382; http://dx.doi.org/10.1002/prot.22561
20. Krallinger M, Vazquez M, Leitner F, Salgado D, Chatr-Aryamontri A, Winter A, Perfetto L, Briganti L, Licata L, Iannuccelli M, et al. The Protein-Protein Interaction tasks of BioCreative III: classification/ranking of articles and linking bio-ontology concepts to full text. BMC Bioinformatics 2011; 12(Suppl 8):S3; PMID:22151929; http://dx.doi. org/10.1186/1471-2105-12-S8-S3
21. Stolovitzky G, Prill RJ, Califano A. Lessons from the DREAM2 Challenges. Ann N Y Acad Sci 2009; 1158:159-95; PMID:19348640; http://dx.doi. org/10.1111/j.1749-6632.2009.04497.x
22. Marbach D, Prill RJ, Schaffter T, Mattiussi C, Floreano D, Stolovitzky G. Revealing strengths and weaknesses of methods for gene network inference. Proc Natl Acad Sci U S A 2010; 107:6286-91; PMID:20308593; http://dx.doi.org/10.1073/pnas.0913357107
23. Benjamini Y, Hochberg Y. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. J R Stat Soc, B 1995; 57:289-300