Statistical strategies for avoiding false discoveries in metabolomics and related experiments

Metabolomics

Volumn 2, Issue 4, 2006, Pages 171-196

  Broadhurst, David I ^a   Kell, Douglas B ^a  

^a UNIVERSITY OF MANCHESTER   (United Kingdom)

Author keywords

Bias; Bonferroni correction; Credit assignment; Cross validiation; False discovery; Hypothesis testing; Machine learning; Over fitting; Receiver operator characteristic; Statistical power; Statistics; Visualisation

Indexed keywords

EID: 33846240326     PISSN: 15733882     EISSN: 15733890     Source Type: Journal    
DOI: 10.1007/s11306-006-0037-z     Document Type: Article

References (246)

1. Adriaans P., Zantinge D. (1996) Data Mining. Addison-Wesley, Harlow, Essex
2. Alsberg B.K., Kell D.B., Goodacre R. (1998) Variable selection in discriminant partial least-squares analysis. Anal. Chem. 70: 4126-4133
3. Alsberg B.K., Woodward A.M., Winson M.K., Rowland J., Kell D.B. (1997) Wavelet denoising of infrared spectra. Analyst 122: 645-652
4. Altman D.G. (2001) Systematic reviews of evaluations of prognostic variables. BMJ 323: 224-228
5. Altman D.G., Deeks J.J. (2002) Meta-analysis, Simpson's paradox, and the number needed to treat. BMC Med. Res. Methodol. 2: 3
6. Anthony M., Biggs N. (1992) Computational Learning Theory. Cambridge University Press, Cambridge
7. Baggerly K.A., Morris J.S., Coombes K.R. (2004) Reproducibility of SELDI-TOF protein patterns in serum: Comparing datasets from different experiments. Bioinformatics 20: 777-785
8. Baker S.G. (2003) The central role of receiver operating characteristic (ROC) curves in evaluating tests for the early detection of cancer. J. Natl. Cancer Inst. 95: 511-515
9. Baldi P., Long A.D. (2001) A Bayesian framework for the analysis of microarray expression data: Regularized t-test and statistical inferences of gene changes. Bioinformatics 17: 509-519
10. Barrow J.D., Silk J. (1995) The Left Hand of Creation: The Origin and Evolution of The Expanding Universe. Penguin, London
11. Bellman R. (1961) Adaptive Control Processes: A Guided Tour. Princeton University Press, Princeton, NJ
12. Benjamini Y., Hochberg Y. (1995) Controlling the false discovery rate - a practical and powerful approach to multiple testing. J. Roy. Stat. Soc. B Met. 57: 289-300
13. Bennett K., Demiriz A. (1998) Semi-supervised support vector machines. Adv. Neural Inf. Proc. Syst. 12: 368-374
14. Bernardo J.M., Smith A.F.M. (2000) Bayesian Theory. Wiley, Chichester
15. Berry D.A. (1996) Statistics: A Bayesian Perspective. Duxbury Press, Belmont
16. Berry M.J.A., Linoff G.S. (2000) Mastering the Art of Data Mining. Wiley, New York
17. Bezdek J.C. and Pal, S.K. (Eds) (1992). Fuzzy Models for Pattern recognition: Methods That Search for Structures In Data. IEEE Press., New York
18. Bland J.M., Altman D.G. (1995) Multiple significance tests: The Bonferroni method. BMJ 310: 170
19. Bland M. (2000) An Introduction to Medical Statistics. Oxford University Press, Oxford
20. Box G.E.P., Hunter W.G., Hunter J.S. (1978) Statistics for Experimenters. Wiley, New York
21. Bradford Hill A., Hill I.D. (1991) Bradford Hill's Principles of medical statistics 12. Edward Arnold, London
22. Breiman L. (1966) The heuristics of instability in model selection. Ann. Statist. 24: 2350-2381
23. Breiman L. (2001) Statistical modeling: The two cultures. Stat. Sci. 16: 199-215
24. Brenner H., Gefeller O. (1997) Variation of sensitivity, specificity, likelihood ratios and predictive values with disease prevalence. Stat. Med. 16: 981-91
25. Brent R. (1999) Functional genomics: Learning to think about gene expression data. Curr. Biol. 9: R338-R341
26. Brent R. (2000) Genomic biology. Cell 100: 169-183
27. Brent R., Lok L. (2005) A fishing buddy for hypothesis generators. Science 308: 504-506
28. Brereton R.G. (2003) Chemometrics: Data Analysis for the Laboratory and Chemical Plant. Wiley, New York
29. Broadhurst D., Goodacre R., Jones A., Rowland J.J. Kell D.B. (1997) Genetic algorithms as a method for variable selection in multiple linear regression and partial least squares regression, with applications to pyrolysis mass spectrometry. Anal. Chim. Acta. 348: 71-86
30. Brown M., Dunn W.B., Ellis D.I., Goodacre R., Handl J., Knowles J.D., O'Hagan S., Spasic I., Kell D.B. (2005) A metabolome pipeline: From concept to data to knowledge. Metabolomics 1: 35-46
31. Cabena P., Hadjinian P., Stadler R., Verhees J., Zanasi A. (1998) Discovering Data Mining: From Concept to Implementation. Prentice Hall, Englewood Cliffs, NJ
32. Camacho D., de la Fuente A., Mendes P. (2005) The origins of correlations in metabolomics data. Metabolomics 1: 53-63
33. Cascante M., Boros L.G., Comin-Anduix B., de Atauri P., Centelles J.J., Lee P.W. (2002) Metabolic control analysis in drug discovery and disease. Nat. Biotechnol. 20: 243-249
34. Casella G., Berger R.L. (2002) Statistical Inference, 2. Duxbury, Pacific Grove, CA
35. Catchpole G.S., Beckmann M., Enot D.P., Mondhe M., Zywicki B., Taylor J., Hardy N., Smith A., King R.D., Kell D.B., Fiehn O., Draper J. (2005) Hierarchical metabolomics demonstrates substantial compositional similarity between genetically modified and conventional potato crops. Proc. Natl. Acad. Sci. 102: 14458-14462
36. Chatfield C. (1995) Model uncertainty, data mining and statistical inference. J. Roy. Stat. Soc. Ser. A 158: 419-466
37. Chen M., Hofestädt R. (2006) A medical bioinformatics approach for metabolic disorders: Biomedical data prediction, modeling, and systematic analysis. J. Biomed. Inform. 39: 147-159
38. Chen V.C.P., Tsui K.L., Barton R.R., Meckesheimer M. (2006) A review on design, modeling and applications of computer experiments. IIE Trans. 38: 273-291
39. Cleveland W.S. (1993) Visualizing Data. Hobart Press, Summit, NJ
40. Cleveland W.S. (1994) The Elements of Graphing Data. Hobart Press, Summit, NJ
41. Coello Coello C.A., van Veldhuizen D.A., Lamont G.B. (2002) Evolutionary Algorithms for Solving Multi-Objective Problems. Kluwer Academic Publishers, New York
42. Conover W.J. (1980) Practical Nonparametric Statistics. Wiley, New York
43. Cook R.J., Farewell V.T. (1996) Multiplicity considerations in the design and analysis of clinical trials. J. Roy. Stat. Soc. A 159: 93-110
44. Cornfield J. (1966) Sequential trials, sequential analysis and likelihood rinciple. Am. Stat. 20: 18-23
45. Cornish-Bowden A., Cárdenas M.L. (2000) From genome to cellular phenotype-a role for metabolic flux analysis? Nat. Biotechnol. 18: 267-269
46. Crary S.B. (2002) Design of computer experiments for metamodel generation. Analog. Integr. Circ. Sig. Proc. 32: 7-16
47. Cui X., Churchill G.A. (2003) Statistical tests for differential expression in cDNA microarray experiments. Genome Biol. 4: 210
48. Dasgupta P., Chakrabarti P.P., DeSarkar S.C. (1999) Multiobjective Heuristic Search. Vieweg, Braunschweig
49. Deb K. (2001) Multi-Objective Optimization Using Evolutionary Algorithms. Wiley, New York
50. Deming S.N., Morgan S.L. (1993) Experimental Design: A Chemometric Approach. Elsevier, Amsterdam
51. Demiriz A., Bennett K., Embrechts M.J. (1999) Semi-supervised clustering using genetic algorithms. In Dagli C.H., Buczak A.L., Ghosh J., Embrechts M.J., Ersoy O. (Eds.), Intelligent Engineering Systems Through Artificial Neural Networks. ASME Press, New York, pp. 809-814
52. di Bernardo D., Thompson M.J., Gardner T.S., Chobot S.E., Eastwood E.L., Wojtovich A.P., Elliott S.J., Schaus S.E., Collins J.J. (2005) Chemogenomic profiling on a genome-wide scale using reverse-engineered gene networks. Nat. Biotechnol. 23: 377-383
53. Diamandis E.P. (2004) Analysis of serum proteomic patterns for early cancer diagnosis: Drawing attention to potential problems. J. Natl. Cancer Inst. 96: 353-356
54. Duda R.O., Hart P.E., Stork D.E. (2001) Pattern Classification, 2. John Wiley, London
55. Duesberg P., Stindl R., Hehlmann R. (2000) Explaining the high mutation rates of cancer cells to drug and multidrug resistance by chromosome reassortments that are catalyzed by aneuploidy. Proc. Natl. Acad. Sci. USA 97: 14295-14300
56. Eades P. (1984) A heuristic for graph drawing. Congressus Numerantium 42: 149-160
57. Ebbels T.M.D., Buxton B.F., Jones D.T. (2006) springScape: Visualisation of microarray and contextual bioinformatic data using spring embedding an 'information landscape'. Bioinformatics 22, e99-e108
58. Edwards A.W.F. (1992) Likelihood. Johns Hopkins University Press, Baltimore
59. Edwards D. (2000) Introduction to Graphical Modeling. 2nd ed. Springer, Berlin
60. Efron B., Gong G. (1983) A Leisurely Look at the Bootstrap, the Jackknife, and Cross-Validation. Am. Stat. 37: 36-48
61. Efron B., Tibshirani R. (2002) Empirical Bayes methods and false discovery rates for microarrays. Genet. Epidemiol. 23: 70-86
62. Efron B., Tibshirani R.J. (1993) Introduction to the Bootstrap. Chapman and Hall, London
63. Egan J.P. (1975) Signal Detection Theory and ROC Analysis. Academic Press, New York
64. Ein-Dor L., Zuk O., Domany E. (2006) Thousands of samples are needed to generate a robust gene list for predicting outcome in cancer. Proc. Natl. Acad. Sci USA 103: 5923-5928
65. Eriksson L., Johansson E., Kettaneh-Wold N., Wold S. (2001) Multi- and Megavariate Data Analysis: Principles and Applications. Umetrics Academy, Umeå
66. Evans W.E., Johnson J.A. (2001) Pharmacogenomics: The inherited basis for interindividual differences in drug response. Annu. Rev. Genomics. Hum. Genet. 2: 9-39
67. Evans W.E., Relling M.V. (1999) Pharmacogenomics: Translating functional genomics into rational therapeutics. Science 286: 487-491
68. Evans W.E., Relling M.V. (2004) Moving towards individualized medicine with pharmacogenomics. Nature 429: 464-468
69. Everitt B.S. (1993) Cluster Analysis. Edward Arnold, London
70. Farnum M.A., DesJarlais, R. and Agrafiotis, D.K. (2003). Molecular diversity in Gasteiger, J. (Ed.), Handbook of Cheminformatics: Vol 4 From Data to Knowledge. Wiley/VCH, Weinheim, pp. 1640-1686
71. Fell D.A. (1996) Understanding the Control of Metabolism. Portland Press, London
72. Fielding A.H., Bell J.F. (1997) A review of methods for the assessment of prediction errors in conservation presence/absence models. Environ. Conserv. 24: 38-49
73. Fortner B. (1995) The Data Handbook. 2nd ed. Springer, New York
74. Frey H.C., Patil S.R. (2002) Identification and review of sensitivity analysis methods. Risk Anal. 22: 553-578
75. Friendly M. (2000) Visualising Categorical Data. SAS Institute, Cary, NC
76. Fruchterman T.M.J., Reingold E.M. (1991) Graph Drawing by Force-Directed Placement. Software - practice & experience 21: 1129-1164
77. Gansner E.R., North S.C. (2000) An open graph visualization system and its applications to software engineering. Software: Practice and Experience 30: 1203-1233
78. Gardner M.J., Altman D.G. (1989) Statistics with Confidence: Confidence Intervals And Statistical Guidelines. BMJ, London
79. Gillet V.J., Khatib W., Willett P., Fleming P.J., Green D.V.S. (2002) Combinatorial library design using a multiobjective genetic algorithm. J. Chem. Inf. Comput. Sci. 42: 375-385
80. Goble C.A., Stevens R., Ng G., Bechhofer S., Paton N.W., Baker P.G., Peim M., Brass A. (2001) Transparent access to multiple bioinformatics information sources. IBM. Syst. J. 40: 532-551
81. Goffeau A., Barrell B.G., Bussey H., Davis R.W., Dujon B., Feldmann H., Galibert F., Hoheisel J.D., Jacq C., Johnston M., Louis E.J., Mewes H.W., Murakami Y., Philippsen P., Tettelin H., Oliver S.G. (1996) Life With 6000 Genes. Science 274: 546-567
82. Golbraikh A., Tropsha A. (2002) Beware of q2!. J. Mol. Graph Model 20: 269-276
83. Golub T.R., Slonim D.K., Tamayo P., Huard C., Gaasenbeek M., Mesirov J.P., Coller H., Loh M.L., Downing J.R., Caligiuri M.A., Bloomfield C.D., Lander E.S. (1999) Molecular classification of cancer: Class discovery and class prediction by gene expression monitoring. Science 286: 531-537
84. Goodacre R., Kell D.B. (2003) Evolutionary computation for the interpretation of metabolome data. In Harrigan G.G., Goodacre R. (Eds.), Metabolic Profiling: Its Role in Biomarker Discovery and Gene Function Analysis. Kluwer Academic Publishers, Boston, pp. 239-256
85. Goodacre R., Neal M.J., Kell D.B. (1996) Quantitative analysis of multivariate data using artificial neural networks: A tutorial review and applications to the deconvolution of pyrolysis mass spectra. Z. Bakteriol. 284: 516-539
86. Goodman S.N., Royall R. (1988) Evidence and scientific research. Am. J. Publ. Health 78: 1568-1574
87. Greenaway W., May J., Scaysbrook T., Whatley F.R. (1991) Identification by gas chromatography-mass spectrometry of 150 compounds in propolis. Z. Naturforsch. C 46: 111-121
88. Grimes D.S. (2006) Are statins analogues of vitamin D? Lancet 368: 83-6
89. Hand D., Mannila H., Smyth P. (2001) Principles of Data Mining. MIT Press, Cambridge, MA
90. Handl, J., Kell, D.B. and Knowles, J. (2006). Multiobjective optimization in bioinformatics and computational biology. IEEE Trans Comput Biol Bioinformatics (in the press)
91. Handl, J. and Knowles, J. (2004). Evolutionary Multiobjective Clustering. PPSN VIII, LNCS 3242, 1081-1091 (see http://dbk.ch.umist.ac.uk/Papers/HandlKnowlesPPSN-webversion.pdf)
92. Handl, J. and Knowles, J. (2006a) An evolutionary approach to multiobjective clustering. IEEE Trans Evol Comput (in press)
93. Handl, J. and Knowles, J. (2006b). Semi-supervised feature selection via multiobjective optimization. International Joint Conference on Neural Networks (IJCNN 2006). Proc WCCI 2006, IEEE Press, pp. 6351-6358
94. Handl J., Knowles J., Kell D.B. (2005) Computational cluster validation in post-genomic data analysis. Bioinformatics 21: 3201-3212
95. Hanley J.A., McNeil B.J. (1982) The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143: 29-36
96. Harrigan G.G., LaPlante R.H., Cosma G.N., Cockerell G., Goodacre R., Maddox J.F., Luyendyk J.P., Ganey P.E., Roth R.A. (2004) Application of high-throughput Fourier-transform infrared spectroscopy in toxicology studies: Contribution to a study on the development of an animal model for idiosyncratic toxicity. Toxicol. Lett. 146: 197-205
97. Hastie T., Tibshirani R., Friedman J. (2001) The Elements Of Statistical Learning: Data Mining, Inference and Prediction. Springer-Verlag, Berlin
98. Heinrich R., Schuster S. (1996) The Regulation Of Cellular Systems. Chapman & Hall, New York
99. Hicks C.R., Turner K.V. Jr (1999) Fundamental Concepts in the Design of Experiments. 5th ed. Oxford University Press, Oxford
100. Hollander M., Wolfe D.A. (1973) Nonparametric Statistical Methods. Wiley, New York
101. Horchner U., Kalivas J.H. (1995) Further investigation on a comparative study of simulated annealing and genetic algorithm for wavelength selection. Anal. Chim. Acta. 311: 1-13
102. Horning E.C., Horning M.G. (1971) Metabolic profiles: Gas-phase methods for analysis of metabolites. Clin Chem 17: 802-809
103. Hubert L., Arabie P. (1985) Comparing partitions. J. Classif. 2: 193-218
104. Hutchinson A. (1994) Algorithmic Learning. Clarendon Press, Oxford
105. Ioannidis J.P. (2005a) Contradicted and initially stronger effects in highly cited clinical research. JAMA 294: 218-228
106. Ioannidis J.P. (2005b) Why most published research findings are false. PLoS Med. 2, e124
107. Ioannidis J.P., Ntzani E.E., Trikalinos T.A., Contopoulos-Ioannidis D.G. (2001) Replication validity of genetic association studies. Nat. Genet. 29: 306-309
108. Ioannidis J.P., Trikalinos T.A. (2005) Early extreme contradictory estimates may appear in published research: The Proteus phenomenon in molecular genetics research and randomized trials. J. Clin. Epidemiol. 58: 543-549
109. Ioannidis J.P., Trikalinos T.A., Ntzani E.E., Contopoulos-Ioannidis D.G. (2003) Genetic associations in large versus small studies: An empirical assessment. Lancet 361: 567-571
110. Jarvis R.M., Goodacre R. (2005) Genetic algorithm optimization for pre-processing and variable selection of spectroscopic data. Bioinformatics 21: 860-868
111. Jellum E., Bjornson I., Nesbakken R., Johansson E., Wold S. (1981) Classification of human cancer cells by means of capillary gas chromatography and pattern recognition analysis. J. Chromatogr. 217: 231-237
112. Jensen F.V. (2001) Bayesian Networks and Decision Graphs. Springer, Berlin
113. Jolliffe I.T. (1986) Principal Component Analysis. Springer-Verlag, New York
114. Judson R. (1997) Genetic algorithms and their use in chemistry. Rev. Comput. Chem. 10: 1-73
115. Jung S.H. (2005) Sample size for FDR-control in microarray data analysis. Bioinformatics 21: 3097-104
116. Kamada T., Kawai S. (1989) An algorithm for drawing general undirected graphs. Inf. Proc. Lett. 31: 7-15
117. Kannel W.B. (1995) Range of serum cholesterol values in the population developing coronary artery disease. Am. J. Cardiol. 76: 69C-77C
118. Kell D.B. (2G02a) Genotype:phenotype mapping: genes as computer programs. Trends. Genet. 18: 555-559
119. Kell D.B. (2002b) Metabolomics and machine learning: Explanatory analysis of complex metabolome data using genetic programming to produce simple, robust rules. Mol. Biol. Rep. 29: 237-41
120. Kell D.B. (2004) Metabolomics and systems biology: Making sense of the soup. Curr. Op. Microbiol. 7: 296-307
121. Kell D.B. (2006) Metabolomics, modelling and machine learning in systems biology: Towards an understanding of the languages of cells. The 2005 Theodor Bücher lecture. FEBS J. 273: 873-894
122. Kell D.B., Brown M., Davey H.M., Dunn W.B., Spasic I., Oliver S.G. (2005) Metabolic footprinting and Systems Biology: The medium is the message. Nat. Rev. Microbiol. 3: 557-565
123. Kell D.B., Darby R.M., Draper J. (2001) Genomic computing: Explanatory analysis of plant expression profiling data using machine learning. Plant. Physiol. 126: 943-951
124. Kell D.B., King R.D. (2000) On the optimization of classes for the assignment of unidentified reading frames in functional genomics programmes: The need for machine learning. Trends Biotechnol. 18: 93-98
125. Kell D.B., Knowles J.D. (2006) The role of modeling in systems biology. In Szallasi Z., Stelling J., Periwal V. (Eds.), System Modeling in Cellular Biology: From Concepts to Nuts and Bolts. MIT Press, Cambridge, pp. 3-18
126. Kell D.B., Oliver S.G. (2004) Here is the evidence, now what is the hypothesis? The complementary roles of inductive and hypothesis-driven science in the post-genomic era. Bioessays 26: 99-105
127. Kell D.B., Sonnleitner B. (1995) GMP - Good Modelling Practice: An essential component of good manufacturing practice. Trends Biotechnol. 13: 481-492
128. Kell, D.B. and Welch, G.R. (1991). No turning back, Reductonism and Biological Complexity. Times Higher Educational Supplement 9th August, 15
129. Kell D.B., Westerhoff H.V. (1986) Metabolic control theory: Its role in microbiology and biotechnology. FEMS Microbiol. Rev. 39: 305-320
130. Kemp, C., Griffiths, T., Stromsten, S. and Tenenbaum, J.B. (2003) Semi-supervised learning with trees. Adv. Neural Inf Proc Syst 16
131. Kenny, L.C., Dunn, W.B., Ellis, D.I., Myers, J., Baker, P.N., The GOPEC Consortium and Kell, D.B. (2005) Novel biomarkers for pre-eclampsia detected using metabolomics and machine learning. Metabolomics 1, 227-234 - online DOI: 10.1007/s11306-005-0003-1
132. Kim S.K., Lund J., Kiraly M., Duke K., Jiang M., Stuart J.M., Eizinger A., Wylie B.N., Davidson G.S. (2001) A gene expression map for Caenorhabditis elegans. Science 293: 2087-2092
133. Kirkwood B.R., Sterne J.A.C. (2003) Essential Medical Statistics. Blackwell, Oxford
134. Kirschenlohr H.L., Griffin J.L., Clarke S.C., Rhydwen R., Grace A.A., Schofield P.M., Brindle K.M., Metcalfe J.C. (2006) Proton NMR analysis of plasma is a weak predictor of coronary artery disease. Nat. Med. 12: 705-710
135. Knowles, J.D. and Hughes, E.J. (2005). Multiobjective optimization on a budget of 250 evaluations. Evolutionary Multi-Criterion Optimization (EMO 2005), LNCS 3410, 176-190 http://dbk.ch.umist.ac.uk/knowles/ pubs.html
136. Knowles, J.D., Watson, R.A. and Corne, D.W. (2001). Reducing local optima in single-objective problems by multi-objectivization in E. Zitzler et al., (ed.), Proc. 1st Int. Conf. on Evolutionary Multi-criterion Optimization (EMO-01), Springer, Berlin, pp. 269-283
137. Kohavi, R. (1995). A study of cross-validation and bootstrap for accuracy estimation and model selection. Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, pp. 1137-1143
138. Kohonen T. (1989) Self-Organization and Associative Memory. Springer-Verlag, Berlin
139. Kose F., Weckwerth W., Linke T., Fiehn O. (2001) Visualizing plant metabolomic correlation networks using clique-metabolite matrices. Bioinformatics 17: 1198-1208
140. Koza J.R. (1992) Genetic Programming: On The Programming of Computers by Means Of Natural Selection. MIT Press, Cambridge, Mass
141. Koza J.R., Keane M.A., Streeter M.J., Mydlowec W., Yu J., Lanza G. (2003) Genetic Programming: Routine Human-Competitive Machine Intelligence. Kluwer, New York
142. Kruse R., Gebhardt J., Klawonn F. (1994) Foundations of Fuzzy Systems. John Wiley, Chichester
143. Kruskal, J.B. and Seery, J.B. (1980). Designing network diagrams. Proc. 1st General Conf. on Social Graphics, pp. 22-50
144. Krzanowski W.J. (1988) Principles of Multivariate Analysis: A User's Perspective. Oxford Univeristy Press, Oxford
145. Langdon W.B. (1998) Genetic Programming And Data Structures: Genetic Programming + Data Structures = Automatic Programming!. Kluwer, Boston
146. Langley P., Simon H.A., Bradshaw G.L., Zytkow J.M. (1987) Scientific Discovery: Computational Exploration Of The Creative Processes. MIT Press, Cambridge, MA
147. Leon A.C. (2004) Multiplicity-adjusted sample size requirements: A strategy to maintain statistical power with Bonferroni adjustments. J. Clin. Psychiatry 65: 1511-1514
148. Li H.-X., Yen V.C. (1995) Fuzzy Sets And Fuzzy Decision-Making. CRC Press, Boca Raton, Florida
149. Li, T., Zhu, S., Li, Q., and Ogihara, M. (2003). Gene functional classification by semi-supervised learning from heterogeneous data. Proc ACM Symp. Appl. Computing. pp. 78-82
150. Liang Y., Kelemen A. (2006) Associating phenotypes with molecular events: Recent statistical advances and challenges underpinning microarray experiments. Funct. Integr Genomics 6: 1-13
151. Linden A. (2006) Measuring diagnostic and predictive accuracy in disease management: An introduction to receiver operating characteristic (ROC) analysis. J. Eval. Clin. Pract. 12: 132-139
152. Lucasius C.B., Beckers M.L.M., Kateman G. (1994) Genetic algorithms in wavelength selection - a comparative-study. Analytica Chimica Acta 286: 135-153
153. Lucasius C.B., Kateman G. (1994) Understanding and using genetic algorithms. 2. Representation, configuration and hybridization. Chemometrics and Intelligent Laboratory Systems 25: 99-145
154. Mackay D.J.C. (2003) Information Theory, Inference and Learning Algorithms. Cambridge University Press, Cambridge
155. Manly B.F.J. (1994) Multivariate Statistical Methods: A Primer. Chapman and Hall, London
156. Martens H., Næs T. (1989) Multivariate Calibration. John Wiley, Chichester
157. Metz C.E. (1978) Basic principles of ROC analysis. Semin Nucl Med 8: 283-98
158. Michalewicz Z., Fogel D.B. (2000) How to Solve it: Modern Heuristics. Springer-Verlag, Heidelberg
159. Michalski R.S., Bratko I., Kubat M. (Eds) (1998) Machine Learning and Data Mining. Methods and applications, Wiley, Chichester
160. Michie D., Spiegelhalter D.J., Taylor C.C. (eds) (1994) Machine Learning Neural and Statistical Classification. Ellis Horwood, Chichester
161. Miller A.J. (1990) Subset Selection in Regression. Chapman and Hall, London
162. Mitchell T.M. (1997) Machine Learning. McGraw Hill, New York
163. Montgomery D.C. (2001) Design and Analysis of Experiments. 5th edition. Wiley, Chichester
164. Myers R.H., Montgomery D.C. (1995) Response Surface Methodology: Process and Product Optimization using Designed Experiments. Wiley, New York
165. Natarajan S., Glick H., Criqui M., Horowitz D., Lipsitz S.R., Kinosian B. (2003) Cholesterol measures to identify and treat individuals at risk for coronary heart disease. Am. J. Prev. Med. 25: 50-7
166. Needham C.J., Bradford J.R., Bulpitt A.J., Westhead D.R. (2006) Inference in Bayesian networks. Nat. Biotechnol. 24: 51-53
167. Ntzani E.E., Ioannidis J.P. (2003) Predictive ability of DNA microarrays for cancer outcomes and correlates: An empirical assessment. Lancet 362: 1439-44
168. O'Hagan S., Dunn W.B., Brown M., Knowles J.D., Kell D.B. (2005) Closed-loop, multiobjective optimisation of analytical instrumentation: gas-chromatography-time-of-flight mass spectrometry of the metabolomes of human serum and of yeast fermentations. Anal. Chem. 77: 290-303
169. Oakley J.E., O'Hagan A. (2004) Probabilistic sensitivity analysis of complex models: A Bayesian approach. JR Stat. Soc. A 66: 751-769
170. Obuchowski N.A., Lieber M.L., Wians F.H. Jr. (2004) ROC curves in clinical chemistry: Uses, misuses, and possible solutions. Clin. Chem. 50: 1118-25
171. Oinn T., Addis M., Ferris J., Marvin D., Senger M., Greenwood M., Carver T., Glover K., Pocock M.R., Wipat A., Li P. (2004) Taverna: A tool for the composition and enactment of bioinformatics workflows. Bioinformatics 20: 3045-3054
172. Oinn T., Li P., Kell D., Goble C., Goderis A., Greenwood M., Hull D., Stevens R., Turi D., Zhao J. (2006) Taverna/Mygrid: Aligning a Workflow System with the Life Sciences Community Workflows for eScience. Springer, Guildford, pp. 299-318
173. Oliver S.G., Winson M.K., Kell D.B., Baganz F. (1998) Systematic functional analysis of the yeast genome. Trends Biotechnol. 16: 373-378
174. Pearl J. (1988) Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, San Francisco
175. Pearl J. (2000) Causality: Models, Reasoning and Inference. Cambridge University Press, Cambridge
176. Peleg M., Yeh I., Altman R.B. (2002) Modelling biological processes using workflow and Petri Net models. Bioinformatics 18: 825-37
177. Perneger T.V. (1998) What's wrong with Bonferroni adjustments. BMJ 316: 1236-8
178. Petricoin E.F. III, Ardekani A.M., Hitt B.A., Levine P.J., Fusaro V.A., Steinberg S.M., Mills G.B., Simone C., Fishman D.A., Kohn E.C., Liotta L.A. (2002) Use of proteomic patterns in serum to identify ovarian cancer. Lancet 359: 572-577
179. Potter S.C., Clarke L., Curwen V., Keenan S., Mongin E., Searle S.M., Stabenau A., Storey R., Clamp M. (2004) The Ensembl analysis pipeline. Genome Res. 14: 934-941
180. Raamsdonk L.M., Teusink B., Broadhurst D., Zhang N., Hayes A., Walsh M., Berden J.A., Brindle K.M., Kell D.B., Rowland J.J., Westerhoff H.V., van Dam K., Oliver S.G. (2001) A functional genomics strategy that uses metabolome data to reveal the phenotype of silent mutations. Nat. Biotechnol. 19: 45-50
181. Ramoni M., Sabastini P. (1998) Theory and Practice of Bayesian Belief Networks. Edward Arnold, London
182. Ransohoff D.F. (2004) Rules of evidence for cancer molecular-marker discovery and validation. Nat. Rev. Cancer 4: 309-314
183. Ransohoff D.F. (2005) Bias as a threat to the validity of cancer molecular-marker research. Nat. Rev. Cancer 5: 142-149
184. Ransohoff D.F., Feinstein A.R. (1978) Problems of spectrum and bias in evaluating the efficacy of diagnostic tests. N. Engl. J. Med. 299: 926-930
185. Rapp P.E. (1993) Chaos in the neurosciences: Cautionary tales from the frontier. Biologist 40: 89-94
186. Raubertas R.F., Rodewald L.E., Humiston S.G., Szilagyi P.G. (1994) ROC curves for classification trees. Med. Decis. Making 14: 169-174
187. Reiner A., Yekutieli D., Benjamini Y. (2003) Identifying differentially expressed genes using false discovery rate controlling procedures. Bioinformatics 19: 368-375
188. Ressom H.W., Varghese R.S., Abdel-Hamid M., Eissa S.A., Saha D., Goldman L., Petricoin E.F., Conrads T.P., Veenstra T.D., Loffredo C.A., Goldman R. (2005) Analysis of mass spectral serum profiles for biomarker selection. Bioinformatics 21: 4039-4045
189. Rifai N., Gillette M.A., Carr S.A. (2006) Protein biomarker discovery and validation: The long and uncertain path to clinical utility. Nat. Biotechnol. 24: 971-983
190. Ringuest J.L. (1992) Multiobjective Optimization: Behavioral and Computational Considerations. Kluwer Academic Publishers, Dordrecht
191. Romano P., Marra D., Milanesi L. (2005) Web services and workflow management for biological resources. BMC Bioinformatics 6(Suppl 4), S24
192. Rothman K.J., Greenland S. (1998) Modern Epidemiology. 2nd ed. Lippincott, Williams & Wilkins, Philadelphia
193. Rowland J.J. (2003) Model selection methodology in supervised learning with evolutionary computation. Biosystems 72: 187-196
194. Royall R. (1997) Statistical Evidence: A Likelihood Paradigm. Chapman and Hall/CRC, London
195. Rud O.P. (2001) Data Mining Cookbook. Wiley, New York
196. Sacks J., Welch W., Mitchell T., Wynn H. (1989) Design and analysis of computer experiments (with discussion). Statist Sci 4: 409-435
197. Saltelli A., Tarantola S., Campolongo F., Ratt M. (2004) Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models. Wiley, New York
198. Sammon J.W. Jr. (1969) A nonlinear mapping for data structure analysis. IEEE Trans. Computers C-18: 401-409
199. Schena M. (Ed) (2000) Microarray Biochip Technology. Eaton Publishing, Natick, MA
200. Seasholtz M.B., Kowalski B. (1993) The parsimony principle applied to multivariate calibration. Anal. Chim. Acta 277: 165-177
201. Seber G.A.F., Wild C.J. (1989) Nonlinear Regression. Wiley, New York
202. Sehgal M.S., Gondal I., Dooley L.S. (2005) Collateral missing value imputation: A new robust missing value estimation algorithm for microarray data. Bioinformatics 21: 2417-2423
203. Shaffer R.E., Small G.W. (1997) Learning optimization from nature - genetic algorithms and simulated annealing. Anal. Chem. 69, A236-A242
204. Sharp S.J., Thompson S.G., Altman D.G. (1996) The relation between treatment benefit and underlying risk in meta-analysis. BMJ 313: 735-738
205. Shipley B. (2001) Cause and Correlation in Biology: A User's Guide to Path Analysis, Structural Equations and Causal Inference. Cambridge University Press, Cambridge
206. Sokal R.R., Rohlf F.J. (1995) Biometry. 3rd edition. Freeman, New York
207. Stephan C., Wesseling S., Schink T., Jung K. (2003) Comparison of eight computer programs for receiver-operating characteristic analysis. Clin. Chem. 49: 433-439
208. Steuer R. (2006) On the analysis and interpretation of correlations in metabolomic data. Brief Bioinform. 7: 151-158
209. Steuer R., Kurths J., Fiehn O., Weckwerth W. (2003) Observing and interpreting correlations in metabolomic networks. Bioinformatics 19: 1019-1026
210. Stevens R., McEntire R., Goble C., Greenwood M., Zhao J., Wipat A., Li P. (2004) myGrid and the drug discovery process. DDT Biosilico. 4: 140-148
211. Storey J.D. (2002) A direct approach to false discovery rates. J. Roy. Stat. Soc. B 64: 479-498
212. Storey J.D., Tibshirani R. (2003) Statistical significance for genomewide studies. Proc. Natl. Acad. Sci. USA 100: 9440-5
213. Tas A.C., van der Greef J. (1994) Mass spectrometric profiling and pattern recognition. Mass Spectrum Rev. 13: 155-181
214. Todd J.A. (2006) Statistical false positive or true disease pathway? Nat. Genet. 38: 731-733
215. Troyanskaya O., Cantor M., Sherlock G., Brown P., Hastie T., Tibshirani R., Botstein D., Altman R.B. (2001) Missing value estimation methods for DNA microarrays. Bioinformatics 17: 520-525
216. Tu Y., Stolovitzky G., Klein U. (2002) Quantitative noise analysis for gene expression microarray experiments. Proc. Natl. Acad. Sci. USA 99: 14031-14036
217. Tufte E.R. (2001) The Visual Display of Quantitative Information. 2nd ed. Graphics Press, Cheshire, CT
218. Tukey J.W. (1977) Exploratory Data Analysis. Addison-Wesley, Reading, MA
219. Urbanczyk-Wochniak E., Luedemann A., Kopka J., Selbig J., Roessner-Tunali U., Willmitzer L., Fernie A.R. (2003) Parallel analysis of transcript and metabolic profiles: A new approach in systems biology. EMBO Rep 4: 989-993
220. Valiant L.G. (1984) A theory of the learnable. Comm ACM 27: 1134-1142
221. van't Veer L.J., Dai H., van de Vijver M.J., He Y.D., Hart A.A., Mao M., Peterse H.L., van der Kooy K., Marton M.J., Witteveen A.T., Schreiber G.J., Kerkhoven R.M., Roberts C., Linsley P.S., Bernards R., Friend S.H. (2002) Gene expression profiling predicts clinical outcome of breast cancer. Nature 415: 530-536
222. van de Vijver M.J., He Y.D., van't Veer L.J., Dai H., Hart A.A., Voskuil D.W., Schreiber G.J., Peterse J.L., Roberts C., Marton M.J., Parrish M., Atsma D., Witteveen A., Glas A., Delahaye L., van der Velde T., Bartelink H., Rodenhuis S., Rutgers E.T., Friend S.H., Bernards R. (2002) A gene-expression signature as a predictor of survival in breast cancer. N. Engl. J. Med. 347: 1999-2009
223. van Rijsbergen C. (1979) Information Retrieval. Butterworth, London
224. Van Veldhuizen D.A., Lamont G.B. (2000) Multiobjective evolutionary algorithms: Analyzing the state-of-the-art. Evol Comput 8: 125-147
225. Vapnik V.N. (1998) Statistical Learning Theory. Wiley, New York
226. von Mering C., Krause R., Snel B., Cornell M., Oliver S.G., Fields S., Bork P. (2002) Comparative assessment of large-scale data sets of protein-protein interactions. Nature 417: 399-403
227. Wacholder S., Chanock S., Garcia-Closas M., El Ghormli L., Rothman N. (2004) Assessing the probability that a positive report is false: An approach for molecular epidemiology studies. J. Natl. Cancer Inst. 96: 434-442
228. Wang Y., Klijn J.G., Zhang Y., Sieuwerts A.M., Look M.P., Yang F., Talantov D., Timmermans M., Meijer-van Gelder M.E., Yu J., Jatkoe T., Berns E.M., Atkins D., Foekens J.A. (2005) Gene-expression profiles to predict distant metastasis of lymph-node-negative primary breast cancer. Lancet 365: 671-679
229. Weckwerth W., Morgenthal K. (2005) Metabolomics: From pattern recognition to biological interpretation. Drug Discov. Today 10: 1551-1558
230. Weiss S.H., Kulikowski C.A. (1991) Computer Systems that Learn: Classification and Prediction Methods from Statistics, Neural Networks, Machine Learning, and Expert Systems. Morgan Kaufmann Publishers, San Mateo, CA
231. Weiss S.M., Indurkhya N. (1998) Predictive Data Mining. Morgan Kaufmann, San Francisco
232. Westerhoff H.V., Kell D.B. (1987) Matrix method for determining the steps most rate-limiting to metabolic fluxes in biotechnological processes. Biotechnol. Bioeng. 30: 101-107
233. White H. (1992) Artificial Neural Networks: Approximation and Learning Theory. Blackwell, Oxford
234. White T.A., Kell D.B. (2004) Comparative genomic assessment of novel broad-spectrum targets for antibacterial drugs. Comp. Func. Genomics 5: 304-327
235. Wilkinson L. (1999) The Grammar of Graphics. Springer-Verlag, New York
236. Williamson P.R., Gamble C., Altman D.G., Hutton J.L. (2005) Outcome selection bias in meta-analysis. Stat. Methods Med. Res. 14: 515-524
237. Wold S., Trygg J., Berglund A., Antti H. (2001) Some recent developments in PLS modeling. Chemometr. Intell. Lab Syst. 58: 131-150
238. Woodward M. (2000) Epidemiology: Study Design and Data analysis. Chapman and Hall/CRC, London
239. Xie Y., Pan W., Khodursky A.B. (2005) A note on using permutation-based false discovery rate estimates to compare different analysis methods for microarray data. Bioinformatics 21: 4280-4288
240. Zadeh L.A. (1965) Fuzzy sets. Information and Control 8: 338-353
241. Zhang J.H., Chung T.D.Y., Oldenburg K.R. (1999) A simple statistical parameter for use in evaluation and validation of high throughput screening assays. J. Biomol. Screen. 4: 67-73
242. Zhou X., Wang X., Dougherty E.R. (2003) Missing-value estimation using linear and non-linear regression with Bayesian gene selection. Bioinformatics 19: 2302-2307
243. Zhou X.H., Obuchowski N.A., McClish D.K. (2002) Statistical Methods in Diagnostic Medicine. Wiley, New York
244. Zitzler E. (1999) Evolutionary Algorithms for Multiobjective Optimization: Methods And Applications. Shaker Verlag, Aachen
245. Zupan J., Gasteiger J. (1993) Neural Networks for Chemists. Verlag Chemie, Weinheim
246. Zweig M.H., Campbell G. (1993) Receiver-Operating Characteristic (ROC) plots - a fundamental evaluation tool in clinical medicine. Clin. Chem. 39: 561-577