Multiple hypothesis testing in genomics

Statistics in Medicine

Volumn 33, Issue 11, 2014, Pages 1946-1978

  Goeman, Jelle J ^a,b   Solari, Aldo ^c  

^a RADBOUD UNIVERSITY MEDICAL CENTER   (Netherlands)

^b LEIDEN UNIVERSITY MEDICAL CENTER   (Netherlands)

^c UNIVERSITY OF MILANO BICOCCA   (Italy)

Author keywords

Bonferroni; False discovery proportion; False discovery rate; Familywise error rate; FDR

Indexed keywords

DNA; MESSENGER RNA; MICRORNA;

ARTICLE; COMPUTER PROGRAM; GENE EXPRESSION; GENETIC SCREENING; GENETIC SELECTION; GENOMICS; MICROARRAY ANALYSIS; MOLECULAR BIOLOGY; RNA SEQUENCE; STATISTICAL ANALYSIS; VALIDATION PROCESS; GENE EXPRESSION PROFILING; GENETIC ASSOCIATION; HUMAN; PROCEDURES;

DATA INTERPRETATION, STATISTICAL; GENE EXPRESSION PROFILING; GENOME-WIDE ASSOCIATION STUDY; GENOMICS; HUMANS;

EID: 84899985461     PISSN: 02776715     EISSN: 10970258     Source Type: Journal    
DOI: 10.1002/sim.6082     Document Type: Article

References (99)

1. Dudoit S, Shaffer J, Boldrick J. Multiple hypothesis testing in microarray experiments. Statistical Science 2003; 18(1):71-103.
2. Benjamini Y. Simultaneous and selective inference: current successes and future challenges. Biometrical Journal 2010; 52(6):708-721.
3. Farcomeni A. A review of modern multiple hypothesis testing, with particular attention to the false discovery proportion. Statistical Methods in Medical Research 2008; 17(4):347-388.
4. Roquain E. Type I error rate control for testing many hypotheses: a survey with proofs. Journal de la Societé Française de Statistique 2011; 153(2):3-38.
5. Dudoit S, vander Laan M. Multiple Testing Procedures with Applications to Genomics. Springer Verlag: New York, 2008.
6. Dmitrienko A, DAgostino R. Traditional multiplicity adjustment methods in clinical trials. Statistics in Medicine 2013; 32(29):5172-5218.
7. Cox D. A remark on multiple comparison methods. Technometrics 1965; 7(2):223-224.
8. Benjamini Y, Yekutieli D. False discovery rate-adjusted multiple confidence intervals for selected parameters. Journal of the American Statistical Association 2005; 100(469):71-81.
9. Bender R, Lange S. Adjusting for multiple testing-when and how? Journal of Clinical Epidemiology 2001; 54(4):343-349.
10. Ioannidis J. Why most published research findings are false. PLoS Medicine 2005; 2(8):e124.
11. Rothstein H, Sutton A, Borenstein M. Publication Bias in Meta-Analysis. Wiley Online Library: New York, 2005.
12. Goeman J, Solari A. Multiple testing for exploratory research. Statistical Science 2011; 26(4):584-597.
13. Van deVijver MJ, He YD, van't Veer LJ, Dai H, Hart AAM, Voskuil DW, Schreiber GJ, Peterse JL, Roberts C, Marton MJ, Parrish M, Atsma D, Witteveen A, Glas A, Delahaye L, van der Velde T, Bartelink H, Rodenhuis S, Rutgers ET, Friend SH, Bernards R. A gene-expression signature as a predictor of survival in breast cancer. New England Journal of Medicine 2002; 347(25):1999-2009.
14. Rosenwald A, Wright G, Chan WC, Connors JM, Campo E, Fisher RI, Gascoyne RD, Muller-Hermelink HK, Smeland EB, Giltnane JM, Hurt EM, Zhao H, Averett L, Yang L, Wilson WH, Jaffe ES, Simon R, Klausner RD, Powell J, Duffey PL, Longo DL, Greiner TC, Weisenburger DD, Sanger WG, Dave BJ, Lynch JC, Vose J, Armitage JO, Montserrat E, López-Guillermo A, Grogan TM, Miller TP, LeBlanc M, Ott G, Kvaloy S, Delabie J, Holte H, Krajci P, Stokke T, Staudt LM, for the Lymphoma/Leukemia Molecular Profiling Project. The use of molecular profiling to predict survival after chemotherapy for diffuse large-B-cell lymphoma. New England Journal of Medicine 2002; 346(25):1937-1947.
15. Devlin B, Roeder K. Genomic control for association studies. Biometrics 1999; 55(4):997-1004.
16. Blanchard G, Dickhaus T, Hack N, Konietschke F, Rohmeyer K, Rosenblatt J, Scheer M, Werft W. μtoss-multiple hypothesis testing in an open software system. Journal of Machine Learning Research: Workshop and Conference Proceedings, Vol. 11, Berlin, 2010; 12-19.
17. Westfall P, Tobias R, Wolfinger R. Multiple Comparisons and Multiple Tests using SAS. SAS Publishing: Cary, NC, 2011.
18. Benjamini Y, Hochberg Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological) 1995; 57(1):289-300.
19. Finner H, Roters M. On the false discovery rate and expected type I errors. Biometrical Journal 2001; 43(8):985-1005.
20. Storey J. The positive false discovery rate: a Bayesian interpretation and the q-value. The Annals of Statistics 2003; 31(6):2013-2035.
21. Scott J, Berger J. An exploration of aspects of Bayesian multiple testing. Journal of Statistical Planning and Inference 2006; 136(7):2144-2162.
22. Hommel G. Tests of the overall hypothesis for arbitrary dependence structures. Biometrical Journal 1983; 25(5):423-430.
23. Guo W, BhaskaraRao M. On control of the false discovery rate under no assumption of dependency. Journal of Statistical Planning and Inference 2008; 138(10):3176-3188.
24. Finner H, Dickhaus T, Roters M. Dependency and false discovery rate: asymptotics. The Annals of Statistics 2007; 35: 1432-1455.
25. Sarkar S. Two-stage stepup procedures controlling FDR. Journal of Statistical Planning and Inference 2008; 138(4):1072-1084.
26. Simes R. An improved Bonferroni procedure for multiple tests of significance. Biometrika 1986; 73(3):751-754.
27. Sarkar S. On the Simes inequality and its generalization. IMS Collections Beyond Parametrics in Interdisciplinary Research: Festschrift in Honor of Professor Pranab K. Sen 2008; 1: 231-242.
28. Rødland E. Simes' procedure is 'valid on average'. Biometrika 2006; 93(3):742-746.
29. Sarkar S. FDR-controlling stepwise procedures and their false negatives rates. Journal of Statistical Planning and Inference 2004; 125(1):119-137.
30. Reiner-Benaim A. FDR control by the BH procedure for two-sided correlated tests with implications to gene expression data analysis. Biometrical Journal 2007; 49(1):107-126.
31. Kim K, Van DeWiel M. Effects of dependence in high-dimensional multiple testing problems. BMC Bioinformatics 2008; 9(1):114.
32. Yekutieli D. Comments on: control of the false discovery rate under dependence using the bootstrap and subsampling. Test 2008; 17(3):458-460.
33. Good P. Permutation Tests. Wiley Online Library: New York, 2000.
34. Pesarin F. Multivariate Permutation Tests: With Applications in Biostatistics. Wiley: Chichester, 2001.
35. Westfall PH, Troendle JF. Multiple testing with minimal assumptions. Biometrical Journal 2008; 50(5):745-755.
36. Huang Y, Xu H, Calian V, Hsu J. To permute or not to permute. Bioinformatics 2006; 22(18):2244-2248.
37. Goeman J, Solari A. The sequential rejection principle of familywise error control. The Annals of Statistics 2010; 38(6):3782-3810.
38. Hoggart C, Clark T, DeIorio M, Whittaker J, Balding D. Genome-wide significance for dense SNP and resequencing data. Genetic Epidemiology 2008; 32(2):179-185.
39. Perneger T. What's wrong with Bonferroni adjustments. British Medical Journal 1998; 316(7139):1236-1238.
40. Westfall PH, Wolfinger RD. Multiple tests with discrete distributions. The American Statistician 1997; 51(1):3-8.
41. Sidak Z. Rectangular confidence regions for the means of multivariate normal distributions. Journal of the American Statistical Association 1967; 62(318):626-633.
42. Rothman K. No adjustments are needed for multiple comparisons. Epidemiology 1990; 1(1):43-46.
43. Goeman J, Van DeGeer S, DeKort F, VanHouwelingen H. A global test for groups of genes: testing association with a clinical outcome. Bioinformatics 2004; 20(1):93-99.
44. Hummel M, Meister R, Mansmann U. Globalancova: exploration and assessment of gene group effects. Bioinformatics 2008; 24(1):78-85.
45. Holm S. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 1979; 6(2):65-70.
46. Hochberg Y. A sharper Bonferroni procedure for multiple tests of significance. Biometrika 1988; 75(4):800-802.
47. Hommel G. A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika 1988; 75(2):383-386.
48. Hommel G. A comparison of two modified Bonferroni procedures. Biometrika 1989; 76(3):624-625.
49. Marcus R, Peritz E, Gabriel K. On closed testing procedures with special reference to ordered analysis of variance. Biometrika 1976; 63(3):655-660.
50. Westfall P, Young S. Resampling-Based Multiple Testing: Examples and Methods for P-Value Adjustment. Wiley-Interscience: New York, 1993.
51. Meinshausen N, Maathuis M, Bühlmann P. Asymptotic optimality of the Westfall-Young permutation procedure for multiple testing under dependence. The Annals of Statistics 2012; 39(6):3369-3391.
52. Ge Y, Dudoit S, Speed T. Resampling-based multiple testing for microarray data analysis. Test 2003; 12(1):1-77.
53. VanWieringen W, Van DeWiel M, Van DerVaart A. A test for partial differential expression. Journal of the American Statistical Association 2008; 103(483):1039-1049.
54. Phipson B, Smyth G. Permutation P-values should never be zero: calculating exact P-values when permutations are randomly drawn. Statistical Applications in Genetics and Molecular Biology 2010; 9(1):39.
55. Benjamini Y, Yekutieli D. The control of the false discovery rate in multiple testing under dependency. Annals of Statistics 2001; 29(4):1165-1188.
56. Benjamini Y, Hochberg Y. On the adaptive control of the false discovery rate in multiple testing with independent statistics. Journal of Educational and Behavioral Statistics 2000; 25(1):60-83.
57. Benjamini Y, Krieger A, Yekutieli D. Adaptive linear step-up procedures that control the false discovery rate. Biometrika 2006; 93(3):491-507.
58. Blanchard G, Roquain E. Adaptive false discovery rate control under independence and dependence. The Journal of Machine Learning Research 2009; 10: 2837-2871.
59. Sarkar S, Guo W, Finner H. On adaptive procedures controlling the familywise error rate. Journal of Statistical Planning and Inference 2012; 142(1):65-78.
60. Blanchard G, Roquain E. Two simple sufficient conditions for FDR control. Electronic Journal of Statistics 2008; 2: 963-992.
61. Yekutieli D, Benjamini Y. Resampling-based false discovery rate controlling multiple test procedures for correlated test statistics. Journal of Statistical Planning and Inference 1999; 82(1-2):171-196.
62. Romano J, Shaikh A, Wolf M. Control of the false discovery rate under dependence using the bootstrap and subsampling. Test 2008; 17(3):417-442.
63. Troendle J. Stepwise normal theory multiple test procedures controlling the false discovery rate. Journal of Statistical Planning and Inference 2000; 84(1-2):139-158.
64. Ge Y, Sealfon S, Speed T. Some step-down procedures controlling the false discovery rate under dependence. Statistica Sinica 2008; 18(3):881-904.
65. Korn E, Troendle J, McShane L, Simon R. Controlling the number of false discoveries: application to high-dimensional genomic data. Journal of Statistical Planning and Inference 2004; 124(2):379-398.
66. Troendle J. Comments on: control of the false discovery rate under dependence using the bootstrap and subsampling. Test 2008; 17(3):456-457.
67. Goeman J, Mansmann U. Multiple testing on the directed acyclic graph of gene ontology. Bioinformatics 2008; 24(4):537-544.
68. Posch M, Zehetmayer S, Bauer P. Hunting for significance with the false discovery rate. Journal of the American Statistical Association 2009; 104(486):832-840.
69. Storey J. A direct approach to false discovery rates. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 2002; 64(3):479-498.
70. Storey J, Taylor J, Siegmund D. Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: a unified approach. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 2004; 66(1):187-205.
71. Schwartzman A, Lin X. The effect of correlation in false discovery rate estimation. Biometrika 2011; 98(1):199-214.
72. Pounds S, Cheng C. Improving false discovery rate estimation. Bioinformatics 2004; 20(11):1737-1745.
73. Owen A. Variance of the number of false discoveries. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 2005; 67(3):411-426.
74. Qiu X, Yakovlev A. Some comments on instability of false discovery rate estimation. Journal of Bioinformatics and Computational Biology 2006; 4(5):1057-1068.
75. Tusher V, Tibshirani R, Chu G. Significance analysis of microarrays applied to the ionizing radiation response. Proceedings of the National Academy of Sciences 2001; 98(9):5116-5121.
76. Efron B, Tibshirani R, Storey J, Tusher V. Empirical Bayes analysis of a microarray experiment. Journal of the American Statistical Association 2001; 96(456):1151-1160.
77. Efron B. Microarrays, empirical Bayes and the two-groups model. Statistical Science 2008; 23: 1-22.
78. Morris C. Comment: microarrays, empirical Bayes and the two-groups model. Statistical Science 2008; 23: 34-40.
79. Qiu X, Klebanov L, Yakovlev A. Correlation between gene expression levels and limitations of the empirical Bayes methodology in microarray data analysis. Statistical Applications in Genetics and Molecular Biology 2005; 4(1):34.
80. Pawitan Y, Calza S, Ploner A. Estimation of false discovery proportion under general dependence. Bioinformatics 2006; 22(24):3025-3031.
81. Efron B. Correlation and large-scale simultaneous significance testing. Journal of the American Statistical Association 2007; 102(477):93-103.
82. Efron B. Correlated z-values and the accuracy of large-scale statistical estimates. Journal of the American Statistical Association 2010; 105(491):1042-1055.
83. Meinshausen N. False discovery control for multiple tests of association under general dependence. Scandinavian Journal of Statistics 2006; 33(2):227-237.
84. Goeman J, Solari A. Rejoinder. Statistical Science 2011; 26(4):608-612.
85. Lee W, Gusnanto A, Salim A, Magnusson P, Sim X, Tai E, Pawitan Y. Estimating the number of true discoveries in genome-wide association studies. Statistics in Medicine 2011; 31(11-12):1177-1189.
86. Hommel G, Kropf S. Tests for differentiation in gene expression using a data-driven order or weights for hypotheses. Biometrical Journal 2005; 47(4):554-562.
87. Finos L, Salmaso L. FDR- and FWE-controlling methods using data-driven weights. Journal of Statistical Planning and Inference 2007; 137(12):3859-3870.
88. Talloen W, Clevert DA, Hochreiter S, Amaratunga D, Bijnens L, Kass S, Göhlmann HW. I/NI-calls for the exclusion of non-informative genes: a highly effective filtering tool for microarray data. Bioinformatics 2007; 23(21):2897-2902.
89. Hackstadt A, Hess A. Filtering for increased power for microarray data analysis. BMC Bioinformatics 2009; 10(1):11.
90. Bourgon R, Gentleman R, Huber W. Independent filtering increases detection power for high-throughput experiments. Proceedings of the National Academy of Sciences 2010; 107(21):9546-9551.
91. van Iterson M, Boer J, Menezes R. Filtering, FDR and power. BMC Bioinformatics 2010; 11(1):450.
92. Meinshausen N. Hierarchical testing of variable importance. Biometrika 2008; 95(2):265-278.
93. Yekutieli D. Hierarchical false discovery rate-controlling methodology. Journal of the American Statistical Association 2008; 103(481):309-316.
94. Läuter J, Horn F, Rosołowski M, Glimm E. High-dimensional data analysis: selection of variables, data compression and graphics-application to gene expression. Biometrical Journal 2009; 51(2):235-251.
95. Benjamini Y, Bogomolov M. Selective inference on multiple families of hypotheses. Journal of the Royal Statistical Society, Series B 2014. in press.
96. Goeman J, Finos L. The inheritance procedure: multiple testing of tree-structured hypotheses. Statistical Applications in Genetics and Molecular Biology 2012; 11(1):1-18.
97. Crager M. Gene identification using true discovery rate degree of association sets and estimates corrected for regression to the mean. Statistics in Medicine 2010; 29(1):33-45.
98. Efron B. Tweedies formula and selection bias. Journal of the American Statistical Association 2011; 106(496):1602-1614.
99. Zhong H, Prentice R. Correcting 'winner's curse' in odds ratios from genomewide association findings for major complex human diseases. Genetic Epidemiology 2010; 34(1):78-91.