A review of modern multiple hypothesis testing, with particular attention to the false discovery proportion

Statistical Methods in Medical Research

Volumn 17, Issue 4, 2008, Pages 347-388

  Farcomeni, Alessio ^a  

^a SAPIENZA UNIVERSITY OF ROME   (Italy)

Author keywords

[No Author keywords available]

Indexed keywords

ANALYTICAL ERROR; BAYES THEOREM; BOOTSTRAPPING; CANCER GENETICS; COLON CANCER; DATA ANALYSIS; DNA MICROARRAY; HUMAN; HYPOTHESIS; INTERMETHOD COMPARISON; LYMPHATIC LEUKEMIA; MEDICAL RESEARCH; MYELOID LEUKEMIA; NULL HYPOTHESIS; REVIEW; SIMULATION; STATISTICAL ANALYSIS; STATISTICAL MODEL;

BAYES THEOREM; FALSE POSITIVE REACTIONS; MODELS, THEORETICAL;

EID: 55249117952     PISSN: 09622802     EISSN: None     Source Type: Journal    
DOI: 10.1177/0962280206079046     Document Type: Review

References (141)

1. Worsley KJ, Marrett S., Neelin P., Vandal AC, Friston KJ, Evans AC A unified statistical approach for determining significant signals in images of cerebral activation. Human Brain Mapping 1996; 4: 58-73.
2. Ellis SP, Underwood MD, Arango V. Mixed models and multiple comparisons in analysis of human neurochemical maps. Psychiat Res-Neuroim 2000; 9: 111-19.
3. Merriam EP, Genovese CR, Colby CL Spatial updating in human parietal cortex. Neuron 2003; 39: 361-73.
4. Logan BR, Rowe DB An evaluation of thresholding techniques in fMRI analysis. Neuroimage 2004; 22: 95-108.
5. Drigalenko EI, Elston RC False discoveries in genome scanning. Genetics Epidemiology 1997; 14: 779-84.
6. Weller JI, Song JZ, Heyen DW A new approach to the problem of multiple comparisons in the genetic dissection of complex traits. Genetics 1998; 150(4): 1699-706.
7. Heyen DW, Weller JI, Ron M. A genome scan for QTL influencing milk production and health traits in dairy cattle. Physiological Genomics 1999; 1(3): 165-75.
8. Bovenhuis H., Spelman RJ Selective genotyping to detect quantitative trait loci for multiple traits in outbred populations. Journal of Dairy Science 2000; 83(1): 173-80.
9. Mosig MO, Lipkin E., Khutoreskaya G., Tchourzyna E., Soller M., Fridmann AA Whole genome scan for quantitative trait loci affecting milk protein percentage in Israeli-Holstein cattle, by means of selective milk DNA pooling in a daughter design, using an adjusted false discovery rate criterion. Genetics 2001; 157: 1683-98.
10. Reiner A., Yekutieli D., Benjamini Y. Identifying differentially expressed genes using false discovery rate controlling procedures. Bioinformatics 2003; 19: 368-75.
11. Dudoit S., Shaffer PJ, Boldrick JC Multiple hypothesis testing in microarray experiments. Statistical Science 2003; 18: 71-103.
12. Sebastiani P., Gussoni E., Kohane IS, Ramoni MF Statistical challenges in functional genomics. Statistical Science 2003; 18: 33-70.
13. Khatri P., Babyyak M., Croughwell ND Temperature during coronary artery bypass surgery affects quality of life. Annals of Thoracical Surgery 2001; 71(1): 110-16.
14. Schlaeppi M., Edwards K., Fuller RW Patient perception of the Diskus inhaler: A comparison with the Turbuhaler inhaler. British Journal of Clinical Practice 1996; 50: 14-19.
15. Ottenbacher KJ Quantitative evaluation of multiplicity in epidemiology and public health research. American Journal of Epidemiology 1998; 147: 615-19.
16. Vedantham K., Brunet A., Boyer R. Post-traumatic stress disorder, trauma exposure, and the current health of Canadian bus drivers. Canadian Journal of Psychiatrics 2001; 46(2): 149-55.
17. Schaffer CM, Green PE Cluster-based market segmentation: Some further comparisons of alternative approaches. Journal of Market Research Sociology 1998; 40: 155-63.
18. George EI The variable selection problem. Journal of the American Statistical Association 2000; 95(452): 1304-08.
19. George EI, Foster DP Calibration and empirical Bayes variable selection. Biometrika 2000; 87(4): 731-47.
20. Ip EH Testing for local dependency in dichotomous and polytomous item response models. Psychometrika 2001; 66(1): 109-32.
21. Green SB, Babyak MA Control of type I errors with multiple tests constraints in structural equation modeling. Multivariable Behavioural Research 1997; 32: 39-51.
22. Yekutieli D., Reiner-Benaim A., Benjamini Y., Elmer GI, Kafkafi N., Letwin NE, Lee NH Approaches to multiplicity issues in complex research in microarray analysis. Statistica Neerlandica 2006; 60: 414-37.
23. Abramovich F., Benjamini Y. Adaptive thresholding of wavelet coefficients. Computational Statistics and Data Analysis 1996; 22: 351-61.
24. Abramovich F., Benjamini Y., Donoho D., Johnstone I. Adapting to unknown sparsity by controlling the false discovery rate. Annals of Statistics 2006; 34: 584-653.
25. Holm S. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 1979; 6: 65-70.
26. Westfall PH, Kropf S., Finos L. Weighted FWE-controlling methods in high-dimensional situations. In Benjamini Y, Bretz F, Sarkar S, eds. Recent developments in multiple comparison procedures. vol. 47. Institute of Mathematical Statistics Lecture Notes-Monograph Series, 2004: 143-54.
27. Benjamini Y., Hochberg Y. Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society (Ser B) 1995; 57: 289-300.
28. Seeger P. A note on a method for the analysis of significance en masse. Technometrics 1968; 10: 586-93.
29. van der Laan MJ, Dudoit S., Pollard KS Augmentation procedures for control of the generalized family-wise error rate and tail probabilities for the proportion of false positives. Statistical Applications in Genetics and Molecular Biology 2004; 3(1).
30. Genovese CR, Wasserman L. Exceedance control of the false discovery proportion. Journal of the American Statistical Association 2006; 101: 1408-17.
31. Efron B., Tibshirani R. Empirical Bayes methods and false discovery rates for microarrays. Genetics Epidemiology 2002; 23: 70-86.
32. Storey JD A direct approach to false discovery rates. Journal of the Royal Statistical Society (Ser B) 2002; 64: 479-98.
33. Lehmann EL, Romano JP Generalizations of the familywise error rate. Annals of Statistics 2005; 33: 1138-54.
34. Sarkar SK Stepup procedures controlling generalized FWER and generalized FDR. Department of Statistics, Temple University, 2005.
35. Storey JD The positive false discovery rate: A Bayesian interpretation and the q-value. Annals of Statistics 2003; 31: 2013-35.
36. Benjamini Y., Hochberg Y. Multiple Hypothesis Testing with weights. Scandinavian Journal of Statistics 1997; 24: 407-18.
37. Genovese CR, Roeder K., Wasserman L. False discovery control with p-value weighting. Biometrika 2006; 93: 509-24.
38. Genovese CR, Wasserman L. Operating characteristics and extensions of the FDR procedure. Journal of the Royal Statistical Society (Ser B) 2002; 64: 499-518.
39. Sarkar SK FDR-controlling stepwise procedures and their false negatives rates. Journal of Statistical Planning and Inference 2004; 125: 119-37.
40. Westfall PH, Young SS Resampling-based multiple testing: Examples and methods for p-value adjustment. Wiley, 1993.
41. Miller RG Simultaneous statistical inference. Wiley, 1981.
42. Diaconis P. Theories of data analysis: From magical thinking through classical statistics. In Hoaglin DC, Mosteller F, Tukey JW, eds. Exploring Data Tables, Trends, and Shapes. Wiley, 1985.
43. Ahmed SW Issues arising in the application of Bonferroni procedures in federal surveys. In 1991 ASA Proceedings of the Survey Research Methods Section, 1991. 344-49.
44. Wright SP Adjusted p-values for simultane ous inference. Biometrics 1992; 48: 1005-10.
45. Dudoit S., van der Laan MJ, Pollard KS Multiple Testing. Part I. Single-step procedures for control of general Type I error rates. Statistical Applications in Genetics and Molecular Biology 2004; 3(1).
46. Hochberg Y., Tamhane AC Multiple comparisons procedures. Wiley, 1987.
47. Finner H., Roters M. Multiple hypotheses testing and expected number of Type I errors. Annals of Statistics 2002; 30: 220-38.
48. Pesarin F. Multivariate permutation tests with applications to biostatistics. Wiley, 2001.
49. Efron B., Tibshirani R. An introduction to the Bootstrap. Springer-Verlag, 1993.
50. Troendle K, McShane. An example of slow convergence of the Bootstrap in high dimensions. American Statistician 2004; 58: 25-9.
51. Pollard KS, van der Laan MJ Choice of a null distribution in resampling-based multiple testing. Journal of Statistical Planning and Inference 2004; 125: 85-100.
52. Ge Y., Dudoit S., Speed TP Resampling-based multiple testing for microarray data analysis. Test 2003; 12: 1-77.
53. Farcomeni A. Some Results on the control of the false discovery rate under dependence. Scandinavian Journal of Statistics 2006; published ahead of print. doi: 10.1111/j.1467-9469.2006.00530.x
54. Ferreira JA, Zwinderman AH On the Benjamini-Hochberg method. Annals of Statistics 2006; 34: 1827-49.
55. Yekutieli D., Benjamini Y. Resampling-based false discovery rate controlling multiple test procedures for correlated test statistics. Journal of Statistical Planning and Inference 1999; 82: 171-96.
56. Meinert CL Clinical trials design, conduct, and analysis. Oxford University Press, 1986.
57. Pocock SJ Clinical trials with multiple outcomes: A statistical perspective on their design, analysis, and interpretation. Controlled Clinical Trials 1997; 18: 530-45.
58. Chi GYH. Multiple testings: Multiple comparisons and multiple endpoints. Drug Information Journal 1998; 32: 1347S-62S.
59. Moyé LA P-value interpretation and alpha allocation in clinical trials. Annals of Epidemiology 1998; 8: 351-57.
60. Moyé LA Alpha calculus in clinical trials: Considerations and commentary for the new millennium. Statistics in Medicine 2000; 19: 767-79.
61. O'Brien PC Procedures for comparing samples with multiple endpoints. Biometrics 1984; 40: 1079-87. With errata: Biometrics 1995; 51: 1580-81.
62. Pocock SJ, Geller NL, Tsiatis AA The analysis of multiple endpoints in clinical trials. Biometrics 1987; 43: 487-98.
63. Follmann D. Multivariate tests for multiple endpoints in clinical trials. Statistics in Medicine 1995; 14: 1163-75.
64. Läuter J. Exact t and F tests for analyzing studies with multiple endpoints. Biometrics 1996; 52: 964-70.
65. Wei LJ, Lachin JM Two-sample asymptotially distribution-free tests for incomplete multivariate observations. Journal of the American Statistical Association 1984; 79: 653-61.
66. Wei LJ, Lin DY, Weissfeld L. Regression analysis of multivariate incomplete failure time data by modeling marginal distributions. Journal of the American Statistical Association 1989; 84: 1065-73.
67. Lehmacher W., Wassmer G., Reitmeir P. Procedures for two-sample comparisons with multiple endpoints controlling the experimentwise error rate. Biometrics 1991; 47: 511-21.
68. Hochberg Y. A sharper Bonferroni procedure for multiple tests of significance. Biometrika 1988; 75: 800-2.
69. van der Laan MJ, Dudoit S., Pollard KS Multiple Testing. Part II. Step-down procedures for control of the family-wise error rate. Statistical Applications in Genetics and Molecular Biology 2004; 3(1).
70. Sidak Z. Rectangular confidence regions for the means of multivariate normal distributions. Journal of the American Statistical Association 1967; 62: 626-33.
71. Sidak Z. On probabilities of rectangles in multivariate Student distributions: Their dependence on correlations. Annals of Mathematical Statistics 1971; 42: 169-75.
72. Finner H., Roters M. Asymptotic comparison of step-down and step-up multiple test procedures based on exchangeable test statistics. Annals of Statistics 1998; 26: 505-24.
73. Dunnet CW, Tamhane AC A Step-Up Multiple Test Procedure. Journal of the American Statistical Association 1992; 87: 162-70.
74. Seneta E., Chen JT A sequentially rejective test procedure. Theory of Stochastic Processes 1997; 3: 393-402.
75. Seneta E., Chen JT Simple stepwise tests of hypotheses and multiple comparisons. International Statistical Review 2005; 73: 21-34.
76. Shaffer J. Multiple hypothesis testing. Annals of Review Psychology 1995; 46: 561 - 84.
77. Simes RJ An improved Bonferroni procedure for multiple tests of significance. Biometrika 1986; 73: 751-54.
78. Benjamini Y., Hochberg Y. The adaptive control of the false discovery rate in multiple hypothesis testing with independent test statistics. Journal of Educational Behaviour Statistics 2000; 25: 60-83.
79. Benjamini Y., Liu W. A step-down multiple hypotheses testing procedure that controls the false discovery rate under independence. Journal of Statistical Planning and Inference 1999; 82: 163-70.
80. Storey JD, Taylor JE, Siegmund D. Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: A unified approach. Journal of the Royal Statistical Society (Ser B) 2004; 66: 187-205.
81. Genovese CR, Wasserman L. A stochastic process approach to false discovery control. Annals of Statistics 2004; 32: 1035-61.
82. Sarkar SK Some Results on false discovery rate in stepwise multiple testing procedures. Annals of Statistics 2002; 30: 239-57.
83. Benjamini Y., Krieger AM, Yekutieli D. Adaptive linear step up procedures that control the false discovery rate. Biometrika 2006; 93: 491-507.
84. Tusher VG, Tibshirani R., Chu G. Significance analysis of microarrays applied to the ionizing radiation response. PNAS 2001; 98: 5116-21.
85. Storey JD, Tibshirani R. Estimating false discovery rates under dependence, with applications to DNA microarrays. Department of Statistics, Stanford University, 2001.
86. Schweder T., Spjøtvoll E. Plots of p-values to evaluate many hypotheses simultaneously. Biometrika 1982; 69: 493-502.
87. Turkheimer FE, Smith CB, Schmidt K. Estimation of the number of 'true' null hypotheses in multivariate analysis of neuroimaging data. NeuroImage 2001; 13: 920-30.
88. Swanepoel JWH. The limiting behavior of a modified maximal symmetric 2s-spacing with applications. Annals of Statistics 1999; 27: 24-35.
89. Meinshausen N., Rice J. Estimating the proportion of false null hypotheses among a large number of independently tested hypotheses. Annals of Statistics 2006; 34: 373-393.
90. Langass M., Lindqvist BH, Ferkingstad E. Estimating the proportion of true null hypotheses, with application to DNA microarray data. Journal of the Royal Statistical Society (Ser B) 2005; 67: 555-72.
91. Miller CJ, Genovese C., Nichol RC, Wasserman L., Connolly A., Reichart D., Hopkins A., Schneider J., Moore A. Controlling the false-discovery rate in astrophysical data analysis. Astronological Journal 2001; 122: 3492-505.
92. Owen AB Variance of the number of false discoveries. Journal of the Royal Statistical Society (Ser B) 2005; 67: 411-26.
93. van der Laan MJ, Birkner MD, Hubbard AE Empirical Bayes and resampling based multiple testing procedure controlling tail probability of the proportion of false positives. Statistical Applications in Genetics and Molecular Biology 2005; 4(1).
94. R Development Core Team. R. A language and environment for statistical computing. R Foundation for Statistical Computing, 2004.
95. Perone Pacifico M., Genovese C., Verdinelli I., Wasserman L. False discovery control for random fields. Journal of the American Statistical Association 2004; 99: 1002-14.
96. Green PJ, Silverman BW Nonparametric regression and generalized linear models: A Roughness Penalty Approach. Chapman & Hall, 1994.
97. Berry DA, Hochberg Y. Bayesian perspectives on multiple comparisons. Journal of Statistical Planning and Inference 1999; 82: 215-77.
98. Berry DA Multiple comparisons, multiple tests, and data dredging: A Bayesian perspective. In Bernardo J, DeGroot M, Lindley D, Smith A, eds. Bayesian statistics. vol. 3. Oxford University Press, 1988: 79-94.
99. Scott JG, Berger JO An exploration of aspects of Bayesian multiple testing. Journal of Statistical Planning and Inference 2006; 136: 2144-62.
100. Bayarri MJ, Berger JO The interplay of Bayesian and frequentist analysis. Statistical Science 2004; 19: 58-80.
101. Duncan DB A Bayesian approach to multiple comparisons. Technomerics 1965; 7: 171-222.
102. Shaffer JP A semi-Bayesian study of Duncan's Bayesian multiple comparison procedure. Journal of Statistical Planning and Inference 1999; 82: 197-213.
103. Leibermann B. Contemporary problems in statistics. Oxford University Press, 1971.
104. Finner H. Stepwise multiple test procedures and control of directional errors. Annals of Statistics 1999; 27: 274-89.
105. Shaffer JP Multiplicity, directional (Type III) errors, and the null hypothesis. Psychological Methods 2002; 7: 356-69.
106. Robert CP, Casella G. Monte Carlo statistical methods. Springer-Verlag, 1999.
107. Efron B., Tibshirani R., Storey JD, Tusher V. Empirical Bayes analysis of a microarray experiment. Journal of the American Statistical Association 2001; 96: 1151-60.
108. Bernardo JM, Smith AFM. Bayesian theory. Wiley and Sons, 1994.
109. Kass RE, Wasserman L. A reference Bayesian test for nested hypotheses and its relationship to the Schwartz criterion. Journal of the American Statistical Association 1995; 90(431): 928-39.
110. Berger JO, Boukai B., Wang Y. Unified frequentist and Bayesian testing of a precise hypothesis. Statistical Science 1997; 12(3): 133-60.
111. Jeffreys H. Theory of probability. Oxford University Press, 1961.
112. Sarkar SK Some probability inequalities for ordered MTP2 random variables: A proof of the Simes conjecture. Annals of Statistics 1998; 26: 494-504.
113. Sarkar SK, Chang CK The Simes method for multiple hypothesis testing with positively dependent test statistics. Journal of the American Statistical Association 1997; 92: 1601-8.
114. Jogdeo K. Association and probability inequalities. Annals of Statistics 1977; 5: 495-504.
115. van der Laan MJ, Bryan J. Gene expression analysis with the parametric Bootstrap. Biostatistics 2000; 1: 1-19.
116. Benjamini Y., Yekutieli D. The control of the false discovery rate in multiple testing under dependency. Annals of Statistics 2001; 29: 1165-88.
117. Hommel G. Tests of the overall hypothesis for arbitrary dependence structures. Biomedical Journal 1983; 25: 423-30.
118. Esary JD, Proschan F., Walkup DW Association of random variables, with applications. Annals of Mathematical Statistics 1967; 38: 1466-74.
119. Farcomeni A. More powerful control of the false discovery rate under dependence. Statistical Methods & Applications 2006; 15: 43-73.
120. Bickel DR On 'Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates': Does a large number of tests obviate confidence intervals of the FDR? Office of Biostatistics and Bioinformatics, Medical College of Georgia, 2004.
121. Storey JD, Tibshirani R. Statistical significance for genome-wide studies. In Proceedings of the National Academy of Sciences 100, 2003: 9440-5.
122. Neuhaus KL, Von Essen R., Tebbe U., Vogt A., Roth M., Riess M., Niederer W., Forycki F., Wirtzfeld A., Mauurer W., Limbourg P., Merx W., Haerten K. Improved thrombolysis in acute myocaridial infarction front-loaded administration of Alteplase: Results of the rt-PA-APSAC patency study (TAPS). JAmCollCard 1992; 19: 885-91.
123. Amaratunga D., Cabrera J. Exploration and analysis of DNA microarray and protein array data. Wiley, 2004.
124. Parmigiani G., Garret ES, Irizarry R., Zeger SL The analysis of gene expression data: Methods and software. Springer, 2003.
125. Brown PO, Botstein D. Exporing the new world of genome with DNA microarrays. Nature Genetics 1999; 21: 33-7.
126. Duggan D., Bittner M., Chen Y., Meltzer P., Trent J. Expression profiling usign cDNA microarrays. Nature Genetics 1999; 21: 10-14.
127. Bolsover SR, Hyams J., Jones S., Shepard EA, White HA From genes to cells. Wiley, 1997.
128. Garret RH, Grisham CM Principles of byochemistry. Brooks/Cole, 2002.
129. Tseng G., Oh M., Rohlin L., Liao J., Wong W. Issues on cDNA microarray analysis: Quality filtering, channel normalization, models of variations and assessment of gene effects. Nucleic Acids Research 2001; 29: 2549-57.
130. Yang YH, Dudoit S., Luu P., Speed TP Normalization for cDNA microarray data. SPIE BiOS 2001; 2001.
131. Durbin BP, Rocke DM Variance stabilizing transformations for two-color microarrays. Bionformatics 2004; 20: 660-7.
132. Zweiger G. Transducing the genome: Information, anarchy and revolution in the biomedical sciences. McGraw-Hill, 2001.
133. Kaufman L., Rousseeuw PJ Finding groups in data. Wiley, 1990.
134. Alon U., Barkai N., Notterman DA, Gish K., Ybarra S., Mack D., Levine AJ Broad patterns of gene expression revealed by clustering analysis of tumor and normal colon tissue probed by oligonucleotide arrays. Proceedings National Academic Science in the USA 1999; 96: 6745-50.
135. Golub TR, Slonim DK, Tamayo P., Huard C., Gaasenbeek M., Mesirov HJP and Coller. Molecular classification of cancer: Class discovery and class prediction by gene expression monitoring. Science 1999; 286: 531-7.
136. Dudoit S., Fridlyand J., Speed TP Comparison of discrimination methods for the classification of tumors using gene expression data. Journal of the American Statistical Association 2002; 97: 77-87.
137. Cover T., Hart P. Nearest neighbor pattern classification. IEEE Transactions on Information Theory 1967; IT-13: 21-7.
138. Schervish M. p-values: What they are and What they are not? American Statistician 1996; 50: 203-6.
139. Bayarri MJ, Berger JO p-values for composite null models. Journal of the American Statistical Association 2000; 95: 1127-42.
140. Cabras S. Control of the false discovery rate with frequentist p-values in microarray data analysis. Universit̀ degli studi di Firenze, 2004.
141. Müller P., Parmigiani G., Robert C., Rousseau J. Optimal sample size for multiple testing: The case of gene expression microarrays. Journal of the American Statistical Association 2004; 99: 990-1001.