Innovated higher criticism for detecting sparse signals in correlated noise

Annals of Statistics

Volumn 38, Issue 3, 2010, Pages 1686-1732

  Hall, Peter ^a,b   Jin, Jiashun ^c  

^a UNIVERSITY OF MELBOURNE   (Australia)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c CARNEGIE MELLON UNIVERSITY   (United States)

Author keywords

Adding noise; Cholesky factorization; Empirical process; Innovation; Multiple hypothesis testing; Sparse normal means; Spectral density; Toeplitz matrix

Indexed keywords

EID: 77951460458     PISSN: 00905364     EISSN: None     Source Type: Journal    
DOI: 10.1214/09-AOS764     Document Type: Article

References (55)

1. ABRAMOVICH, F., BENJAMINI, Y., DONOHO, D. and JOHNSTONE, I. (2006). Adapting to unknown sparsity by controlling the false discovery rate. Ann. Statist. 34 584-653. MR2281879
2. BENJAMINI, Y. andHOCHBERG, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. Roy. Statist. Soc. Ser. B 57 289-300. MR1325392
3. BENJAMINI, Y. and YEKUTIELI, D. (2001). The control of the false discovery rate in multiple testing under dependency. Ann. Statist. 29 1165-1188. MR1869245
4. BICKEL, P. and LEVINA, E. (2008). Regularized estimation of large covariance matrices. Ann. Statist. 36 199-227. MR2387969
5. BÖTTCHER, A. and SILBERMANN, B. (1998). Introduction to Large Truncated Toeplitz Matrices. Springer, New York.
6. BROCKWELL, P. and DAVIS, R. (1991). Time Series and Methods, 2nd ed. Springer, New York. MR1093459
7. BROWN, B.W. and RUSSELL, K. (1997). Methods correcting for multiple testing: Operating characteristics. Stat. Med. 16 2511-2528.
8. CAI, T., JIN, J. and LOW, M. (2007). Estimation and confidence sets for sparse normalmixtures. Ann. Statist. 35 2421-2449. MR2382653
9. CAI, T. and SUN, W. (2009). Simultaneous testing of grouped hypotheses: Finding needles in multiple haystacks. J. Amer. Statist. Assoc. 104 1467-1481.
10. CAYON, L., JIN, J. and TREASTER, A. (2005). Higher criticism statistic: Detecting and identifying non-Gaussianity in the WMAP first year data. Monthly Notes of the Royal Astronomical Society 362 826-832.
11. CHEN, L., TONG, T. and ZHAO, H. (2005). Considering dependence among genes and markers for false discovery control in eQTL mapping. Bioinformatics 24 2015-2022.
12. CLARKE, S. and HALL, P. (2009). Robustness of multiple testing procedures against dependence. Ann. Statist. 37 332-358. MR2488354
13. COHEN, A., SACKROWITZ, H. B. and XU, M. (2009). A new multiple testing method in the dependent case. Ann. Statist. 37 1518-1544. MR2509082
14. CSÖRGÖ, M., CSÖRGÖ, S., HORVATH, L. and MASON, D. (1986). Weighted empirical and quantile processes. Ann. Probab. 14 31-85. MR0815960
15. COVER, T. M. and THOMAS. J. A. (2006). Elementary Information Theory. Wiley, Hoboken, NJ. MR2239987
16. CRUZ, M., CAYON, L., MARTÍNEZ-GONZÁLEZ, E., VIEVA, P. and JIN, J. (2007). The non- Gaussian cold spot in the 3-year WMAP data. Astrophys. J. 655 11-20.
17. DELAIGLE, A. and HALL, J. (2009). Higher criticism in the context of unknown distribution, non-independence and classification. In Platinum Jubilee Proceedings of the Indian Statistical Institute (N. S. Narasimha Sastry, T. S. S. R. K. Rao, M. Delampady and B. Rajeev, eds.) 109-138. World Scientific, Hackensack, NJ.
18. DONOHO, D. and JIN, J. (2004). Higher criticism for detecting sparse heterogeneous mixtures. Ann. Statist. 32 962-994. MR2065195
19. DONOHO, D. and JIN, J. (2006). Asymptotic minimaxity of False Discovery Rate thresholding for sparse exponential data. Ann. Statist. 34 2980-3018. MR2329475
20. DONOHO, D. and JIN, J. (2008). Higher criticism thresholding: Optimal feature selection when useful features are rare and weak. Proc. Natl. Acad. Sci. USA 105 14790-14795.
21. DONOHO, D. and JIN, J. (2009). Higher criticism thresholding achieves optimal phase diagram. Phil. Trans. R. Soc. A 367 4449-4470.
22. DUNNETT, C.W. and TAMHANE, A. C. (1995). Step-up testing of parameters with unequally correlated estimates. Biometrics 51 217-227.
23. EFRON, B. (2007). Correlation and large-scale simultaneous significance testing. J. Amer. Statist. Assoc. 102 93-103. MR2293302
24. FINNER, H. and ROTERS, M. (1998). Asymptotic comparison of step-down and step-up multiple test procedures based on exchangeable test statistics. Ann. Statist. 26 505-524. MR1626043
25. GENOVESE, C. andWASSERMAN, L. (2004). A stochastic process approach to false discovery control. Ann. Statist. 32 1035-1061. MR2065197 (Pubitemid 41250292)
26. GOEMAN, J., VAN DE GEER, S., DE KORT, F. and VAN HOUWELINGEN, H. (2004). A global test for groups of genes: Testing association with a clinical outcome. Bioinformatics 20 93-99.
27. GOEMAN, J., VAN DE GEER, S. and VAN HOUWELINGEN, H. (2006). Testing against a high dimensional alternative. J. R. Stat. Soc. Ser. B Stat. Methodol. 68 477-493. MR2278336
28. GRÖCHENIG, K. and LEINERT, M. (2006). Symmetry and inverse-closedness of matrix algebra and functional calculus for infinite matrices. Trans. Amer. Math. Soc. 358 2695-2711. MR2204052
29. HALL, P., PITTELKOW, Y. and GHOSH, M. (2008). Theoretical measures of relative performance of classifiers for high dimensional data with small sample sizes. J. R. Stat. Soc. Ser. B Stat. Methodol. 70 159-173. MR2412636
30. HALL, P. and JIN, J. (2008). Properties of higher criticism under strong dependence. Ann. Statist. 36 381-402. MR2387976
31. HORN, R. A. and JOHNSON, C. R. (2006). Matrix Analysis. Cambridge Univ. Press, Cambridge. MR1084815
32. INGSTER, Y. I. (1997). Some problems of hypothesis testing leading to infinitely divisible distribution. Math. Methods Statist. 6 47-69. MR1456646
33. INGSTER, Y. I. (1999). Minimax detection of a signal for l p n -balls. Math. Methods Statist. 7 401-428. MR1680087
34. JAFFARD, S. (1990). Propriétés des matrices "bien localisées" près de leur diagonale et quelques applications. Ann. Inst. H. Poincaré Anal. Non Linéaire 7 461-476. MR1138533
35. JAGER, L. andWELLNER, J. (2007). Goodness-of-fit tests via phi-divergences. Ann. Statist. 35 2018-2053. MR2363962
36. JIN, J. (2004). Detecting a target in very noisy data from multiple looks. In A Festschrift to Honor Herman Rubin (A. Dasgupta, ed.). Institute of Mathematical Statistics Lecture Notes-Monograph Series 45 255-286. IMS, Beachwood, OH. MR2126903
37. JIN, J. (2006). Higher criticism statistic: Theory and applications in non-Gaussian detection. In Statistical Problems in Particle Physics, Astrophysics And Cosmology (L. Lyons and M. K. Ünel, eds.). Imperial College Press, London. MR2270215
38. JIN, J. (2007). Proportion of nonzero normal means: Universal oracle equivalences and uniformly consistent estimators. J. R. Stat. Soc. Ser. B Stat. Methodol. 70 461-493. MR2420411
39. JIN, J. (2009). Impossibility of successful classification when useful features are rare and weak. Proc. Natl. Acad. Sci. USA 106 8859-8864. MR2520682
40. JIN, J. and CAI, T. (2007). Estimating the null and the proportion of non-null effects in largescale multiple comparisons. J. Amer. Statist. Assoc. 102 496-506. MR2325113
41. KUELBS, J. and VIDYASHANKAR, A. N. (2009). Asymptotic inference for high dimensional data. Ann. Statist. To appear.
42. MANSILLA, R., DE CASTILLO, N.,GOVEZENSKY, T.,MIRAMONTES, P., JOSÉ, M. and COHO, G. (2004). Long-range correlation in the whole human genome. Available at http://arxiv. org/pdf/q-bio/0402043v1.
43. MESSER, P. W. and ARNDT, P. F. (2006). CorGen-measuring and generating long-range correlations for DNA sequence analysis. Nucleic Acids Research 34 W692-W695.
44. JIN, J., STARCK, J.-L., DONOHO, D., AGHANIM, N. and FORNI, O. (2005). Cosmological non-Gaussian signature detection: Comparing performance of different statistical tests. EURASIP J. Appl. Signal Process. 15 2470-2485. MR2210857
45. MEINSHAUSEN, M. and RICE, J. (2006). Estimating the proportion of false null hypotheses among a large number of independently tested hypotheses. Ann. Statist. 34 373-393. MR2275246
46. OLEJNIK, S., LI, J. M., SUPATTATHUM, S. and HUBERTY, C. J. (1997). Multiple testing and statistical power with modified Bonferroni procedures. J. Educ. Behav. Statist. 22 389-406.
47. ROM, D. M. (1990). A sequentially rejective test procedure based on a modified Bonferroni inequality. Biometrika 77 663-665. MR1087860
48. SARKAR, S. K. and CHANG, C. K. (1997). The Simes method for multiple hypothesis testing with positively dependent test statistics. J. Amer. Statist. Assoc. 92 1601-1608. MR1615269
49. SHROACK, G. and WELLNER, J. (1986). Empirical Processes with Applications to Statistics. Wiley, New York. MR0838963
50. STRASSER, H. (1998). Differentiability of statistical experiments. Statist. Decisions 16 113-130. MR1621858
51. SUN, Q. (2005). Wiener's lemma for infinite matrices with polynomial off-diagonal decay. C. R. Math. Acad. Sci. Paris 340 567-570. MR2138705
52. TUKEY, J. W. (1989). Higher criticism for individual significances in several tables or parts of tables. Internal working paper, Princeton Univ.
53. WIENER, N. (1949). Extrapolation, Interpolation, and Smoothing of Stationary Time Series. Wiley, New York.
54. WU, W. B. (2008). On false discovery control under dependence. Ann. Statist. 36 364-380. MR2387975
55. ZYGMUND, A. (1959). Trigonometric Series, 2nd ed. Cambridge Univ. Press, New York. MR0107776