Stochastic expansions using continuous dictionaries: Lévy adaptive regression kernels

Annals of Statistics

Volumn 39, Issue 4, 2011, Pages 1916-1962

  Wolpert, Robert L ^a   Clyde, Merlise A ^a   Tu, Chong ^b  

^a Duke University   (United States)

^b PIMCO   (United States)

Author keywords

Bayes; Besov; Kernel regression; Lark; L ; Nonparametric regression; Relevance vector machine; Reversible jump Markov chain Monte Carlo; Splines; Support vector machine; Vy random field; Wavelets.

Indexed keywords

EID: 80052832466     PISSN: 00905364     EISSN: None     Source Type: Journal    
DOI: 10.1214/11-AOS889     Document Type: Article

References (70)

1. ABRAMOVICH, F., SAPATINAS, T. and SILVERMAN, B. W. (1998). Wavelet thresholding via a Bayesian approach. J. R. Stat. Soc. Ser. B Stat. Methodol. 60 725-749. MR1649547 (Pubitemid 128656809)
2. ABRAMOVICH, F., SAPATINAS, T. and SILVERMAN, B. W. (2000). Stochastic expansions in an overcomplete wavelet dictionary. Probab. Theory Related Fields 117 133-144. MR1759511
3. ABRAMOWITZ, M. and STEGUN, I. A. (1964). Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables. National Bureau of Standards Applied Mathematics Series 55. U.S. Government Printing Office, Washington, DC. MR0167642
4. ANTONIAK, C. E. (1974). Mixtures of Dirichlet processes with applications to Bayesian nonparametric problems. Ann. Statist. 2 1152-1174. MR0365969
5. CANDÈS, E. and TAO, T. (2007). The Dantzig selector: Statistical estimation when p is much larger than n. Ann. Statist. 35 2313-2351. MR2382644
6. CHEN, S. S., DONOHO, D. L. and SAUNDERS, M. A. (1998). Atomic decomposition by basis pursuit. SIAM J. Sci. Comput. 20 33-61. MR1639094 (Pubitemid 128689501)
7. CHILÈS, J.-P. and DELFINER, P. (1999). Geostatistics: Modeling Spatial Uncertainty. Wiley, New York. MR1679557
8. CHOI, T. and SCHERVISH, M. J. (2007). On posterior consistency in nonparametric regression problems. J. Multivariate Anal. 98 1969-1987. MR2396949
9. CHOUDHURI, N., GHOSAL, S. and ROY, A. (2004). Bayesian estimation of the spectral density of a time series. J. Amer. Statist. Assoc. 99 1050-1059. MR2109494 (Pubitemid 39665950)
10. CHU, J.-H., CLYDE, M. A. and LIANG, F. (2009). Bayesian function estimation using continuous wavelet dictionaries. Statist. Sinica 19 1419-1438. MR2589190
11. CHU, C.-K. andMARRON, J. S. (1991). Choosing a kernel regression estimator (with discussion). Statist. Sci. 6 404-436. MR1146907
12. CLYDE, M. A. and WOLPERT, R. L. (2007). Nonparametric function estimation using overcomplete dictionaries. In Bayesian Statistics 8 (J. M. Bernardo, M. J. Bayarri, J. O. Berger, A. P. Dawid, D. Heckerman, A. F. M. Smith and M. West, eds.) 91-114. Oxford Univ. Press, Oxford. MR2433190
13. CONT, R. and TANKOV, P. (2004). Financial Modelling with Jump Processes. Chapman & Hall/CRC, Boca Raton, FL. MR2042661
14. CRISTIANINI, N. and SHAWE-TAYLOR, J. (2000). An Introduction to Support Vector Machines and Other Kernel-based Learning Methods. Cambridge Univ. Press, Cambridge.
15. DAUBECHIES, I. (1988). Orthonormal bases of compactly supported wavelets. Comm. Pure Appl. Math. 41 909-996. MR0951745
16. DAUBECHIES, I. (1992). Ten Lectures on Wavelets. CBMS-NSF Regional Conference Series in Applied Mathematics 61. SIAM, Philadelphia, PA. MR1162107
17. DENISON, D. G. T., MALLICK, B. K. and SMITH, A. F. M. (1998). Automatic Bayesian curve fitting. J. R. Stat. Soc. Ser. B Stat. Methodol. 60 333-350. MR1616029 (Pubitemid 128656787)
18. DENISON, D. G. T., HOLMES, C. C., MALLICK, B. K. and SMITH, A. F. M. (2002). Bayesian Methods for Nonlinear Classification and Regression. Wiley, Chichester. MR1962778
19. DIMATTEO, I., GENOVESE, C. R. and KASS, R. E. (2001). Bayesian curve-fitting with free-knot splines. Biometrika 88 1055-1071. MR1872219 (Pubitemid 33771551)
20. DONOHO, D. L. and ELAD, M. (2003). Optimally sparse representation in general (nonorthogonal) dictionaries via l1 minimization. Proc. Natl. Acad. Sci. USA 100 2197-2202 (electronic). MR1963681 (Pubitemid 36297476)
21. DONOHO, D. L. and JOHNSTONE, I. M. (1994). Ideal spatial adaptation by wavelet shrinkage. Biometrika 81 425-455. MR1311089
22. ESCOBAR, M. D. (1994). Estimating normal means with a Dirichlet process prior. J. Amer. Statist. Assoc. 89 268-277. MR1266299
23. ESCOBAR, M. D. andWEST, M. (1995). Bayesian density estimation and inference using mixtures. J. Amer. Statist. Assoc. 90 577-588. MR1340510
24. FERGUSON, T. S. (1973). A Bayesian analysis of some nonparametric problems. Ann. Statist. 1 209-230. MR0350949
25. FERGUSON, T. S. (1974). Prior distributions on spaces of probability measures. Ann. Statist. 2 615-629. MR0438568
26. GHOSAL, S. and VAN DER VAART, A. (2007). Convergence rates of posterior distributions for noni. i.d. observations. Ann. Statist. 35 192-223. MR2332274
27. GILKS, W. R., RICHARDSON, S. and SPIEGELHALTER, D. J., eds. (1996). Markov Chain Monte Carlo in Practice. Chapman and Hall, London. MR1397966
28. GREEN, P. J. (1995). Reversible jump Markov chain Monte Carlo computation and Bayesian model determination. Biometrika 82 711-732. MR1380810
29. HIGDON, D. M. (1998). A process-convolution approach to modeling temperatures in the North Atlantic ocean. Environ. Ecol. Stat. 5 173-190.
30. HIGDON, D., SWALL, J. and KERN, J. (1999). Non-stationary spatial modeling. In Bayesian Statistics 6 (J. M. Bernardo, J. O. Berger, A. P. Dawid and A. F. M. Smith, eds.) 761-768. Oxford Univ. Press, Oxford.
31. JACOD, J. and SHIRYAEV, A. N. (1987). Limit Theorems for Stochastic Processes. Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences] 288. Springer, Berlin. MR0959133
32. JOHNSTONE, I. M. and SILVERMAN, B. W. (2004). Needles and straw in haystacks: Empirical Bayes estimates of possibly sparse sequences. Ann. Statist. 32 1594-1649. MR2089135 (Pubitemid 41250278)
33. JOHNSTONE, I. M. and SILVERMAN, B. W. (2005a). EBayesThresh: R programs for empirical Bayes thresholding. Journal of Statistical Software 12 1-38.
34. JOHNSTONE, I. M. and SILVERMAN, B. W. (2005b). Empirical Bayes selection of wavelet thresholds. Ann. Statist. 33 1700-1752. MR2166560 (Pubitemid 41423984)
35. JORDAN, M. I. (2010). Hierarchical models, nested models and completely random measures. In Frontiers of Statistical Decision Making and Bayesian Analysis: In Honor of James O. Berger (M.-H. Chen, D. K. Dey, P. Müller, D. Sun and K. Ye, eds.) 207-217. Springer, New York. MRMR2766461
36. KHINCHINE, A. Y. and LÉVY, P. (1936). Sur les lois stables. C. R. Math. Acad. Sci. Paris 202 374-376.
37. KINGMAN, J. F. C. (1967). Completely random measures. Pacific J. Math. 21 59-78. MR0210185
38. KWAPIEŃ, S. and WOYCZYŃSKI, W. A. (1992). Random Series and Stochastic Integrals: Single and Multiple. Birkhäuser, Boston, MA. MR1167198
39. LAW, M. H. and KWOK, J. T. (2001). Bayesian support vector regression. In Proceedings of the Eighth International Workshop on Artificial Intelligence and Statistics (AISTATS) 239-244. Key West, FL.
40. LIANG, F.,MUKHERJEE, S. and WEST, M. (2007). The use of unlabeled data in predictive modeling. Statist. Sci. 22 189-205. MR2408958
41. MACEACHERN, S. N. (1994). Estimating normal means with a conjugate style Dirichlet process prior. Comm. Statist. Simulation Comput. 23 727-741. MR1293996
42. MACEACHERN, S. N. (1998). Computational methods for mixture of Dirichlet process models. In Practical Nonparametric and Semiparametric Bayesian Statistics (D. K. Dey, P. Müller and D. Sinha, eds.). Lecture Notes in Statist. 133 23-43. Springer, New York. MR1630074
43. MALLAT, S. G. and ZHANG, Z. (1993). Matching pursuit with time-frequency dictionaries. IEEE Trans. Signal Process 41 3397-3415.
44. MÜLLER, P. and QUINTANA, F. A. (2004). Nonparametric Bayesian data analysis. Statist. Sci. 19 95-110. MR2082149 (Pubitemid 39056879)
45. NIKOL'SKǏI, S. M. (1975). Approximation of Functions of Several Variables and Imbedding Theorems. Die Grundlehren der Mathematischen Wissenschaften 205 Springer, New York. Translated from the Russian by John M. Danskin, Jr. MR0374877
46. PILLAI, N. S. (2008). Lévy random measures: Posterior consistency and applications. Ph.D. dissertation, Dept. Statist. Sci., Duke Univ. Available at http://stat.duke.edu/people/theses/PillaiNS.html.
47. PILLAI, N. S.,WU, Q., LIANG, F.,MUKHERJEE, S. andWOLPERT, R. L. (2007). Characterizing the function space for Bayesian kernel models. J. Mach. Learn. Res. 8 1769-1797 (electronic). MR2332448 (Pubitemid 47258008)
48. R DEVELOPMENT CORE TEAM (2004). R: A language and environment for statistical computing. R foundation for statistical computing. Available at http://www.R-project.org.
49. RAJPUT, B. S. and ROSIńSKI, J. (1989). Spectral representations of infinitely divisible processes. Probab. Theory Related Fields 82 451-487. MR1001524
50. REED, M. C. and SIMON, B. (1975). Methods of Modern Mathematical Physics, Vol. II: Fourier Analysis, Self-Adjointness. Academic Press, New York.
51. SATO, K.-I. (1999). Lévy Processes and Infinitely Divisible Distributions. Cambridge Studies in Advanced Mathematics 68. Cambridge Univ. Press, Cambridge. Translated from the 1990 Japanese original. Revised by the author. MR1739520
52. SCHMIDT, G., MATTERN, R. and SCHÜLER, F. (1981). Biomechanical investigation to determine physical and traumatological differentiation criteria for the maximum load capacity of head and vertebral column with and without protective helmet under the effects of impact. EEC research program on biomechanics of impacts, final report, phase III, Project 65, Institut für Rechtsmedizin, Univ. Heidelberg, Germany.
53. SILVERMAN, B. W. (1985). Some aspects of the spline smoothing approach to nonparametric regression curve fitting. J. R. Stat. Soc. Ser. B Stat. Methodol. 47 1-52. MR0805063
54. SISSON, S. A. (2005). Transdimensional Markov chains: A decade of progress and future perspectives. J. Amer. Statist. Assoc. 100 1077-1089. MR2201033 (Pubitemid 41276350)
55. SMITH, M. and KOHN, R. (1996). Nonparametric regression using Bayesian variable selection. J. Econometrics 75 317-343. (Pubitemid 126372818)
56. SOBOLEV, S. L. (1991). Some Applications of Functional Analysis in Mathematical Physics. Translations of Mathematical Monographs 90. Amer. Math. Soc., Providence, RI. MR1125990
57. SOLLICH, P. (2002). Bayesian methods for support vector machines: Evidence and predictive class probabilities. Machine Learning 46 21-52. (Pubitemid 34129962)
58. STEIN, M. L. (1999). Interpolation of Spatial Data: Some Theory for Kriging. Springer, New York. MR1697409
59. TIBSHIRANI, R. (1996). Regression shrinkage and selection via the lasso. J. R. Stat. Soc. Ser. B Stat. Methodol. 58 267-288. MR1379242
60. TIPPING, M. E. (2001). Sparse Bayesian learning and the relevance vector machine. J. Mach. Learn. Res. 1 211-244. MR1875838 (Pubitemid 33687203)
61. TRIEBEL, H. (1992). Theory of Function Spaces. II. Monographs in Mathematics 84. Birkhäuser, Basel. MR1163193
62. TU, C. (2006). Nonparametric modelling using Lévy process priors with applications for function estimation, time series modeling and spatio-temporal modeling. Ph.D. dissertation, Dept. Statist. Sci., Duke Univ. Available at http://www.stat.duke.edu/people/theses/TuC.html.
63. U.S. EPA. (2007). Air Quality System (AQS). Available at http://www.epa.gov/ttn/airs/airsaqs/.
64. VIDAKOVIC, B. (1999). Statistical Modeling by Wavelets. Wiley, New York. MR1681904
65. WAHBA, G. (1992). Multivariate function and operator estimation, based on smoothing splines and reproducing kernels. In Nonlinear Modeling and Forecasting: Proceedings of the Workshop on Nonlinear Modeling and Forecasting held September, 1990, in Santa Fe, New Mexico (M. Casdagli and S. G. Eubank, eds.). SFI Studies in the Sciences of Complexity XII 95-112. Addison- Wesley, Redwood, CA.
66. WEST, M. (2003). Bayesian factor regression models in the "large p, small n" paradigm. In Bayesian Statistics 7 (J. M. Bernardo et al., eds.) 733-742. Oxford Univ. Press, New York. MR2003537
67. WOLFE, P. J., GODSILL, S. J. and NG, W.-J. (2004). Bayesian variable selection and regularization for time-frequency surface estimation. J. R. Stat. Soc. Ser. B Stat. Methodol. 66 575-589. MR2088291 (Pubitemid 39015902)
68. WOLPERT, R. L., ICKSTADT, K. and HANSEN, M. B. (2003). A nonparametric Bayesian approach to inverse problems. In Bayesian Statistics 7 (J. M. Bernardo et al., eds.) 403-417. Oxford Univ. Press, New York. MR2003186
69. WOLPERT, R. L. and TAQQU, M. S. (2005). Fractional Ornstein-Uhlenbeck Lévy processes and the Telecom process: Upstairs and downstairs. Signal Processing 85 1523-1545. (Pubitemid 40762451)
70. ZOLOTAREV, V. M. (1986). One-dimensional Stable Distributions. Translations of Mathematical Monographs 65. Amer. Math. Soc., Providence, RI. MR0854867