Bayesian support vector regression using a unified loss function

IEEE Transactions on Neural Networks

Volumn 15, Issue 1, 2004, Pages 29-44

  Chu, Wei ^a   Keerthi, S Sathiya ^b   Ong, Chong Jin ^b  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

^b NATIONAL UNIVERSITY OF SINGAPORE   (Singapore)

Author keywords

Automatic relevance determination; Bayesian inference; Gaussian processes; Model selection; Nonquadratic loss function; Support vector regression

Indexed keywords

COMPUTER SIMULATION; ERROR ANALYSIS; MATHEMATICAL MODELS; MAXIMUM LIKELIHOOD ESTIMATION; PROBLEM SOLVING; QUADRATIC PROGRAMMING; REGRESSION ANALYSIS;

AUTOMATIC RELEVANCE DETERMINATION; BAYESIAN INFERENCE; GAUSSIAN PROCESSES; MODEL SELECTION; NONQUADRATIC LOSS FUNCTION; SUPPORT VECTOR REGRESSION;

NEURAL NETWORKS;

ARTICLE; BAYES THEOREM; REGRESSION ANALYSIS;

BAYES THEOREM; REGRESSION ANALYSIS;

EID: 1242331293     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2003.820830     Document Type: Article

References (41)

1. D. J. C. MacKay, "A practical Bayesian framework for back propagation networks," Neural Computation, vol. 4, no. 3, pp. 448-472, 1992.
2. R. M. Neal, "Bayesian Training of Backpropagation Networks by the Hybrid Monte Carlo Method," Department of Statistics, University of Toronto, Tech. Rep.CRG-TR-92-1, 1992.
3. W. L. Buntine and A. S. Weigend, "Bayesian back-propagation," Complex Systems, vol. 5, no. 6, pp. 603-643, 1991.
4. C. M. Bishop, Neural Networks for Pattern Recognition. London, U.K.: Oxford Univ. Press, 1995.
5. D. J. C. MacKay, "Probable networks and plausible predictions - a review of practical Bayesian methods for supervised neural networks," Network: Computation in Neural Syst., vol. 6, no. 3, pp. 469-505, 1995.
6. J. Lampinen and A Vehtari, "Bayesian approach for neural networks - reviews and case studies," Neural Netw., vol. 14, pp. 257-274, 2001.
7. R. M. Neal, Bayesian Learning for Neural Networks, ser. Lecture Notes in Statistics, U.K.: Springer, 1996.
8. C. K. I. Williams and C. E. Rasmussen, "Gaussian processes for regression," in Advances in Neural Information Processing Systems, D. S. Touretzky, M. C. Mozer, and M. E. Hasselmo, Eds. Cambridge, MA: MIT Press, 1996, vol. 8, pp. 598-604.
9. C. K. I. Williams, "Prediction with Gaussian processes: from linear regression to linear prediction and beyond," in Learning and Inference in Graphical Models. Norwell, MA: Kluwer, 1998.
10. V. N. Vapnik, The Nature of Statistical Learning Theory. New York: Springer-Verlag, 1995.
11. G. Wahba, Spline Models for Observational Data, ser. CBMS-NSF Regional Conference Series in Applied Mathematics. Philadelphia, PA: SIAM, 1990, vol. 59.
12. J. T. Kwok, "The evidence framework applied to support vector machines," IEEE Trans. Neural Networks, vol. 11, pp. 1162-1173, Sept. 2000.
13. M. Seeger, "Bayesian model selection for support vector machines, Gaussian processes and other kernel classifiers," in Advances in Neural Information Processing Systems, vol. 12, 2000, pp. 603-609.
14. P. Sollich, "Bayesian methods for support vector machines: evidence and predictive class probabilities," Machine Learning, vol. 46, pp. 21-52, 2002.
15. M. H. Law and J. T. Kwok, "Bayesian support vector regression," in Proc. 8th Int. Workshop on Artificial Intelligence and Statistics (AIS-TATS), Key West, FL, 2001, pp. 239-244.
16. J. B. Gao, S. R. Gunn, C. J. Harris, and M. Brown, "A probabilistic framework for SVM regression and error bar estimation," Machine Learning, vol. 46, pp. 71-89, Mar. 2002.
17. C. Saunders, A. Gammerman, and V. Vovk, "Ridge regression learning algorithm in dual variables," in Proc. 15th Int. Conference on Machine Learning, 1998, pp. 515-521.
18. C. A. Micchelli, "Interpolation of scatter data: distance matrices and conditionally positive definite functions," Constructive Approximation, vol. 2, pp. 11-22, 1986.
19. T. Evgeniou, M. Pontil, and T. Poggio, A Unified Framework for Regularization Networks and Support Vector Machines, MIT, Cambridge, MA, 1999. A.I. Memo 1654.
20. G. S. Kimeldorf and G. Wahba, "Some results on Tchebycheffian spline function," J. Math. Analysis Applicat., vol. 33, pp. 82-95, 1971.
21. B. Schölkopf, R. Herbrich, and A. J. Smola, "A generalized representer theorem," in Proc. Annu. Conf. Computational Learning Theory, 2001.
22. P. J. Huber, Robust Statistics. New York: Wiley, 1981.
23. M. Pontil, S. Mukherjee, and F. Girosi, On the Noise Model of Support Vector Regression, MIT, Artificial Intelligence Lab., 1998. A.I. Memo 1651.
24. W. Chu, S. S. Keerthi, and C. J. Ong, "A unified loss function in Bayesian framework for support vector regression," in Proc. 18th Int. Conf. Machine Learning, 2001, http://guppy.mpe.nus.edu.sg/~mpessk/svm/icml.pdf., pp. 51-58.
25. R. Fletcher, Practical Methods of Optimization. New York: Wiley, 1987.
26. R. J. Vanderbei, Linear Programming: Foundations and Extensions, 2nd ed. Norwell, MA: Kluwer, June 2001, vol. 37, Int. Series in Operations Research and Management Science.
27. A. J. Smola and B. Schölkopf, "A Tutorial on Support Vector Regression,", GMD First, Technical Report NC2-TR-1998-030, Oct. 1998.
28. S. K. Shevade, S. S. Keerthi, C. Bhattacharyya, and K. R. K. Murthy, "Improvements to the SMO algorithm for SVM regression," IEEE Trans. Neural Networks, vol. 11, pp. 1188-1194, Sept. 2000.
29. W. Chu, "Bayesian approach to support vector machines," Ph.D. dissertation, Nat. Univ. Singapore, Jan. 2003.
30. D. G. Luenberger, Linear and Nonlinear Programming, 2nd ed. Menlo Park, CA: Addison-Wesley, 1984.
31. S. S. Keerthi and S. K. Shevade, "SMO algorithm for least squares SVM formulations," Neural Computation, vol. 15, no. 2, pp. 487-507, Feb. 2003.
32. T. S. Jebara and T. S. Jaakkola, "Feature selection and dualities in maximum entropy discrimination," in Proc. 16th Conf. Uncertainty in Artificial Intelligence (UAI-2000), San Francisco, CA, 2000, pp. 291-300.
33. J. Weston, S. Mukherjee, O. Chapelle, M. Pontil, and T. Poggio, "Feature selection in SVMs," in Advances in Neural Information Processing Systems, T. Leen, T. Dietterich, and V. Tresp, Eds. Cambridge, MA: MIT Press, 2001, vol. 13.
34. D. J. C. MacKay, "Bayesian methods for backpropagation networks," Models of Neural Networks III, pp. 211-254, 1994.
35. B. Schölkopf and A. J. Smola, "New Support Vector Machines,", GMD FIRST, NeuroCOLT2 Tech. Rep. NC2-TR-1998-031, vol. 1998.
36. M. H. Law and J. T. Kwok, "Applying the Bayesian evidence framework to ν-support vector regression," in Proc. 12th Eur. Conf. Machine Learning, Freiburg, Germany, 2001, pp. 312-323.
37. M. Opper and D. Saad, Advanced Mean Field Methods - Theory and Practice. London, U.K.: MIT Press, Feb. 2001.
38. S. Duane, A. D. Kennedy, and B. J. Pendleton, "Hybrid Monte Carlo," Phys. Lett. B, vol. 195, no. 2, pp. 216-222, 1987.
39. R. H. Byrd, P. Lu, and J. Nocedal, "A limited memory algorithm for bound constrained optimization," SIAM J. Sci. Stat. Comput., vol. 16, no. 5, pp. 1190-1208, 1995.
40. K.-R. Müller, A. Smola, G. Rätsch, B. Schölkopf, J. Kohlmorgen, and V. Vapnik, "Using support vector machines for time series prediction," in Proc. Int. Conf. Artificial Neural Networks ICANN '97, W. Gerstner, A. Germond, M. Hasler, and J.-D. Nicoud, Eds. Berlin: springer, 1997, pp. 999-1004.
41. M. E. Tipping, "The relevance vector machine," in Advances in Neural Information Processing Systems 12, S. A. Solla, T. K. Leen, and K.-R. Mller, Eds. Cambridge, MA: MIT Press, 2000, pp. 652-658.