Efficient learning and feature selection in high-dimensional regression

Neural Computation

Volumn 22, Issue 4, 2010, Pages 831-886

  Ting, Jo Anne ^a   D'Souza, Aaron ^b   Vijayakumar, Sethu ^a   Schaal, Stefan ^c,d  

^a UNIVERSITY OF EDINBURGH   (United Kingdom)

^b GOOGLE INC   (United States)

^c UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

^d ADVANCED TELECOMMUNICATIONS RESEARCH INSTITUTE INTERNATIONAL   (Japan)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHM; ARTICLE; ARTIFICIAL NEURAL NETWORK; AUTOMATED PATTERN RECOGNITION; HUMAN; LEARNING; PHYSIOLOGY; STATISTICAL MODEL;

ALGORITHMS; HUMANS; LEARNING; LINEAR MODELS; NEURAL NETWORKS (COMPUTER); PATTERN RECOGNITION, AUTOMATED;

EID: 77953353840     PISSN: 08997667     EISSN: 1530888X     Source Type: Journal    
DOI: 10.1162/neco.2009.02-08-702     Document Type: Article

References (54)

1. Belsley, D.A., Kuh, E., & Welsch, R.E. (1980). Regression diagnostics: Identifying influential data and sources of collinearity. New York:Wiley.
2. Bishop, C.M., & Tipping, M.E. (2000). Variational relevance vector machines. In C. Boutilier & M. Goldszmidt (Eds.), Proceedings of the 16th Conference in Uncertainty in Artificial Intelligence (pp. 46-53). San Francisco: Morgan Kaufmann.
3. Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273-297.
4. Derksen, S., & Keselman, H. (1992). Backward, forward and stepwise automated subset selection algorithms: Frequency of obtaining authentic and noise variables. British Journal of Mathematical and Statistical Psychology, 45, 265-282.
5. Efron, B., Hastie, T., Johnstone, L., & Tibshirani, R. (2004). Least angle regression. Annals of Statistics, 32, 407-499.
6. Everitt, B.S. (1984). An introduction to latent variable models. London: Chapman and Hall.
7. Fahlman, S.E., & Lebiere, C. (1989). The cascade-correlation learning architecture. In D. Touretzty (Ed.), Advances in neural information processing systems, 2. Cambridge, MA: MIT Press.
8. Feder, M., &Weinstein, E. (1988). Parameter estimation of superimposed signals using the EM algorithm. IEEE Transactions on Acoustics, Speech and Signal Processing, 36(4), 477-490.
9. Frank, I., & Friedman, J. (1993). A statistical view of some chemometric regression tools. Technometrics, 35, 109-135.
10. Friedman, J.H., Bentley, J.L., & Finkel, R.A. (1977). An algorithm for finding best matches in logarithmic expected time. ACM Transactions onMathematical Software, 3, 209-226.
11. Friedman, J., Hastie, T., & Tibshirani, R. (2007). Pathwise coordinate optimization (Tech. Rep.). Stanford, CA: Department of Statistics, Stanford University.
12. Fukumizu, K., Bach, F.R., & Jordan, M.I. (2006). Kernel dimensionality reduction in regression (Tech. Rep.). Berkeley: Department of Statistics, University of California, Berkeley.
13. Gelman, A., Carlin, J., Stern, H., & Rubin, D. (2000). Bayesian data analysis. London: Chapman and Hall.
14. Ghahramani, Z., & Beal, M. (2000a). Graphical models and variational methods. In D. Saad & M. Opper (Eds.), Advanced mean field methods: Theory and practice. Cambridge, MA: MIT Press.
15. Ghahramani, Z., & Beal, M.J. (2000b). Variational inference for Bayesian mixtures of factor analyzers. In S.A. Solla, T.K. Leen, & K.-R. M̈ uller (Eds.), Advances in neural information processing systems, 12. Cambridge, MA: MIT Press.
16. Ghahramani, Z., &Hinton, G.E. (1997). The EM algorithm for mixtures of factor analyzers (Tech. Rep. No CRG-TR-96-1). Toronto: University of Toronto.
17. Goodman, J. (2004). Exponential priors formaximum entropy models. In Proceedings of the AnnualMeeting of the Association for Computational Linguistics. San Francisco: Morgan Kaufmann.
18. Gray, A.G., & Moore, A. (2001). N-body problems in statistical learning. In T.K. Leen, T.G. Dietterich, & V. Tresp (Eds.), Advances in neural information processing systems, 13. Cambridge, MA:MIT Press.
19. Hastie, T., & Tibshirani, R. (1990). Generalized additive models. London: Chapman and Hall.
20. Hastie, T., & Tibshirani, R. (2000). Bayesian backfitting. Statistical Science, 15(3), 196-223.
21. Jaakkola, T.S., & Jordan, M.I. (2000). Bayesian parameter estimation via variational methods. Statistics and Computing, 10(1), 25-37.
22. Jeffreys, H. (1946).An invariant form for the prior probability in estimation problems. Journal of the Royal Statistical Society. Series A, 186, 453-461.
23. Kakei, S., Hoffman, D., & Strick, P. (1999). Muscle and movement representations in the primary motor cortex. Science, 285, 2136-2139.
24. Keerthi, S.S., Chapelle, O., & DeCoste, D. (2006). Building support vector machines with reduced classifier complexity. Journal of Machine Learning Research, 7, 1493-1515.
25. Kim, S.-J., Koh, K., Lustig, M., Boyd, S., & Gorinevsky, D. (2007). A method for large-scale l1-regularized least squares. IEEE Journal on Selected Topics in Signal Processing, 1(4), 606-617.
26. Komarek, P., & Moore, A. (2000). A dynamic adaptation of AD-trees for efficient machine learning on large data sets. In Proceedings of the 17th International Conference on Machine Learning. San Francisco: Morgan Kaufmann.
27. Krishnapuram, B., Carin, L., Figueiredo, M.A., & Hartemink, A.J. (2005). Sparse multinomial logistic regression: Fast algorithms and generalization bounds. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(6), 957-968.
28. Lecun, Y., Bottou, A., Bengio, Y., &Haffner, P. (1998).Gradient-based learning applied to document recognition. In Proceedings of the IEEE. Piscataway, NJ: IEEE.
29. Lee, S., Lee, H., Abbeel, P., & Ng, A. (2006). Efficient L1-regularized logistic regresssion. In Proceedings of the 21st National Conference on Artificial Intelligence: Menlo Park, CA: AAAI Press.
30. Lokhorst, J. (1999). The LASSO and generalized linear models (Tech, Rep.). South Adelaide: Department of Statistics, University of South Adelaide, South Australia, Australia.
31. MacKay, D.J.C. (1999). Comparison of approximate methods for handling hyperparameters. Neural Computation, 11(5), 1035-1068.
32. Massey, W. (1965). Principal component regression in exploratory statistical research. Journal of the American Statistical Association, 60, 234-246.
33. Moore, A. (2000). The anchors hierarchy: Using the triangle inequality to survive high dimensional data. In C. Boutilier & M. Goldszmidt (Eds.), Proceedings of the 16th Conference in Uncertainty in Artificial Intelligence (pp. 397-405). San Francisco: Morgan Kaufmann.
34. Moore, A., & Lee, M.S. (1998). Cached sufficient statistics for efficient machine learning with large datasets. Journal of Artificial Intelligence Research, 8, 67-91.
35. Neal, R. (1994). Bayesian learning for neural networks. Unpublished doctoral dissertation, University of Toronto.
36. Ng, A.Y. (2004). Feature selection, L1 vs. L2 regularization, and rotational invariance. In Proceedings of the 21st International Conference on Machine Learning. New York: ACM Press.
37. Omohundro, S.M. (1990). Bumptrees for efficient function, constraint and classification learning. InD. Touretzky&R. Lippmann (Eds.), Advances in neural information processing systems, 3. Cambridge, MA:MIT Press.
38. Parisi, G. (1988). Statistical field theory. Reading, MA: Addison-Wesley.
39. R̈atsh, G., Onoda, T., & M̈ uller, K.-R. (2001). Soft margins for AdaBoost. Machine Learning, 42(3), 287-320.
40. Ripley, B.D. (1996). Pattern recognition and neural networks. Cambridge: Cambridge University Press.
41. Rustagi, J. (1976). Variational methods in statistics. Orlando, FL: Academic Press.
42. Schaal, S., Vijayakumar, S., & Atkeson, C. (1998). Local dimensionality reduction. In M. Jordan, M. Kearns, & S. Solla (Eds.), Advances in neural information processing systems, 10. Cambridge, MA: MIT Press.
43. Sergio, L., & Kalaska, J. (1998). Changes in the temporal pattern of primary motor cortex activity in a directional isometric force versus limb movement task. Journal of Neurophysiology, 80, 1577-1583.
44. Shevade, S., & Keerthi, S. (2003). A simple and efficient algorithm for gene selection using sparse logistic regression. Bioinformatics, 19(17), 2246-2253.
45. Tibshirani, R. (1996). Regression shrinkage and selection via the LASSO. Journal of Royal Statistical Society, Series B, 58(1), 267-288.
46. Ting, J., D'Souza, A., & Schaal, S. (2006). Bayesian regression with input noise for high dimensional data. In Proceedings of the 23rd InternationalConference onMachine Learning (pp. 937-944). New York: ACM.
47. Ting, J., D'Souza, A., Yamamoto, K., Yoshioka, T., Hoffman, D., Kakei, S., et al. (2005). Predicting EMG data from M1 neurons with variational Bayesian least squares. In Y.Weiss, B. Scḧolkopf, & J. Platt (Eds.), Advances in neural information processing systems, 18. Cambridge, MA: MIT Press.
48. Tipping, M.E. (2001). Sparse Bayesian learning and the relevance vector machine. Journal of Machine Learning Research, 1, 211-244.
49. Tipping, M.E., &Bishop, C.M. (1999).Mixtures of probabilistic principal component analyzers. Neural Computation, 11(2), 443-482.
50. Tipping, M.E., & Faul, A.C. (2003). Fast marginal likelihood maximisation for sparse Bayesian methods. In C.M. Bishop & B.J. Frey (Eds.), Proceedings of the 9th International Workshop on Artificial Intelligence and Statistics. N.p.: Society for Artificial Intelligence and Statistics.
51. Vijayakumar, S., & Schaal, S. (2000). Locally weighted projection regression: Incremental real time learning in high dimensional space. In Proceedings of the 17th International Conference onMachine Learning (pp. 1079-1086). San Francisco:Morgan Kaufmann.
52. Wang, L., Zhu, J., & Zou, H. (2006). The doubly regularized support vector machine. Statistica Sinica, 16, 589-615.
53. Williams, C.K.I., & Rasmussen, C.E. (1996). Gaussian processes for regression. In D.S. Touretzky, M.C. Mozer, & M.E. Hasselmo (Eds.), Advances in neural information processing systems, 8. Cambridge, MA: MIT Press.
54. Wold, H. (1975). Soft modeling by latent variables: The nonlinear iterative partial least squares approach. In J. Gani (Ed.), Perspectives in probability and statistics: Papers in honor of M.S. Bartlett. London: Academic Press.