Many regression algorithms, one unified model: A review

Neural Networks

Volumn 69, Issue , 2015, Pages 60-79

  Stulp, Freek ^a,b   Sigaud, Olivier ^c  

^a ENSTA PARISTECH   (France)

^b INRIA   (France)

^c SORBONNE UNIVERSITÉ   (France)

Author keywords

Gaussian mixture regression; Gaussian process regression; Locally weighted regression; Radial basis function networks; Regression

Indexed keywords

ALGORITHMS; FUNCTIONS; NEURAL NETWORKS; REGRESSION ANALYSIS;

FIRST PRINCIPLES; FUNCTION REPRESENTATIONS; GAUSSIAN MIXTURES; GAUSSIAN PROCESS REGRESSION; LOCALLY WEIGHTED REGRESSION; PROCESS OF LEARNING; REGRESSION; REGRESSION ALGORITHMS;

RADIAL BASIS FUNCTION NETWORKS;

ALGORITHM; BACK PROPAGATION; EXTREME LEARNING MACHINE; GAUSSIAN MIXTURE REGRESSION; GAUSSIAN PROCESS REGRESSION; INCREMENTAL RANDOM FEATURES REGULARIZED LEAST SQUARES; INCREMENTAL SPARSE SPECTRUM GAUSSIAN PROCESS REGRESSION; KERNEL RIDGE REGRESSION; LOCALLY WEIGHTED PROJECTION REGRESSION; LOCALLY WEIGHTED REGRESSION; MATHEMATICAL MODEL; MODEL TREE REGRESSION; MULTI LAYER FEEDFORWARD NETWORK; NONLINEAR REGRESSION ANALYSIS; PRIORITY JOURNAL; RADIAL BASED FUNCTION; RECEPTIVE FIELD WEIGHTED REGRESSION; RECURRENT NEURAL NETWORK; REGRESSION ANALYSIS; REGRESSION WITH RADIAL BASIS FUNCTION NETWORK; REVIEW; STATISTICAL MODEL; SUPPORT VECTOR MACHINE; XCS FOR FUNCTIONS; ARTIFICIAL NEURAL NETWORK; HUMAN; NORMAL DISTRIBUTION; THEORETICAL MODEL;

ALGORITHMS; HUMANS; LINEAR MODELS; MODELS, THEORETICAL; NEURAL NETWORKS (COMPUTER); NORMAL DISTRIBUTION;

EID: 84931291727     PISSN: 08936080     EISSN: 18792782     Source Type: Journal    
DOI: 10.1016/j.neunet.2015.05.005     Document Type: Review

References (75)

1. Alain, G., & Bengio, Y. (2013). What regularized auto-encoders learn from the data generating distribution. arxiv:1211.4246v4.
2. Atkeson C.G., Schaal S. Memory-based neural networks for robot learning. Neurocomputing 1995, 9(3):243-269.
3. Augusteijn M.F., Harrington T.P. Evolving transfer functions for artificial neural networks. Neural Computing & Applications 2004, 13(1):38-46.
4. Bengio Y. Learning deep architectures for AI. Foundations and Trends in Machine Learning 2009, 2(1):1-127.
5. Bengio Y., Courville A., Vincent P. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence 2013, 35(8):1798-1828.
6. Bishop C.M. Neural networks for pattern recognition 1995, Oxford University Press.
7. Bishop C.M. Model-based machine learning. Philosophical Transactions of the Royal Society A 2013, 371:1-17.
8. Bishop C.M., et al. Pattern recognition and machine learning. Vol. 1 2006, Springer, New York.
9. Bloomfield P., Steiger W. Least absolute deviations curve-fitting. SIAM Journal on Scientific Computing 1980, 1(2):290-301.
10. Breiman L., Friedman J., Olshen R., Stone C. Classification and regression trees 1984, Wadsworth and Brooks, Monterey, CA.
11. Butz M.V., Herbort O. Context-dependent predictions and cognitive arm control with XCSF. Conference on genetic and evolutionary computation 2008, 1357-1364. ACM.
12. Butz M.V., Pedersen G.K.M., Stalph P.O. Learning sensorimotor control structures with XCSF: redundancy exploitation and dynamic control. Conference on genetic and evolutionary computation 2009, 1171-1178. ACM.
13. Calinon S. Robot programming by demonstration 2009, EPFL/CRC Press.
14. Calinon S., Guenter F., Billard A. On learning, representing, and generalizing a task in a humanoid robot. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 2007, 37(2):286-298.
15. Cederborg, T., Li, M., Baranes, A., & Oudeyer, P.-Y. (2010). Incremental local online Gaussian mixture regression for imitation learning of multiple tasks. In IEEE/RSJ international conference on intelligent robots and systems (pp. 267-274).
16. Cohen S., Intrator N. A hybrid projection-based and radial basis function architecture: initial values and global optimisation. Pattern Analysis & Applications 2002, 5(2):113-120.
17. Dauphin Y.N., Pascanu R., Gulcehre C., Cho K., Ganguli S., Bengio Y. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. Advances in neural information processing systems 2014, 2933-2941.
18. Dempster A.P., Laird N.M., Rubin D.B. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society. Series B (Methodological) 1977, 1-38.
19. Dorffner G. A unified framework for MLPs and RBFNs: Introducing conic section function networks. Cybernetics and Systems 1994, 25(4):511-554.
20. Droniou, A., Ivaldi, S., Padois, V., & Sigaud, O. (2012). Autonomous online learning of velocity kinematics on the iCub: A comparative study. In Intelligent robots and systems, IROS, 2012 IEEE/RSJ international conference on (pp. 3577-3582).
21. Droniou, A., Ivaldi, S., Stalph, P., Butz, M., & Sigaud, O. (2012). Learning velocity kinematics: Experimental comparison of on-line regression algorithms. In Proceedings robotica (pp. 15-20).
22. Ebden M. Gaussian processes for regression: A quick introduction. Tech. rep 2008, Department on Engineering Science, University of Oxford.
23. Fisher R. Statistical methods for research workers 1925, Oliver & Boyd.
24. Friedman J.H., Stuetzle W. Projection pursuit regression. Journal of the American Statistical Association 1981, 76(376):817-823.
25. Geladi P., Kowalski B. Partial least squares regression: A tutorial. Analytica Chimica Acta 1986, 185:1-17.
26. Ghahramani Z., Jordan M.I. Supervised learning from incomplete data via an EM approach. Advances in neural information processing systems, Vol. 6 1993, 120-127.
27. Gijsberts, A., & Metta, G. (2011). Incremental learning of robot dynamics using random features. In IEEE international conference on robotics and automation(pp. 951-956).
28. Gijsberts A., Metta G. Real-time model learning using incremental sparse spectrum Gaussian process regression. Neural Networks 2012.
29. Grollman, D., & Jenkins, O.C. (2008). Sparse incremental learning for interactive robot control policy estimation. In IEEE international conference on robotics and automation (pp. 3315-3320).
30. Hersch M., Guenter F., Calinon S., Billard A. Dynamical system modulation for robot learning via kinesthetic demonstrations. IEEE Transactions on Robotics 2008, 24(6):1463-1467.
31. Hochreiter S., Bengio Y., Frasconi P., Schmidhuber J. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. A field guide to dynamical recurrent neural networks 2001, IEEE Press. K. Kremer (Ed.).
32. Huang G.-B., Chen L. Enhanced random search based incremental extreme learning machine. Neurocomputing 2008, 71(16-18):3460-3468.
33. Huang G.-B., Chen L., Siew C.K. Universal approximation using incremental constructive feedforward networks with random hidden nodes. IEEE Transactions on Neural Networks 2006, 17(4):879-892.
34. Huang G.-B., Li M.-B., Chen L., Siew C.K. Incremental extreme learning machine with fully complex hidden nodes. Neurocomputing 2008, 71(4-6):576-583.
35. Huang G.-B., Wang D.H., Lan Y. Extreme learning machines: a survey. International Journal of Machine Learning and Cybernetics 2011, 2(2):107-122.
36. Huang G.-B., Zhu Q.-Y., Siew C.-K. Extreme learning machine: theory and applications. Neurocomputing 2006, 70(1):489-501.
37. Ijspeert A.J., Nakanishi J., Hoffmann H., Pastor P., Schaal S. Dynamical movement primitives: learning attractor models for motor behaviors. Neural Computing 2013, 25(2):328-373.
38. Lammert, A.C., Goldstein, L., & Iskarous, K. (2010). Locally-weighted regression for estimating the forward kinematics of a geometric vocal tract model. In INTERSPEECH (pp. 1604-1607).
39. Marin D., Decock J., Rigoux L., Sigaud O. Learning cost-efficient control policies with XCSF: generalization capabilities and further improvement. Proceedings of the 13th annual conference on genetic and evolutionary computation 2011, 1235-1242. ACM.
40. Meier, F., Hennig, P., & Schaal, S. (2014). Local Gaussian regression. ArXiv Preprint . arxiv:1402.0645.
41. Munzer, T., Stulp, F., & Sigaud, O. (2014). Non-linear regression algorithms for motor skill acquisition: a comparison. In Proceedings JFPDA (pp. 1-16).
42. Neal R.M. Bayesian learning for neural networks 1996, Springer-Verlag.
43. Nguyen-Tuong D., Seeger M., Peters J. Model learning with local gaussian process regression. Advanced Robotics 2009, 23(15):2015-2034.
44. Orr M.J.L., Hallam J., Takezawa K., Murray A.F., Ninomiya S., Oide M., et al. Combining regression trees and radial basis function networks. International Journal of Neural Systems 2000, 10(6):453-465.
45. Park J., Sandberg I.W. Approximation and radial-basis-function networks. Neural Computing 1993, 5(2):305-316.
46. Pascanu, R., Montúfar, G., & Bengio, Y. (2013). On the number of inference regions of deep feed forward networks with piece-wise linear activations. ArXiv Preprint . arxiv:1312.6098.
47. Plackett R.L. Some theorems in least squares. Biometrika 1950, 37:149-157.
48. Poggio T., Girosi F. Networks for approximation and learning. Proceedings of the IEEE 1990, 78(9).
49. Quinlan R.J. Learning with continuous classes. 5th Australian joint conference on artificial intelligence 1992, 343-348. World Scientific, Singapore.
50. Quiñonero Candela J., Rasmussen C.E. A unifying view of sparse approximate Gaussian process regression. Journal of Machine Learning Research 2005, 6:1939-1959.
51. Rahimi A., Recht B. Random features for large-scale kernel machines. Advances in neural information processing systems 2007, 1177-1184.
52. Rahimi A., Recht B. Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. Advances in neural information processing systems, Vol. 21 2008, 1313-1320. URL: http://books.nips.cc/papers/files/nips21/NIPS2008_0885.pdf.
53. Rosenblatt F. The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review 1958, 65(6):386-408.
54. Saunders C., Gammerman A., Vovk V. Ridge regression learning algorithm in dual variables. Proceedings of the 15th international conference on machine learning 1998, 515-521. Morgan Kaufmann.
55. Schaal S., Atkeson C.G. Receptive field weighted regression. Tech. Rep. TR-H-209 1997, ATR Human Information Processing Laboratories.
56. Schmidhuber, J. (2014). Deep learning in neural networks: An overview. ArXiv Preprint . arxiv:1404.7828.
57. Schmidt M. Least squares optimization with l1-norm regularization. Tech. rep., CS542B Project Report 2005.
58. Schmitt M. On the complexity of computing and learning with multiplicative neural networks. Neural Computation 2002, 14(2):241-301.
59. Schölkopf B., Smola A., Williamson R., Bartlett P. New support vector algorithms. Neural Computing 2000, 12(5):1207-1245.
60. Schwenker F., Kestler H.A., Palm G. Three learning phases for radial-basis-function networks. Neural Networks 2001, 14(4-5):439-458.
61. Schwenker F., Kestler H.A., Palm G. Three learning phases for radial-basis-function networks. Neural Networks 2001, 14(4):439-458.
62. Sigaud O., Salaün C., Padois V. On-line regression algorithms for learning mechanical models of robots: a survey. Robotics and Autonomous Systems 2011, 51:1117-1125.
63. Smola A.J., Schölkopf B. A tutorial on support vector regression. Statistics and Computing 2004, 14(3):199-222.
64. Stulp F., Beetz M. Refining the execution of abstract actions with learned action models. Journal of Artificial Intelligence Research (JAIR) 2008, 32:487-523.
65. Stulp F., Theodorou E., Schaal S. Reinforcement learning with sequences of motion primitives for robust manipulation. IEEE Transactions on Robotics 2012, 28(6):1360-1370. king-Sun Fu Best Paper Award of the IEEE Transactions on Robotics for the year 2012.
66. Tibshirani R. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society (Series B) 1996, 58:267-288.
67. Vapnik V. The nature of statistical learning theory 1995, Springer, New York.
68. Vijayakumar, S., & Schaal, S. (2000). Locally weighted projection regression: An O(n) algorithm for incremental real time learning in high dimensional space. In Proceedings of the seventeenth international conference on machine learning: Vol. 1 (pp. 288-293).
69. Wang Y., Witten I.H. Induction of model trees for predicting continuous classes. Poster papers of the 9th European conference on machine learning 1997, Springer.
70. Werbos P.J. Beyond regression: New tools for prediction and analysis in the behavioral sciences 1974, (Ph.D. thesis), Harvard University.
71. Werbos P.J. Generalization of backpropagation with application to a recurrent gas market model. Neural Networks 1988, 1(4):339-356.
72. Widrow B., Greenblatt A., Kim Y., Park D. The no-prop algorithm: A new learning algorithm for multilayer neural networks. Neural Networks 2013, 37:182-188.
73. Williams C.K.I. Computation with infinite neural networks. Neural Computation 1998, 10(5):1203-1216.
74. Williams C.K., Rasmussen C.E. Gaussian processes for machine learning 2006, MIT Press.
75. Wilson D.R., Martinez T.R. The general inefficiency of batch training for gradient descent learning. Neural Networks 2003, 16(10):1429-1451.