Robust sequential learning of feedforward neural networks in the presence of heavy-tailed noise

Neural Networks

Volumn 63, Issue , 2015, Pages 31-47

  Vuković, Najdan ^a   Miljković, Zoran ^a  

^a UNIVERSITY OF BELGRADE   (Serbia)

Author keywords

Feedforward neural networks; Heavy tailed noise; Inverse Wishart distribution; Robust extended Kalman filter; Sequential learning; Structured variational approximation

Indexed keywords

ESTIMATION; EXTENDED KALMAN FILTERS; ITERATIVE METHODS; LEARNING ALGORITHMS; PROBABILITY DISTRIBUTIONS; RANDOM PROCESSES; SPURIOUS SIGNAL NOISE; STATISTICS; STOCHASTIC MODELS; STOCHASTIC SYSTEMS;

HEAVY-TAILED NOISE; INVERSE WISHART DISTRIBUTIONS; ROBUST EXTENDED KALMAN FILTERS; SEQUENTIAL LEARNING; STRUCTURED VARIATIONAL APPROXIMATIONS;

FEEDFORWARD NEURAL NETWORKS;

ARTICLE; ARTIFICIAL NEURAL NETWORK; BAYES THEOREM; CALCULATION; CONTROLLED STUDY; COVARIANCE; LEARNING ALGORITHM; MATHEMATICAL COMPUTING; NOISE; NOISE MEASUREMENT; PROBABILITY; PROCESS DEVELOPMENT; ALGORITHM; FEEDBACK SYSTEM;

ALGORITHMS; BAYES THEOREM; FEEDBACK; NEURAL NETWORKS (COMPUTER);

EID: 84912000366     PISSN: 08936080     EISSN: 18792782     Source Type: Journal    
DOI: 10.1016/j.neunet.2014.11.001     Document Type: Article

References (62)

1. Agamennoni G., Nebot E.M. Robust estimation in non-linear state-space models with state-dependent noise. IEEE Transactions on Signal Processing 2014, 62(8):2165-2175.
2. Agamennoni G., Nieto J., Nebot E. An outlier-robust Kalman filter. IEEE international conference on robotics and automation 2011, (IEEE) Institute of Electrical and Electronics Engineers, USA.
3. Agamennoni G., Nieto J., Nebot E. Approximate inference in state-space models with heavy-tailed noise. IEEE Transactions on Signal Processing 2012, 60(10):5024-5037.
4. Andrieu C., De Freitas N., Doucet A. Robust full Bayesian learning for radial basis networks. Neural Computation 2001, 13(10):2359-2407.
5. Archambeau C., Verleysen M. Robust Bayesian clustering. Neural Networks 2007, 20(1):129-138.
6. Bache K., Lichman M. UCI machine learning repository 2013, University of California, School of Information and Computer Science., Irvine, CA, http://archive.ics.uci.edu/ml (last date of access: February 10, 2014).
7. Barber D. Bayesian reasoning and machine learning 2012, Cambridge University Press.
8. Beal M.J. Variational algorithms for approximate Bayesian inference 2003, (Doctoral dissertation), University of London.
9. Billings S.A., Wei H.L., Balikhin M.A. Generalized multiscale radial basis function networks. Neural Networks 2007, 20(10):1081-1094.
10. Bishop C. Pattern recognition and machine learning 2006, Springer, Berlin.
11. Box G., Tiao G. Bayesian inference in statistical analysis 1973, John Wiley & Sons.
12. Chandola V., Banerjee A., Kumar V. Anomaly detection: A survey. ACM Computing Surveys (CSUR) 2009, 41(3):15.
13. Chatzis S.P., Korkinof D., Demiris Y. The one-hidden layer non-parametric Bayesian kernel machine. 23rd IEEE International conference on tools with artificial intelligence (ICTAI) 2011, 825-831. IEEE.
14. Chuang C.C., Jeng J.T. CPBUM neural networks for modeling with outliers and noise. Applied Soft Computing 2007, 7(3):957-967.
15. Chuang C.C., Jeng J.T., Lin P.T. Annealing robust radial basis function networks for function approximation with outliers. Neurocomputing 2004, 56:123-139.
16. Chuang C.C., Lee Z.J. Hybrid robust support vector machines for regression with outliers. Applied Soft Computing 2011, 11(1):64-72.
17. Chuang C.C., Su S.F., Chen S.S. Robust TSK fuzzy modeling for function approximation with outliers. IEEE Transactions on Fuzzy Systems 2001, 9(6):810-821.
18. Chuang C.C., Su S.F., Jeng J.T., Hsiao C.C. Robust support vector regression networks for function approximation with outliers. IEEE Transactions on Neural Networks 2002, 13(6):1322-1330.
19. Connor J.T., Martin R.D., Atlas L.E. Recurrent neural networks and robust time series prediction. IEEE Transactions on Neural Networks 1994, 5(2):240-254.
20. Doucet A., De Freitas N., Gordon N.J. Sequential Monte Carlo methods in practice 2001, Springer-Verlag, New York.
21. Durović Ž.M., Kovačević B.D. Robust estimation with unknown noise statistics. IEEE Transactions on Automatic Control 1999, 44(6):1292-1296.
22. Fu Y.Y., Wu C.J., Jeng J.T., Ko C.N. ARFNNs with SVR for prediction of chaotic time series with outliers. Expert Systems with Applications 2010, 37(6):4441-4451.
23. Gu Y., Liu J., Chen Y., Jiang X., Yu H. TOSELM: Timeliness online sequential extreme learning machine. Neurocomputing 2014, 128:119-127.
24. Guo L., Hao J.H., Liu M. An incremental extreme learning machine for online sequential learning problems. Neurocomputing 2014, 128:50-58.
25. Guvenir, H.A., & Uysal, I. (2000). Bilkent University Function Approximation Repository. (last date of access: February 10, 2014). http://funapp.cs.bilkent.edu.tr.
26. Hodge V.J., Austin J. A survey of outlier detection methodologies. Artificial Intelligence Review 2004, 22(2):85-126.
27. Huang G.B., Saratchandran P., Sundararajan N. An efficient sequential learning algorithm for growing and pruning RBF (GAP-RBF) networks. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 2004, 34(6):2284-2292.
28. Huang G.B., Saratchandran P., Sundararajan N. A generalized growing and pruning RBF (GGAP-RBF) neural network for function approximation. IEEE Transactions on Neural Networks 2005, 16(1):57-67.
29. Huang G.B., Zhu Q.Y., Siew C.K. Extreme learning machine: theory and applications. Neurocomputing 2006, 70(1):489-501.
30. Huber P.J. Robust statistics 2011, Springer, Berlin, Heidelberg.
31. Huynh H.T., Won Y. Regularized online sequential learning algorithm for single-hidden layer feedforward neural networks. Pattern Recognition Letters 2011, 32(14):1930-1935.
32. Jeng J.T., Chuang C.C., Tao C.W. Hybrid SVMR-GPR for modeling of chaotic time series systems with noise and outliers. Neurocomputing 2010, 73(10):1686-1693.
33. Ko C.N. Identification of nonlinear systems with outliers using wavelet neural networks based on annealing dynamical learning algorithm. Engineering Applications of Artificial Intelligence 2012, 25(3):533-543.
34. Lee C.C., Chiang Y.C., Shih C.Y., Tsai C.L. Noisy time series prediction using M-estimator based robust radial basis function neural networks with growing and pruning techniques. Expert Systems with Applications 2009, 36(3):4717-4724.
35. Lee C.C., Chung P.C., Tsai J.R., Chang C.I. Robust radial basis function neural networks. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 1999, 29(6):674-685.
36. Li D., Han M., Wang J. Chaotic time series prediction based on a novel robust echo state network. IEEE Transactions on Neural Networks and Learning Systems 2012, 23(5):787-799.
37. Liang N.Y., Huang G.B., Saratchandran P., Sundararajan N. A fast and accurate online sequential learning algorithm for feedforward networks. IEEE Transactions on Neural Networks 2006, 17(6):1411-1423.
38. Mahdi R.N., Rouchka E.C. Reduced HyperBF networks: Regularization by explicit complexity reduction and scaled Rprop based training. IEEE Transactions on Neural Networks 2011, 22(5):673-686.
39. Markou M., Singh S. Novelty detection: a review-part 1: statistical approaches. Signal Processing 2003, 83(12):2481-2497.
40. Markou M., Singh S. Novelty detection: a review-part 2: neural network based approaches. Signal Processing 2003, 83(12):2499-2521.
41. Miljković, Z., & Aleksendrić, D. (2009). Artificial neural networks-solved examples with theoretical background. University of Belgrade-Faculty of Mechanical Engineering, Belgrade, (in Serbian).
42. Miljković Z., Vuković N., Mitić M., Babić B. New hybrid vision-based control approach for automated guided vehicles. The International Journal of Advanced Manufacturing Technology 2013, 66(1-4):231-249.
43. Minka T.P. Expectation propagation for approximate Bayesian inference. Proceedings of the seventeenth conference on uncertainty in artificial intelligence 2001, 362-369. Morgan Kaufmann Publishers Inc.
44. Nikolaev N.Y., de Menezes L.M. Sequential Bayesian kernel modelling with non-Gaussian noise. Neural Networks 2008, 21(1):36-47.
45. Nishida K., Yamauchi K., Omori T. An online learning algorithm with dimension selection using minimal hyper basis function networks. Systems and Computers in Japan 2006, 37(11):11-21.
46. Pernía-Espinoza A.V., Ordieres-Meré J.B., Martínez-de-Pisón F.J., González-Marcos A. TAO-robust backpropagation learning algorithm. Neural Networks 2005, 18(2):191-204.
47. Poggio, T., & Girosi, F. (1989). A theory of networks for approximation and learning. A. I. Memo 1140, MIT.
48. Poggio T., Girosi F. Networks for approximation and learning. Proceedings of the IEEE 1990, 1481-1497.
49. Rusiecki A. Robust LTS backpropagation learning algorithm. Computational and ambient intelligence 2007, 102-109. Springer, Berlin, Heidelberg.
50. Rusiecki A. Robust learning algorithm based on LTA estimator. Neurocomputing 2013, 624-632. Special Issue: Image Feature Detection and Description.
51. Sarkka S., Nummenmaa A. Recursive noise adaptive Kalman filtering by variational Bayesian approximations. IEEE Transactions on Automatic Control 2009, 54(3):596-600.
52. Schick I.C., Mitter S.K. Robust recursive estimation in the presence of heavy-tailed observation noise. The Annals of Statistics 1994, 22(2):1045-1080.
53. Simon D. Training radial basis neural networks with the extended Kalman filter. Neurocomputing 2002, 48(1):455-475.
54. Soria-Olivas E., Gomez-Sanchis J., Jarman I.H., Vila-Frances J., Martinez M., Magdalena J.R., et al. BELM: Bayesian extreme learning machine. IEEE Transactions on Neural Networks 2011, 22(3):505-509.
55. Stanković S.S., Kovačević B.D. Analysis of robust stochastic approximation algorithms for process identification. Automatica 1986, 22(4):483-488.
56. Ting J.A., Theodorou E., Schaal S. A Kalman filter for robust outlier detection. IEEE/RSJ international conference on intelligent robots and systems, 2007 2007, 1514-1519. IEEE.
57. Tzikas D.G., Likas C.L., Galatsanos N.P. The variational approximation for Bayesian inference. IEEE Signal Processing Magazine 2008, 25(6):131-146.
58. Vuković N. Machine learning of intelligent mobile robot based on artificial neural networks 2012, (Doctoral dissertation), University of Belgrade-Faculty of Mechanical Engineering (in Serbian). 10.2298/BG20120928VUKOVIC.
59. Vuković N., Miljković Z. A growing and pruning sequential learning algorithm of hyper basis function neural network for function approximation. Neural Networks 2013, 46:210-226.
60. Yang Y.K., Sun T.Y., Huo C.L., Yu Y.H., Liu C.C., Tsai C.H. A novel self-constructing radial basis function neural-fuzzy system. Applied Soft Computing 2013, 13(5):2390-2404.
61. Yohai V.J., Zamar R.H. High breakdown-point estimates of regression by means of the minimization of an efficient scale. Journal of the American Statistical Association 1988, 83(402):406-413.
62. Zhu J., Hoi S., Lyu M.T. Robust regularized kernel regression. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 2008, 38(6):1639-1644.