Derivative-free optimization and neural networks for robust regression

Optimization

Volumn 61, Issue 12, 2012, Pages 1467-1490

  Beliakov, Gleb ^a   Kelarev, Andrei ^a   Yearwood, John ^b  

^a DEAKIN UNIVERSITY   (Australia)

^b UNIVERSITY OF BALLARAT   (Australia)

Author keywords

global optimization; least trimmed squares; neural networks; non smooth optimization; robust regression

Indexed keywords

EID: 84868710324     PISSN: 02331934     EISSN: 10294945     Source Type: Journal    
DOI: 10.1080/02331934.2012.674946     Document Type: Article

References (61)

1. Bagirov, A. 2002. A method for minimization of quasidifferentiable functions. Optim. Methods Software, 17: 31-60.
2. G. Beliakov, Parallel calculation of the median and order statistics on GPUs with application to robust regression, Tech. Rep., Deakin University, arxiv:1104.2732, 2011
3. Beliakov, G, Johnstone, M and Nahavandi, S. 2012. Computing of high breakdown regression estimators without sorting on graphics processing units. Computing, doi:10.1007/s00607-011-0183-7
4. Beliakov, G and Kelarev, A. 2012. Global non-smooth optimization in robust multivariate regression. Optim. Methods Software, doi:10.1080/10556788.2011.614609
5. G. Beliakov, A. Kelarev, and J. Yearwood, Robust artificial neural networks and outlier detection, Tech. Rep. Deakin University, arxiv:1110.0169, 2011
6. Beliakov, G and Ugon, J. 2007. Implementation of novel methods of global and non-smooth optimization: GANSO programming library. Optimization, 56: 543-546.
7. T. Bernholt, Robust estimators are hard to compute. Tech. Rep. University of Dortmund, 2005. Available athttp://www.statistik.tu-dortmund.de/fileadmin/userupload/Lehrstuehle/MSind/SFB475/2005/tr52-05.pdf
8. Cederman, D and Tsigas, P. 2009. GPU-Quicksort: A practical quicksort algorithm for graphics processors. ACM J. Exp. Algor., 14: 1.4.1-1.4.24.
9. Cerioli, A. 2010. Multivariate outlier detection with high-breakdown estimators. J. Amer. Stat. Assoc., 105: 147-156.
10. Chen, DS and Jain, RC. 1994. A robust back-propagation learning algorithm for function approximation. IEEE Trans. Neural Networks, 5: 467-479.
11. Chuang, C-C, Jeng, J-T and Lin, P-T. 2004. Annealing robust radial basis function networks for function approximation with outliers. Neurocomputing, 56: 123-139.
12. Chuang, CC, Su, SF and Hsiao, CC. 2000. The annealing robust backpropagation (ARBP) learning algorithm. IEEE Trans. Neural Networks, 11: 1067-1077.
13. Chuong, CC and Jeng, JT. 2007. CPBUM neural network for modeling with outliers and noise. Appl. Soft Comput., 7: 957-967.
14. El-Melegy, M, Essai, M and Ali, A. 2009. "Robust training of artificial feedforward neural networks". In Foundations of Computational Intelligence, Volume 201 of Studies in Computational Intelligence, Edited by: Hassanien, A., Abraham, A., Vasilakos, A. and Pedrycz, W. 217-242. Berlin, Heidelberg: Springer.
15. Gervini, D and Yohai, VJ. 2002. A class of robust and fully efficient regression estimators. Ann. Stat., 30: 583-616.
16. Hadi, AS and Luceño, A. 1997. Maximum trimmed likelihood estimators: A unified approach, examples, and algorithms. Comput. Stat. Data Anal., 25: 251-272.
17. Hagan, MT and Menhaj, MB. 1994. Training feedforward networks with the Marquardt algorithm. IEEE Trans. Neural Networks, 5: 989-993.
18. Hampel, FR. 1971. A general qualitative definition of robustness. Ann. Math. Stat., 42: 1887-1896.
19. Hawkins, DM and Olive, DJ. 1999. Applications and algorithms for least trimmed sum of absolute deviations regression. Comput. Stat. Data Anal., 32: 119-134.
20. J. Hoberock and N. Bell, Thrust: A parallel template library, 2010. Available athttp://www.meganewtons.com/
21. Hornik, K, Stinchcombe, M and White, H. 1989. Multilayer feedforward networks are universal approximators. Neural Networks, 2: 359-366.
22. Huber, PJ. 2003. Robust Statistics, New York: Wiley.
23. Igel, C and Hüsken, M. 2003. Empirical evaluation of the improved Rprop learning algorithms. Neurocomputing, 50: 105-123.
24. Ilonen, J, Kamarainen, J-K and Lampinen, J. 2003. Differential evolution training algorithm for feed-forward neural networks. Neural Process. Lett., 17: 93-105.
25. J.-T. Jeng, C-T. Chuang, and C.-C. Chuang, Least trimmed squares based CPBUM neural networks. International Conference on System Science and Engineering, IEEE, 2011, pp. 187-192
26. Jeng, JT and Lee, TT. 2000. Control of magnetic bearing systems via the Chebyshev polynomial-based unified model (CPBUM) neural network. IEEE Trans. Man Cybernetics, Part B: Cybernet., 30: 85-92.
27. Lee, C-C, Chung, P-C, Tsai, J-R and Chang, C-I. 1999. Robust radial basis function neural networks. IEEE Trans. Syst. Man Cybernet. Part B Cybernet., 29: 674-685.
28. Liano, K. 1996. Robust error measure for supervised neural network learning with outliers. IEEE Trans. Neural Networks, 7: 246-250.
29. Makela, MM and Neittaanmaki, P. 1992. Nonsmooth Optimization: Analysis and Algorithms With Applications to Optimal Control, River Edge, NJ: World Scientific.
30. Mammadov, MA, Rubinov, A and Yearwood, J. 2005. "Dynamical systems described by relational elasticities with applications to global optimization". In Continuous Optimisation: Current Trends and Modern Applications, Edited by: Rubinov, A. and Jeyakumar, V. 365-385. New York: Springer.
31. Mammadov, M, Rubinov, A and Yearwood, J. 2007. The study of drug-reaction relationships using global optimization techniques. Optim. Methods Software, 22: 99-126.
32. Maronna, R, Martin, R and Yohai, V. 2006. Robust Statistics: Theory and Methods, New York: Wiley.
33. Masters, T. 1995. Advanced Algorithms for Neural Networks: A C++ Sourcebook, New York: Wiley.
34. Moré, JJ and Wild, SM. 2009. Benchmarking derivative-free optimization algorithms. SIAM J. Optim., 20: 172-191.
35. Nelder, JA and Mead, R. 1965. A simplex method for function minimization. Comput. J., 7: 308-313.
36. Nguyen, TD and Welsch, W. 2010. Outlier detection and least trimmed squares approximation using semi-definite programming. Comput. Stat. Data Anal., 12: 3212-3226.
37. Nunkesser, R and Morell, O. 2010. An evolutionary algorithm for robust regression. Comput. Stat. Data Anal., 54: 3242-3248.
38. NVIDIA. Available athttp://developer.nvidia.com/object/cuda_4_0_rc_downloads.html
39. NVIDIA. Tesla datasheet. Available athttp://www.nvidia.com/docs/io/43395/nv_ds_tesla_psc_us_nov08_lowres.pdf
40. NVIDIA. Available athttp://developer.download.nvidia.com/compute/cuda/1_1/website/dataparallelalgorithms.html
41. Pati, YC and Krishnaprasad, PS. 1993. Analysis and synthesis of feedforward neural networks using discrete affine wavelet transformations. IEEE Trans. Neural Networks, 4: 72-85.
42. Powell, MJD. 2006. "The NEWUOA software for unconstrained optimization without derivatives". In in Large-Scale Nonlinear Optimization, Nonconvex Optimization and its Applications, 255-297. New York: Springer.
43. L. Prechelt, PROBEN 1 - A set of benchmarks and benchmarking rules for neural network training algorithms. Tech. Rep. 21/94, Universität Karlsruhe, D-76128 Jarlsruhem Germany, September 1994. Available athttp://digbib.ubka.unikarlsruhe.de/eva/ira/1994/21
44. L. Prechelt, PROBEN 1-A standardized benchmark collection for neural network algorithms, 1994. Available from ftp://ftp.ira.uka.de/pub/neuron/proben1.tar.gz
45. Prechelt, L. 1998. Automatic early stopping using cross validation: Quantifying the criteria. Neural Networks, 11: 761-767.
46. Press, AH, Teukolsky, SA, Vetterling, WT and Flannery, BP. 2002. Numerical Recipes in C: The Art of Scientific Computing, New York: Cambridge University Press.
47. R software, 2012. Available athttp://www.r-project.org/
48. Rousseeuw, PJ. 1984. Least median of squares regression. J. Amer. Stat. Assoc., 79: 871-880.
49. Rousseeuw, PJ and Leroy, AM. 2003. Robust Regression and Outlier Detection, New York: Wiley.
50. Rousseeuw, PJ and Van Driessen, K. 2006. Computing LTS regression for large data sets. Data Mining Knowledge Discov., 12: 29-45.
51. Rusiecki, A. 2007. "Robust LTS backpropagation learning algorithm". In Computational and Ambient Intelligence, Volume 4507 of Lecture Notes in Computer Science, Edited by: Sandoval, F., Prieto, A., Cabestany, J. and Graãa, M. Vol. 4507, 102-109. Heidelberg: Springer, Berlin.
52. Rusiecki, A. 2008. "Robust MCD-based backpropagation learning algorithm". In Artificial Intelligence and Soft Computing, ICAISC 2008, Volume 5097 of Lecture Notes in Computer Science, Edited by: Rutkowski, L., Tadeusiewicz, R., Zadeh, L. and Zurada, J. 154-163. Heidelberg: Springer, Berlin.
53. Schyns, M, Haesbroeck, G and Critchley, F. 2010. RelaxMCD: Smooth optimisation for the minimum covariance determinant estimator. Comput. Stat. Data Anal., 54: 843-857.
54. Sengupta, S, Harris, M, Zhang, Y and Owens, JD. 2007. Scan primitives for GPU computing. Graphics hardware, ACM, San Diego, CA,: 97-106.
55. Shieh, H-L, Yang, Y-K, Chang, P-L and Jeng, J-T. 2009. Robust neural-fuzzy method for function approximation. Expert Syst. Appl., 36: 6903-6913.
56. Sintorn, E and Assarsson, U. 2008. Fast parallel GPU-sorting using a hybrid algorithm. J. Parallel Distributed Comput., 68: 1381-1388.
57. Smola, AJ and Schölkopf, B. 1998. From regularzation operators to support vector kernels. Neural Inf. Process. Syst., 10: 343-349.
58. Stromberg, AJ, Hossjer, O and Hawkins, DM. 2000. The least trimmed differences regression estimator and alternatives. J. Amer. Stat. Assoc., 95: 853-864.
59. Stromberg, AJ and Ruppert, D. 1992. Breakdown in nonlinear regression. J. Amer. Stat. Assoc., 87: 991-997.
60. Tomandl, D and Schober, A. 2001. A modified general regression neural network (MGRNN) with new, efficient training algorithms as a robust black box-tool for data analysis. Neural Networks, 14: 1023-1034.
61. Zukerman, M, Mammadov, M, Tan, L, Ouveysi, I and Lachlan, AL. 2008. To be fair or efficient or a bit of both. Comput. Oper. Res., 35: 3787-3806.