Smoothing neural network for constrained non-lipschitz optimization with applications

IEEE Transactions on Neural Networks and Learning Systems

Volumn 23, Issue 3, 2012, Pages 399-411

  Bian, Wei ^a,b   Chen, Xiaojun ^b  

^a HARBIN INSTITUTE OF TECHNOLOGY   (China)

^b HONG KONG POLYTECHNIC UNIVERSITY   (Hong Kong)

Author keywords

Image and signal restoration; non Lipschitz optimization; smoothing neural network; stationary point; variable selection

Indexed keywords

ACCUMULATION POINTS; LIPSCHITZ PROPERTY; NONCONVEX FUNCTIONS; OBJECTIVE FUNCTIONS; OPTIMIZATION PROBLEMS; SIGNAL RESTORATION; STATIONARY POINTS; VARIABLE SELECTION;

BLIND SOURCE SEPARATION; DIFFERENTIAL EQUATIONS; FUNCTIONS; IMAGE RECONSTRUCTION; NEURAL NETWORKS; NUMBER THEORY;

CONSTRAINED OPTIMIZATION;

ARTIFICIAL NEURAL NETWORK; THEORETICAL MODEL;

MODELS, THEORETICAL; NEURAL NETWORKS (COMPUTER);

EID: 84865698786     PISSN: 2162237X     EISSN: 21622388     Source Type: Journal    
DOI: 10.1109/TNNLS.2011.2181867     Document Type: Article

References (49)

1. A. Cichocki and R. Unbehauen, Neural Networks for Optimization and Signal Processing. New York: Wiley, 1993.
2. N. Kalouptsidis, Signal Processing Systems, Theory and Design. New York: Wiley, 1997.
3. Z. G. Hou, M. M. Gupta, P. N. Nikiforuk, M. Tan, and L. Cheng, "A recurrent neural network for hierarchical control of interconnected dynamic systems," IEEE Trans. Neural Netw., vol. 18, no. 2, pp. 466-481, Mar. 2007.
4. Y. S. Xia, J. Wang, and L. M. Fok, "Grasping-force optimization for multifingered robotic hands using a recurrent neural network," IEEE Trans. Robot. Autom., vol. 20, no. 3, pp. 549-554, Jun. 2004.
5. X. B. Gao and L. Z. Liao, "A new projection-based neural network for constrained variational inequalities," IEEE Trans. Neural Netw., vol. 20, no. 3, pp. 373-388, Mar. 2009.
6. Y. S. Xia, G. Feng, and J. Wang, "A novel recurrent neural network for solving nonlinear optimization problems with inequality constraints," IEEE Trans. Neural Netw., vol. 19, no. 8, pp. 1340-1353, Aug. 2008.
7. J. L. Liang, Z. D. Wang, and X. H. Liu, "State estimation for coupled uncertain stochastic networks with missing measurements and timevarying delays: The discrete-time case," IEEE Trans. Neural Netw., vol. 20, no. 5, pp. 781-793, May 2009.
8. B. Y. Zhang, S. Y. Xu, G. D. Zong, and Y. Zou, "Delay-dependent exponential stability for uncertain stochastic Hopfield neural networks with time-varying delays," IEEE Trans. Circuits Syst. I, vol. 56, no. 6, pp. 1241-1247, Jun. 2009.
9. D. W. Tank and J. J. Hopfield, "Simple 'neural' optimization networks: An A/D converter, signal decision circuit, and a linear programming circuit," IEEE Trans. Circuits Syst., vol. 33, no. 5, pp. 533-541, May 1986.
10. M. P. Kennedy and L. O. Chua, "Neural networks for nonlinear programming," IEEE Trans. Circuit Syst., vol. 35, no. 5, pp. 554-562, May 1988.
11. M. Forti, P. Nistri, and M. Quincampoix, "Generalized neural network for nonsmooth nonlinear programming problems," IEEE Trans. Circuits Syst. I, vol. 51, no. 9, pp. 1741-1754, Sep. 2004.
12. X. P. Xue and W. Bian, "Subgradient-based neural networks for nonsmooth convex optimization problems," IEEE Trans. Circuits Syst. I, vol. 55, no. 8, pp. 2378-2391, Sep. 2008.
13. G. Li, S. Song, and C. Wu, "Generalized gradient projection neural networks for nonsmooth optimization problems," Sci. Chin. Inf. Sci., vol. 53, no. 5, pp. 990-1005, 2010.
14. Q. S. Liu and J. Wang, "Finite-time convergent recurrent neural network with a hard-limiting activation function for constrained optimization with piecewise-linear objective functions," IEEE Trans. Neural Netw., vol. 22, no. 4, pp. 601-613, Apr. 2011.
15. L. Cheng, Z. G. Hou, Y. Z. Lin, M. Tan, W. C. Zhang, and F. X. Wu, "Recurrent neural network for non-smooth convex optimization problems with application to the identification of genetic regulatory networks," IEEE Trans. Neural Netw., vol. 22, no. 5, pp. 714-726, May 2011.
16. M. Forti, P. Nistri, and M. Quincampoix, "Convergence of neural networks for programming problems via a nonsmooth Łojasiewicz inequality," IEEE Trans. Neural Netw., vol. 17, no. 6, pp. 1471-1486, Nov. 2006.
17. W. Bian and X. P. Xue, "Subgradient-based neural networks for nonsmooth nonconvex optimization problems," IEEE Trans. Neural Netw., vol. 20, no. 6, pp. 1024-1038, Jun. 2009.
18. W. L. Lu and J. Wang, "Convergence analysis of a class of nonsmooth gradient systems," IEEE Trans. Circuits Syst. I, vol. 55, no. 11, pp. 3514-3527, Dec. 2008.
19. M. Forti, M. Grazzini, P. Nistri, and L. Pancioni, "Generalized Lyapunov approach for convergence of neural networks with discontinuous or non-Lipschitz activations," Phys. D, vol. 214, no. 1, pp. 88-99, Feb. 2006.
20. M. Forti, "M-matrices and global convergence of discontinuous neural networks," Int. J. Circuit Theory Appl., vol. 35, no. 2, pp. 105-130, Mar.-Apr. 2007.
21. X. Chen, M. K. Ng, and C. Zhang. (2010). Nonconvex-p-Regularization and Box Constrained Model for Image Restoration [Online]. Available: http://www.polyu.edu.hk/ama/staff/xjchen/nonconvex10Sept2011.pdf
22. M. Nikolova, M. K. Ng, S. Q. Zhang, and W. K. Ching, "Efficient reconstruction of piecewise constant images using nonsmooth nonconvex minimization," SIAM J. Imag. Sci., vol. 1, no. 1, pp. 2-25, Mar. 2008.
23. N. Meinshausen and B. Yu, "Lasso-type recovery of sparse representations for high-dimensional data," Ann. Stat., vol. 37, no. 1, pp. 246-270, 2009.
24. R. Tibshirani, "Regression shrinkage and selection via the lasso," J. Royal Stat. Soc. Ser. B, vol. 58, no. 1, pp. 267-288, 1996.
25. R. Saab, O. Yilmaz, M. J. Mckeown, and R. Abugharbieh, "Underdetermined anechoic blind source separation via-q-basis-pursuit with q-1," IEEE Trans. Signal Process., vol. 55, no. 8, pp. 4004-4017, Aug. 2007.
26. V. Saligrama and M. Q. Zhao, "Thresholded basis pursuit: LP algorithm for order-wise optimal support recovery for sparse and approximately sparse signals from noisy random measurements," IEEE Trans. Inf. Theory, vol. 57, no. 3, pp. 1567-1586, Mar. 2011.
27. D. L. Donoho, "Neighborly polytopes and sparse solutions of underdetermined linear equations," Dept. Stat., Standford University, Standford, CA, Tech. Rep., 2005.
28. L. B. Montefusco, D. Lazzaro, and S. Papi, "Nonlinear filtering for sparse signal recovery from incomplete measurements," IEEE Trans. Signal Process., vol. 57, no. 7, pp. 2494-2502, Jul. 2009.
29. Y. C. Eldar and M. Mishali, "Robust recovery of signals from a structured union of subspaces," IEEE Trans. Inf. Theory, vol. 55, no. 11, pp. 5302-5316, Nov. 2009.
30. D. L. Donoho, "Compressed sensing," IEEE Trans. Inf. Theory, vol. 52, no. 4, pp. 1289-1306, Apr. 2006.
31. R. Chartrand and V. Staneva, "Restricted isometry properties and nonconvex compressive sensing," Inverse Probl., vol. 24, no. 3, pp. 1-14, 2008.
32. X. Chen, F. Xu, and Y. Ye, "Lower bound theory of nonzero entries in solutions of l2-lp minimization," SIAM J. Sci. Comput., vol. 32, no. 5, pp. 2832-2852, 2010.
33. X. Chen, D. Ge, Z. Wang, and Y. Ye. (2011). Complexity of Unconstrained L2-L p Minimization [Online]. Available: http://www.tju.edu. cn/science/xydt/tztg/201109/P020110908302116195306.pdf
34. P. Huber, Robust Estimation. New York: Wiley, 1981.
35. J. Q. Fan and H. Peng, "Nonconcave penalized likelihood with a diverging number of parameters," Ann. Stat., vol. 32, no. 3, pp. 928-961, 2004.
36. A. Auslender, "How to deal with the unbounded in optimization: Theory and algorithms," Math. Program., vol. 79, nos. 1-3, pp. 3-18, 1997.
37. W. Bian and X. Chen. (2011). Smoothing Dynamical Approach to Nonsmooth, Nonconvex Constrained Minimization [Online]. Available: http://www.polyu.edu.hk/ ama/staff/xjchen/MPB2012.pdf
38. J. Kreimer and R. Y. Rubinstein, "Nondifferentiable optimization via smooth approximation: General analytical approach," Ann. Oper. Res., vol. 39, no. 1, pp. 97-119, 1992.
39. R. T. Rockafellar and R. J.-B. Wets, Variational Analysis. Berlin, Germany: Springer-Verlag, 1998.
40. F. H. Clarke, Optimization and Nonsmooth Analysis. New York: Wiley, 1983.
41. J.-P. Aubin and A. Cellina, Differential Inclusion: Set-Valued Maps and Viability Theory. New York: Springer-Verlag, 1984.
42. D. Kinderlehrer and G. Stampacchia, An Introduction to Variational Inequalities and Their Applications. New York: SIAM, 1987.
43. Z. H. Yuan, L. H. Huang, D. W. Hu, and B. W. Liu, "Convergence of nonautonomous Cohen-Grossberg-type neural networks with variable delays," IEEE Trans. Neural Netw., vol. 19, no. 1, pp. 140-147, Jan. 2008.
44. B. Y. Zhang, S. Y. Xu, G. D. Zong, and Y. Zou, "Delay-dependent exponential stability for uncertain stochastic Hopfield neural networks with time-varying delays," IEEE Trans. Circuits Syst. I, vol. 56, no. 6, pp. 1241-1247, Jun. 2009.
45. D. Betounes, Differential Equations: Theory and Applications. New York: Springer-Verlag, 2009.
46. Y. Xiang, S. K. Ng, and V. K. Nguyen, "Blind separation of mutually correlated sources using precoders," IEEE Trans. Neural Netw., vol. 21, no. 1, pp. 82-90, Jan. 2010.
47. C. Blake and C. Merz. (1998). Repository of Machine Learning Databases[DB/OL]. Dept. Inf. Comput. Sci., Univ. California, Irvine, CA [Online]. Available: http://archive.ics.uci.edu/ml/
48. M. L. Overton and R. S. Womersley, "Optimality conditions and duality theory for minimizing sums of the largest eigenvalues of symmetric matrices," Math. Program., vol. 62, nos. 1-3, pp. 321-357, 1993.
49. X. Chen, R. S. Womersley, and J. Ye, "Minimizing the condition number of a Gram matrix," SIAM J. Optim., vol. 21, no. 1, pp. 127-148, 2011.