Convergence and rate analysis of neural networks for sparse approximation

IEEE Transactions on Neural Networks and Learning Systems

Volumn 23, Issue 9, 2012, Pages 1377-1389

  Balavoine, Aurèle ^a   Romberg, Justin ^a   Rozell, Christopher J ^a  

^a GEORGIA INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

Exponential convergence; global stability; locally competitive algorithm; Lyapunov function; nonsmooth objective; sparse approximation

Indexed keywords

COMPETITIVE ALGORITHMS; EXPONENTIAL CONVERGENCE; GLOBAL STABILITY; NONSMOOTH OBJECTIVE; SPARSE APPROXIMATIONS;

LIFE CYCLE; LYAPUNOV FUNCTIONS; NEURAL NETWORKS; SIGNAL PROCESSING;

APPROXIMATION ALGORITHMS;

ALGORITHM; ARTIFICIAL NEURAL NETWORK; AUTOMATED PATTERN RECOGNITION; COMPUTER SIMULATION; PROCEDURES; SAMPLE SIZE; STATISTICAL MODEL;

ALGORITHMS; COMPUTER SIMULATION; MODELS, STATISTICAL; NEURAL NETWORKS (COMPUTER); PATTERN RECOGNITION, AUTOMATED; SAMPLE SIZE;

EID: 84870977024     PISSN: 2162237X     EISSN: 21622388     Source Type: Journal    
DOI: 10.1109/TNNLS.2012.2202400     Document Type: Article

References (45)

1. A. Balavoine, C. J. Rozell, and J. Romberg, "Global convergence of the locally competitive algorithm," in Proc. IEEE Digital Signal Process. Workshop, Jan. 2011, pp. 431-436.
2. M. Elad, M. A. T. Figueiredo, and Y. Ma, "On the role of sparse and redundant representations in image processing," Proc. IEEE, vol. 98, no. 6, pp. 972-982, Jun. 2010.
3. B. A. Olshausen and D. J. Field, "Sparse coding of sensory inputs," Current Opinion Neurobiol., vol. 14, no. 4, pp. 481-487, Aug. 2004.
4. B. A. Olshausen and D. J. Field, "Emergence of simple-cell receptive field properties by learning a sparse code for natural images," Nature, vol. 381, no. 6583, pp. 607-609, Jun. 1996.
5. B. A. Olshausen and D. J. Field, "Sparse coding with an overcomplete basis set: A strategy employed by V1?" Vis. Res., vol. 37, no. 23, pp. 3311-3325, Dec. 1997.
6. E. J. Candes, J. Romberg, and T. Tao, "Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information," IEEE Trans. Inf. Theory, vol. 52, no. 2, pp. 489-509, Feb. 2006.
7. D. L. Donoho, "Compressed sensing," IEEE Trans. Inf. Theory, vol. 52, no. 4, pp. 1289-1306, Apr. 2006.
8. S.-J. Kim, K. Koh, M. Lustig, S. Boyd, and D. Gorinevsky, "An interior-point method for large-scale L1-regularized least squares," IEEE J. Sel. Topics Signal Process., vol. 1, no. 4, pp. 606-617, Dec. 2007.
9. M. A. T. Figueiredo, R. D. Nowak, and S. J.Wright, "Gradient projection for sparse reconstruction: Application to compressed sensing and other inverse problems," IEEE J. Sel. Topics Signal Process., vol. 1, no. 4, pp. 586-598, Dec. 2007.
10. I. Daubechies, M. Defrise, and C. De Mol, "An iterative thresholding algorithm for linear inverse problems with a sparsity constraint," Commun. Pure Appl. Math., vol. 57, no. 11, pp. 1413-1457, Aug. 2004.
11. A. Cichocki and R. Unbehauen, Neural Networks for Optimization and Signal Processing. New York: Wiley, 1993.
12. J. J. Hopfield, "Neural networks and physical systems with emergent collective computational abilities," Proc. Nat. Acad. Sci., vol. 79, no. 8, pp. 2554-2558, Apr. 1982.
13. C. Twigg and P. Hasler, "Configurable analog signal processing," Digital Signal Process., vol. 19, no. 6, pp. 904-922, Dec. 2009.
14. C. J. Rozell, D. H. Johnson, R. G. Baraniuk, and B. A. Olshausen, "Sparse coding via thresholding and local competition in neural circuits," Neural Comput., vol. 20, no. 10, pp. 2526-2563, Oct. 2008.
15. J. P. Hespanha, "Uniform stability of switched linear systems: Extensions of LaSalle's invariance principle," IEEE Trans. Autom. Control, vol. 49, no. 4, pp. 470-482, Apr. 2004.
16. S. Liu and J. Wang, "A simplified dual neural network for quadratic programming with its KWTA application," IEEE Trans. Neural Netw., vol. 17, no. 6, pp. 1500-1510, Nov. 2006.
17. H. Yang and T. S. Dillon, "Exponential stability and oscillation of Hopfield graded response neural network," IEEE Trans. Neural Netw., vol. 5, no. 5, pp. 719-729, Sep. 1994.
18. X.-B. Liang and J. Wang, "A recurrent neural network for nonlinear optimization with a continuously differentiable objective function and bound constraints," IEEE Trans. Neural Netw., vol. 11, no. 6, pp. 1251-1262, Nov. 2000.
19. M. Forti and A. Tesi, "Absolute stability of analytic neural networks: An approach based on finite trajectory length," IEEE Trans. Circuits Syst. I, Reg. Papers, vol. 51, no. 12, pp. 2460-2469, Dec. 2004.
20. W. Lu and J. Wang, "Convergence analysis of a class of nonsmooth gradient systems," IEEE Trans. Circuits Syst. I, Reg. Papers, vol. 55, no. 11, pp. 3514-3527, Dec. 2008.
21. Y. Xia, G. Feng, and J. Wang, "A recurrent neural network with exponential convergence for solving convex quadratic program and related linear piecewise equations," Neural Netw., vol. 17, no. 7, pp. 1003-1015, Sep. 2004.
22. L. V. Ferreira, E. Kaszkurewicz, and A. Bhaya, "Support vector classifiers via gradient systems with discontinuous righthand sides," Neural Netw., vol. 19, no. 10, pp. 1612-1623, Dec. 2006.
23. M. Forti and A. Tesi, "New conditions for global stability of neural networks with application to linear and quadratic programming problems," IEEE Trans. Circuits Syst. I, Fundam. Theory Appl., vol. 42, no. 7, pp. 354-366, Jul. 1995.
24. L. Li and L. Huang, "Dynamical behaviors of a class of recurrent neural networks with discontinuous neuron activations," Appl. Math. Model., vol. 33, no. 12, pp. 4326-4336, Dec. 2009.
25. M. Forti, P. Nistri, and M. Quincampoix, "Generalized neural network for nonsmooth nonlinear programming problems," IEEE Trans. Circuits Syst. I, Reg. Papers, vol. 51, no. 9, pp. 1741-1754, Sep. 2004.
26. M. Forti, P. Nistri, and M. Quincampoix, "Convergence of neural networks for programming problems via a nonsmooth Łojasiewicz inequality," IEEE Trans. Neural Netw., vol. 17, no. 6, pp. 1471-1486, Nov. 2006.
27. X. Xue and W. Bian, "Subgradient-based neural networks for nonsmooth convex optimization problems," IEEE Trans. Circuits Syst. I, Reg. Papers, vol. 55, no. 8, pp. 2378-2391, Sep. 2008.
28. Q. S. Liu and J. Wang, "Finite-time convergent recurrent neural network with a hard-limiting activation function for constrained optimization with piecewise-linear objective functions," IEEE Trans. Neural Netw., vol. 22, no. 4, pp. 601-613, Apr. 2011.
29. H. Lin and P. J. Antsaklis, "Stability and stabilizability of switched linear systems: A survey of recent results," IEEE Trans. Autom. Control, vol. 54, no. 2, pp. 308-322, Feb. 2009.
30. W. Maass, "On the computational power of winner-take-all," Neural Comput., vol. 12, no. 11, pp. 2519-2535, 2000.
31. B. K. Natarajan, "Sparse approximate solutions to linear systems," SIAM J. Comput., vol. 24, no. 2, pp. 227-234, 1995.
32. S. S. Chen, D. L. Donoho, and M. A. Saunders, "Atomic decomposition by basis pursuit," SIAM Rev., vol. 43, no. 1, pp. 129-159, Mar. 2001.
33. D. L. Donoho and J. Tanner, "Sparse nonnegative solutions of underdetermined linear equations by linear programming," Proc. Nat. Acad. Sci. United States Amer., vol. 102, no. 27, pp. 9446-9451, 2005.
34. D. C. Knill and R. Pouget, "The Bayesian brain: The role of uncertainty in neural coding and computation," Trends Neurosci., vol. 27, no. 12, pp. 712-719, 2004.
35. M. Rehn and F. T. Sommer, "A network that uses few active neurones to code visual input predicts the diverse shapes of cortical receptive fields," J. Comput. Neurosci., vol. 22, no. 2, pp. 135-146, Apr. 2007.
36. E. C. Smith and M. S. Lewicki, "Efficient auditory coding," Nature, vol. 439, no. 7079, pp. 978-982, Feb. 2006.
37. S. Fischer, R. Redondo, L. Perrinet, and G. Cristóbal, "Sparse approximation of images inspired from the functional architecture of the primary visual areas," EURASIP J. Adv. Signal Process., vol. 2007, no. 1, pp. 1-17, 2007.
38. L. Perrinet, "Feature detection using spikes: The greedy approach," J. Physiol. (Paris), vol. 98, nos. 4-6, pp. 530-539, Jul. 2004.
39. J. M. Bioucas-Dias and M. A. T. Figueiredo, "A new TwIST: Two-step iterative shrinkage/thresholding algorithms for image restoration," IEEE Trans. Image Process., vol. 16, no. 12, pp. 2992-3004, Dec. 2007.
40. F. H. Clarke, Optimization and Nonsmooth Analysis. Philadelphia, PA: SIAM, Jan. 1987.
41. R. A. DeCarlo, M. S. Branicky, S. Pettersson, and B. Lennartson, "Perspectives and results on the stability and stabilizability of hybrid systems," Proc. IEEE, vol. 88, no. 7, pp. 1069-1082, Jul. 2000.
42. A. Bacciotti and L. Rosier, Liapunov Functions and Stability in Control Theory. New York: Springer-Verlag, 2005.
43. J.-J. Fuchs, "On sparse representations in arbitrary redundant bases," IEEE Trans. Inf. Theory, vol. 50, no. 6, pp. 1341-1344, Jun. 2004.
44. J. Bolte, A. Daniilidis, and A. Lewis, "The Łojasiewicz inequality for nonsmooth subanalytic functions with applications to subgradient dynamical systems," SIAM J. Optim., vol. 17, no. 4, pp. 1205-1223, 2007.
45. E. J. Candes, J. K. Romberg, and T. Tao, "Stable signal recovery from incomplete and inaccurate measurements," Commun. Pure Appl. Math., vol. 59, no. 8, pp. 1207-1223, Aug. 2006.