Robust and fast learning of sparse codes with stochastic gradient descent

IEEE Journal on Selected Topics in Signal Processing

Volumn 5, Issue 5, 2011, Pages 1048-1060

  Labusch, Kai ^a   Barth, Erhardt ^a   Martinetz, Thomas ^a  

^a UNIVERSITY OF LÜBECK   (Germany)

Author keywords

Dictionary learning; K SVD; matching pursuit; Method of Optimal Directions (MOD); neural gas; sparse coding

Indexed keywords

DICTIONARY LEARNING; K-SVD; MATCHING PURSUIT; NEURAL GAS; OPTIMAL DIRECTION; SPARSE CODING;

ALGORITHMS; OPTIMIZATION; STOCHASTIC SYSTEMS; TIME VARYING NETWORKS;

IMAGE MATCHING;

EID: 80051734077     PISSN: 19324553     EISSN: None     Source Type: Journal    
DOI: 10.1109/JSTSP.2011.2149496     Document Type: Article

References (22)

1. B. A. Olshausen and D. J. Field, "Emergence of simple-cell receptive field properties by learning a sparse code for natural images," Nature, no. 381, pp. 607-609, 1996. (Pubitemid 26177476)
2. K. Labusch, E. Barth, and T. Martinetz, "Simple method for highperformance digit recognition based on sparse coding," IEEE Trans. Neural Netw., vol. 19, no. 11, pp. 1985-1989, Nov. 2008.
3. D. Donoho and X. Huo, "Uncertainty principles and ideal atomic decomposition," IEEE Trans. Inf. Theory, vol. 47, no. 7, pp. 2845-2862, Nov. 1999.
4. S. S. Chen, D. L. Donoho, and M. A. Saunders, "Atomic decomposition by basis pursuit," SIAM Rev. vol. 43, no. 1, pp. 129-159, 2001 [Online]. Available: http://dx.doi.org/10.1137/S003614450037906X (Pubitemid 32406896)
5. M. Elad and A. Bruckstein, "A generalized uncertainty principle and sparse representation in pairs of bases," IEEE Trans. Inf. Theory, vol. 48, no. 9, pp. 2558-2567, Sep. 2002. (Pubitemid 35032680)
6. D. Donoho and M. Elad, "Optimally sparse representation in general (non-orthogonal) dictionaries via l1 minimization," Proc. Nat. Acad. Sci., vol. 100, pp. 2197-2202, 2003. (Pubitemid 36297476)
7. R. Gribonval and M. Nielsen, "Sparse decompositions in unions of bases," IEEE Trans. Inf. Theory, vol. 49, no. 12, pp. 3320-3325, Dec. 2003.
8. J. A. Tropp, "Greed is good: Algorithmic results for sparse approximation," IEEE Trans. Inf. Theory, vol. 50, no. 10, pp. 2231-2242, Oct. 2004.
9. E. Candes and T. Tao, "Decoding by linear programming," IEEE Trans. Inf. Theory, vol. 51, no. 12, pp. 4203-4215, Dec. 2005. (Pubitemid 41800353)
10. D. L. Donoho, M. Elad, and V. N. Temlyakov, "Stable recovery of sparse overcomplete representations in the presence of noise," IEEE Trans. Inf. Theory, vol. 52, no. 1, pp. 6-18, Jan. 2006. (Pubitemid 43263116)
11. E. Candes, J. Romberg, and T. Tao, "Stable signal recovery from incomplete and inaccurate measurements," Commun. Pure Appl. Math., vol. 59, pp. 1207-1223, 2006. (Pubitemid 43988295)
12. A. M. Bruckstein, D. Donoho, and M. Elad, "From sparse solutions of systems of equations to sparse modeling of signals and images," SIAM REVIEW, vol. 51, no. 1, pp. 34-81, 2009.
13. K. Engan, S. O. Aase, and J. Hakon Husoy, "Method of optimal directions for frame design," in Proc. Int. Conf. Acoust., Speech, Signal Process., (ICASSP'99), Washington, DC, 1999, pp. 2443-2446.
14. M. Aharon, M. Elad, and A. Bruckstein, "K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation," IEEE Trans. Signal Process., vol. 54, no. 11, pp. 4311-4322, Dec. 2006. (Pubitemid 44637761)
15. K. Labusch, E. Barth, and T. Martinetz, "Sparse coding neural gas: Learning of overcomplete data representations," Neurocomputing, vol. 72, no. 7-9, pp. 1547-1555, 2009.
16. K. Labusch, E. Barth, and T. Martinetz, "Bag of pursuits and neural gas for improved sparse coding," in Proc. 19th Int. Conf. Comput. Statist., G. Saporta, Ed., 2010, pp. 327-336.
17. G. Davis, S. Mallat, and M. Avellaneda, "Greedy adaptive approximation," J. Constr. Approx., vol. 13, pp. 57-89, 1997.
18. S. Mallat and Z. Zhang, "Matching pursuits with time-frequency dictionaries," IEEE Trans. Signal Process., vol. 41, no. 12, pp. 3397-3415, Dec. 1993.
19. Y. Pati, R. Rezaiifar, and P. Krishnaprasad, "Orthogonal matching pursuit: Recursive function approximation with applications towavelet decomposition," in Proc. 27th Annu. Asilomar Conf. Signals, Syst., Nov. 1993.
20. L. Rebollo-Neira and D. Lowe, "Optimized orthogonal matching pursuit approach," IEEE Signal Process. Lett., vol. 9, no. 4, pp. 137-140, Apr. 2002. (Pubitemid 34711502)
21. T. Martinetz and K. Schulten, "A 'Neural-Gas Network' learns topologies," Artif. Neural Netw., vol. I, pp. 397-402, 1991.
22. T. Martinetz, S. Berkovich, and K. Schulten, "'Neural-gas' network for vector quantization and its application to time-series prediction," IEEE Trans. Neural Netw., vol. 4, no. 4, pp. 558-569, Jul. 1993.