Dictionary learning algorithms for sparse representation

Neural Computation

Volumn 15, Issue 2, 2003, Pages 349-396

  Kreutz Delgado, Kenneth ^a   Murray, Joseph F ^a   Rao, Bhaskar D ^a   Engan, Kjersti ^b   Lee, Te Won ^c   Sejnowski, Terrence J ^c  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b UNIVERSITY OF STAVANGER   (Norway)

^c HOWARD HUGHES MEDICAL INSTITUTE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; LEARNING; PHYSIOLOGY; STATISTICS;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; LEARNING; STOCHASTIC PROCESSES;

EID: 0037313218     PISSN: 08997667     EISSN: None     Source Type: Journal    
DOI: 10.1162/089976603762552951     Document Type: Article

References (58)

1. Adler, J., Rao, B., & Kreutz-Delgado, K. (1996). Comparison of basis selection methods. In Proceedings of the 30th Asilomar Conference on Signals, Systems, and Computers. Pacific Grove, CA: IEEE.
2. Attias, H. (1999). Independent factor analysis. Neural Computation, 11(4), 803-851.
3. Basilevsky, A. (1994). Statistical factor analysis and related methods: Theory and applications. New York: Wiley.
4. Bell, A., & Sejnowski, T. (1995). An information-maximization approach to blind separation and blind deconvolution. Neural Computation, 7, 1129-1159.
5. Coifman, R., & Wickerhauser, M. (1992). Entropy-based algorithms for best basis selection. IEEE Transactions on Information Theory, IT-38(2), 713-718.
6. Comon, P. (1994). Independent component analysis: A new concept? Signal Processing, 36, 287-314.
7. Deco, G., & Obradovic, D. (1996). An information-theoretic approach to neural computing. Berlin: Springer-Verlag.
8. Dhrymes, P. (1984). Mathematics for econometrics (2nd ed.). New York: Springer-Verlag.
9. Dirac, P. A. M. (1958). The principles of quantum mechanics. Oxford: Clarendon Press.
10. Donoho, D. (1994). On minimum entropy segmentation. In C. Chui, L. Montefusco, & L. Puccio (Eds.), Wavelets: Theory, algorithms, and applications (pp. 233-269). San Diego, CA: Academic Press.
11. Donoho, D. (1995). De-noising by soft-thresholding. IEEE Trans. on Information Theory, 41, 613-627.
12. Engan, K. (2000). Frame based signal representation and compression. Unpublished doctoral dissertation, Stavanger Universty College, Norway.
13. Engan, K., Rao, B., & Kreutz-Delgado, K. (1999). Frame design using FOCUSS with method of optimal directions (MOD). In Proc. NORSIG-99. Tronheim, Norway: IEEE.
14. Engan, K., Rao, B. D., & Kreutz-Delgado, K. (2000). Regularized FOCUSS for subset selection in noise. In Proc. NORSIG 2000, Sweden, June, 2000 (pp. 247-250). Kolmarden, Sweden: IEEE.
15. Field, D. (1994). What is the goal of sensory coding. Neural Computation, 6, 559-599.
16. Girolami, M. (2001). A variational method for learning sparse and overcomplete representations. Neural Computation, 13, 2517-2532.
17. Gorodnitsky, I., George, J., & Rao, B. (1995). Neuromagnetic source imaging with FOCUSS: A recursive weighted minimum norm algorithm. Journal of Electroencephalography and Clinical Neurophysiology, 95(4), 231-251.
18. Gorodnitsky, I., & Rao, B. (1997). Sparse signal reconstruction from limited data using FOCUSS: A re-weighted minimum norm algorithm. IEEE Trans. Signal Processing, 45(3), 600-616.
19. Hansen, P. C., & O'Leary, D. P. (1993). The use of the L-curve in the regularization of discrete ill-posed problems. SIAM J. Set. Comput., 14, 1487-1503.
20. Hyvärinen, A., Cristescu, R., & Oja, E. (1999). A fast algorithm for estimating overcomplete ICA bases for image windows. In Proc. Int. Joint Conf. on Neural Networks (IJCNN'99) (pp. 894-899). Washington, DC.
21. Kalouptsidis, T., & Theodoridis, S. (1993). Adaptive system identification and signal processing algorithms. Upper Saddle River, NJ: Prentice Hall.
22. Kassam, S. (1982). Signal detection in non-gaussian noise. New York: Springer-Verlag.
23. Khalil, H. (1996). Nonlinear systems (2nd ed). Upper Saddle River, NJ: Prentice Hall.
24. Kreutz-Delgado, K., & Rao, B. (1997). A general approach to sparse basis selection: Majorization, concavity, and affine Scaling (Full Report with Proofs) (Center for Information Engineering Report No. UCSD-CIE-97-7-1). San Diego: University of California. Available on-line at: http://dsp.ucsd.edu/̃kreutz.
25. Kreutz-Delgado, K., & Rao, B. (1998a). A new algorithm and entropy-like measures for sparse coding. In Proc. 5th Joint Symp. Neural Computation. La Jolla, CA.
26. Kreutz-Delgado, K., & Rao, B. (1998b). Gradient factorization-based algorithm for best-basis selection. In Proceedings of the 8th IEEE Digital Signal Processing Workshop. New York: IEEE.
27. Kreutz-Delgado, K., & Rao, B. (1998c). Measures and algorithms for best basis selection. In Proceedings of the 1998 ICASSP. New York: IEEE.
28. Kreutz-Delgado, K., & Rao, B. (1999). Sparse basis selection, ICA, and majorization: Towards a unified perspective. In Proc. 1999 ICASSP. Phoenix, AZ.
29. Kreutz-Delgado, K., Rao, B., & Engan, K. (1999). Novel algorithms for learning overcomplete Dictionaries. In Proc. 1999 Asilomar Conference. Pacific Grove, CA: IEEE.
30. Kreutz-Delgado, K., Rao, B., Engan, K., Lee, T.-W., & Sejnowski, T. (1999a). Convex/Schur-convex (CSC) log-priors and sparse coding. In Proc. 6th Joint Symposium on Neural Computation. Pasadena, CA.
31. Kreutz-Delgado, K., Rao, B., Engan, K., Lee, T.-W., & Sejnowski, T. (1999b). Learning overcomplete dictionaries: Convex/Schur-convex (CSC) log-priors, factorial codes, and independent/dependent component analysis (I/DCA). In Proc. 6th Joint Symposium on Neural Computation. Pasadena, CA.
32. Lee, T.-W., Girolami, M., & Sejnowski, T. (1999). Independent component analysis using an extended infomax algorithm for mixed sub-gaussian and super-gaussian sources. Neural Computation, 11(2), 417-441.
33. Lewicki, M. S., & Olshausen, B. A. (1999). A probabilistic framework for the adaptation and comparison of image codes. J. Opt. Soc. Am. A, 16(7), 1587-1601.
34. Lewicki, M. S., & Sejnowski, T. J. (2000). Learning overcomplete representations. Neural Computation, 12(2), 337-365.
35. Mallat, S., & Zhang, Z. (1993). Matching pursuits with time-frequency dictionaries. IEEE Trans. ASSP, 41(12), 3397-3415.
36. Marshall, A., & Olkin, I. (1979). Inequalities: Theory of majorization and its applications. San Diego, CA: Academic Press.
37. Murray, J. F., & Kreutz-Delgado, K. (2001). An improved FOCUSS-based learning algorithm for solving sparse linear inverse problems. In Conference Record of the 35th Asilomar Conference on Signals, Systems and Computers. Pacific Grove, CA: IEEE.
38. Nadal, J.-P., & Parga, N. (1994). Nonlinear neurons in the low noise limit: A factorial code maximizes information transfer. Network, 5, 565-581.
39. Olshausen, B., & Field, D. (1996). Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381, 607-609.
40. Olshausen, B., & Field, D. (1997). Sparse coding with an overcomplete basis set: A strategy employed by V1? Vision Research, 37, 3311-3325.
41. Pham, D. (1996). Blind separation of instantaneous mixture of sources via an independent component analysis. IEEE Trans. Signal Processing, 44(11), 2768-2779.
42. Popovic, Z., & Sjöstrand, J. (2001). Resolution, separation of retinal ganglion cells, and cortical magnification in humans. Vision Research, 41, 1313-1319.
43. Rao, B. (1997). Analysis and extensions of the FOCUSS algorithm. In Proceedings of the 1996 Asilomar Conference on Circuits, Systems, and Computers (Vol. 2, pp. 1218-1223). New York: IEEE.
44. Rao, B. (1998). Signal processing with the sparseness constraint. In Proc. 1998 ICASSP. New York: IEEE.
45. Rao, B. D., Engan, K., Cotter, S. F., & Kreutz-Delgado, K. (2002). Subset selection in noise based on diversity measure minimization. Manuscript submitted for publication.
46. Rao, B., & Gorodnitsky, I. (1997). Affine scaling transformation based methods for computing low complexity sparse solutions. In Proc. 1996 ICASSP (Vol. 2, pp. 1783-1786). New York: IEEE.
47. Rao, B., & Kreutz-Delgado, K. (1997). Deriving algorithms for computing sparse solutions to linear inverse problems. In J. Neyman (Ed.), Proceedings of the 1997 Asilomar Conference on Circuits, Systems, and Computers. New York: IEEE.
48. Rao, B., & Kreutz-Delgado, K. (1998a). Basis selection in the presence of noise. In Proceedings of the 1998 Asilomar Conference. New York: IEEE.
49. Rao, B., & Kreutz-Delgado, K. (1998b). Sparse solutions to linear inverse problems with multiple measurement vectors. In Proceedings of the 8th IEEE Digital Signal Processing Workshop. New York: IEEE.
50. Rao, B., & Kreutz-Delgado, K. (1999). An affine scaling methodology for best basis selection. IEEE Trans. Signal Processing, 1, 187-202.
51. Roberts, S. (1998). Independent component analysis: Source assessment and separation, a Bayesian approach. IEE (Brit.) Proc. Vis. Image Signal Processing, 145(3), 149-154.
52. Rockafellar, R. (1970). Convex analysis. Princeton, NJ: Princeton University Press.
53. Ruderman, D. (1994). The statistics of natural images. Network: Computation in Neural Systems, 5, 517-548.
54. Wang, K., Lee, C., & Juang, B. (1997). Selective feature extraction via signal decomposition. IEEE Signal Processing Letters, 4(1), 8-11.
55. Watanabe, S. (1981). Pattern recognition as a quest for minimum entropy. Pattern Recognition, 13(5), 381-387.
56. Wickerhauser, M. (1994). Adapted wavelet analysis from theory to software. Wellesley, MA: A. K. Peters.
57. Zhu, S., Wu, Y., & Mumford, D. (1997). Minimax entropy principle and its application to texture modeling. Neural Computation, 9, 1627-1660.
58. Zibulevsky, M., & Pearlmutter, B. A. (2001). Blind source separation by sparse decomposition in a signal dictionary. Neural Computation, 13(4), 863-882.