Energy-based models for sparse overcomplete representations

Journal of Machine Learning Research

Volumn 4, Issue 7-8, 2004, Pages 1235-1260

  Teh, Yee Whye ^a   Welling, Max ^a   Osindero, Simon ^b   Hinton, Geoffrey E ^a  

^a UNIVERSITY OF TORONTO   (Canada)

^b UNIVERSITY COLLEGE LONDON   (United Kingdom)

Author keywords

Density Estimation; Independent Components Analysis; Overcomplete Representations; Sparse Representations

Indexed keywords

ALGORITHMS; BLIND SOURCE SEPARATION; ESTIMATION; FUNCTIONS; MATHEMATICAL MODELS; PROBABILITY DISTRIBUTIONS; PROBLEM SOLVING;

DENSITY ESTIMATION; OVERCOMPLETE REPRESENTATION; SPARSE REPRESENTATION;

INDEPENDENT COMPONENT ANALYSIS;

EID: 8344290493     PISSN: 15324435     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Conference Paper

References (33)

1. S. A. Abdallah and M. D. Plumbley. If edges are the independent components of natural images, what are the independent components of natural sounds? In International Conference On Independent Component Analysis and Blind Signal Separation, 2001.
2. S. Amari, A. Cichocki, and H. Yang. A new algorithm for blind signal separation. In Advances in Neural Information Processing Systems, volume 8, pages 757-763, 1996.
3. H. Attias. Independent Factor Analysis. Neural Computation, 11:803-851, 1999.
4. M. S. Bartlett, J. R. Movellan, and T. J. Sejnowski. Face recognition by independent component analysis. IEEE Transactions on Neural Networks, 2002. In Press.
5. A. J. Bell and T. J. Sejnowski. An information maximisation approach to blind separation and blind deconvolution. Neural Computation, 7:1129-1159, 1995.
6. J. F. Cardoso. Infomax and maximum likelihood for blind source separation. IEEE Signal Processing Letters, 4(4):112-114, 1997.
7. S. S. Chen, D. L. Donoho, and M. A. Saunders. Atomic decomposition by basis pursuit. SIAM Journal on Scientific Computing, 20(1):33-61, 1998.
8. P. Comon. Independent component analysis: A new concept? Signal Processing, 36:287-314, 1994.
9. S. Della Pietra, V. Della Pietra, and J. Lafferty. Inducing features of random fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(4):380-393, 1997.
10. M. Girolami. A variational method for learning overcomplete representations. Neural Computation, 13:2517-2532, 2001.
11. G. E. Hinton. Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8): 1771-1800, 2002.
12. G. E. Hinton and Y. W. Teh. Discovering multiple constraints that are frequently approximately satisfied. In Proceedings of the Seventh Conference on Uncertainty in Artificial Intelligence, pages 227-234, 2001.
13. G. E. Hinton, Y. W. Teh, M. Welling, and S. Osindero. Contrastive backpropagation. In preparation, 2004.
14. A. Hyvarinen and M. Inki. Estimating overcomplete independent component bases for image windows. Journal of Mathematical Imaging and Vision, 2002. in press.
15. A. Hyvrinen and P. O. Hoyer. A two-layer sparse coding model learns simple and complex cell receptive fields and topography from natural images. Vision Research, 41(18):2413-2423, 2001.
16. M. S. Lewicki and B. A. Olshausen. A probabilistic framework for the adaptation and comparison of image codes. Journal of the Optical Society of America A: Optics, Image Science, and Vision, 16(7): 1587-1601, 1999.
17. M. S. Lewicki and T. J. Sejnowski. Learning overcomplete representations. Neural Computation, 12:337-365, 2000.
18. D. MacKay. Maximum likelihood and covariant algorithms for independent components analysis. Available electronically at http://www.inference.phy.cam. ac.uk/mackay/abstracts/ica.html, 1996.
19. D. MacKay. Failures of the one-step learning algorithm. Available electronically at http://www.inference.phy.cam.ac.uk/mackay/abstracts/gbm.html, 2001.
20. S. G. Mallat and Z. Zhang. Matching pursuits with time-frequency dictionaries. IEEE Transactions in Signal Processing, 41(12):3397-3414, 1993.
21. R. M. Neal. Probabilistic inference using Markov chain Monte Carlo methods. Technical Report CRG-TR-93-1, University of Toronto, Department of Computer Science, 1993.
22. B. A. Olshausen and D. J. Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381:607-609, 1996.
23. B. A. Olshausen and D. J. Field. Sparse coding with an overcomplete basis set: A strategy employed by VI. Vision Research, 37:3311-3325, 1997.
24. B. A. Olshausen and K. J. Millman. Learning sparse codes with a mixture-of-gaussians prior. In Advances in Neural Information Processing Systems, volume 12, pages 841-847, 2000.
25. B. A. Pearlmutter and L. C. Parra. A context-sensitive generalization of ICA. In Proceedings of the International Conference on Neural Information Processing, 1996.
26. D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning internal representations by error propagation. In D. E. Rumelhart, J. L. McClelland, and the PDP Research Group, editors, Parallel Distributed Processing: Explorations in The Microstructure of Cognition. Volume 1: Foundations. The MIT Press, 1986.
27. O. Shriki, H. Sompolinsky, and D. D. Lee. An information maximization approach to overcomplete and recurrent representations. In Advances in Neural Information Processing Systems, pages 612-618, 2002.
28. E. P. Simoncelli, W. T. Freeman, E. H. Adelson, and D. J. Heeger. Shiftable multi-scale transforms. IEEE Transactions Information Theory, 38(2):587-607, 1992.
29. J. H. van Hateren and A. van der Schaaf. Independent component filters of natural images compared with simple cells in primary visual cortex. Proceedings of the Royal Society of London B, 265: 359-366, 1998.
30. M. Welling, G. E. Hinton, and S. Osindero. Learning sparse topographic representations with products of student-t distributions. In Advances in Neural Information Processing Systems, 2003.
31. P. J. Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE,78(10):1550-1560, 1990.
32. C. K. I. Williams and F. V. Agakov. An analysis of contrastive divergence learning in gaussian boltzmann machines. Technical Report EDI-INF-RR-0120, Institute for Adaptive and Neural Computation, University of Edinburgh, 2002.
33. S. C. Zhu, Y. N. Wu, and D. Mumford. Minimax entropy principle and its application to texture modelling. Neural Computation, 9(8), 1997.