Expectation truncation and the benefits of preselection in training generative models

Journal of Machine Learning Research

Volumn 11, Issue , 2010, Pages 2855-2900

  Lúcke, Jörg ^a   Eggert, Julian ^b  

^a FRANKFURT INSTITUTE FOR ADVANCED STUDIES   (Germany)

^b HONDA RESEARCH INSTITUTE EUROPE GMBH   (Germany)

Author keywords

Component extraction; Deterministic approximations; Generative models; Maximum likelihood; Multiple cause models; Variational EM

Indexed keywords

COMPONENT EXTRACTION; DETERMINISTIC APPROXIMATION; GENERATIVE MODEL; MULTIPLE-CAUSE MODELS; VARIATIONAL EM;

BLIND SOURCE SEPARATION; COST REDUCTION; FACTORIZATION; MATRIX ALGEBRA;

MAXIMUM LIKELIHOOD;

EID: 78349238081     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (40)

1. P. Berkes, R. E. Turner, and M. Sahani. A structured model of video reproduces primary visual cortical organisation. PLoS Computational Biology, 5 (9): el000495, 2009.
2. C. M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
3. N. J. Butko and J. Triesch. Learning sensory representations with intrinsic plasticity. Neurocomputing, 70(7-9):1130-1138, 2007.
4. D. Charles, C. Fyfe, D. MacDonald, and J. Koetsier. Unsupervised neural networks for the identification of minimum overcomplete basis in visual data. Neurocomputing, 47(1-4):119-143, 2002.
5. P. Comon. Independent component analysis, a new concept? Signal Processing, 36(3):287-314, 1994.
6. P. Dayan and R. S. Zemel. Competition and multiple cause models. Neural Computation, 7:565-579, 1995.
7. A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the EM algorithm (with discussion). Journal of the Royal Statistical Society B, 39:1-38, 1977.
8. P. Földiák. Forming sparse representations by local anti-Hebbian learning. Biological Cybernetics, 64:165-170, 1990.
9. M. Fromer and A. Globerson. An LP View of the M-best MAP problem. In Advances in Neural Information Processing Systems 22, pages 567-575, 2009.
10. G. E. Hinton, P. Dayan, B. J. Frey, and R. M. Neal. The 'wake-sleep' algorithm for unsupervised neural networks. Science, 268:1158-1161, 1995.
11. S. Hochreiter and J. Schmidhuber. Feature extraction through LOCOCODE. Neural Computation, 11:679-714, 1999.
12. P. O. Hoyer. Non-negative sparse coding. a Neural Networks for Signal Processing XII: Proceedings of the IEEE Workshop on Neural Networks for Signal Processing, pages 557-565. IEEE, 2002.
13. A. Hyvarinen, J. Hurri, and P. O. Hoyer. Natural Image Statistics. Springer, Heidelberg London New York, 2009.
14. T. Jaakkola. Tutorial on variational approximation methods. In M. Opper and D. Saad, editors, Advanced mean field methods: theory and practice. MIT Press, 2000.
15. M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul. An introduction to variational methods for graphical models. Machine Learning, 37(2):183-233, 1999.
16. E. Körner, M. O. Gewaltig, U. Körner, A. Richter, and T. Rodemann. A model of computation in neocortical architecture. Neural Networks, 12:989-1005, 1999.
17. U. Köster and A. Hyvärinen. A two-layer ICA-like model estimated by score matching. In Proc. International Conference on Artificial Neural Networks, LNCS 4669, pages 798-807. Springer, 2007.
18. T. W. Lam and H.-F. Ting. Selecting the k largest elements with parity tests. Discrete Appl. Math., 101(1-3):187-196, 2000.
19. D. D. Lee and H. S. Seung. Learning the parts of objects by non-negative matrix factorization. Nature, 401(6755):788-91, 1999.
20. D. D. Lee and H. S. Seung. Algorithm for non-negative matrix factorization. In Advances in Neural Information Processing Systems, volume 13, 2001.
21. T. S. Lee and D. Mumford. Hierarchical Bayesian inference in the visual cortex. J Opt Soc Am A Opt Image Sci Vis, 20(7):1434-1448, 2003.
22. J. Lücke. Hierarchical self-organization of minicolumnar receptive fields. Neural Networks, 17/8-9:1377-1389, 2004. (Pubitemid 39539548)
23. J. Lücke and M. Sahani. Generalized softmax networks for non-linear component extraction. In Proc. International Conference on Artificial Neural Networks, LNCS 4668, pages 657-667. Springer, 2007.
24. J. Lücke and M. Sahani. Maximal causes for non-linear component extraction. Journal of Machine Learning Research, 9:1227-1267, 2008.
25. J. Lücke and C. von der Malsburg. Rapid processing and unsupervised learning in a model of the cortical macrocolumn. Neural Computation, 16:501-533, 2004.
26. J. Lücke, R. Turner, M. Sahani, and M. Henniges. Occlusive components analysis. In Y. Bengio, D. Schuurmans, J. Lafferty, C. K. I. Williams, and A. Culotta, editors, Advances in Neural Information Processing Systems 22, pages 1069-1077, 2009.
27. D. J. C. MacKay. Information Theory, Inference, and Learning Algorithms. Cambridge University Press, 2003. Available from http://www.inference.phy.cam. ac.uk/mackay/itila/.
28. T. P. Minka. Expectation propagation for approximate Bayesian inference. In UAI '01: Proceedings of the 17th Conference in Uncertainty in Artificial Intelligence, pages 362-369, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1-55860-800-1.
29. R. Neal and G. Hinton. A view of the EM algorithm that justifies incremental, sparse, and other variants. In M. I. Jordan, editor, Learning in Graphical Models. Kluwer, y1998.
30. B. A. Olshausen. Probabilistic Models of the Brain: Perception and Neural Function, chapter Sparse Codes and Spikes, pages 257-272. MIT Press, 2002.
31. B. A. Olshausen and D. J. Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381:607-609, 1996.
32. M. Riesenhuber and T. Poggio. Hierarchical models of object recognition in cortex. Nature Neuroscience, 211(11):1019-1025, 1999.
33. M. Sahani. Latent variable models for neural data analysis, 1999. PhD Thesis, Caltech.
34. E. Saund. A multiple cause mixture model for unsupervised learning. Neural Computation, 7:51-71, 1995.
35. M. W. Spratling. Learning image components for object recognition. Journal of Machine Learning Research, 7:793-815, 2006.
36. Y. W. Teh, M. Welling, S. Osindero, and G. E. Hinton. Energy-based models for sparse overcomplete representations. The Journal of Machine Learning Research, 4 (7), 2003.
37. N. Ueda and R. Nakano. Deterministic annealing EM algorithm. Neural Networks, 11(2):271-282, 1998.
38. R. van Rullen and S. J. Thorpe. Rate coding versus temporal order coding: What the retinal ganglion cells tell the visual cortex. Neural Computation, 13(6):1255-1283, 2001.
39. G. Westphal and R. P. Würtz. Combining feature- and correspondence-based methods for visual object recognition. Neural Computation, 21(7):1952-1989, 2009.
40. A. Yuille and D. Kersten. Vision as Bayesian inference: analysis by synthesis? Trends in Cognitive Sciences, 10(7):301-308, 2006.