Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics

Journal of Machine Learning Research

Volumn 13, Issue , 2012, Pages 307-361

  Gutmann, Michael U ^a   Hyvärinen, Aapo ^a  

^a UNIVERSITY OF HELSINKI   (Finland)

Author keywords

Computation; Estimation; Natural image statistics; Partition function; Unnormalized models

Indexed keywords

CLOSED FORM; COMPUTATIONAL PERFORMANCE; ESTIMATION METHODS; FINITE NUMBER; GIBBS DISTRIBUTION; LOGISTIC REGRESSIONS; MAXIMUM LIKELIHOOD ESTIMATOR; MODEL PROBABILITIES; MULTI-LAYER NETWORK; NATURAL IMAGE STATISTICS; NUMERICAL APPROXIMATIONS; OBJECTIVE FUNCTIONS; OBSERVED DATA; PARAMETERIZED; PARTITION FUNCTION; PARTITION FUNCTIONS; PROBABILISTIC MODELS; SAMPLE SIZES; STATISTICAL MODELS; TWO LAYER MODEL;

CALCULATIONS; COMMERCE; ESTIMATION; LOGISTICS; MAXIMUM LIKELIHOOD ESTIMATION; STATISTICS;

PARAMETER ESTIMATION;

EID: 84857892556     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (31)

1. C. Bishop. Neural Networks for Pattern Recognition. Oxford University Press, 1995.
2. C.J. Geyer. On the convergence of Monte Carlo maximum likelihood calculations. Journal of the Royal Statistical Society, Series B (Methodological), 56(1):261-274, 1994.
3. M. Gutmann and A. Hyvärinen. Learning features by contrasting natural images with noise. In Proceedings of the 19th International Conference on Artificial Neural Networks (ICANN), volume 5769 of Lecture Notes in Computer Science, pages 623-632. Springer Berlin / Heidelberg, 2009.
4. M. Gutmann and A. Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (AISTATS), volume 9 of JMLR W&CP, pages 297-304, 2010.
5. T. Hastie, R. Tibshirani, and J.H. Friedman. The Elements of Statistical Learning. Springer, 2009.
6. G. Hinton. Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8):1771-1800, 2002.
7. A. Hyvärinen. Estimation of non-normalized statistical models using score matching. Journal of Machine Learning Research, 6:695-709, 2005.
8. A. Hyvärinen. Optimal approximation of signal priors. Neural Computation, 20:3087-3110, 2008.
9. A. Hyvärinen, P.O. Hoyer, and M. Inki. Topographic independent component analysis. Neural Computation, 13(7):1527-1558, 2001a. (Pubitemid 33595031)
10. A. Hyvärinen, J. Karhunen, and E. Oja. Independent Component Analysis. Wiley-Interscience, 2001b.
11. A. Hyvärinen, J. Hurri, and P.O. Hoyer. Natural Image Statistics. Springer, 2009.
12. Y. Karklin and M. Lewicki. A hierarchical Bayesian model for learning nonlinear statistical regularities in nonstationary natural signals. Neural Computation, 17:397-423, 2005. (Pubitemid 40190481)
13. D. Koller and N. Friedman. Probabilistic Graphical Models. MIT Press, 2009.
14. U. Köster and A. Hyvärinen. A two-layer model of natural stimuli estimated with score matching. Neural Computation, 22(9):2308-2333, 2010.
15. J. Lücke and M. Sahani. Maximal causes for non-linear component extraction. Journal of Machine Learning Research, 9:1227-1267, 2008.
16. R.M. Neal. Handbook of Markov Chain Monte Carlo, chapter MCMC using Hamiltonian Dynamics. Chapman & Hall /CRC Press, 2010.
17. B.A. Olshausen and D.J. Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381(6583):607-609, 1996. (Pubitemid 26177476)
18. S. Osindero and G. Hinton. Modeling image patches with a directed hierarchy of Markov random fields. In Advances in Neural Information Processing Systems 20, pages 1121-1128. MIT Press, 2008.
19. S. Osindero, M. Welling, and G. E. Hinton. Topographic product models applied to natural scene statistics. Neural Computation, 18 (2):381-414, 2006. (Pubitemid 43543842)
20. M. Pihlaja, M. Gutmann, and A. Hyvärinen. A family of computationally efficient and simple estimators for unnormalized statistical models. In Proceedings of the 26th Conference on Uncertainty in Artificial Intelligence (UAI), pages 442-449. AUAI Press, 2010.
21. M.A. Ranzato and G. Hinton. Modeling pixel means and covariances using factorized third-order Boltzmann machines. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2551-2558, 2010.
22. C.E. Rasmussen. Conjugate gradient algorithm, Matlab code version 2006-09-08. Downloaded from http://learning.eng.cam.ac.uk/carl/code/minimize/ minimize.m. 2006.
23. C.P. Robert and G. Casella. Monte Carlo Statistical Methods. Springer, 2nd edition, 2004.
24. N.N. Schraudolph and T. Graepel. Towards stochastic conjugate gradient methods. In Proceedings of the 9th International Conference on Neural Information Processing (ICONIP), volume 2, pages 853-856, 2002.
25. W. Sun and Y. Yuan. Optimization Theory and Methods: Nonlinear Programming. Springer, 2006.
26. Y. Teh, M. Welling, S. Osindero, and G. Hinton. Energy-based models for sparse overcomplete representations. Journal of Machine Learning Research, 4:1235-1260, 2004.
27. T. Tieleman. Training restricted Boltzmann machines using approximations to the likelihood gradient. In Proceedings of the 25th International Conference on Machine Learning, pages 1064-1071, 2008.
28. J. H. van Hateren and A. van der Schaaf. Independent component filters of natural images compared with simple cells in primary visual cortex. Proceedings of the Royal Society of London. Series B: Biological Sciences, 265(1394):359-366, 1998. (Pubitemid 28111919)
29. Z. Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600-612, 2004.
30. L. Wasserman. All of Statistics. Springer, 2004.
31. L. Younes. Parametric inference for imperfectly observed Gibbsian fields. Probability Theory and Related Fields, 82(4):625-645, 1989.