Nonlinear extraction of independent components of natural images using radial gaussianization

Neural Computation

Volumn 21, Issue 6, 2009, Pages 1485-1519

  Lyu, Siwei ^a   Simoncelli, Eero P ^b  

^a State University of New York   (United States)

^b NEW YORK UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; ARTIFICIAL INTELLIGENCE; COMPUTER ASSISTED DIAGNOSIS; HUMAN; NONLINEAR SYSTEM; NORMAL DISTRIBUTION; PRINCIPAL COMPONENT ANALYSIS; SIGNAL PROCESSING;

ARTIFICIAL INTELLIGENCE; HUMANS; IMAGE INTERPRETATION, COMPUTER-ASSISTED; NONLINEAR DYNAMICS; NORMAL DISTRIBUTION; PRINCIPAL COMPONENT ANALYSIS; SIGNAL PROCESSING, COMPUTER-ASSISTED;

EID: 67651065449     PISSN: 08997667     EISSN: 1530888X     Source Type: Journal    
DOI: 10.1162/neco.2009.04-08-773     Document Type: Article

References (91)

1. Abadir, K. M., & Magnus, J. R. (2005). Matrix algebra. Cambridge: Cambridge University Press.
2. Adelson, E. H., Simoncelli, E. P., & Hingorani, R. (1987). Orthogonal pyramid transforms for image coding. In Proc. SPIE Visual Communications and Image Processing II (Vol. 845, pp. 50-58). Bellingham, WA: SPIE.
3. Andrews, D. F., & Mallows, C. L. (1974). Scale mixtures of normal distributions. Journal of the Royal Statistical Society, Series B (Methodological), 36(1), 99-102.
4. Atick, J. J., & Redlich, A. N. (1990). Towards a theory of early visual processing. Neural Computation, 2, 308-320.
5. Attneave, F. (1954). Some informational aspects of visual perception. Psych. Rev., 61, 183-193.
6. Baddeley, R. (1996). Searching for filters with "interesting"output distributions: An uninteresting direction to explore. Network, 7, 409-421.
7. Barlow, H. B. (1961). Possible principles underlying the transformation of sensory messages. In W. A. Rosenblith (Ed.), Sensory communication (pp. 217-234). Cambridge, MA: MIT Press.
8. Bell, A. J., & Sejnowski, T. J. (1997). The "independent components" of natural scenes are edge filters. Vision Research, 37(23), 3327-3338.
9. Bethge, M. (2006). Factorial coding of natural images: How effective are linear models in removing higher-order dependencies? J. Opt. Soc. Am. A, 23(6), 1253-1268.
10. Buccigrossi, R. W., & Simoncelli, E. P. (1999). Image compression via joint statistical characterization in the wavelet domain. IEEE Transactions on Image Processing, 8(12), 1688-1701.
11. Burt, P., & Adelson, E. (1981). The Laplacian pyramid as a compact image code. IEEE Transactions on Communication, 31(4), 532-540.
12. Cardoso, J.-F. (1999). High-order contrasts for independent component analysis. Neural Computation, 11(1), 157-192.
13. Cardoso, J.-F. (2004). Dependence, correlation and gaussianity in independent component analysis. J. Mach. Learn. Res., 4(7-8), 1177-1203.
14. Casella, G., & Berger, R. L. (2001). Statistical inference (2nd ed.). Pacific Grove, CA: Duxbury Press.
15. Chen, S. S., Donoho, D. L., & Saunders, M. A. (1998). Atomic decomposition by basis pursuit. SIAM Journal on Scientific Computing, 20(1), 33-61.
16. Chen, S. S., & Gopinath, R. A. (2000). Gaussianization. In T. K. Leen, T. G. Dietterich, & V. Tresp (Eds.), Advances in neural computation systems, 13 (pp. 423-429). Cambridge, MA: MIT Press.
17. Coifman, R. R., & Wickerhauser, M. V. (1992). Entropy-based algorithms for best basis selection. IEEE Trans. Info. Theory, 38(2), 713-718.
18. Comon, P. (1994). Independent component analysis, a new concept? Signal Process., 36, 387-314.
19. Cover, T., & Thomas, J. (2006). Elements of information theory (2nd ed.). Hoboken, NJ: Wiley-Interscience.
20. Dayan, P., & Abbott, L. F. (2001). Theoretical neuroscience.Cambridge,MA:MITPress.
21. Donoho, D. L., & Elad, M. (2003). Optimally sparse representation in general (nonorthogonal) dictionaries via l1 minimization. Proc. National Academy of Sciences, 100(5), 2197-2202.
22. Fang, K., Kotz, S., & Ng, K. (1990). Symmetric multivariate and related distributions. London: Chapman and Hall.
23. Feller, W. (1968). An introduction to probability theory and its applications (Vol. 1). Hoboken, NJ: Wiley.
24. Field, D. J. (1987). Relations between the statistics of natural images and the response properties of cortical cells. J. Opt. Soc. Am. A, 4(12), 2379-2394.
25. Foley, J. (1994). Human luminance pattern-vision mechanisms: Masking experiments require a new model.J. Opt. Soc. Am. A, 11(6), 1710-1719.
26. Gehler, P., & Welling, M. (2006). Products of "edge-perts."In Y. Weiss, B. Schölkopf, & J. Platt (Eds.), Advances in Neural Information Processing Systems, 18 (419-426). Cambridge, MA: MIT Press.
27. Geisler, W. S., & Albrecht, D. G. (1992). Cortical neurons: Isolation of contrast gain control. Vision Research, 8, 1409-1410.
28. Gluckman, J. M. (2006). Higher order pyramids: An early vision representation. In A. Leonardis, H. Bischof, & A. Prinz (Eds.), European Conference on Computer Vision (ECCV). Berlin: Springer-Verlag.
29. Granlund, G. H. (1978). In search of a general picture processing operator. Computer Graphics and Image Processing, 8(2), 155-173.
30. Heeger, D. J. (1992). Normalization of cell responses in cat striate cortex. Visual Neural Science, 9, 181-198.
31. Huang, J., & Mumford, D. (1999). Statistics of natural images and models. In IEEE International Conference on Computer Vision and Pattern Recognition. New York: IEEE.
32. Hyvärinen, A. (1999). Fast and robust fixed-point algorithms for independent component analysis. IEEE Transactions on Neural Networks, 10(3), 626-634.
33. Hyvärinen, A., & Hoyer, P. (2000). Emergence of phaseand shift-invariant features by decomposition of natural images into independent feature subspaces. Neural Computation, 12(2), 1705-1720.
34. Hyvärinen, A., Hoyer, P. O., & Inki, M. (2001). Topographic independent component analysis. Neural Computation, 13, 1527-1558.
35. Hyvärinen, A., Hurri, J., & Väyrynen, J. (2003). Bubbles: A unifying framework for low-level statistical properties of natural image sequences. J. Opt. Soc. Am. A, 20(7), 1237-1252.
36. Hyvärinen, A., & Pajunen, P. (1999). Nonlinear independent component analysis: Existence and uniqueness results. Neural Networks, 12(3), 429-439.
37. Jolliffe, I. (2002). Principal component analysis (2nd ed.). Berlin: Springer-Verlag.
38. Karklin, Y., & Lewicki, M. S. (2005). A hierarchical Bayesian model for learning non-linear statistical regularities in non-stationary natural signals. Neural Computation, 17(2), 397-423.
39. Kingman, J. F. C. (1963). Random walks with spherical symmetry. Acta Math., 109(9), 11-53.
40. Koenderink, J. J. (1984). The structure of images. Biological Cybernetics, 50, 363-370.
41. Kohonen, T. (1996). Emergence of invariant-feature detectors in the adaptivesubspace self-organizing map. Biological Cybernetics, 75(5), 281-291.
42. Kraskov, A., Stögbauer, H., & Grassberger, P. (2004). Estimating mutual information. Phys. Rev. E, 69(6), 66-82.
43. Learned-Miller, E. G., & Fisher, J. W. (2000). ICA using spacings estimates of entropy. Journal of Machine Learning Research, 4(1), 1271-1295.
44. Lewicki, M. S. (2002). Efficient coding of natural sounds. Nature Neuroscience, 5(4), 356-363.
45. Lewicki, M. S., & Sejnowski, T. J. (2000). Learning overcomplete representations. Neural Computation, 12, 337-365.
46. Lyu, S., & Simoncelli, E. P. (2007). Statistically and perceptually motivated nonlinear image representation. In Proceedings of the IS&T/SPIE 19th Annual Symposium of Electronic Imaging. Bellingham, WA: SPIE.
47. Lyu, S., & Simoncelli, E. P. (2008). Nonlinear extraction of "independent components" of elliptically symmetric densities using radial Gaussianization (Tech. Rep. TR2008-911). New York: Courant Institute of Mathematical Sciences, New York University.
48. Lyu, S., & Simoncelli, E. P. (2009a). Modeling multiscale subbands of photographic images with fields of gaussian scale mixtures. IEEE Trans. Patt. Analysis and Machine Intelligence, 31(4), 693-706.
49. Lyu, S., & Simoncelli, E. P. (2009b). Reducing statistical dependencies in natural signals using radial gaussianization. In D. Koller, D. Schuurman, Y. Bengio, & L. Bottou (Eds.), Advances in neural information processing systems, 21.Cambridge, MA: MIT Press.
50. Mallat, S. G. (1989). A theory for multiresolution signal decomposition: The wavelet representation. IEEE Trans. Patt. Analysis and Machine Intelligence, 11, 674-693.
51. Mallat, S., & Zhang, Z. (1993). Matching pursuits with time-frequency dictionaries. IEEE Trans. Signal Proc., 41(12), 3397-3415.
52. Malo, J., Epifanio, I., Navarro, R., & Simoncelli, E. P. (2006). Nonlinear image representation for efficient perceptual coding. IEEE Trans. Image Processing, 15(1), 68-80.
53. Malo, J., Navarro, R., Epifanio, I., Ferri, F., & Artigas, J. (2000). Non-linear invertible representation for joint statistical and perceptual feature decorrelation. In Proc. SPR+SSPR (pp. 658-667). Berlin: Springer-Verlag.
54. Mante, V., Bonin, V., & Carandini, M. (2008). Functional mechanisms shaping lateral geniculate responses to artificial and natural stimuli. Neuron, 58, 625-638.
55. Manzotti, A., Pérez, F. J., & Quiroz, A. J. (2002). A statistic for testing the null hypothesis of elliptical symmetry. Journal of Multivariate Analysis, 81(2), 274-285.
56. Mika, S., Schölkopf, B., Smola, A. J., Müller, K.-R., Scholz, M., & Rätsch, G. (1999). Kernel PCA and de-noising in feature spaces. In M. S. Kearns, S. A. Solla, & D. A. Cohn (Eds.), Advances in neural information processing systems, 11.Cambridge, MA: MIT Press
57. Nash, D., & Klamkin, M. S. (1976). A spherical characterization of the normal distribution. Journal of Multivariate Analysis, 55, 56-158.
58. Nolan, J. P. (2009). Stable distributions-Models for heavy tailed data. Boston: Birkhäuser. In progress, Chapter 1 online at http://academic2.american.edu/~jpnolan.
59. Olshausen, B. A., & Field, D. J. (1996). Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381, 607-609.
60. Olshausen, B. A., & Field, D. J. (1997). Sparse coding with an overcomplete basis set: A strategy employed by V1? Vision Research, 37, 3311-3325.
61. Parra, L., Spence, C., & Sajda, P. (2001). Higher-order statistical properties arising from the non-stationarity of natural signals. In T. K. Leen, T. G. Dietterich, & V. Tresp (Eds.), Advances in neural information processing systems, 13 (pp. 786-792). Cambridge, MA: MIT Press.
62. Portilla, J., Strela, V., Wainwright, M. J., & Simoncelli, E. P. (2003). Image denoising using a scale mixture of gaussians in the wavelet domain. IEEE Trans. Image Processing, 12(11), 1338-1351.
63. Ruderman, D. L. (1996). The statistics of natural images. Network: Computation in Neural Systems, 5, 517-548.
64. Ruderman, D. L., & Bialek, W. (1994). Statistics of natural images: Scaling in the woods. Phys. Rev. Letters, 73(6), 814-817.
65. Schwartz, O., & Simoncelli, E. P. (2001). Natural signal statistics and sensory gain control. Nature Neuroscience, 4(8), 819-825.
66. Sendur, L., & Selesnick, I. W. (2002). Bivariate shrinkage functions for wavelet-based denoising exploiting interscale dependency. IEEE Trans. Signal Proc., 50(11), 2744-2756.
67. Shan, H., Zhang, L., & Cottrell, G. W. (2007). Recursive ICA. In B. Schölkopf, J. Platt, & T. Hoffman (Eds.), Advances in neural information processing systems, 19 (pp. 1273-1280). Cambridge, MA: MIT Press.
68. Shapley, R., & Enroth-Cugell, C. (1984). Visual adaptation and retinal gain control. Progress in Retinal Research, 3, 263-346.
69. Simoncelli, E. P. (1997). Statistical models for images: Compression, restoration and synthesis. In Proc 31st Asilomar Conf. on Signals, Systems and Computers (Vol. 1, pp. 673-678). Washington, DC: IEEE Computer Society.
70. Simoncelli, E. P., & Olshausen, B. (2001). Natural image statistics and neural representation. Annual Review of Neuroscience, 24, 1193-1216.
71. Srinivasan, M. V., Laughlin, S. B., & Dubs, A. (1982). Predictive coding: A fresh view of inhibition in the retina. J. R. Soc. Lond. B, 216, 427-459.
72. Srivastava, A., Liu, X., & Grenander, U. (2002). Universal analytical forms for modeling image probability. IEEE Trans. Patt. Anal. Mach. Intell., 24(9), 1200-1214.
73. Studeny, M., & Vejnarova, J. (1998). The multiinformation function as a tool for measuring stochastic dependence. In M. I. Jordan (Ed.), Learning in graphical models (pp. 261-297). Dordrecht: Kluwer.
74. Teh, Y., Welling, M., & Osindero, S. (2003). Energy-based models for sparse overcomplete representations. Journal of Machine Learning Research, 4, 1235-1260.
75. Teo, P. C., & Heeger, D. J. (1994). Perceptual image distortion. In IEEE Intl. Conf. on Image Proc. (pp. 982-986). New York: IEEE.
76. Turner, R., & Sahani, M. (2008). Modeling natural sounds with modulation cascade processes. In J. Platt, D. Koller, Y. Singer, & S. Roweis (Eds.), Advances in neural information processing systems, 20. Cambridge, MA: MIT Press.
77. Valerio, R., & Navarro, R. (2003). Optimal coding through divisive normalization models of V1 neurons. Network: Computation in Neural Systems, 14, 579-593.
78. van der Schaaf, A., & van Hateren, J. H. (1996). Modelling the power spectra of natural images: Statistics and information. Vision Research, 28(17), 2759-2770.
79. van Hateren, J. H., & Ruderman, D. L. (1998). Independent component analysis of natural image sequences yields spatio-temporal filters similar to simple cells in primary visual cortex. Proc.R.Soc.Lond.B,265, 2315-2320.
80. Vasicek, O. (1976). A test for normality based on sample entropy. Journal of the Royal Statistical Society, Series B, 38(1), 54-59.
81. Wainwright, M. J., Schwartz, O., & Simoncelli, E. P. (2002). Natural image statistics and divisive normalization: Modeling nonlinearity and adaptation in cortical neurons. In R. Rao, B. Olshausen, & M. Lewicki (Eds.), Probabilistic models of the brain: Perception and neural function (pp. 203-222). Cambridge, MA: MIT Press.
82. Wainwright, M. J., & Simoncelli, E. P. (2000). Scale mixtures of gaussians and the statistics of natural images. In S. A. Solla, T. K. Leen, & K.-R. Müller (Eds.), Advances in neural information processing systems, 12 (pp. 855-861). Cambridge, MA: MIT Press.
83. Wainwright, M. J., Simoncelli, E. P., & Willsky, A. S. (2001). Random cascades on wavelet trees and their use in modeling and analyzing natural imagery. Applied and Computational Harmonic Analysis, 11(1), 89-123.
84. Watson, A., & Solomon, J. (1997). A model of visual contrast gain control and pattern masking. J. Opt. Soc. Amer. A, 14, 2379-2391.
85. Wegmann, B., & Zetzsche, C. (1990). Statistical dependence between orientation filter outputs used in an human vision based image code. In Proc. Visual Comm. and Image Processing (Vol. 1360, pp. 909-922). Bellingham, WA: SPIE.
86. Yao, K. (1973). A representation theorem and its applications to spherically-invariant random processes. IEEE Trans. on Information Theory, 19(9), 600-608.
87. Zetzsche, C., & Barth, E. (1990). Fundamental limits of linear filters in the visual processing of two-dimensional signals. Vision Research, 30, 1111-1117.
88. Zetzsche, C., & Krieger, G. (1999). The atoms of vision: Cartesian or polar? J. Opt. Soc. Am. A, 16(7), 1554-1565.
89. Zetzsche, C., & Schönecker, W. (1987). Orientation selective filters lead to entropy reduction in the processing of natural images. Perception, 16, 229.
90. Zetzsche, C., Wegmann, B., & Barth, E. (1993). Nonlinear aspects of primary vision: Entropy reduction beyond decorrelation. In Intl. Symposium, Society for Information Display (Vol. 24, pp. 933-936). Campbell, CA: Society for Information Display.
91. Zhaoping, L. (2006). Theoretical understanding of the early visual processes by data compression and data selection. Network: Computation in Neural Systems, 17(4), 301-334.