Dependency reduction with divisive normalization: Justification and effectiveness

Neural Computation

Volumn 23, Issue 11, 2011, Pages 2942-2973

  Lyu, Siwei ^a  

^a State University of New York   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ANIMAL; BIOLOGICAL MODEL; BRAIN; HUMAN; LETTER; NERVE CELL; NONLINEAR SYSTEM; PHYSIOLOGY;

ANIMALS; BRAIN; HUMANS; MODELS, NEUROLOGICAL; NEURONS; NONLINEAR DYNAMICS;

EID: 80053960644     PISSN: 08997667     EISSN: 1530888X     Source Type: Journal    
DOI: 10.1162/NECO_a_00197     Document Type: Letter

References (82)

1. Andrews, D. F., & Mallows, C. L. (1974). Scale mixtures of normal distributions. Journal of the Royal Statistical Society. Series B (Methodological), 36(1), 99-102.
2. Atick, J. J. (1992). Network Could information theory provide an ecological theory of sensory processing? Network, 3, 213-251.
3. Atick, J. J., Li, Z., & Redlich, A. N. (1992). Understanding retinal color coding from first principles. Neural Computation, 4, 559-572.
4. Atick, J. J., & Redlich, A. N. (1992).What does the retina know about natural scenes? Neural Computation, 4, 196-210.
5. Attneave, F. (1954). Some informational aspects of visual perception. Psych. Rev., 61, 183-193.
6. Baddeley, R. (1996). Searching for filters with "interesting" output distributions: An uninteresting direction to explore. Network, 7, 409-421.
7. Barlow, H. B. (1961). Possible principles underlying the transformation of sensory messages. In W. A. Rosenblith (Ed.), Sensory Communication (pp. 217-234). Cambridge, MA: MIT Press.
8. Barlow, H. (2001). Redundancy reduction revisited. Network, 12, 241-253.
9. Bell, A. J.,&Sejnowski, T. J. (1997). The "independent components" of natural scenes are edge filters. Vision Research, 37(23), 3327-3338.
10. Bethge,M. (2006). Factorial coding of natural images:Howeffective are linearmodels in removing higher-order dependencies? J. Opt. Soc. Am. A, 23(6), 1253-1268.
11. Brady, N., & Field, D. J. (2000). Local contrast in natural images: Normalisation and coding efficiency. Perception, 29, 1041-1055.
12. Burt, P., & Adelson, E. (1981). The Laplacian pyramid as a compact image code. IEEE Transactions on Communication, 31(4), 532-540.
13. Carandini, M.,&Heeger,D. J. (1994). Summation and division by neurons in primate visual cortex. Science, 264, 1333-1336.
14. Carandini, M.,Heeger,D. J.,&Senn,W. (2002).Asynaptic explanation of suppression in visual cortex. Journal of Neuroscience, 22(22), 10053-10065.
15. Chantas, G.,Galatsanos, N., Likas,A.,&Saunders, M. (2008). Bayesian image restoration based on a product of t-distributions image prior. IEEE Transactions on Image Processing, 17(10), 1795-1820.
16. Chichilnisky, E. (2001). A simple white noise analysis of neuronal light responses. Network, 12, 199-213.
17. Costa, J., Hero, A., & Vignat, C. (2003). On solutions to multivariate maximum α-entropy problems. In Proceedings of Energy Minimization Methods in Computer Vision and Pattern Recognition. New York: Springer.
18. Cover, T., & Thomas, J. (2006). Elements of information theory (2nd ed.). Hoboken, NJ: Wiley-Interscience.
19. Deneve, S., Pouget, A., & Latham, P. (1999). Divisive normalization, line attractor networks and ideal observers. In M. Kearns, S. A. Solla, & D. A. Cohn (Eds.), Advances in neural processing systems 11 (pp. 104-110). Cambridge,MA: MIT Press.
20. Eichhorn, J., Sinz, F., & Bethge, M. (2009). Natural image coding in V1: How much use is orientation selectivity? PLoS Computational Biology, 5(4), 1-16.
21. Elderton,W. (1953). Frequency curves and correlations. General Books LLC.
22. Feller,W. (1968). An Introduction to probability theory and its applications. Hoboken, NJ: Wiley.
23. Field, D. J. (1987). Relations between the statistics of natural images and the response properties of cortical cells. Journal of the Optical Society of America A, 4(12), 2379-2394.
24. Foley, J. (1994). Human luminance pattern mechanisims: Masking experiments require a newmodel. J. of Opt. Soc. of Amer. A, 11(6), 1710-1719.
25. Gao, D., & Vasconcelos, N. (2009). Decision-theoretic saliency: Computational principles, biological plausibility, and implications for neurophysiology and psychophysics. Neural Computation, 21, 239-271.
26. Garner,W. R. (1962). Uncertainty and structure as psychological concepts. Hoboken, NJ: Wiley.
27. Guerrero-Cusumano, J.-L. (1996). A measure of total variability for the multivariate t distribution with applications to finance. Information Sciences, 92(1-4), 47-63.
28. Heeger,D. J. (1992). Normalization of cell responses in cat striate cortex. Visual Neural Science, 9, 181-198.
29. Holt, G. R., & Koch, C. (1997). Shunting inhibition does not have a divisive effect on firing rates. Neural Computation, 9, 1001-1013.
30. Jarrett, K., Kavukcuoglu, K., Ranzato, M., & LeCun, Y. (2009).What is the best multistage architecture for object recognition? In Proceedings of the 12th IEEE Conference on Computer Vision. Piscataway, NJ: IEEE.
31. Johannesma, P. (1972). The pre-response stimulus ensemble of neurons in the cochlear nucleus. In Symposium on Hearing Theory (pp. 58-69). Eindhoven, Holland: IPO.
32. Kac, M. (1939). On a characterization of the normal distribution. American Journal of Mathematics, 61(3), 726-728.
33. Kotz, S., & Nadarajah, S. (2004). Multivariate t distributions and their applications. Cambridge: Cambridge University Press.
34. Laparra, V., Mu{ogonek} noz-Mari, J., &Malo, J. (2010). Divisive normalization image quality metric revisited. Journal of the Optical Society of America A, 27(4), 852-864.
35. Lee, J., & Maunsell, J. H. R. (2009). A normalizationmodel of attentional modulation of single unit responses. PLoS One, 4(2), 1-13.
36. Lewicki, M. S. (2002). Efficient coding of natural sounds. Nature Neuroscience, 5(4), 356-363.
37. Li, Q., & Wang, Z. (2008). General-purpose reduced-reference image quality assessment based on perceptually and statistically motivated image representation. In IEEE 15th Int'l. Conf. on Image Proc. (Vol. 15, pp. 1192-1195). Piscataway, NJ: IEEE.
38. Lyu, S. (2009). An implicit Markov random field model for natural images in multiscale oriented representations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE.
39. Lyu, S. (2010). Divisive normalization: Justification and effectiveness as efficient coding transform. In D. Koller, D. Schuurmans, Y. Bengio, & L. Bottou (Eds.). Advances in neural information processing systems, 21. Cambridge, MA: MIT Press.
40. Lyu, S., & Simoncelli, E. P. (2008). Nonlinear image representation using divisive normalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE.
41. Lyu, S., & Simoncelli, E. P. (2009a). Modeling multiscale subbands of photographic images with fields of gaussian scale mixtures. In IEEE Trans. Patt. Analysis and Machine Intelligence, 31(4), 693-706.
42. Lyu, S., & Simoncelli, E. P. (2009b). Nonlinear extraction of "independent components" of natural images using radial gaussianization. Neural Computation, 18(6), 1-35.
43. Malo, J., Epifanio, I., Navarro, R., & Simoncelli, E. P. (2006). Non-linear image representation for efficient perceptual coding. IEEE Transactions on Image Processing, 15(1), 68-80.
44. Malo, J., & Laparra, V. (2010). Psychophysically tuned divisive normalization factorizes the PDF of natural images. Neural Computation, 22, 3179-3206.
45. Mante, V., Bonin, V., & Carandini,M. (2008). Functional mechanisms shaping lateral geniculate responses to artificial and natural stimuli. Neuron, 58, 625-638.
46. McCann, J. (2005). Rendering high-dynamic range images: Algorithms that mimic human vision. In Proc. AMOS Technical Conf. (pp. 19-28) Maui: Maui Economic Development Board Publications.
47. Nash, D., & Klamkin, M. S. (1976). A spherical characterization of the normal distribution. Journal of Multi-Variate Analysis, 55, 56-158.
48. Olsen, S. R., Bhandawat,V.,&Wilson, R. I. (2010).Divisive normalization in olfactory population codes. Neuron, 66(2), 287-299.
49. Olshausen, B. A., & Field, D. J. (1996). Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381, 607-609.
50. Paninski, L. (2003). Estimation of entropy and mutual information. Neural Comput., 15(6), 1191-1253.
51. Pillow, J. W., & Simoncelli, E. P. (2006). Dimensionality reduction in neural models: An information theoretic generalization of spike-triggered average and covariance analysis. Journal of Vision, 6, 414-428.
52. Reynolds, J. H.,&Heeger,D. J. (2009). The normalizationmodel of attention. Neuron, 61, 168-185.
53. Ringach, D. L. (2010). Population coding under normalization. Vision Research, 50, 2223-2232.
54. Roth, S.,&Black,M. (2005). Fields of experts:Aframework for learning image priors. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (Vol. 2, pp. 860-867). Piscataway, NJ: IEEE.
55. Ruderman, D. L., Cronin, T. W., & Chiao, C.-C. (1998). Statistics of cone responses to natural images: Implications for visual coding. Journal of the Optical Society of America A, 15(8), 2036-2045.
56. Rust, N. C., Schwartz, O., Movshon, J. A., & Simoncelli, E. P. (2005). Spatiotemporal elements of macaque V1 receptive fields. Neuron, 46(6), 945-956.
57. Schwartz, O., Sejnowski, T. J., & Dayan, P. (2005). Assignment of multiplicative mixtures in natural images. In L. K. Saul, Y. Weiss, & L. Bottou (Eds.), Advances in neural information processing systems, 17. Cambridge, MA: MIT Press.
58. Schwartz, O., Sejnowski, T. J., & Dayan, P. (2006). Soft mixer assignment in a hierarchical model of natural scene statistics. Neural Computation, 18, 2680-2718.
59. Schwartz, O., & Simoncelli, E. P. (2001a). Natural signal statistics and sensory gain control. Nature Neuroscience, 4(8), 819-825.
60. Schwartz, O., & Simoncelli, E. P. (2001b). Natural sound statistics and divisive normalization in the auditory system. In T. K. Leen, T. G. Dietterich, & V. Tresp (Eds.), Advances in neural information processing systems, 13 (pp, 166-172). Cambridge, MA: MIT Press.
61. Shapley, R., & Enroth-Cugell, C. (1984). Visual adaptation and retinal gain control. Progress in Retinal Research, 3, 263-346.
62. Simoncelli, E. P., & Buccigrossi, R.W. (1997). Embedded wavelet image compression based on a joint probabilitymodel. In Proc 4th IEEE Int'l Conf on Image Proc. (Vol. I, pp. 640-643). Piscataway, NJ: IEEE.
63. Simoncelli, E. P., & Freeman, W. T. (1995). The steerable pyramid: A flexible architecture for multi-scale derivative computation. In Proceedings of the IEEE International Conference on Image Processing (Vol. 3, pp. 444-447). Piscataway, NJ: IEEE.
64. Simoncelli, E. P., & Heeger, D. J. (1998). A model of neuronal responses in visual area MT. Vision Research, 38(5), 743-761.
65. Sinz, F. H., & Bethge, M. (2009). The conjoint effect of divisive normalization and orientation selectivity on redundancy reduction. In D. Koller, D. Schuurmans, Y. Bengio, & L. Bottou (Eds.), Advances in neural information processing systems, 21. Cambridge, MA: MIT Press.
66. Solomon, S., Lee, B., & Sun, H. (2006). Suppressive surrounds and contrast gain in magnocellular pathway retinal gangalion cells of macaque. Journal of Neuroscience, 26(34), 8715-8726.
67. Studeny, M., & Vejnarova, J. (1998). The multi-information function as a tool for measuring stochastic dependence. In M. I. Jordan (Ed.), Learning in graphical models (pp. 261-297). Dordrecht: Kluwer.
68. Valerio, R., & Navarro, R. (2003a). Input-output statistical independence in divisive normalization models of V1 neurons. Network, 14(4), 733-745.
69. Valerio, R., & Navarro, R. (2003b). Optimal coding through divisive normalization models of V1 neurons. Network, 14(3), 579-593.
70. van der Schaaf, A., & van Hateren, J. H. (1996). Modelling the power spectra of natural images: Statistics and information. Vision Research, 28(17), 2759-2770.
71. Vasicek, O. (1976). A test for normality based on sample entropy. Journal of the Royal Statistical Society, Series B, 38(1), 54-59.
72. Wainwright, M. J., Schwartz, O., & Simoncelli, E. P. (2002). Natural image statistics and divisive normalization: Modeling nonlinearity and adaptation in cortical neurons. In R.P.N. Rao, B. A. Olshausen, & M. S. Lewicki (Eds.), Probabilistic models of the brain: Perception and neural function (pp. 203-222). Cambridge, MA: MIT Press.
73. Wainwright, M. J., & Simoncelli, E. P. (2000). Scale mixtures of Gaussians and the statistics of natural images. In S. A., Solla, T. K. Leen, & K.-R. M̈ uller (Eds.), Advances in neural information processing systems, 12 (pp. 855-861). Cambridge, MA: MIT Press.
74. Watanabe, S. (1960). Information theoretical analysis ofmultivariate correlation. IBM Journal of Research and Development, 4, 66-82.
75. Watson, A., & Solomon, J. (1997). A model of visual contrast gain control and pattern masking. J. Opt. Soc. Amer. A, 14, 2379-2391.
76. Wegmann, B.,&Zetzsche,C. (1990). Statistical dependence between orientation filter outputs used in an human vision based image code. In Proc. Visual Comm. and Image Processing (Vol. 1360, pp. 909-922). Bellingham,WA: SPIE.
77. Welling, M., Hinton, G. E., & Osindero, S. (2002). Learning sparse topographic representations with products of Student-t distributions. In S. Becker, S. Thr ̈ un, & K. Obermayer (Eds.), Advances in neural information processing systems (pp. 1111-1117). Cambridge, MA: MIT Press.
78. Zellner, A. (1971). An introduction to bayesian inference in econometrics. Hoboken, NJ: Wiley-Interscience.
79. Zetzsche, C., & Barth, E. (1990). Fundamental limits of linear filters in the visual processing of two-dimensional signals. Vision Research, 30, 1111-1117.
80. Zetzsche, C., & Krieger, G. (1999). The atoms of vision: Cartesian or polar? Journal of the Optical Society of America, A, 16, 1554-1565.
81. Zoccolan, D., Cox, D., & DiCarlo, J. (2005). Multiple object response normalization in monkey inferotemporal cortex. Journal of Neuroscience, 25(36), 8150-8164.
82. Zografos, K. (1999). On maximum entropy characterization of Pearson's type II and VII multivariate distributions. Journal of Multivariate Analysis, 71, 67-75.