Estimates of the information content and dimensionality of natural scenes from proximity distributions

Journal of the Optical Society of America A: Optics and Image Science, and Vision

Volumn 24, Issue 4, 2007, Pages 922-941

  Chandler, Damon M ^a   Field, David J ^b  

^a Oklahoma State University   (United States)

^b Department of Molecular Biology and Genetics   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EXTRAPOLATION; IMAGE PROCESSING; MATHEMATICAL MODELS; POWER SPECTRUM; WHITE NOISE;

EXTRAPOLATION MODEL; IMAGE PATCHES; PROXIMITY DISTRIBUTIONS;

GAUSSIAN NOISE (ELECTRONIC);

ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; AUTOMATED PATTERN RECOGNITION; COMPUTER ASSISTED DIAGNOSIS; COMPUTER SIMULATION; IMAGE ENHANCEMENT; INFORMATION RETRIEVAL; INFORMATION SCIENCE; METHODOLOGY; STATISTICAL DISTRIBUTION; STATISTICAL MODEL; THREE DIMENSIONAL IMAGING;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; COMPUTER SIMULATION; IMAGE ENHANCEMENT; IMAGE INTERPRETATION, COMPUTER-ASSISTED; IMAGING, THREE-DIMENSIONAL; INFORMATION STORAGE AND RETRIEVAL; INFORMATION THEORY; MODELS, STATISTICAL; PATTERN RECOGNITION, AUTOMATED; STATISTICAL DISTRIBUTIONS;

EID: 34248388588     PISSN: 10847529     EISSN: None     Source Type: Journal    
DOI: 10.1364/JOSAA.24.000922     Document Type: Article

References (52)

1. D. J. Field, "Relations between the statistics of natural images and the response properties of cortical cells," J. Opt. Soc. Am. A 4, 2379-2394 (1987).
2. J. J. Atick, "Could information theory provide an ecological theory of sensory processing?" Network 3, 213-251 (1992).
3. D. Kersten, "Predictability and redundancy of natural images," J. Opt. Soc. Am. A 4, 2395-2400 (1987).
4. O. Schwartz and E. P. Simoncelli, "Natural signal statistics and sensory gain control," Nat. Neurosci. 4, 819-825 (2001).
5. E. P. Simoncelli and B. A. Olshausen, "Natural image statistics and neural representation," Annu. Rev. Neurosci. 24, 1193-1216 (2001).
6. W. S. Geisler, J. S. Perry, B. J. Super, and D. P. Gallogly, "Edge co-occurence in natural images predicts contour grouping performance," Vision Res. 41, 711-724 (2001).
7. D. J. Field, "Scale-invariance and self-similar 'Wavelet' transforms: an analysis of natural scenes and mammalian visual systems," in Wavelets, Fractals and Fourier Transforms: New Developments and New Applications (Oxford U. Press, 1993), pp. 151-193.
8. Y. Petrov and L. Zhoaping, "Local correlations, information redundancy, and the sufficient pixel depth in natural images," J. Opt. Soc. Am. A 20, 56-66 (2003).
9. T. S. Lee, D. Mumford, R. Romero, and V. A. F. Lamme, "The role of the primary visual cortex in higher level vision," Vision Res. 38, 2429-2454 (1998).
10. B. A. Olshausen and D. J. Field, "Sparse coding with an overcomplete basis set: a strategy employed by V1?" Vision Res. 37, 3311-3325 (1996).
11. A. J. Bell and T. J. Sejnowski, "The independent components of natural scenes are edge filters," Vision Res. 37, 3327-3338 (1997).
12. W. B. Pennebaker and J. L. Mitchell, The JPEG Still Image Data Compression Standard (Van Nostrand Reinhold, 1993).
13. International Organization for Standardization, "Information technology - JPEG 2000 image coding system: core coding system," Tech. Rep. ISO/IEC FDIS15444-1:2000 (International Organization for Standardization, 2000).
14. N. G. Deriugin, "The power spectrum and the correlation function of the television signal," Telecommun. 1, 1-12 (1957).
15. D. L. Ruderman and W. Bialek, "Statistics of natural images: scaling in the woods," Phys. Rev. Lett. 73, 814-817 (1994).
16. J. Minguillon and J. Pujol, "Uniform quantization error for Laplacian sources with applications to JPEG standard," in Mathematics of Data/Image Coding, Compression, and Encryption, M. S. Schmalz, ed., Proc. SPIE 3456, 77-88 (1998).
17. M. Wainwright, E. P. Simoncelli, and A. Willsky, "Random cascades on wavelet trees and their use in modeling and analyzing natural imagery," Appl. Comput. Harmon. Anal. 11, 89-123 (2001).
18. L. F. Kozachenko and N. N. Leonenko, "A statistical estimate for the entropy of a random vector," Probl. Inf. Transm. 23, 9-16 (1987).
19. J. D. Victor, "Binless strategies for estimation of information from neural data," Phys. Rev. E 66, 051903 (2002).
20. A. Kraskov, H. Stgbauer, and P. Grassberger, "Estimating mutual information," Phys. Rev. E 69, 066138 (2004).
21. C. E. Shannon, "A mathematical theory of communication," Bell Syst. Tech. J. 27, 623-656 (1948).
22. T. M. Cover and J. A. Thomas, Elements of Information Theory, Wiley Series in Telecommunications (Wiley, 1991).
23. S. Verdu and T. Han, "The role of the asymptotic equipartition property in noiseless source coding," IEEE Trans. Inf. Theory 43, 847-857 (1997).
24. E. W. Weisstein, "Birthday problem," from MathWorld - A Wolfram Web Resource, http://mathworld.wolfram.com/BirthdayProblem.html.
25. I. Nemenman, W. Bialek, and R. de Ruyter van Steveninck, "Entropy and information in neural spike trains: progress on the sampling problem," Phys. Rev. E 69, 056111 (2004).
26. Z. E. Schnabel, "The estimation of total fish population of a lake," Am. Math. Monthly 45, 348-352 (1938).
27. I. Nemenman, F. Shafee, and W. Bialek, "Entropy and inference, revisited," in Advances in Neural Information Processing Systems, Vol. 14, T. G. Dietterich, S. Becker, and Z. Ghahramani, eds. (MIT Press, 2002).
28. J. Guckenheimer and G. Buzyna, "Dimension measurements for geostrophic turbulence," Phys. Rev. Lett. 51, 1438-1441 (1983).
29. R. Badii and A. Politi, "Statistical description of chaotic attractors: the dimension function," J. Stat. Phys. 40, 725-750 (1985).
30. P. Grassberger, "Generalizations of the Hausdorff dimension of fractal measures," Phys. Lett. A 107, 101-105 (1985).
31. K. Pettis, T. Bailey, A. K. Jain, and R. Dubes, "An intrinsic dimensionality estimator from near-neighbor information," IEEE Trans. Pattern Anal. Mach. Intell. 1, 25-36 (1979).
32. W. van de Water and P. Schram, "Generalized dimensions from near-neighbor information," Phys. Rev. A 37, 3118-3125 (1988).
33. J. B. Tenenbaum, V. de Silva, and J. C. Langford, "A global geometric framework for nonlinear dimensionality reduction," Science 290, 2319-2323 (2000).
34. S. T. Roweis and L. K. Saul, "Nonlinear dimensionality reduction by locally linear embedding," Science 290, 2323-2326 (2000).
35. J. Theiler, "Estimating fractal dimension," J. Opt. Soc. Am. A 7, 1055-1073 (1990).
36. K. L. Clarkson, "Nearest neighbor searching and metric space dimensions," in Nearest-Neighbor Methods for Learning and Vision: Theory and Practice, G. Shakhnarovich, T. Darrell, and P. Indyk, eds. (MIT Press, 2006), Chap. 2, pp. 15-59.
37. In this paper, we estimate differential entropy assuming that the original images are drawn from an underlying continuous distribution. Under high-rate quantization, the discrete entropy H is related to the differential entropy h by H ≈ h + log Δ, where Δ is the quantization step size (here Δ = 1, log Δ = 0); see Refs. 19 and 22.
38. J. H. van Hateren and A. van der Schaaf, "Independent component filters of natural images compared with simple cells in primary visual cortex," Proc. R. Soc. London, Ser. B 265, 359-366 (1998).
39. R. M. Gray and D. L. Neuhoff, "Quantization," IEEE Trans. Inf. Theory 44, 2325-2384 (1998).
40. see http://redwood.psych.cornell.edu/proximity/.
41. J. Kybic, "High-dimensional mutual information estimation for image registration," in Proceedings of IEEE International Conference on Image Processing (IEEE, 2004), pp. 1779-1782.
42. As long as the means are identical, the distance between two patches does not depend on the (common) mean of the underlying Gaussian from which the pixels are drawn.
43. E. W. Weisstein, "Chi-squared distribution," from MathWorld - A Wolfram Web Resource, http://mathworld.wolfram.com/ Chi-SquaredDistribution.html.
44. Although the pixel values of the spectrum-equalized noise patches were correlated (and were therefore statistically dependent), the real and imaginary components of the DFT coefficients of each block were independent. Accordingly, the entropy of the spectrum-equalized noise was computed by summing the individual entropies of the real and imaginary part of each DFT coefficient; the individual entropies were computed via Eq. (9).
45. J. A. Nelder and R. Mead, "A simplex method for function minimization," J. Comput. 7, 308-313 (1965).
46. D. J. Field, "What is the goal of sensory coding?" Neural Comput. 6, 559-601 (1994).
47. J. R. Parks, "Prediction and entropy of half-tone pictures," Behav. Sci. 10, 436-445 (1965).
48. N. S. Tzannes, R. V. Spencer, and A. Kaplan, "On estimating the entropy of random fields," Inf. Control. 16, 1-6 (1970).
49. A. B. Lee, K. S. Pedersen, and D. Mumford, "The nonlinear statistics of high-contrast patches in natural images," Int. J. Comput. Vis. 54, 83-103 (2003).
50. J. Costa and A. O. Hero, "Geodesic entropic graphs for dimension and entropy estimation in manifold learning," IEEE Trans. Signal Process. 52, 2210-2221 (2004).
51. B. V. Dasarathy, Nearest Neighbour (NN) Norms: NN Pattern Classification Techniques (IEEE, 1973).
52. G. Shakhnarovich, T. Darrell, and P. Indyk, Nearest-Neighbor Methods in Learning and Vision: Theory and Practice (MIT Press, 2006).