Automatic foveation for video compression using a neurobiological model of visual attention

IEEE Transactions on Image Processing

Volumn 13, Issue 10, 2004, Pages 1304-1318

  Itti, Laurent ^a  

^a UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; EYE MOVEMENTS; IMAGE CODING; IMAGE QUALITY; MATHEMATICAL MODELS; TRACKING (POSITION);

FOVEATION; NEUROBIOLOGICAL MODEL; SALIENCY INTERACTION; VIDEO CLIPS; VISUAL ATTENTION;

IMAGE COMPRESSION;

ADULT; ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; ATTENTION; AUTOMATED PATTERN RECOGNITION; BIOLOGICAL MODEL; BIOMIMETICS; CLINICAL TRIAL; COMPARATIVE STUDY; COMPUTER GRAPHICS; COMPUTER INTERFACE; COMPUTER SIMULATION; EYE MOVEMENT; FEMALE; HUMAN; IMAGE ENHANCEMENT; INFORMATION PROCESSING; MALE; MATHEMATICAL COMPUTING; METHODOLOGY; PHYSIOLOGY; REPRODUCIBILITY; RETINA FOVEA; SENSITIVITY AND SPECIFICITY; SIGNAL PROCESSING; VALIDATION STUDY; VIDEORECORDING; VISION;

ADULT; ALGORITHMS; ARTIFICIAL INTELLIGENCE; ATTENTION; BIOMIMETICS; COMPUTER GRAPHICS; COMPUTER SIMULATION; DATA COMPRESSION; EYE MOVEMENTS; FEMALE; FOVEA CENTRALIS; HUMANS; IMAGE ENHANCEMENT; MALE; MODELS, NEUROLOGICAL; NUMERICAL ANALYSIS, COMPUTER-ASSISTED; PATTERN RECOGNITION, AUTOMATED; REPRODUCIBILITY OF RESULTS; SENSITIVITY AND SPECIFICITY; SIGNAL PROCESSING, COMPUTER-ASSISTED; USER-COMPUTER INTERFACE; VIDEO RECORDING; VISUAL PERCEPTION;

EID: 4544260101     PISSN: 10577149     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIP.2004.834657     Document Type: Article

References (74)

1. B. Wandell, Foundations of Vision. Sunderland, MA: Sinauer, 1995.
2. J. M. Shapiro, "Embedded image coding using zerotrees of wavelet coefficients," IEEE Trans. Signal Processing, vol. 41, pp. 3445-3462, Dec. 1993.
3. D. Taubman and A. Zakhor, "Multirate 3-D subband coding of video," IEEE Trans. Image Processing, vol. 3, pp. 572-588, Sept. 1994.
4. Z. Wang, L. G. Lu, and A. C. Bovik, "Foveation scalable video coding with automatic fixation selection," IEEE Trans. Image Processing, vol. 12, pp. 243-254, Feb. 2003.
5. P. M. J. van Diepen, P. De Graef, and J. van Rensbergen, "On-line control of moving masks and windows on a complex background using the atvista videographics adapter," Behav: Res. Meth., vol. 26, pp. 454-60, 1994.
6. P. T. Kortum and W. S. Geisler, "Implementation of a foveated image-coding system for bandwidth reduction of video images," Proc. SPIE, vol. 2657, pp. 350-360, 1996.
7. N. Doulamis, A. Doulamis, D. Kalogeras, and S. Kollias, "Improving the performance of MPEG coders using adaptive regions of interest," IEEE Trans. Circuits Syst. Video Technol., vol. 8, pp. 928-934, Dec. 1998.
8. S. Lee, A. C. Bovik, and Y. Y. Kim, "Low delay foveated visual communications over wireless channels," in Proc. IEEE Int. Conf. Image Processing, 1999, pp. 90-94.
9. U. Rauschenbach and H. Schumann, "Demand-driven image transmission with levels of detail and regions of interest," Comput. Grpah., vol. 23, no. 6, pp. 857-866, 1999.
10. E. M. Reingold and L. C. Loschky, "Saliency of peripheral targets in gaze-contingent multiresolutional displays," Behavior Res. Meth., Instrum. Comput., vol. 34, no. 4, pp. 491-499, 2002.
11. D. J. Parkhurst and E. Niebur, "Variable-resolution displays: A theoretical, practical, and behavioral evaluation," Human Factors, vol. 44, no. 4, pp. 611-629, 2002.
12. W. S. Geisler and J. S. Perry, "A real-time foveated multi-resolution system for low-bandwidth video communication," Proc. SPIE, vol. 3299, pp. 294-305, 1998.
13. J. H. Goldberg and J. C. Schryver, "Eye-gaze-contingent control of the computer interface: Methodology and example for zoom detection," Behav. Res. Meth., Instrum. Comput., vol. 27, no. 3, pp. 338-350, Aug. 1995.
14. L. B. Stelmach, W. J. Tam, and P. J. Hearty, "Static and dynamic spatial resolution in image coding: An investigation of eye movements," Proc. SPIE, vol. 1453, pp. 147-152, 1991.
15. A. T. Duchowski and B. H. McCormick, "Pre-attentive considerations for gaze-contingent image processing," Proc. SPIE, vol. 2411, pp. 128-139, 1995.
16. A. T. Duchowski, "Modeling visual attention for gaze-contingent video processing," in Proc. 9th Image and Multidim.l Signal Proc. Workshop, 1996, pp. 130-131.
17. W. Osberger and A. J. Maeder, "Automatic identification of perceptually important regions in an image using a model of the human visual system," in Proc. Int. Conf. Pattern Recognition, Aug. 1998, pp. 701-704.
18. X. Yang and K. Ramchandran, "A low-complexity region-based video coder using backward morphological motion field segmentation," IEEE Trans. Image Processing, vol. 8, pp. 332-345, Mar. 1999.
19. F. Stentiford, "An estimator for visual attention through competitive novelty with application to image compression," in Proc. Picture Coding Symp., Apr. 2001, pp. 101-104.
20. W. Osberger, A. J. Maeder, and N. Bergmann, "Perceptually based quantization technique for MPEG encoding," Proc. SPIE, vol. 3299, pp. 148 159, 1998.
21. M. W. Marcellin, M. J. Gormish, A. Bilgin, and M. R Boliek, "An overview of JPEG-2000," in Proc. Data Compression Conf., 2000, pp. 523-544.
22. E. Mandel and P. Penev, "Facial feature tracking and pose estimation in video sequences by factorial coding of the low-dimensional entropy manifolds due to the partial symmetries of faces," in Proc. 25th IEEE Int. Conf., Acoustics, Speech, and Signal Processing, vol. IV, June 2000, pp. 2345-2348.
23. C. M. Privitera and L. W. Stark, "Algorithms for defining visual regions-of-interest: Comparison with eye fixations," IEEE Trans. Pattern Anal. Machine Intell., vol. 22, pp. 970-982, Sept. 2000.
24. W. M. Osberger and A. M. Rohaly, "Automatic identification of regions of interest in complex video sequences," Proc. SPIE, vol. 4299, pp. 361-372, 2001.
25. D. Parkhurst, K. Law, and E. Niebur, "Modeling the role of salience in the allocation of overt visual attention," Vis. Res., vol. 42, no. 1, pp. 107-123, 2002.
26. A. Yarbus, Eye Movements and Vision. New York: Plenum, 1967.
27. J. M. Henderson and A. Hollingworth, "High-level scene perception," Annu. Rev. Psychol., vol. 50, pp. 243-271, 1999.
28. R. A. Rensink, "The dynamic representation of scenes," Vis. Cognit., vol. 7, pp. 17-42, 2000.
29. L. Itti and C. Koch, "Computational modeling of visual attention," Nat. Rev. Neurosci., vol. 2, no. 3, pp. 194-203, Mar. 2001.
30. L. Itti, "Modeling primate visual attention," in Computational Neuroscience: A Comprehensive Approach, J. Feng, Ed. Boca Raton, FL: CRC, 2003, pp. 607-626.
31. M. C. Potter and E. I. Levy, "Recognition memory for a rapid sequence of pictures," J. Exp. Psychol., vol. 81, no. 1, pp. 10-15, July 1969.
32. A. Friedman, "Framing pictures: The role of knowledge in automatized encoding and memory for gist," J. Exp. Psychol. Gen., vol. 108, pp. 316-355, Sept. 1979.
33. I. Biederman, R. C. Teitelbaum, and R. J. Mezzanotte, "Scene perception: A failure to find a benefit from prior expectancy or familiarity," J. Exp. Psychol. Learning Memory Cognit., vol. 9, no. 3, pp. 411-429, July 1983.
34. A. Oliva and P. G. Schyns, "Coarse blobs or fine edges? Evidence that information diagnosticity changes the perception of complex visual stimuli," Cognit. Psychol., vol. 34, no. 1, pp. 72-107, Oct. 1997.
35. A. Hollingworth and J. M. Henderson, "Does consistent scene context facilitate object perception?," J. Exp. Psychol. Gen., vol. 127, no. 4, pp. 398-415, Dec. 1998.
36. A. Torralba, "Modeling global scene factors in attention," J. Opt. Soc:. Amer A, vol. 20, no. 7, pp. 1407-1418, July 2003.
37. J. Moran and R. Desimone, "Selective attention gates visual processing in the extrastriate cortex," Science, vol. 229, no. 4715, pp. 782-4, 1985.
38. B. Motter, "Neural correlates of attentive selection for color or luminance in extrastriate area V4," J. Neurosci., vol. 14, no. 4, pp. 2178-89, 1994.
39. Y. Yeshurun and M. Carrasco, "Attention improves or impairs visual performance by enhancing spatial resolution," Nature, vol. 396, no. 6706, pp. 72-75, Nov. 1998.
40. S. Treue and J. C. M. Trujillo, "Feature-based attention influences motion processing gain in macaque visual cortex," Nature, vol. 399, no. 6736, pp. 575-579, Jun 1999.
41. M. Ito and C. D. Gilbert, "Attention modulates contextual influences in the primary visual cortex of alert monkeys," Neuron, vol. 22, no. 3, pp. 593-604, Mar. 1999.
42. J. H. Reynolds, T. Pasternak, and R. Desimone, "Attention increases sensitivity of V4 neurons," Neuron, vol. 26, no. 3, pp. 703-714, June 2000.
43. J. M. Wolfe, K. R. Cave, and S. L. Franzel, "Guided search: An alternative to the feature integration model for visual search," J. Exp. Psychol. Hum. Percept. Perform., vol. 15, no. 3, pp. 419-33, 1989.
44. J. M. Wolfe, "Visual memory: What do you know about what you saw?," Curr. Biol., vol. 8, no. 9, pp. R303-4, 1998.
45. M. K. Tanenhaus, M. J. Spivey-Knowlton, K. M. Eberhard, and J. C. Sedivy, "Integration of visual and linguistic information in spoken language comprehension," Science, vol. 268, no. 5217, pp. 1632-1634, June 1995.
46. C. F. Nodine and E. A. Krupinski, "Perceptual skill, radiology expertise, and visual test performance with nina and waldo," Acad. Radiol., vol. 5, no. 9, pp. 603-612, Sept. 1998.
47. F. J. Moreno, R. Reina, V. Luis, and R. Sabido, "Visual search strategies in experienced and inexperienced gymnastic coaches," Percept. Mot. Skills, pt. 1, vol. 95, no. 3, pp. 901-902, Dec. 2002.
48. G. J. Savelsbergh, A. M. Williams, J. V. der Kamp, and P. Ward, "Visual search, anticipation and expertise in soccer goalkeepers," J. Sports Sci., vol. 20, no. 3, pp. 279-287, Mar. 2002.
49. G. E. Legge, T. A. Hooven, T. S. Klitz, J. S. S. Mansfield, and B. S. Tjan, "Mr. Chips 2002: New insights from an ideal-observer model of reading," Vis. Res., vol. 42, no. 18, pp. 2219-2234, Aug. 2002.
50. D. Peebles and P. C. Cheng, "Modeling the effect of task and graphical representation on response latency in a graph reading task," Human Factors, vol. 45, no. 1, pp. 28-46, 2003.
51. A. Hollingworth, C. C. Williams, and J. M. Henderson, "To see and remember: Visually specific information is retained in memory from previously attended objects in natural scenes," Psychon. Bull. Rev., vol. 8, no. 4, pp. 761-768, Dec. 2001.
52. J. M. Henderson and A. Hollingworth, "Global transsaccadic change blindness during scene perception," Psychol. Sci., vol. 14, no. 5, pp. 493-497, Sept. 2003.
53. D. Noton and L. Stark, "Scanpaths in eye movements during pattern perception," Science, vol. 171, no. 968, pp. 308-11, 1971.
54. M. Carreiras and C. Clifton, "Another word on parsing relative clauses: Eyetracking evidence from spanish and english," Mem. Cognit., vol. 27, no. 5, pp. 826-833, Sept. 1999.
55. L. Itti, C. Koch, and E. Niebur, "A model of saliency-based visual attention for rapid scene analysis," IEEE Trans. Pattern Anal. Machine Intell., vol. 20, pp. 1254-1259, Nov. 1998.
56. L. Itti and C. Koch, "A saliency-based search mechanism for overt and covert shifts of visual attention," Vision Res., vol. 40, no. 10-12, pp. 1489-1506, May 2000.
57. L. Itti, "Feature combination strategies for saliency-based visual attention systems," J. Electron. Imag., vol. 10, no. 1, pp. 161-169, Jan. 2001.
58. C. Koch and S. Ullman, "Shifts in selective visual attention: Toward the underlying neural circuitry," Human Neurobiol., vol. 4, no. 4, pp. 219-27, 1985.
59. S. W. Kuffler, "Discharge patterns and functional prganization of mammalian retina," J. Neurophysiol., vol. 16, pp. 37-68, 1953.
60. D. H. Hubel and T. N. Wiesel, "Receptive fields, binocular interaction and functional architecture in the cat's visual cortex," J. Physiol. (London), vol. 160, pp. 106-54, 1962.
61. L. Itti, N. Dhavale, and F. Pighin, "Realistic avatar eye and head animation using a neurobiological model of visual attention," Proc. SPIE, vol. 5200, pp. 64-78, Aug. 2003.
62. A. M. Treisman and G. Gelade, "A feature-integration theory of attention," Cognit. Psychol., vol. 12, no. 1, pp. 97-136, 1980.
63. G. Borgefors, "Distance transformations in digital images," CVGIP: Image Understanding, vol. 54, no. 2, p. 301, 1991.
64. Online. Available: http://iLab.usc.edu/toolkit/
65. D.L. Sparks, "The brainstem control of saccadic eye movements," Nat. Rev. Neurosci., vol. 3, no. 12, pp. 952-964, 2002.
66. S. Maskell, M. Rollason, D. Salmond, and N. Gordon, "Efficient particle filtering for multiple target tracking with application to tracking in structured images," Proc. SPIE, vol. 4728, 2002.
67. D. Walther, L. Itti, M. Riesenhuber, T. Poggio, and C. Koch, "Attentional selection for object recognition - A gentle way," Lecture Notes Comput. Sci., vol. 2525, pp. 472-479, Nov. 2002.
68. Online. Available: http://bmrc.berkeley.edu/research/mpeg/mpegcode.html
69. Online. Available: http://www.mplayerhq.hu
70. Online. Available: http://Ilab.usc.edu/bu/co/
71. D. M. Stampe, "Heuristic filtering and reliable calibration methods for video based pupil tracking systems," Behavior Res. Meth., Instrum. Comput., vol. 25, no. 2, pp. 137-142, 1993.
72. S. A. Finney, "Real-time data collection in linux: A case study," Behavior Res. Meth., Instrum. Comput., vol. 33, pp. 167-173, 2001.
73. F. L. Bookstein, "Principal warps: Thin-plate splines and the decomposition of deformations," IEEE Trans. Pattern Anal. Machine Intell., vol. 11, pp. 567-585, June 1989.
74. V. Navalpakkam and L. Itti, "A goal oriented attention guidance model," Lecture Notes Comput. Sci., vol. 2525, pp. 453-461, Nov.