Combining sensory information: Mandatory fusion within, but not between, senses

Science

Volumn 298, Issue 5598, 2002, Pages 1627-1630

  Hillis, J H ^a,d   Ernst, M O ^b   Banks, M S ^a   Landy, M S ^c  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b MAX PLANCK INSTITUTE FOR BIOLOGICAL CYBERNETICS   (Germany)

^c NEW YORK UNIVERSITY   (United States)

^d UNIVERSITY OF PENNSYLVANIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DATA PROCESSING; OBJECT RECOGNITION; VISION;

SENSORY INFORMATION;

SENSORY PERCEPTION;

SENSORY SYSTEM;

ACCOMMODATION; ARTICLE; ASSOCIATION; DISCRIMINATION LEARNING; EYE; PRIORITY JOURNAL; PROPRIOCEPTION; SENSORY STIMULATION; TACTILE DISCRIMINATION; VISION;

CUES; FORM PERCEPTION; HUMANS; MATHEMATICS; STEREOGNOSIS; TOUCH; VISION DISPARITY; VISUAL PERCEPTION;

EID: 2242435339     PISSN: 00368075     EISSN: None     Source Type: Journal    
DOI: 10.1126/science.1075396     Document Type: Article

References (20)

1. Our interest is in the manner in which the nervous system resolves discrepancies at any given moment between sensory measurements. We assume the sensory systems under examination are well calibrated, so their signals will, on average, agree with one another. They will, however, disagree from one measurement to the next due to random measurement error.
2. J. J. Clark, A. L. Yuille, Data Fusion for Sensory Information Systems (Kluwer Academic, Boston, MA, 1990).
3. M. S. Landy, L. T. Maloney, E. B. Johnston, M. J. Young, Vision Res. 35, 389 (199S).
4. Z. Ghahramani, D. M. Wolpert, M. I. Jordan, in Self-Organization, Computational Maps, and Motor Control, P. G. Morasso, V. Sanguineti, Eds. (Elsevier, Amsterdam, 1997) pp. 117-147.
5. M. O. Ernst, M. S. Banks, Nature 451, 429 (2002).
6. M. S. Landy, H. Kojima, J. Opt. Soc. Am. 18, 2307 (2001).
7. Metamers are composite stimuli that cannot be discriminated even though their constituents can be. The classic example is the inability to discriminate a yellow light consisting of one wavelength from another yellow light consisting of red added to green.
8. W. Richards, Sens. Processes 3, 207 (1979).
9. B. T. Backus, in Advances in Neural Information Processing Systems, vol. 14, T. G. Dietterich, S. Becker, Z. Ghahramani, Eds. (MIT, Cambridge, MA, 2002) part VII, chap. 1.
10. E. B. Johnston, B. G. Cumming, M. S. Landy, Vision Res. 34, 2259 (1994).
11. B. T. Backus, M. S. Banks, R. van Ee, J. A. Crowell, Vision Res. 39, 1143 (1999).
12. E. Brenner, W. J. M. van Damme, Vision Res. 39, 975 (1999).
13. With two independent estimators, there are two chances for discriminating the odd stimulus. For each value of the comparison, each estimator has a likelihood of discriminating the comparison from the standard. Complete predictions for independent estimators would therefore include probability summation: P(estimator 1 discrimates odd) + P(estimator 2 discrimates odd) - P(both discriminate odd). Predicted thresholds that include probability summation would be a rectangle like the one in Fig. 1B, but with rounded corners. Inclusion of probability summation does not affect our interpretation of the data, because probability summation would not produce the observed asymmetries between thresholds in the four quadrants (Figs. 2 and 3).
14. From among the large set of possible tasks and cues, we have examined a small subset. We chose tasks that were natural for participants to perform and pairs of cues that were approximately equally reliable to better show the gains and losses predicted by MLE.
15. Material and methods are available as supporting online material on Science Online.
16. The participants' phenomenology was instructive. They reported using a difference in perceived size when the comparison stimulus was in the cues-consistent quadrants (1 and 3). This percept is well modeled by the equations for combined estimation (Eqs. 1 and 2). Participants' reports were less consistent with stimuli in the cues-inconsistent quadrants (2 and 4). Sometimes they used a difference in perceived size, but frequently they noticed the conflict between the visually and haptically specified sizes and used the perceived conflict to make the oddity discrimination. The phenomenology is consistent with the hypothesis that participants used three estimators in performing the oddity discrimination: two single-cue estimators and a combined estimator.
17. D. C. Knill, Vision Res. 38, 1683 (1998).
18. M. de Berg, M. van Kreveld, M. Overmars, O. Schwarzkopf, Computational Geometry: Algorithms and Applications (Springer-Verlag, New York, ed. 2, 2000).
19. d); the slant cues were truly fused. They made the discrimination in the cues-inconsistent quadrants by use of another calculation - shape constancy - that allowed them to perceive the objective shape of the texture on the surface. If this hypothesis is correct, metamers (cases in which discrimination is poorer than predicted by single-cue estimates) occurred when the stimulus made the shape-constancy judgment difficult.
20. We thank S. Watt, S. Gepshtein, and B. Backus for comments. This work was supported by the Max-Planck Society and by research grants from Air Force Office of Scientific Research (F49620-98), NIH (EY08266 and EY12851), NSF (DBS-9309820) and by an equipment grant from Silicon Graphics.