Attention via synchrony: Making use of multimodal cues in social learning

IEEE Transactions on Autonomous Mental Development

Volumn 1, Issue 1, 2009, Pages 55-67

  Rolf, Matthias ^a   Hanheide, Marc ^a   Rohlfing, Katharina J ^a  

^a BIELEFELD UNIVERSITY   (Germany)

Author keywords

Attention; Infant directed communication; Multimodality; Social learning; Synchrony

Indexed keywords

ATTENTION; INFANT-DIRECTED COMMUNICATION; MULTIMODALITY; SOCIAL LEARNING; SYNCHRONY;

EDUCATION;

EID: 77955681043     PISSN: 19430604     EISSN: None     Source Type: Journal    
DOI: 10.1109/TAMD.2009.2021091     Document Type: Article

References (46)

1. W. Quine, Word and Object. Cambridge, MA: MIT Press, 1960.
2. M. Tomasello, "Could we please lose the mapping metaphor, please?," Behav. Brain Sci., vol. 24, no. 6, pp. 1119-1120, 2001.
3. C. Sinha, "Grounding, mapping and acts of meaning," in Cognitive Linguistics: Foundations, Scope and Methodology, T. Janssen and G. Re-deker, Eds. Berlin, Germany: Mouton deGruyter, 1999, pp. 223-255.
4. P. Zukow-Goldring, "Socio-perceptual bases for the emergence of language: An alternative to innatist approaches," Develop. Psychobiol., vol. 23, no. 7, pp. 705-726, 1990.
5. P. Zukow-Goldring, "Assisted imitation: Affordances, effectivities, and the mirror system in early language development," in From Action to Language, M. A. Arbib, Ed. Cambridge, U.K.: Cambridge Univ. Press, 2006, pp. 469-500.
6. L. Gogate, L. Bahrick, and J. Watson, "A study of multimodal motherese: The role of temporal synchrony between verbal labels and gestures," Child Develop., vol. 71, no. 4, pp. 878-894, Jul./Aug. 2000.
7. L. J. Gogate and L. E. Bahrick, "Intersensory redundancy and 7-month-old infants' memory for arbitrary syllable-object relations," Infancy, vol. 2, no. 2, pp. 219-231, 2001.
8. L. J. Gogate, A. S. Walker-Andrews, and L. E. Bahrick, "The intersen-sory origins of word-comprehension: An ecological-dynamic systems view," Develop. Sci., vol. 4, no. 1, pp. 1-18, 2001.
9. L. Bahrick, R. Lickliter, and R. Flom, "Intersensory redundancy guides the development of selective attention, perception, and cognition in infancy," Current Direct. Psychol. Sci., 2004.
10. J. Driver and C. Spence, "Multisensory perception: Beyond modularity and convergence,"Current Biol., vol. 10, no. 20, pp. R731-R735, 2000.
11. J. Vroomen and B. de Gelder, "Sound enhances visual perception: Cross-modal effects of auditory organization on vision," J. Exper. Psychol. Human Percept. Perform., vol. 26, no. 5, pp. 1583-1590, 2000.
12. C. Binnie, A. Montgomery, and P. Jackson, "Auditory and visual contributions to the perception of consonants," J. Speech, Lang., Hearing Res., vol. 17, no. 4, pp. 619-630, 1974.
13. B. Dodd, "The role of vision in the perception of speech," Perception, vol. 6, no. 1, pp. 31-40, 1977.
14. M. J. Mendelson and M. M. Haith, "The relation between audition and vision in the human newborn," Mono. Soc. Res. Child Develop., vol. 41, no. 4, 1976.
15. P. Kaplan, K. Fox, D. Scheuneman, and L. Jenkins, "Cross-modal facilitation of infant visual fixation: Temporal and intensity effects," Infant Behavior Develop., vol. 14, no. 1, pp. 83-109, 1991.
16. A. Meltzoff and P. Kuhl, "Faces and speech: Intermodal processing of biologically relevant signals in infants and adults," in The Development of Intersensory Perception: Comparative Perspectives, D. Lewkowicz and R. Lickliter, Eds. Philadelphia, PA: Lawrence Erlbaum, 1994.
17. P. Zukow-Goldring, Rader, and N. de Villiers, "Caregiver gestures cultivate a shared understanding: Assisted imitation and early word learning," in Intermodal Action Struct. Conf., Bielefeld, Germany, Jul. 2008.
18. Y. Nagai and K. Rohlfing, "Can motionese tells infants and robots, "Whatto imitate"?," in Proc. 4th Int. Symp. Imitation Animals Artifacts, Apr. 2007.
19. K. Rohlfing, J. Fritsch, B. Wrede, and T. Jungmann, "How can multimodal cues from child-directed interaction reduce learning complexity in robots?," Adv. Robot., vol. 20, no. 10, pp. 1183-1199, 2006.
20. J. M. Wolfe and T. S. Horowitz, "What attributes guide the deployment of visual attention and how do they do it?," Nature Rev. Neurosc., vol. 5, pp. 495-501, 2004.
21. D. J. Lewkowicz, "The development of intersensory temporal perception : An epigenetic systems/limitations view," Psychol. Bull., vol. 126, no. 2, pp. 281-308, 2000.
22. C. Spence and S. Squire, "Multisensory integration: Maintaining the perception of synchrony," Current Biol., vol. 13, no. 13, pp. R519-R521, 2003.
23. J. Hershey and J. Movellan, "Audio-vision: Using audio-visual synchrony to locate sounds," Adv. Neural Inf. Process. Syst., vol. 12, pp. 813-819, 2000.
24. D. Matatyaho, Z. Mason, and L. Gogate, "Word learning by eight-month-old infants: The role of object motion and synchrony," in Proc. 7th Int. Conf. Epigenetic Robot., 2007.
25. V. Virsu, H. Oksanen-Hennah, A. Vedenpää, P. Jaatinen, and P. Lahti-Nuuttila, "Simultaneity learning in vision, audition, tactile sense and their cross-modal combinations," Exper. Brain Res., vol. 186, no. 4, pp. 525-537, Apr. 2008.
26. A. Treisman and G. Gelade, "A feature-integration theory of attention," Cogn. Psychol., vol. 12, pp. 97-136, 1980.
27. G. Monaci and P. Vandergheynst, "Audiovisual gestalts," in Proc. 2006 Conf. Comput. Vision Pattern Recognit. Workshop, 2006.
28. H. Bredin and G. Chollet, "Measuring audio and visual speech synchrony: Methods and applications," in Proc. IET Int. Conf. Visual Inf. Eng., 2006, pp. 255-260.
29. C. G. Prince, G. J. Hollich, N. A. Helder, E. J. Mislivec, A. Reddy, S. Salunke, and N. Memon, "Taking synchrony seriously: A perceptual-level model of infant synchrony detection," in Proc. 4th Int. Workshop Epigenetic Robot., 2004, pp. 89-96.
30. M. Slaney and M. Covell, "FaceSync: A linear operator for measuring synchronization of video facial images and audio tracks," in Proc. NIPS, 2000, pp. 814-820.
31. E. Kidron, Y. Schechner, and M. Elad, "Cross-modal localization via sparsity," IEEE Trans. Signal Process., vol. 55, pp. 1390-1404, 2005.
32. C. Chibelushi, F. Deravi, and J. Mason, "A review of speech-based bimodal recognition," IEEE Trans. Multimedia, vol. 4, pp. 23-37, Mar. 2002.
33. L. Itti, C. Koch, and E. Niebur, "A model of saliency-based visual attention for rapid scene analysis," IEEE Trans. Pattern Anal. Machine Intell., vol. 20, no. 11, pp. 1254-1259, 1998.
34. M. I. Posner, "Orienting of attention," Quart. J. Exper. Psychol., vol. 32, no. 1, pp. 3-25, 1980.
35. R. J. Brand, D. A. Baldwin, and L. A. Ashburn, "Evidence for 'motionese': Modifications in mothers' infant-directed action," Develop. Sci., vol. 5, no. 1, pp. 72-83, 2002.
36. P. F. Dominey and C. Dodane, "Indeterminacy in language acquisition: The role of child directed speech and joint attention," J. Neurolinguist., vol. 17, no. 2-3, pp. 121-145, 2004.
37. R. Cooper, J. Abraham, S. Berman, and M. Staska, Infant Behavior Develop., vol. 20, no. 4, pp. 477-488, 1997.
38. A. Cangelosi, T. Belpaeme, G. Sandini, G. Metta, L. Fadiga, G. Sagerer, K. Rohlfing, B. Wrede, S. Nolfi, D. Parisi, C. Nehaniv, K. Dautenhahn, J. Saunders, K. Fischer, J. Tani, and D. Roy, "The iTalk project: Integration and transfer of action and language knowledge," in Proc. 3rd ACM/IEEE Int. Conf. Human Robot Interaction (HRI 2008), Amsterdam, The Netherlands, Mar. 2008.
39. D. A. Baldwin, "Joint attention: Its origins and role in development," in Understanding the Link Between Joint Attention and Language. Philadelphia, PA: Lawrence Erlbaum, 1995, pp. 131-158.
40. F. Kaplan and V. V. Hafner, "The challenges of joint attention," Interact. Studies, vol. 7, no. 35, pp. 135-169, 2006.
41. L. Itti and C. Koch, "Computational modelling of visual attention," Nature Rev. Neurosci., vol. 3, no. 2, pp. 194-203, Mar. 2001.
42. J. M. Wolfe, "Guided search 2.0: A revised model of visual search," Psychon. Bull. Rev., vol. 1, no. 2, pp. 202-238, 1994.
43. B. Wrede, K. J. Rohlfing, M. Hanheide, and G. Sagerer, "Towards learning by interacting," in Creating Brain-Like Intelligence, B. Send-hoff, E. Koerner, O. Sporns, H. Ritter, and K. Doya, Eds. Berlin, Heidelberg, Germany: Springer, 2009, pp. 139-150.
44. A. Lockerd and C. Breazeal, "Tutelage and socially guided robot learning," in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS 2004), 2004, vol. 4, pp. 3475-3480.
45. J. Peltason, F. H. Siepmann, B. W. Thorsten, P. Spexard, M. Hanheide, and E. A. Topp, "Mixed-initiative in human augmented mapping," in Proc. Int. Conf. Robot. Automat., 2009.
46. P. G. Zukov, "Socio-perceptual basis for the emergence of language: An alternative to innatist approaches," Develop. Psychobiol., vol. 23, no. 7, pp. 705-726, 1990.