A Multimodal Learning Interface for Grounding Spoken Language in Sensory Perceptions

ACM Transactions on Applied Perception

Volumn 1, Issue 1, 2004, Pages 57-80

  Yu, Chen ^a   Ballard, Dana H ^a  

^a University of Rochester   (United States)

Author keywords

Cognitive modeling; Experimentation; Human Factors; Multimodal interaction; Multimodal learning

Indexed keywords

EID: 84867456688     PISSN: 15443558     EISSN: 15443965     Source Type: Journal    
DOI: 10.1145/1008722.1008727     Document Type: Article

References (56)

1. Adams, R. and Bischof, L. 1994. Seeded region growing. IEEE Trans. Pattern Anal. Mach. Intell. 16, 6 (June), 641-647.
2. Bailey, D. 1997. When Push Comes to Shove: A Computational Model of the Role of Motor Control in the Acquisition of Action Verbs. Ph.D. thesis, Computer Science Division, University of California Berkeley.
3. Baldwin, D. A., Markman, E. M., Bill, B., Desjardins, R. N., Irwin, J. M., and Tidball, G. 1996. Infant's reliance on a social criterion for establishing word-object relations. Child Dev. 67, 3135-3153.
4. Ballard, D. H., Hayhoe, M. M., Pook, P. K., and Rao, R. P. N. 1997. Deictic codes for the embodiment of cognition. Behav. Brain Sci. 20, 1311-1328.
5. Ballard, D. H. and Yu, C. 2003. A multimodal learning interface for word acquisition. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing, Hong Kong.
6. Baron-Cohen, S. 1995. Mindblindness: An Essay on Autism and Theory of Mind. MIT Press, Cambridge, MA.
7. Bates, E. and Goodman, J. C. 1999. On the emergence of grammar from the lexicon. In The Emergence of Language, B. M. Whinney, Ed. Lawrence Erlbaum Associates, New Jersy
8. Bloom, P. 2000. How Children Learn the Meanings of Words. The MIT Press, Cambridge, MA.
9. Brent, M. R. 1999. Speech segmentation and word discovery: A computational perspective. Trends Cogn. Sci. 3, 8, 294-301.
10. Brent, M. R. and Cartwright, T. A. 1996. Distributional regularity and phonotactic constraints are useful for segmentation. Cognition 61, 93-125.
11. Brown, P. F., Pietra, S., Pietra, V., and Mercer, R. L. 1993. The mathematics of statistical machine translation: Parameter estimation. Comput. Linguist. 19, 2.
12. Campbell, L., Becker, D., Azarbayejani, A., Bobick, A., and Pentland, A. 1996. Invariant features for 3-D gesture recognition. In Second International Workshop on Face and Gesture Recognition, Killington, VT. 157-162.
13. Gardenfors, P. 1999. Some Tenets of Cognitive Semantics. John Benjamins.
14. Gillette, J., Gleitman, H., Gleitman, L., and Lederer, A. 1999. Human simulations of vocabulary learning. Cognition 73, 135-176.
15. Gopnik and Meltzoof. 1997. Words, Thought and Theories. MIT Press, Cambridge, MA.
16. Harnad, S. 1990. The symbol grounding problem. Physica D 42, 335-346.
17. Hartigan, J. 1975. Clustering Algorithms. Wiley, New York.
18. Hayhoe, M. 2000. Visual routines: A functional account of vision. Visual Cognition 7, 43-64.
19. Jusczyk, P. W. 1997. The Discovery of Spoken Language. MIT Press, Cambridge, MA.
20. Jusczyk, P. W. and Aslin, R. N. 1995. Infants detection of the sound patterns of words in fluent speech. Cogn. Psychol.
21. KEIL. 1989. Concepts, Kinds, and Cognitive Development. MIT Press, Cambridge, MA.
22. Kuniyoshi, Y. and Inoue, H. 1993. Qualitative recognition of ongoing human action sequences. In Proceedings of the IJCAI93, Chambery, France. 1600-1609.
23. Ladefoged, P. 1993. A Course in Phonetics. Harcourt Brace Jovanovich, Orlando, FL.
24. Lakoff, G. and Johnson, M. 1980. Metaphors We Live By. University of Chicago Press, Chicago.
25. Land, M., Mennie, N., and Rusted, J. 1999. The roles of vision and eye movements in the control of activities of daily living. Perception 28, 1311-1328.
26. Land, M. F. and Hayhoe, M. 2001. In what ways do eye movements contribute to everyday activities? Vision Research 41, 3559-3565.
27. Landau, B., Smith, L., and Jones, S. 1998. Object perception and object naming in early development. Trends Cogn. Sci.
28. Lippmann, R. P. 1989. Review of neural networks for speech recognition. Neural Comput. 1, 1, 1-38.
29. Mel, B. W. 1997. Seemore: Combining color, shape, and texture histogramming in a neurally inspired approach to visual object recognition. Neural Comput. 9, 777-804.
30. Meyer, A. S., Sleiderink, A. M., and Levelt, W. J. 1998. Viewing and naming objects: Eye movements during noun phrase production. Cognition 66, B25-B33.
31. Oates, T., Firoiu, L., and Cohen, P. R. 1999. Clustering time series with hidden Markov models and dynamic time warping. In Proceedings of the IJCAI-99 Workshop on Neural, Symbolic and Reinforcement Learning Methods for Sequence Learning, Stockholm, Sweden. 17-21.
32. Oviatt, S. 2002. Multimodal interfaces. In Handbook of Human-Computer Interaction, J. Jacko and A. Sears, Eds. Lawrence Erlbaum, New Jersey.
33. Quinn, P., Eimas, P., and Rosenkrantz, S. 1993. Evidence for representations of perceptually similar natural categories by 3-month old and 4-month old infants. Perception 22, 375-463.
34. Rabiner, L. R. and Juang, B. 1989. A tutorial on hidden Markov models and selected applications in speech recognition. Proc. IEEE 77, 2, 257-286.
35. Regier, T. 1996. The Human Semantic Potential: Spatial Language and Constrained Connectionism. MIT Press, Cambridge, MA.
36. Regier, T. 2003. Emergent constraints on word-learning: A computational review. Trends Cogn. Sci. 7, 263-268.
37. Robinson, T. 1994. An application of recurrent nets to phone probability estimation. IEEE Trans. NeuralNetw. 5, 2, 298-305.
38. Roy, D. 2002. Learning visually grounded words and syntax of natural spoken language. Evol. Commun. 4, 1.
39. Roy, D. and Pentland, A. 2002. Learning words from sights and sounds: A computational model. Cogn. Sci. 26, 1, 113-146.
40. Salvucci, D. D. and Anderson, J. 1998. Tracking eye movement protocols with cognitive process models. In Proceedings of the Twentieth Annual Conference of the Cognitive Science Society. LEA, Mahwah, NJ, 923-928.
41. Schiele, B. and Crowley, J. L. 2000. Recognition without correspondence using multidimensional receptive field histograms. Int. J. Comput. Vis. 36, 1, 31-50.
42. Schyns, P. and Rodet, L. 1997. Categorization creates functional features. J. Exp. Psychol. Learn. Mem. Cogn. 23, 681-696.
43. Siskind, J. M. 1995. Grounding language in perception. Artif. Intell. Rev. 8, 371-391.
44. Siskind, J. M. 2001. Grounding the lexical semantics of verbs in visual perception using force dynamics and event logic. J. Artif. Intell. Res. 15, 31-90.
45. Smith, L., Jones, S., and Landau, B. 1996. Naming in young children: A dumb attentional mechanism? Congition 60, 2.
46. Smyth, P. 1997. Clustering sequences with hidden markov models. In Advances in Neural Information Processing Systems, M. C. Mozer, M. I. Jordan, and T. Petsche, Eds. MIT Press, Cambridge, MA, 648.
47. Steels, L. and Vogt, P. 1997. Grounding adaptive language game in robotic agents. In Proceedings of the 4th European Conference on Artificial Life, C. Husbands and I. Harvey, Eds. MIT Press, London.
48. Swain, M. J. and Ballard, D. 1991. Color indexing. Int. J. Comput. Vis. 7, 11-32.
49. Tomasello, M. 2000. Perceiving intentions and learning words in the second year of life. In Language Acquisition and Conceptual Development, M. Bowerman and S. Levinson, Eds. Cambridge University Press.
50. Wang, S. and Siskind, J. M. 2003. Image segmentation with ratio cut. IEEE Trans. Pattern Anal. Mach. Intell. 25, 6, 675-690.
51. Weng, J., McClelland, J., Pentland, A., Sporns, O., Stockman, I., Sur, M., and Thelen, E. 2001. Artificial intelligence: Autonomous mental development by robots and animals. Science 291, 5504, 599-600.
52. Yu, C. and Ballard, D. H. 2002a. Learning to recognize human action sequences. In IEEE Proceedings of the 2nd International Conference on Development and Learning, Boston, MA. 28-34.
53. Yu, C. and Ballard, D. H. 2002b. Understanding human bahaviors based on eye-head-hand coordination. In 2nd Workshop on Biologically Motivated Computer Vision, Tubingen, Germany.
54. Yu, C. and Ballard, D. H. 2003. Exploring the role of attention in modeling embodied language acquisition. In Proceedings of the Fifth International Conference on Cognitive Modeling, Bamburg, Germany.
55. Yu, C., Ballard, D. H., and Aslin, R. N. 2003. The role of embodied intention in early lexical acquisition. In Proceedings the Twenty Fifth Cognitive Science Society Annual Meetings, Boston, MA.
56. Yu, C., Ballard, D. H., and Zhu, S. 2002. Attentional object spotting by integrating multisensory input. In Proceedings of the 4th International Conference on Multimodal Interface, Pittsburg, PA.