Semiotic schemas: A framework for grounding language in action and perception

Artificial Intelligence

Volumn 167, Issue 1-2, 2005, Pages 170-205

  Roy, Deb ^a  

^a MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

Action; Cross modal; Embodied; Grounding; Language; Meaning; Multimodal; Perception; Representation; Schemas; Semiotic; Situated

Indexed keywords

ARTIFICIAL INTELLIGENCE; LEARNING SYSTEMS; ROBOTS; SENSORY PERCEPTION;

ACTION; CROSS-MODAL; EMBODID; GROUNDING; LANGUAGE; MEANING; MULTIMODAL; REPRESENTATION; SCHEMAS; SITUATED;

SEMIOTICS;

EID: 24144445149     PISSN: 00043702     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.artint.2005.04.007     Document Type: Article

References (78)

1. J. Allen Maintaining knowledge about temporal intervals Comm. ACM 26 1983 832 843
2. M.A. Arbib Schema theory M.A. Arbib The Handbook of Brain Theory and Neural Networks second ed. 2003 MIT Press Cambridge, MA 993 998
3. M.A. Arbib, T. Iberall, and D. Lyons Schemas that integrate vision and touch for hand control Vision, Brain, and Cooperative Computation 1987 MIT Press Cambridge, MA 489 510
4. D. Bailey, J. Feldman, S. Narayanan, and G. Lakoff Embodied lexical development Proceedings of the Nineteenth Annual Meeting of the Cognitive Science Society 1997 Erlbaum Mahwah, NJ
5. D. Bailey, When push comes to shove: A computational model of the role of motor control in the acquisition of action verbs, PhD thesis, Computer science division, EECS Department, University of California at Berkeley, 1997
6. K. Barnard, P. Duygulu, D. Forsyth, N. de Freitas, D. Blei, and M. Jordan Matching words and pictures J. Machine Learning Res. 3 2003 1107 1135
7. L. Barsalou Perceptual symbol systems Behavioural and Brain Sciences 22 1999 577 609
8. J. Barwise, and J. Perry Situations and Attitudes 1983 MIT-Bradford
9. E. Bates The Emergence of Symbols 1979 Academic Press New York
10. M.H. Bickhard Representational content in humans and machines J. Experimental Theoret. Artificial Intelligence 5 1993 285 333
11. A. Cangelosi, K. Coventry, R. Rajapakse, D. Joyce, A. Bacon, L. Richards, S. Newstead, Grounding language in perception: A connectionist model of spatial terms and vague quantifiers, in: Ninth Neural Computation and Psychology Workshop, Plymouth, UK, 2004
12. A. Cangelosi, S. Harnad, The adaptive advantage of symbolic theft over sensorimotor toil: Grounding language in perceptual categories, Evolution of Communication, 2000, 2001
13. J. Coelho, J. Piaterand, and R. Grupen Developing haptic and visual perceptual categories for reaching and grasping with a humanoid robot Proceedings of the First IEEE-RAS International Conference on Humanoid Robots 2000 IEEE-RAS
14. P. Cohen, Projections as concepts, in: Proceedings of the 2nd European Conference on Cognitive Science, 1997, pp. 56-60
15. P. Cohen, T. Oates, C. Beal, and N. Adams, Contentful mental states for robot baby, in: Proceedings of the Eighteenth National Conference on Artificial Intelligence, Edmonton, AB, 2002
16. K. Coventry, and S. Garrod Saying, Seeing and Acting 2004 Psychology Press
17. D. Dennett Consciousness Explained 1991 Little, Brown and Company
18. D. Dennett Real patterns J. Philos. 88 1 1991 27 51
19. G. Drescher Made-up Minds 1991 MIT Press Cambridge, MA
20. F.I. Dretske Knowledge and the Flow of Information 1981 MIT Press Cambridge, MA
21. J. Feldman, G. Lakoff, D. Bailey, S. Narayanan, T. Regier, and A. Stolcke Lzero: The first five years Artificial Intelligence Rev. 10 1996 103 129
22. P. Gärdenfors Conceptual Spaces: The Geometry of Thought 2000 MIT Press Cambridge, MA
23. J.J. Gibson The Ecological Approach to Visual Perception 1979 Erlbaum Mahwah, NJ
24. A. Glenberg, and M. Kaschak Grounding language in action Phychonomic Bull. Rev. 9 3 2002 558 565
25. P. Gorniak, and D. Roy Grounded semantic composition for visual scenes J. Artificial Intelligence Res. 21 2004 429 470
26. P. Grice Logic and conversation P. Cole J. Morgan Syntax and Semantics: vol. 3, Speech Act 1975 Academic Press New York 43 58
27. S. Harnad The symbol grounding problem Physica D 42 1990 335 346
28. A. Herskovits Language, spatial cognition, and vision Spatial and Temporal Reasoning 1997 Kluwer Academic Dordrecht Chapter 6
29. G. Herzog, and P. Wazinski VIsual TRAnslator: Linking perceptions and natural language descriptions Artificial Intelligence Rev. 8 1994 175 187
30. D. Isla and B. Blumberg, Object persistence for synthetic creatures, in: Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, 2002
31. R. Jackendoff Foundations of Language 2002 Oxford University Press Oxford
32. P.N. Johnson-Laird Mental Models: Towards a Cognitive Science of Language, Inference, and Consciousness 1983 Cambridge University Press Cambridge
33. J. Juster, D. Roy, Elvis: Situated speech and gesture understanding for a robotic chandilier, in: Proceedings of the International Conference on Multimodal Interfaces, 2004
34. I. Kant Critique of Pure Reason Section A137 1781 Cambridge University Press Cambridge 1998
35. B. Kuipers The spatial semantic hierarchy Artificial Intelligence 119 2000 191 233
36. G. Lakoff, and M. Johonson Metaphors We Live By 1980 University of Chicago Press Chicago
37. D. Lenat Cyc: A large-scale investment in knowledge infrastructure Comm. ACM 38 11 1995 33 38
38. H.J. Levesque Making believers out of computers Artificial Intelligence 30 1 1986 81 108
39. M. Littman, R. Sutton, and S. Singh Predictive representations of state Advances in Neural Information Processing Systems 2002 MIT Press Cambridge, MA
40. G. Miller Wordnet: A lexical database for English Comm. ACM 38 11 1995 39 41
41. G. Miller, and P. Johnson-Laird Language and Perception 1976 Harvard University Press Harvard
42. R.G. Millikan Varieties of Meaning 2004 MIT Press Cambridge, MA
43. M. Minsky A framework for representing knowledge P. Winston The Psychology of Computer Vision 1975 McGraw-Hill New York 211 277
44. S. Narayanan, Moving right along: A computational model of metaphoric reasoning about events, in: Proceedings of the National Conference on Artificial Intelligence AAAI-99, Orlando, FL, 1999
45. U. Neisser Cognition and Reality 1976 Freeman New York
46. K. Nelson Concept, word, and sentence: Interrelations in acquisition and development Psychological Rev. 81 1974 267 285
47. C.K. Odgen, and I.A. Richards The Meaning of Meaning 1923 Harcourt
48. C.S. Peirce How to make our ideas clear Popular Science Monthly 12 1878 286 302
49. C.S. Peirce Logic as semiotic: The theory of signs Philosophical Writings of Peirce 1897 Dover 1940
50. J. Piaget The Construction of Reality in the Child 1954 Ballentine
51. K. Plunkett, C. Sinha, M.F. Moller, and O. Strandsby Symbol grounding or the emergence of symbols? vocabulary growth in children and a connectionist net Connection Sci. 4 3-4 1992 293 312
52. K. Hsiao, N. Mavridis, D. Roy, Coupling perception and simulation: Steps towards conversational robotics, in: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, Las Vegas, 2003
53. D. McDermott Artificial intelligence meets natural stupidity J. Haugeland Mind Design 1981 MIT Press Cambridge, MA 143 160 Chapter 5
54. H. Putnam The meaning of 'meaning' Philosophical Papers: vol. 2. Mind, Language and Reality 1975 Cambridge University Press Cambridge 215 271
55. D.A. Randell, Z. Cui, and A.G. Cohn A spatial logic based on regions and connection Proceedings 3rd International Conference on Knowledge Representation and Reasoning 1992 Morgan Kaufmann San Mateo, CA 165 176
56. T. Regier The Human Semantic Potential 1996 MIT Press Cambridge, MA
57. T. Regier, and L. Carlson Grounding spatial language in perception: An empirical and computational investigation J. Experimental Psychol. 130 2 2001 273 298
58. E. Reiter, and S. Sripada Human variation and lexical choice Computational Linguistics 22 2002 545 553
59. A. Rosenblueth, N. Wiener, and J. Bigelow Behavior, purpose and teleology Philos. Sci. 10 1943 18 24
60. D. Roy, Learning words from sights and sounds: A computational model, PhD thesis, Massachusetts Institute of Technology, 1999
61. D. Roy Learning visually-grounded words and syntax for a scene description task Computer Speech and Language 16 3 2002 353 385
62. D. Roy Grounding words in perception and action: Computational insights Trends Cognitive Sci. 9 8 2005
63. D. Roy, P. Gorniak, N. Mukherjee, J. Juster, A trainable spoken language understanding system for visual object selection, in: International Conference of Spoken Language Processing, 2002
64. D. Roy, K. Hsiao, and N. Mavridis Mental imagery for a conversational robot IEEE Trans. Systems Man Cybernet. B 34 3 2004 1374 1383
65. D. Roy, and N. Mukherjee Towards situated speech understanding: Visual context priming of language models Computer Speech and Language 19 2 2005 227 248
66. D. Roy, and A. Pentland Learning words from sights and sounds: A computational model Cognitive Sci. 26 1 2002 113 146
67. D. Rumelhart, and A. Ortony The representation of knowledge in memory R.C. Anderson R.J. Spiro W.E. Montague Schooling and the Acquisition of Knowledge 1977 Erlbaum Hillsdale, NJ 99 136
68. R. Schank, and R. Abelson Scripts, Plans, Goals and Understanding 1977 Erlbaum Hilsdale, NJ
69. J. Siskind, Naive physics, event perception, lexical semantics, and language acquisition, PhD thesis, Massachusetts Institute of Technology, 1992
70. J. Siskind Grounding the lexical semantics of verbs in visual perception using force dynamics and event logic J. Artificial Intelligence Res. 15 2001 31 90
71. A. Sloman, J. Chappell, The altricial-precocial spectrum for robots, in: Proceedings of IJCAI, 2005
72. B.C. Smith On the Origin of Objects 1996 MIT Press Cambridge, MA
73. M. Spivey, M. Tyler, D. Richardson, and E. Young Eye movements during comprehension of spoken scene descriptions Proceedings of the 22nd Annual Conference of the Cognitive Science Society 2000 Erlbaum Mahwah, NJ 487 492
74. L. Steels, and P. Vogt Grounding adaptive language games in robotic agents C. Husbands I. Harvey Proceedings of the 4th European Conference on Artificial Life 1997 MIT Press Cambridge, MA
75. B.E. Stein, and M.A. Meredith The Merging of the Senses 1993 MIT Press Cambridge, MA
76. T. Winograd A Process Model of Language Understanding 1973 Freeman New York pp. 152-186
77. T. Winograd, and F. Flores Understanding Computers and Cognition 1986 Addison-Wesley Reading, MA
78. L. Wittgenstein The Blue and Brown Books 1958 Basil Blackwell Oxford