Language as a multimodal phenomenon: Implications for language learning, processing and evolution

Philosophical Transactions of the Royal Society B: Biological Sciences

Volumn 369, Issue 1651, 2014, Pages

  Vigliocco, Gabriella ^a,b   Perniss, Pamela ^a,b   Vinson, David ^a  

^a Research Department of Cognitive Perceptual and Brain Sciences   (United Kingdom)

^b Deafness   (United Kingdom)

Author keywords

Iconicity; Language development; Language evolution; Language processing; Multimodality of language

Indexed keywords

EID: 84905227007     PISSN: 09628436     EISSN: 14712970     Source Type: Journal    
DOI: 10.1098/rstb.2013.0292     Document Type: Editorial

References (62)

1. Barsalou LW, Simmons WK, Barbey AK, Wilson CD. 2003 Grounding conceptual knowledge in modalityspecific systems. Trends Cogn. Sci. 7, 84-91. (doi: 10. 1016/S1364-6613(02)00029-3).
2. Manusov V, Patterson ML (eds). 2006 The Sage handbook of nonverbal communication. Thousand Oaks, CA: Sage Publications.
3. McGurk H, MacDonald J. 1976 Hearing lips and seeing voices. Nature 264, 746-748. (doi: 10. 1038/264746a0).
4. Rowe M, Goldin-Meadow S. 2009 Differences in early gesture explain SES disparities in child vocabulary size at school entry. Science 323, 951-953. (doi: 10. 1126/science. 1167025).
5. Özçali{dotless}şkan Ş, Goldin-Meadow S. 2005 Do parents lead their children by the hand? J. Child Lang. 32, 481-505. (doi: 10. 1017/S0305000905007002).
6. Church RB, Goldin-Meadow S. 1986 The mismatch between gesture and speech as an index of transitional knowledge. Cognition 23, 43-71. (doi: 10. 1016/0010-0277(86)90053-3).
7. Goldin-Meadow S, Cook SW, Mitchell ZA. 2009 Gesturing gives children new ideas about math. Psychol. Sci. 20, 267-272. (doi: 10. 1111/j. 1467-9280. 2009. 02297. x).
8. Kita S, Özyürek A. 2003 What does cross-linguistic variation in semantic co-ordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking. J. Mem. Lang. 48, 16-32. (doi: 10. 1016/S0749-596X(02)00505-3).
9. Kelly SD, Özyürek A, Maris E. 2010 Two sides of the same coin: speech and gesture mutually interact to enhance comprehension. Psychol. Sci. 21, 260-267. (doi: 10. 1177/0956797609357327).
10. Kutas M, Federmeier KD. 2011 Thirty years and counting: finding meaning in the N400 component of the event-related brain potential (ERP). Ann. Rev. Psychol. 62, 621-647. (doi: 10. 1146/annurev. psych. 093008. 131123).
11. Wu YC, Coulson S. 2005 Meaningful gestures: electrophysiological indices of iconic gesture comprehension. Psychophysiology 42, 654-667. (doi: 10. 1111/j. 1469-8986. 2005. 00356. x).
12. Sandler W. 2009 Symbiotic symbolization by hand and mouth in sign language. Semiotica 174, 241-275. (doi: 10. 1515/semi. 2009. 035).
13. Sandler W, Lillo-Martin D. 2006 Sign language and linguistic universals. Cambridge, UK: Cambridge University Press.
14. Zeshan U. 2004 Interrogative constructions in signed languages: crosslinguistic perspectives. Language 80, 7-39. (doi: 10. 1353/lan. 2004. 0050).
15. Pfau R. 2008 The grammar of headshake: a typological perspective on German sign language negation. Linguist. Amsterdam 1, 37-74.
16. Saussure F. de. 1916 Cours de linguistique générale. Paris, France: Payot.
17. Locke J. 1975 An essay concerning human understanding. Oxford, UK: Clarendon Press. Original work published 1689.
18. Levelt WJM, Roelofs A, Meyer AS. 1999 A theory of lexical access in speech production. Behav. Brain Sci. 22, 1-75. (doi: 10. 1017/S0140525X99001776).
19. Perniss P, Thompson RL, Vigliocco G. 2010 Iconicity as a general property of language: evidence from spoken and signed languages. Front. Psychol. 1, 227. (doi: 10. 3389/fpsyg. 2010. 00227).
20. Köhler W. 1929 Gestalt psychology. New York, NY: Liveright.
21. Maurer D, Pathman T, Mondloch CJ. 2006 The shape of boubas: sound-shape correspondences in toddlers and adults. Dev. Sci. 9, 316-322. (doi: 10. 1111/j. 1467-7687. 2006. 00495. x).
22. Ramachandran VS, Hubbard EM. 2001 Synaesthesia: a window into perception, thought and language. J. Conscious. Stud. 8, 3-34.
23. Ohtake Y, Haryu E. 2013 Investigation of the process underpinning vowel-size correspondence. Jpn Psychol. Res. 55, 390-399. (doi: 10. 1111/jpr. 12029).
24. Westbury C. 2005 Implicit sound symbolism in lexical access: evidence from an interference task. Brain Lang. 93, 10-19. (doi: 10. 1016/j. bandl. 2004.07. 006).
25. Kovic V, Plunkett K, Westermann G. 2010 The shape of words in the brain. Cognition 114, 19-28. (doi: 10. 1016/j. cognition. 2009. 08. 016).
26. Kanero J, Imai M, Okuda J, Okada H, Matsuda T. 2014 How sound symbolism is processed in the brain: a study on Japanese mimetic words. PLoS ONE 9, 1-8. (doi: 10. 1371/journal. pone. 0097905).
27. Imai M, Kita S, Nagumo M, Okada H. 2008 Sound symbolism facilitates early verb learning. Cognition 109, 54-65. (doi: 10. 1016/j. cognition. 2008. 07. 015).
28. Kantartzis K, Imai M, Kita S. 2011 Japanese soundsymbolism facilitates word learning in Englishspeaking children. Cogn. Sci. 35, 575-586. (doi: 10. 1111/j. 1551-6709. 2010. 01169. x).
29. Nygaard LC, Cook AE, Namy LL. 2009 Sound to meaning correspondences facilitate word learning. Cognition 112, 181-186. (doi: 10. 1016/j. cognition. 2009. 04. 001).
30. Thompson RL, Vinson DP, Vigliocco G. 2009 The link between form and meaning in American sign language: lexical processing effects. J. Exp. Psychol. Learn. 35, 550-557. (doi: 10. 1037/a0014547).
31. Vigliocco G, Vinson DP, Woolfe T, Dye MW, Woll B. 2005 Language and imagery: effects of language modality. Proc. R. Soc. B 272, 1859-1863. (doi: 10. 1098/rspb. 2005. 3169).
32. Vinson D, Thompson RL, Skinner R, Vigliocco G. Submitted. A faster path between meaning and form? Iconicity facilitates sign recognition and production in British Sign Language.
33. Thompson RL, Vinson DP, Vigliocco G. 2010 The link between form and meaning in British sign language: effects of iconicity for phonological decisions. J. Exp. Psychol. Learn. 36, 1017-1027. (doi: 10. 1037/a0019339).
34. Baus C, Carreiras M, Emmorey K. 2013 When does iconicity in sign language matter? Lang. Cogn. Proc. 28, 261-271. (doi: 10. 1080/01690965. 2011. 620374).
35. Bosworth R, Emmorey K. 2010 Effects of iconicity and semantic relatedness on lexical access in American sign language. J. Exp. Psychol. Learn. 36, 1573-1581. (doi: 10. 1037/a0020934).
36. Orlansky M, Bonvillian JD. 1984 The role of iconicity in early sign language acquisition. J. Speech Hear. Disord. 49, 287-292. (doi: 10. 1044/jshd. 4903. 287).
37. Lloyd L, Loeding B, Doherty J. 1985 Role of iconicity in sign acquisition: a response to Orlansky and Bonvillian (1984). J. Speech Hear. Disord. 50, 299-301. (doi: 10. 1044/jshd. 5003. 299).
38. Thompson RL, Vinson DP, Woll B, Vigliocco G. 2012 The road to language learning is iconic: evidence from British sign language. Psychol. Sci. 23, 1443-1448. (doi: 10. 1177/0956797612459763).
39. Vinson D, Cormier K, Denmark T, Schembri A, Vigliocco G. 2008 The British sign language (BSL) norms for age of acquisition, familiarity, and iconicity. Behav. Res. Methods 40, 1079-1087. (doi: 10. 3758/BRM. 40. 4. 1079).
40. Klima ES, Bellugi U. 1979 The signs of language. Cambridge, MA: Harvard University Press.
41. Liddell SK. 1980 American sign language syntax. New York, NY: Mouton.
42. Lillo-Martin D, Klima ES. 1990 Pointing out differences: ASL pronouns in syntactic theory. In Theoretical issues in sign language research (eds SD Fischer, P Siple), pp. 191-210. Chicago, IL: University of Chicago Press.
43. Padden CA. 1990 The relation between space and grammar in ASL verb morphology. In Sign language research: theoretical issues (ed. C Lucas), pp. 118-132. Washington, DC: Gallaudet University Press.
44. Aronoff M, Meir I, Sandler W. 2005 The paradox of sign language morphology. Language 81, 301-344. (doi: 10. 1353/lan. 2005. 0043).
45. Emmorey K. 2002 Language, cognition and the brain. Mahwah, NJ: Lawrence Erlbaum Associates.
46. Meier RP. 2002 Why different, why the same? Explaining effects and non-effects of modality upon linguistic structure in sign and speech. In Modality and structure in signed and spoken languages (eds RP Meier, K Cormier, D Quinto-Pozos), pp. 1-25. Cambridge, UK: Cambridge University Press.
47. Liddell SK. 2003 Grammar, gesture, and meaning in American sign language. Cambridge, UK: Cambridge University Press.
48. Kendon A. 2014 Semiotic diversity in utterance production and the concept of 'language'. Phil. Trans. R. Soc. B 369, 20130293. (doi: 10. 1098/rstb. 2013. 0293).
49. Goldin-Meadow S. 2014 Widening the lens: what the manual modality reveals about language, learning and cognition. Phil. Trans. R. Soc. B 369, 20130295. (doi: 10. 1098/rstb. 2013. 0295).
50. Liszkowski U. 2014 Two sources of meaning in infant communication: preceding action contexts and act-accompanying characteristics. Phil. Trans. R. Soc. B 369, 20130294. (doi: 10. 1098/rstb. 2013. 0294).
51. Özyürek A. 2014 Hearing and seeing meaning in speech and gesture: insights from brain and behaviour. Phil. Trans. R. Soc. B 369, 20130296. (doi: 10. 1098/rstb. 2013. 0296).
52. Skipper JI. 2014 Echoes of the spoken past: how auditory cortex hears context during speech perception. Phil. Trans. R. Soc. B 369, 20130297. (doi: 10. 1098/rstb. 2013. 0297).
53. Levinson SC, Holler J. 2014 The origin of human multi-modal communication. Phil. Trans. R. Soc. B 369, 20130302. (doi: 10. 1098/rstb. 2013. 0302).
54. Sereno MI. 2014 Origin of symbol-using systems: speech, but not sign, without the semantic urge. Phil. Trans. R. Soc. B 369, 20130303. (doi: 10. 1098/rstb. 2013. 0303).
55. Monaghan P, Shillcock RC, Christiansen MH, Kirby S. 2014 How arbitrary is language? Phil. Trans. R. Soc. B 369, 20130299. (doi: 10. 1098/rstb. 2013. 0299).
56. Imai M, Kita S. 2014 The sound symbolism bootstrapping hypothesis for language acquisition and language evolution. Phil. Trans. R. Soc. B 369, 20130298. (doi: 10. 1098/rstb. 2013. 0298).
57. Emmorey K. 2014 Iconicity as structure mapping. Phil. Trans. R. Soc. B 369, 20130301. (doi: 10. 1098/rstb. 2013. 0301).
58. Gentner D. 1983 Structure-mapping: a theoretical framework for analogy. Wiley Interdiscip. Rev. Cogn. Soc. 7, 155-170. (doi: 10. 1207/s15516709cog0702_3).
59. Gentner D, Markman AB. 1997 Structure mapping in analogy and similarity. Am. Psychol. 52, 45-56. (doi: 10. 1037/0003-066X. 52. 1. 45).
60. Perniss P, Vigliocco G. 2014 The bridge of iconicity: from a world of experience to the experience of language. Phil. Trans. R. Soc. B 369, 20130300. (doi: 10. 1098/rstb. 2013. 0300).
61. Engel AK, Maye A, Kurthen M, König P. 2013 Where's the action? The pragmatic turn in cognitive science. Trends Cogn. Sci. 17, 202-209. (doi: 10. 1016/j. tics. 2013. 03. 006).
62. Zwaan R. 2014 Embodiment and language comprehension: reframing the discussion. Trends Cogn. Sci. 18, 229-234. (doi: 10. 1016/j. tics. 2014. 02. 008).