(Pea)nuts and bolts of visual narrative: Structure and meaning in sequential image comprehension

Cognitive Psychology

Volumn 65, Issue 1, 2012, Pages 1-38

  Cohn, Neil ^a   Paczynski, Martin ^a   Jackendoff, Ray ^a   Holcomb, Phillip J ^a   Kuperberg, Gina R ^a,b  

^a TUFTS UNIVERSITY   (United States)

^b MASSACHUSETTS GENERAL HOSPITAL   (United States)

Author keywords

Coherence; Comics; Discourse; ERPs; Event related potentials; Film; LAN; N300; N400; Visual language

Indexed keywords

ADULT; ART; ARTICLE; COMPREHENSION; FEMALE; HUMAN; MALE; SEMANTICS; VERBAL COMMUNICATION; VISION;

ADULT; CARTOONS AS TOPIC; COMPREHENSION; FEMALE; HUMANS; MALE; NARRATION; SEMANTICS; VISUAL PERCEPTION;

EID: 84857601593     PISSN: 00100285     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.cogpsych.2012.01.003     Document Type: Article

References (119)

1. Barrett S.E., Rugg M.D. Event-related potentials and the semantic matching of pictures. Brain and Cognition 1990, 14:201-212.
2. Bentin S., McCarthy G., Wood C.C. Event-related potentials, lexical decision and semantic priming. Electroencephalography and Clinical Neurophysiology 1985, 60:343-355.
3. Black J.B., Bower G. Story understanding as problem-solving. Poetics 1980, 9:223-250.
4. Black J.B., Wilensky R. An evaluation of story grammars. Cognitive Science 1979, 3:213-230.
5. Bower G.H., Black J.B., Turner T.J. Scripts in memory for text. Cognitive Psychology 1979, 11(2):120-177.
6. Bransford J.D., Johnson M.K. Contextual prerequisites for understanding: Some investigations of comprehension and recall. Journal of Verbal Learning and Verbal Behavior 1972, 11(6):717-726.
7. Brewer W.F., Lichtenstein E.H. Event schemas, story schemas, and story grammars. Attention and performance IX 1981, 363-379. Erlbaum, Hillsdale. J. Long, A.D. Baddeley (Eds.).
8. Brown G., Yule G. Discourse analysis 1983, Cambridge University Press, Cambridge.
9. Butcher S.H. The poetics of Aristotle 1902, Macmillian and Co. Ltd, London.
10. Carroll J.M., Bever T.G. Segmentation in cinema perception. Science 1976, 191(4231):1053-1055.
11. Chatman S. Story and discourse 1978, Cornell University Press, Ithaca.
12. Chomsky N. Aspects of the theory of syntax 1965, MIT Press, Cambridge, MA.
13. Clark H.H. Difficulties people have in answering the question "Where is it?" Journal of Verbal Learning and Verbal Behavior 1972, 11:265-277.
14. Cohn N. Early writings on visual language 2003, Emaki Productions, Carlsbad, CA.
15. Cohn N. A visual lexicon. Public Journal of Semiotics 2007, 1(1):53-84.
16. Cohn N. The limits of time and transitions: Challenges to theories of sequential image comprehension. Studies in Comics 2010, 1(1):127-147.
17. Cohn, N. (in preparation). Visual narrative structure. Cognitive Science.
18. de Beaugrande R. The story of grammars and the grammar of stories. Journal of Pragmatics 1982, 6:383-422.
19. Delong K.A., Urbach T.P., Kutas M. Probabilistic word pre-activation during language comprehension inferred from electrical brain activity. Nature Neuroscience 2005, 8(8):1117-1121.
20. Federmeier K.D., Kutas M. Right words and left words: Electrophysiological evidence for hemispheric differences in meaning processing. Brain Research: Cognitive Brain Research 1999, 8:373-392.
21. Federmeier K.D., Kutas M. Meaning and modality: Influences of context, semantic memory organization, and perceptual predictability on picture processing. Journal of Experimental Psychology: Learning, Memory, and Cognition 2001, 27(1):202-224.
22. Friederici A.D. Towards a neural basis of auditory sentence processing. Trends in Cognitive Sciences 2002, 6(2):78-84.
23. Ganis G., Kutas M. An electrophysiological study of scene effects on object identification. Cognitive Brain Research 2003, 16(2):123-144.
24. Ganis G., Kutas M., Sereno M.I. The search for "common sense": An electrophysiological study of the comprehension of words and pictures in reading. Journal of Cognitive Neuroscience 1996, 8:89-106.
25. Gee J.P., Grosjean F. Empirical evidence for narrative structure. Cognitive Science 1984, 8:59-85.
26. Genette, G. (1980). In J. E. Lewin (Trans.) Narrative discourse. Ithaca: Cornell University Press.
27. Gernsbacher, M. A. (1983). Memory for surface information in non-verbal stories: Parallels and insights to language processes. Unpublished dissertation, University of Texas at Austin, Austin, Texas.
28. Gernsbacher M.A. Surface information loss in comprehension. Cognitive Psychology 1985, 17:324-363.
29. Gernsbacher M.A. Language comprehension as structure building 1990, Lawrence Erlbaum, Hillsdale, NJ.
30. Gernsbacher M.A., Varner K.R., Faust M. Investigating differences in general comprehension skill. Journal of Experimental Psychology: Learning, Memory, and Cognition 1990, 16:430-445.
31. Glanzer M., Fischer B., Dorfman D. Short-term storage in reading. Journal of Verbal Learning and Verbal Behavior 1984, 17:467-486.
32. Greeno J.G., Noreen D.L. Time to read semantically related sentences. Memory and Cognition 1974, 2:117-120.
33. Gunter T.C., Stowe L.A., Mulder G. When syntax meets semantics. Psychophysiology 1997, 34(6):660-676.
34. Haberlandt K. Components of sentence and word reading times. New methods in reading comprehension research 1984, Erlbaum, Hillsdale, NJ. D.E. Kieras, M.A. Just (Eds.).
35. Hagoort, P., Brown, C. M., & Groothusen, J. (1993). The syntactic positive shift (SPS) as an ERP measure of syntactic processing. In S. M. Garnsey (Ed.), Language and cognitive processes: Event-related brain potentials in the study of language (pp. 439-483). Hove: Lawrence Erlbaum Associates (special issue).
36. Hahne A., Jescheniak J.D. What's left if the jabberwocky gets the semantics? An ERP investigation into semantic and syntactic processing during auditory sentence comprehension. Cognitive Brain Research 2001, 11:199-212.
37. Halliday M.A.K., Hasan R. Cohesion in English 1976, Longman, London.
38. Halliday M.A.K., Hasan R. Language, context, and text: Aspects of language in a social-semiotic perspective 1985, Deakin University Press, Victoria.
39. Haviland S.E., Clark H.H. What's new? Acquiring new information as a process in comprehension. Journal of Verbal Learning and Verbal Behavior 1974, 13:512-521.
40. Hobbs, J. R. (1985). On the coherence and structure of discourse. CSLI Technical report 85-37, Stanford, CA.
41. Jackendoff R. Semantic structures 1990, MIT Press, Cambridge, MA.
42. Jackendoff R. Foundations of language: Brain, meaning, grammar, evolution 2002, Oxford University Press, Oxford.
43. Kaufman A.S., Lichtenberger E.O. Assessing adolescent and adult intelligence 2006, Wiley, Hoboken. 3rd ed.
44. Keenan J.M., Baillet S.D., Brown P. The effect of causal cohesion on comprehension and memory. Journal of Verbal Learning and Verbal Behavior 1984, 23:115-126.
45. Kehler A. Coherence, reference, and the theory of grammar 2002, CSLI Publications, Stanford.
46. Kieras D.E. Good and bad structure in simple paragraphs: Effects on apparent theme, reading time, and recall. Journal of Verbal Learning and Verbal Behavior 1978, 17:13-28.
47. Kintsch W., van Dijk T. Toward a model of text comprehension and production. Psychological Review 1978, 85:363-394.
48. Koelsch S., Gunter T., Friederici A.D., Schröger E. Brain indices of music processing: "Nonmusicians" are musical. Journal of Cognitive Neuroscience 2000, 12(3):520-541.
49. Kunzle D. The history of the comic strip 1973, Vol. 1. University of California Press, Berkeley.
50. Kuperberg G. Neural mechanisms of language comprehension: Challenges to syntax. Brain Research 2007, 1146:23-49.
51. Kuperberg G., Paczynski M., Ditman T. Establishing causal coherence across sentences: An ERP study. Journal of Cognitive Neuroscience 2011, 23(5):1230-1246.
52. Kutas M., Federmeier K.D. Thirty years and counting: Finding meaning in the N400 component of the event-related brain potential (ERP). Annual Review of Psychology 2011, 62(1):621-647.
53. Kutas M., Hillyard S.A. Reading senseless sentences: Brain potential reflect semantic incongruity. Science 1980, 207:203-205.
54. Kutas M., Hillyard S.A. Brain potentials during reading reflect word expectancy and semantic association. Nature 1984, 307:161-163.
55. Kutas M., Van Petten C. Psycholinguistics electrified: Event-related brain potentials investigations. Handbook of psycholinguistics 1994, 83-143. Academic Press, San Diego, CA. M.A. Gernsbacher (Ed.).
56. Lau E.F., Phillips C., Poeppel D. A cortical network for semantics: (De)constructing the N400. Nature Reviews Neuroscience 2008, 9:920-933.
57. Lichtenstein E.H., Brewer W.F. Memory for goal directed events. Cognitive Psychology 1980, 12:412-445.
58. Linde C., Labov W. Spatial networks as a site for the study of language and thought. Language 1975, 51:924-939.
59. Magliano J.P., Miller J., Zwaan R.A. Indexing space and time in film understanding. Applied Cognitive Psychology 2001, 15:533-545.
60. Mandler J.M. A code in the node: The use of story schema in retrieval. Discourse Processes 1978, 1:14-35.
61. Mandler J.M. Stories, scripts, and scenes: Aspects of schema theory 1984, Lawrence Earlbaum Associates, Hillsdale, NJ.
62. Mandler J.M. On the comprehension of temporal order. Language and Cognitive Processes 1986, 1:309-320.
63. Mandler J.M. On the psychological reality of story structure. Discourse Processes 1987, 10:1-29.
64. Mandler J.M., DeForest M. Is there more than one way to recall a story?. Child Development 1979, 50:886-889.
65. Mandler J.M., Goodman M.S. On the psychological validity of story structure. Journal of Verbal Learning and Verbal Behavior 1982, 21:507-523.
66. Mandler J.M., Johnson N.S. Remembrance of things parsed: Story structure and recall. Cognitive Psychology 1977, 9:111-151.
67. Mann W.C., Thompson S.A. Rhetorical structure theory: A theory of text organization 1987, Information Sciences Institute, Marina del Rey, CA.
68. Marslen-Wilson W.D. Functional parallelism in spoken word-recognition. Cognition 1987, 25(1-2):71-102.
69. Marslen-Wilson W.D., Tyler L.K. Processing structure of sentence perception. Nature 1975, 257:784-785.
70. Marslen-Wilson W.D., Tyler L.K. The temporal structure of spoken language understanding. Cognition 1980, 8:1-71.
71. McCloud S. Understanding comics: The invisible art 1993, Harper Collins, New York, NY.
72. McPherson W.B., Holcomb P.J. An electrophysiological investigation of semantic priming with pictures of real objects. Psychophysiology 1999, 36(1):53-65.
73. Morrow D.G., Greenspan S.L., Bower G. Accessibility and situation models in narrative comprehension. Journal of Memory and Language 1987, 26:165-187.
74. Münte T.F., Matzke M., Johannes S. Brain activity associated with syntactic incongruencies in words and psuedo-words. Journal of Cognitive Neuroscience 1997, 9:318-329.
75. Nakazawa J. Manga (comic) literacy skills as determinant factors of manga story comprehension. Manga Studies 2004, 5:7-25.
76. Nakazawa J., Nakazawa S. How do children understand comics? Analysis of comic reading comprehension. Annual of Research in Early Childhood 1993, 15:35-39.
77. Neville H.J., Nicol J.L., Barss A., Forster K.I., Garrett M.F. Syntactically based sentence processing classes: Evidence from event-related brain potentials. Journal of Cognitive Neuroscience 1991, 3(2):151-165.
78. Newtson D., Engquist G. The perceptual organization of ongoing behavior. Journal of Experimental Social Psychology 1976, 12:436-450.
79. Nieuwland M.S., Kuperberg G.R. When the truth is not too hard to handle. Psychological Science 2008, 19(12):1213-1218.
80. Osterhout L., Holcomb P. Event-related potentials elicited by syntactic anomaly. Journal of Memory and Language 1992, 31:758-806.
81. Otten M., Van Berkum J.J.A. What makes a discourse constraining? Comparing the effects of discourse message and scenario fit on the discourse-dependent N400 effect. Brain Research 2007, 1153:166-177.
82. Patel A.D. Language, music, syntax and the brain. Nature Neuroscience 2003, 6(7):674-681.
83. Patel A.D., Gibson E., Ratner J., Besson M., Holcomb P.J. Processing syntactic relations in language and music: An event-related potential study. Journal of Cognitive Neuroscience 1998, 10(6):717-733.
84. Rugg M.D. Event-related potentials in phonological matching tasks. Brain and Language 1984, 23:225-240.
85. Rumelhart D.E. Notes on a schema for stories. Representation and understanding 1975, Academic Press, New York, NY. D. Bobrow, A. Collins (Eds.).
86. Saraceni, M. (2000). Language beyond language: Comics as verbo-visual texts. Unpublished dissertation, University of Nottingham.
87. Saraceni, M. (2001). Relatedness: Aspects of textual connectivity in comics. In J. Baetens (Ed.), The graphic novel (pp. 167-179).
88. Saraceni M. The language of comics 2003, Routeledge, New York, NY.
89. Schank R.C. The structure of episodes in memory. Representation and understanding: Studies in cognitive science 1975, 237-272. Academic Press, New York. D. Bobrow, A. Collins (Eds.).
90. Schank R.C., Abelson R. Scripts, plans, goals and understanding 1977, Lawrence Earlbaum Associates, Hillsdale, NJ.
91. Sitnikova T., Holcomb P.J., Kuperberg G. Neurocognitive mechanisms of human comprehension. Understanding events: How humans see, represent, and act on events 2008, 639-683. Oxford University Press. T.F. Shipley, J.M. Zacks (Eds.).
92. Sitnikova T., Holcomb P.J., Kuperberg G. Two neurocognitive mechanisms of semantic integration during the comprehension of visual real-world events. Journal of Cognitive Neuroscience 2008, 20(11):1-21.
93. Sitnikova T., Kuperberg G., Holcomb P. Semantic integration in videos of real-world events: An electrophysiological investigation. Psychophysiology 2003, 40(1):160-164.
94. Sitnikova T., Paczynski M., Kuperberg G.R. Context motivates comprehenders attempts to make sense of novel events: Evidence from event-related potentials. Journal of Cognitive Neuroscience Supplement 2007.
95. Sitnikova T., West W.C., Kuperberg G., Holcomb P. The neural organization of semantic memory: Electrophysiological activity suggests feature-based segregation. Biological Psychology 2006, 71(3):326-340.
96. Speer N.K., Zacks J.M. Temporal changes as event boundaries: Processing and memory consequences of narrative time shifts. Journal of Memory and Language 2005, 53:125-140.
97. St. George M., Mannes S., Hoffman J.E. Individual differences in inference generation: An ERP analysis. Journal of Cognitive Neuroscience 1997, 9(6):776-787.
98. Stein N.L., Glenn C.G. An analysis of story comprehension in elementary school children. New directions in discourse processing 1979, 53-119. Ablex, Norwood, NJ. R. Freedle (Ed.).
99. Stein N.L., Nezworski T. The effects of organization and instructional set on story memory. Discourse Processes 1978, 1(2):177-193.
100. Thorndyke P. Cognitive structures in comprehension and memory of narrative discourse. Cognitive Psychology 1977, 9:77-110.
101. Tomashevsky B. Thematics. Russian formalist criticism: Four essays 1965, 61-95. University of Nebraska Press, Lincoln. L.T. Lemon, M.I. Reis (Eds.).
102. Trabasso T., Secco T., van den Broek P. Causal cohesion and story coherence. Learning and comprehension of text 1984, 83-111. Erlbaum, Hillsdale, NJ. H. Mandl, N.L. Stein, T. Trabasso (Eds.).
103. Trabasso T., Sperry L.L. Causal relatedness and importance of story events. Journal of Memory and Language 1985, 24:595-611.
104. Trabasso T., van den Broek P. Causal thinking and the representation of narrative events. Journal of Memory and Language 1985, 24:612-630.
105. van Berkum, J. J. A. (2009). The neuropragmatics of "simple" utterance comprehension: An ERP review. In U. Sauerland & K. Yatsushiro (Eds.), Semantics and pragmatics: From experiment to theory (pp. 276-316). Basingstoke: Palgrave.
106. van Berkum J.J.A., Hagoort P., Brown C.M. Semantic integration in sentences and discourse from the N400. Journal of Cognitive Neuroscience 1999, 11:657-671.
107. van Dijk T. Text and context 1977, Longman, London.
108. van Dijk T., Kintsch W. Strategies of discourse comprehension 1983, Academic Press, New York.
109. Van Petten C., Kutas M. Influences of semantic and syntactic context on open- and closed-class words. Memory and Cognition 1991, 19:95-112.
110. West W.C., Holcomb P. Event-related potentials during discourse-level semantic integration of complex pictures. Cognitive Brain Research 2002, 13:363-375.
111. Yang C.L., Perfetti C.A., Schmalhofer F. Event-related potential indicators of text integration across sentence boundaries. Journal of Experimental Psychology: Learning, Memory, and Cognition 2007, 33(1):55-89.
112. Zacks J.M., Magliano J.P. Film, narrative, and cognitive neuroscience. Art and the senses 2011, Oxford University Press, New York. D.P. Melcher, F. Bacci (Eds.).
113. Zacks J.M., Braver T.S., Sheridan M.A., Donaldson D.I., Snyder A.Z., Ollinger J.M., et al. Human brain activity time-locked to perceptual event boundaries. Nature Neuroscience 2001, 4(6):651-655.
114. Zacks J.M., Speer N.K., Reynolds J.R. Segmentation in reading and film comprehension. Journal of Experimental Psychology: General 2009, 138(2):307-327.
115. Zacks J.M., Speer N.K., Swallow K.M., Maley C.J. The brain's cutting-room floor: Segmentation of narrative cinema. Frontiers in Human Neuroscience 2010, 4:1-15.
116. Zacks J.M., Tversky B. Event structure in perception and conception. Psychological Bulletin 2001, 127(1):3-21.
117. Zacks J.M., Tversky B., Iyer G. Perceiving, remembering, and communicating structure in events. Journal of Experimental Psychology 2001, 130(1):29-58.
118. Zwaan R.A. Processing narrative time shifts. Journal of Experimental Psychology: Learning, Memory, and Cognition 1996, 22:1196-1207.
119. Zwaan R.A., Radvansky G.A. Situation models in language comprehension and memory. Psychological Bulletin 1998, 123(2):162-185.