Reading paths and visual perception in multimodal research, psychology and brain sciences

Journal of Pragmatics

Volumn 44, Issue 3, 2012, Pages 315-327

  Hiippala, Tuomo ^a  

^a UNIVERSITY OF HELSINKI   (Finland)

Author keywords

Eye tracking; Genre; Multimodality; Semiotic modes; Visual perception

Indexed keywords

EID: 84857035638     PISSN: 03782166     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.pragma.2011.12.008     Document Type: Article

References (65)

1. Baldry Anthony P., Thibault Paul J. Multimodal Transcription and Text Analysis 2005, Equinox, London.
2. Barthes Roland Image-Music-Text: Essays 1977, Hill and Wang, New York, translated by Stephen Heath.
3. Bateman John A. Towards a grande paradigmatique of film: Christian Metz reloaded. Semiotica 2007, 167(1):13-64.
4. Bateman John A. Multimodality and Genre: A Foundation for the Systematic Analysis of Multimodal Documents 2008, Palgrave Macmillan, London.
5. Bateman John A. Discourse across semiotic modes. Discourse of Course: An Overview of Research in Discourse Studies 2009, 55-66. Benjamins, Amsterdam, Philadelphia. J. Renkema (Ed.).
6. Bateman John A. Film and representation: making filmic meaning. Metarepresentation Self-organization and Art. European Semiotics 2009, 137-162. Lang, Bern. W. Wildgen, B. van Heusden (Eds.).
7. Bateman John A. The decomposability of semiotic modes. Multimodal Studies: Multiple Approaches and Domains 2011, 17-38. Routledge Studies in Multimodality. Routledge, London. K.L. O'Halloran, B.A. Smith (Eds.).
8. Bateman John A., Schmidt Karl-Heinrich Multimodal Film Analysis: How Films Mean. Routledge Studies in Multimodality 2011, London, Routledge.
9. Cheong Yin Yuen The construal of ideational meaning in print advertisements. Multimodal Discourse Analysis: Systemic Functional Perspectives 2004, 163-195. Continuum, London. K.L. O'Halloran (Ed.).
10. Genre and Institutions: Social Processes in the Workplace and School 1997, Continuum, London. F. Christie, J.R. Martin (Eds.).
11. Multiliteracies: Literacy Learning and the Design of Social Futures 2000, Routledge, London. B. Cope, M. Kalantzis (Eds.).
12. Delin Judy L., Bateman John A. Describing and critiquing multimodal documents. Document Design 2002, 3(2):140-155.
13. Elleström Lars The modalities of media: a model for understanding intermedial relations. Media Borders, Multimodality and Intermediality 2010, 11-48. Palgrave, London. L. Elleström (Ed.).
14. Forceville Charles Book review: multimodal transcription and text analysis: a multimedia toolkit and coursebook. Journal of Pragmatics 2007, 39(6):1235-1238.
15. Guo Libo Multimodality in a biology textbook. Multimodal Discourse Analysis: Systemic Functional Perspectives 2004, 196-219. Continuum, London. K.L. O'Halloran (Ed.).
16. Hannus Matti, Hyönä Jukka Utilization of illustrations during learning of science textbook passages among low- and high-ability children. Contemporary Educational Psychology 1999, 24(2):95-123.
17. Hiippala Tuomo The localisation of advertising print media as a multimodal process. Multimodal Texts from Around the World: Linguistic and Cultural Insights 2012, 97-122. Palgrave, London. W.L. Bowcher (Ed.).
18. Hofinger Andrea, Ventola Eija Multimodality in operation: language and picture in a museum. Perspectives on Multimodality 2004, 193-210. Benjamins, Amsterdam. E. Ventola, C. Charles, M. Kaltenbacher (Eds.).
19. Holsanova Jana, Holmberg Nils, Holmqvist Kenneth Reading information graphics: the role of spatial contiguity and dual attentional guidance. Applied Cognitive Psychology 2009, 23:1215-1226.
20. Holsanova Jana, Holmqvist Kenneth Entry points and reading paths on newspaper spreads: comparing a semiotic analysis with eye-tracking measurements. Visual Communication 2006, 5(1):65-93.
21. Holsanova Jana, Nord Andreas Multimodal design: media structures media principles and users meaning-making in printed and digital media. Neue Medie - neue Formate: Ausdifferenzierung und Konvergenz in der Medienkommunikation. Campus 2010, 81-103. Frankfurt, New York. H.-J. Bucher, T. Gloning, K. Lehnen (Eds.).
22. Jewitt Carey An introduction to multimodality. The Routledge Handbook of Multimodal Analysis 2009, 14-27. Routledge, London. C. Jewitt (Ed.).
23. Kaltenbacher Martin Perspectives on multimodality: from the early beginnings to the state of the art. Information Design Journal+Document Design 2004, 12(3):190-207.
24. Kappas Arvid, Olk Bettina The concept of visual competence as seen from the perspective of the psychological and brain sciences. Visual Studies 2008, 23(2):162-173.
25. Kara Pirkko, Koskenniemi Antti, Kovakoski Heikki, Nurmi Matti Värikäs luonto 4 1986, WSOY, Porvoo.
26. Kong Kenneth A taxonomy of the discourse relations between words and visuals. Information Design Journal 2006, 14(3):207-230.
27. Kostelnick Charles, Hassett Michael Shaping Information: The Rhetoric of Visual Conventions 2003, Southern Illinois University Press, Carbondale.
28. Kress Günther Visual and verbal modes of representation in electronically mediated communication: the potentials of new forms of text. Page to Screen: Taking Literacy into the Electronic Era 1998, 53-79. Routledge, London and New York. I. Snyder (Ed.).
29. Kress Günther Literacy in the New Media Age 2003, Routledge, London.
30. Kress Günther What is mode?. The Routledge Handbook of Multimodal Analysis 2009, 54-67. Routledge, London. C. Jewitt (Ed.).
31. Kress Günther, van Leeuwen Theo Reading Images: The Grammar of Visual Design 1996, Routledge, London.
32. Kress Günther, van Leeuwen Theo Front pages: (the critical) analysis of newspaper layout. Approaches to Media Discourse 1998, 186-219. Blackwell, Oxford. Bell, Allan, Garrett, Peter (Eds.).
33. Kress Günther, van Leeuwen Theo Reading Images: The Grammar of Visual Design 2006, Routledge, London. 2nd ed.
34. Kvåle Gunhild Invitasjon til destinasjon: multimodal retorikk i turistkommunikasjon. Skrift/bilde/lyd: Analyse av sammensatte tekster 2010, 39-55. Høyskoleforlaget, Kristiansand. M. Engebretsen (Ed.).
35. Lemke Jay L. Typology, Topology, Topography: Genre Semantics 1999, MS University of Michigan.
36. Lim Fei Victor Developing an integrative multi-semiotic model. Multimodal Discourse Analysis: Systemic Functional Perspectives 2004, 220-246. Continuum, London. K.L. O'Halloran (Ed.).
37. Livingstone Sonia Media literacy and the challenge of new information and communication technologies. The Communication Review 2004, 7:3-14.
38. Mann William C., Thompson Sandra A. Rhetorical structure theory: toward a functional theory of text organization. Text 1988, 8(3):243-281.
39. Martin James R. Analysing genre: functional parameters. Genre and Institutions: Social Processes in the Workplace and School 1997, 3-39. Continuum, London. Christie, Frances, Martin, R. James (Eds.).
40. Martin James R., Rose David Genre Relations: Mapping Culture 2008, Equinox, London.
41. Martin James R., Veel R. Reading Science: Critical and Functional Perspectives on Discourses of Science 1998, Routledge, London.
42. Martinec Radan The social semiotics of text and image in [j]apanese and [e]nglish software manuals and other procedures. Social Semiotics 2003, 13(1):43-69.
43. Martinec Radan, Salway Andrew A system for image-text relations in new (and old) media. Visual Communication 2005, 4(3):337-371.
44. Nisbett Richard E., Miyamoto Yuri The influence of culture: holistic versus analytic perception. Trends in Cognitive Sciences 2005, 9(10):467-473.
45. Nisbett Richard E., Peng Kaiping, Choi Incheol, Norenzayan Ara Culture and systems of thought: holistic versus analytic cognition. Psychological Review 2001, 108(2):291-310.
46. O'Halloran Kay L. Interdependence, interaction and metaphor in multisemiotic texts. Social Semiotics 1999, 9(3):317-354.
47. O'Halloran Kay L. Visual semiosis in film. Multimodal Discourse Analysis: Systemic Functional Perspectives 2004, 109-130. Continuum, London. K.L. O'Halloran (Ed.).
48. O'Halloran Kay L. Systemic functional-multimodal discourse analysis (SF-MDA): constructing ideational meaning using language and visual imagery. Visual Communication 2008, 7(4):443-475.
49. O'Halloran Kay L., Tan Sabine, Smith Bradley A., Podlasov Alexey Challenges in designing digital interfaces for the study of multimodal phenomena. Information Design Journal 2010, 18(1):2-21.
50. O'Halloran Kay L., Tan Sabine, Smith Bradley A., Podlasov Alexey Multimodal analysis within an interactive software environment: critical discourse perspectives. Critical Discourse Studies 2011, 8(2):109-125.
51. O'Toole Michael The Language of Displayed Art 1994, Leicester University Press, London.
52. Parodi Giovanni Research challenges for corpus cross-linguistics and multimodal texts. Information Design Journal 2010, 18(1):69-73.
53. Rayner Keith Eye movements in reading and information processing: 20 years of research. Psychological Bulletin 1998, 124(3):372-422.
54. Reichenberger, Klaus, Rondhuis, Klaas Jan, Kleinz, Jörg, Bateman, John A., 1996. Effective presentation of information through page layout: a linguistically-based approach. Tech. Rep. Arbeitspapiere der GMD 970. Institut für Integrierte Publikations- und Informationssysteme (IPSI), GMD, Darmstadt.
55. Royce Terry D. Synergy on the page: exploring intersemiotic complementarity in page-based multimodal text. JASFL Occasional Papers 1998, 1(1):25-49.
56. Royce Terry D. Intersemiotic complementarity: a framework for multimodal discourse analysis. New Directions in the Analysis of Multimodal Discourse 2007, 63-109. Lawrence Erlbaum, Mahwah, NJ. Royce, D. Terry, Bowcher, L. Wendy (Eds.).
57. Stenglin Mareen Space and communication in exhibitions: unravelling the nexus. The Routledge Handbook of Multimodal Analysis 2009, 272-283. Routledge, London. C. Jewitt (Ed.).
58. Tan Sabine A systemic functional framework for the analysis of corporate television advertisements. The World Told and the World Shown: Multisemiotic Issues 2009, 157-182. Palgrave, London. E. Ventola, A.J.M. Guijarro (Eds.).
59. Thomas, Martin, 2009. Localizing pack messages: a framework for corpus-based cross-cultural multimodal analysis. Ph.D. Thesis. University of Leeds.
60. Tseng Chiaoi, Bateman John A. Chain and choice in filmic narrative: an analysis of multimodal narrative construction in The Fountain. Narrative Revisited: Telling a Story in the Age of New Media. Pragmatics and Beyond, vol. 199 2010, 213-244. John Benjamins, Amsterdam. C.R. Hoffmann (Ed.).
61. van Leeuwen Theo Genre and field in critical discourse analysis: a synopsis. Discourse and Society 1993, 4(2):193-223.
62. van Leeuwen Theo Multimodality genre and design. Discourse in Action: Introducing Mediated Discourse Analysis 2005, 73-94. Routledge, London. S. Norris, R.H. Jones (Eds.).
63. White Paul Grabbing attention: the importance of modal density in advertising. Visual Communication 2010, 9(4):371-397.
64. Williamson Judith Decoding Advertisements: Ideology and Meaning in Advertising 1978, Marion Boyars, London.
65. Yarbus Alfred L. Eye Movements and Vision 1967, Plenum Press, New York.