Challenges in designing digital interfaces for the study of multimodal phenomena

Information Design Journal

Volumn 18, Issue 1, 2010, Pages 2-21

  O'Halloran, Kay ^a   Sabine Tan, ^a   Smith, Bradley ^a   Podlasov, Alexey ^a  

^a NATIONAL UNIVERSITY OF SINGAPORE   (Singapore)

Author keywords

Digital interface design; Digital technology; Multimodal; Social semiotics

Indexed keywords

EID: 77955743556     PISSN: 01425471     EISSN: 1569979X     Source Type: Journal    
DOI: 10.1075/idj.18.1.02hal     Document Type: Article

References (54)

1. Baldry, A. P. (2004). Phase and transition, type and instance: Patterns in media texts as seen through a multimodal concordance. In K. L. O'Halloran (Ed.), Multimodal discourse analysis: Systemic-functional perspectives (pp. 83-108). London; New York: Continuum.
2. Baldry, A. P., & Thibault, P. J. (2006). Multimodal transcription and text analysis. Oakville, CT: Equinox Publishing.
3. Bateman, J. A. (2008). Multimodality and genre: A foundation for the systematic analysis of multimodal documents. Basingstoke; New York: Palgrave MacMillan.
4. Bateman, J. A. (forthcoming). The decomposability of semiotic modes. In Kay L. O'Halloran & B. A. Smith (Eds.), Multimodal studies: Exploring Issues and Domains. New York & London: Routledge.
5. Bishara, N. (2007). "Absolut Anonymous": Self-reference in opaque advertising. In W. Nöth & N. Bishara (Eds.), Self-reference in the media (pp. 79-92). New York: Mouton de Gruyter.
6. Chignell, M. H. (1993). Cooperative human-machine reasoning: Communication through the user interface. In R. J. Jorna, B van Heusden & R. Posner (Eds.), Signs, search and communication: Semiotic aspects of artificial intelligence (pp. 348-368). Berlin; New York: Walter de Gruyter.
7. Constantinou, O. (2005). Multimodal discourse analysis: Media, modes and technologies. Journal of Sociolinguistics, 9/4, 602-618.
8. Djonov, E. (2005). Analysing the organisation of information in websites: From hypermedia design to systemic functional hypermedia discourse analysis. Unpublished doctoral dissertation, University of New South Wales.
9. Eggins, S. (1994). An introduction to systemic functional linguistics. New York: Continuum.
10. Firth, J. R. (1957). Papers in linguistics: 1934-1951. London: Oxford University Press.
11. Firth, J. R. (1968). Selected papers of J. R. Firth: 1952-1959, ed. F. R. Palmer. London; Harlow: Longmans.
12. Halliday, M. A. K. (1973). Explorations in the functions of language. London: Arnold.
13. Halliday, M. A. K. (1978). Language as social semiotic. London: Edward Arnold.
14. Halliday, M. A. K. (1985). Spoken and written language. Waurn Ponds, Victoria: Deakin University Press.
15. Halliday, M. A. K. (1994 [1985]). An introduction to functional grammar. Second Edition. London: Edward Arnold.
16. Hodge, R., & Kress, G. (1988). Social semiotics. Cambridge, UK: Polity Press in association with Basil Blackwell, Oxford, UK.
17. Iedema, R. (2001). Analysing film and television: A social semiotic account of hospital: An unhealthy business. In T. van Leeuwen & C. Jewitt (Eds.), Handbook of Visual Analysis (pp. 183-204). London; Thousand Oaks; New Delhi: Sage Publications.
18. Iedema, R. (2003). Multimodality, resemiotization: Extending the analysis of discourse as multi-semiotic practice. Visual Communication, 2/1, 29-57.
19. Jewitt, C. (2006). Technology, literacy and learning: A multimodal approach. Routledge Falmer.
20. Jewitt, C. (2009). Handbook of multimodal analysis. London: Routledge.
21. Kress, G., & van Leeuwen, T. (2001). Multimodal discourse: The modes and media of contemporary communication. London: Arnold.
22. Kress, G., & van Leeuwen, T. (2006 [1996]). Reading images: The grammar of visual design. London: Routledge.
23. Lemke, J. L. (1998). Multiplying meaning: visual and verbal semiotics in scientific text. In J. R. Martin & R. Veel (Eds.), Reading science (pp. 87-113). London: Routledge.
24. Lemke, J. L. (2002). Travels in hypermodality. Visual Communication, 1/3, 299-325.
25. Lemke, J. L. (2003). Texts and discourses in the technologies of social organization. In G. Weiss & R. Wodak (Eds.), Critical discourse analysis theory and interdisciplinarity (pp. 130-149).
26. Basingstoke, Hampshire [England]; New York: Palgrave Macmillan.
27. LeVine, P., & Scollon, R. (2004). Discourse and technology: Multimodal discourse analysis. Washington, D.C.: Georgetown University Press.
28. Machin, D. (2007). Introduction to multimodal analysis. London: Hodder Arnold; New York: Oxford University Press.
29. Manovich, L. (2009). Cultural analytics: visualizing cultural patterns in the era of "more media" (http://lab.softwarestudies.com/2008/09/ cultural-analytics.html)
30. Martin, J. R., & Rose, D. (2007). Working with discourse: Meaning beyond the clause. Second Edition. London; New York: Continuum.
31. Martinec, R. (2000). Types of process in action. Semiotica, 130, 3/4, 243-268.
32. Martinec, R. (2001). Interpersonal resources in action. Semiotica, 135, 1/4, 117-145.
33. Martinec, R. (2004). Gestures that co-occur with speech as a systematic resource: The realization of experiential meanings in indexes. Social Semiotics, 14/2, 193-213.
34. Martinec, R. (2007). Topics in multimodality. In R. Hasan, C. Matthiessen & J. Webster (Eds.), Continuing discourse on language: A functional perspective (pp. 157-181). London; Oakville: Equinox Pub.
35. Moore, A. (in press). Surgical teams in action: A contextually sensitive approach to modelling body alignment and interpersonal engagement. In A. Baldry & E. Montagna (Eds.), Interdisciplinary perspectives on multimodality: Theory and practice. Campobasso, Italy: Palladino.
36. Norris, S. (2004). Analyzing multimodal interaction: A methodological framework. New York; London: Routledge.
37. O'Halloran, K. L. (2004). Visual semiosis in film. In K. L. O'Halloran (Ed.), Multimodal discourse analysis: Systemic-functional perspectives (pp. 109-130). London; New York: Continuum.
38. O'Halloran, K. L. (2008a). Systemic functional-multimodal discourse analysis (SF-MDA): Constructing ideational meaning using language and visual imagery. Visual Communication, 7/4, 443-475.
39. O'Halloran, K. L. (2008b). Inter-semiotic expansion of experiential meaning: Hierarchical scales and metaphor in mathematics discourse. In E. Ventola & C. Jones (Eds.), New developments in the study of ideational meaning: From language to multimodality (pp. 231-254). London: Equinox.
40. O'Halloran, K. L. (2009a). Multimodal analysis and digital technology. In A. Baldry & E. Montagna (Eds.), Interdisciplinary perspectives on multimodality: Theory and practice. Proceedings of the Third International Conference on Multimodality. Campobasso: Palladino. Retrieved from: http://multimodalanalysis-lab.org/events/publications.html.
41. O'Halloran, K. L. (2009b). Historical changes in the semiotic landscape: From calculation to computation. In C. Jewitt (Ed.), Handbook of multimodal analysis. London: Routledge.
42. O'Toole, M. (1994). The language of displayed art (pp. 98-113). London: Leicester University Press.
43. Rohlfing, K., Loehr, D., Duncan, S., Brown A., Franklin, A., Kimbara, I., Milde, J.-T., Parrill, F., Rose, T., Schmidt, T., Sloetjes, H., Thies, A., & Wellinghoff, S. (2006). Comparison of multimodal annotation tools: Workshop report. Gesprächsforschung - Online-Zeitschrift zur verbalen Interaktion, 7, 99-123. Retrieved from: http://www.gespraechsforschung-ozs.de/ heft2006/tb-rohlfing.pdf
44. Rosenfeld, A., Doermann, D., & DeMenthon, D. (2003). Video mining. Boston: Kluwer Academic.
45. Saussure, F. (1916/1974). Course in general linguistics (trans. Wade Baskin). London: Fontana/Collins.
46. Smith, M. A., & Kanade, T. (2005). Multimodal video characterization and summarization. Boston: Kluwer Academic Publishers.
47. Tan, S. (2009). A systemic functional framework for the analysis of corporate television advertisements. In E. Ventola & A. J. Moya Guijjaro (Eds.), The world told and the world shown: Multisemiotic issues (pp. 157-182). Basingstoke: Palgrave Macmillan.
48. Thibault, P. (1997). Re-reading Saussure: The dynamics of signs in social life. London: Routledge.
49. Thibault, P. J. (2000). The multimodal transcription of a television advertisement: Theory and practice. In A. Baldry (Ed.), Multimodality and multimediality in the distance learning age: Papers in English linguistics (pp. 311-385). Campobasso, Italy: Palladino Editore.
50. Turner, G. (1994). Film languages. In D. Graddol & O. Boyd-Barrett (Eds.), Media texts, authors and readers: A reader (pp. 119-135). Clevedon, England; Philadelphia: Multilingual Matters in association with The Open University.
51. van Leeuwen, T. (1999). Speech, music, sound. Basingstoke; London: Macmillan Press.
52. van Leeuwen, T. (2001). Semiotics and iconography. In T. van Leeuwen & C. Jewitt (Eds.), Handbook of visual analysis (pp. 92-118). London; Thousand Oaks; New Delhi: Sage Publications.
53. van Leeuwen, T. (2005). Introducing social semiotics. New York: Routledge.
54. van Leeuwen, T. (2008). Discourse and practice: New tools for critical discourse analysis. New York: Oxford University Press.