HeadTalk, HandTalk and the corpus: Towards a framework for multi-modal, multi-media corpus development

Corpora

Volumn 4, Issue 1, 2009, Pages 1-32

  Knight, Dawn ^a   Evans, David ^a   Carter, Ronald ^a   Adolphs, Svenja ^a  

^a UNIVERSITY OF NOTTINGHAM   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 85126437033     PISSN: 17495032     EISSN: 17551676     Source Type: Journal    
DOI: 10.3366/E1749503209000203     Document Type: Article

References (119)

1. Abercrombie, D. 1963. 'Conversation and spoken prose', English Language Teaching XVIII (1), pp. 10-16.
2. Adolphs, S. and R. Carter. 2007. 'Beyond the word: new challenges in analysing corpora of spoken English', European Journal of English Studies 11 (2), pp. 133-46.
3. Allwood, J., L. Gr nqvist, E. Ahls n andM. Gunnarssan. 2001. 'Annotations and tools for an activity based spoken language corpus', Proceedings of the Second SIGdialWorkshop of Discourse and Dialogue 16, pp. 1-10. Morristown, New Jersey: Association for Computational Linguistics.
4. Argyle, M. 1988. Bodily Communication. (Second edition.) London: Methuen.
5. Avil s-Arriaga, H.H. and L.E. Sucar. 2002. 'Dynamic Bayesian networks for visual recognition of dynamic gestures', Journal of Intelligent and Fuzzy Systems 12 (3-4), pp. 243-50.
6. Badre, A., M. Guzdial, S. Hudson and P. Santos. 1995. 'A user interface evaluation using synchronized video, visualizations and event trace data', Software Quality Journal 4 (2), pp. 101-13.
7. Baldry, A. and P.J. Thibault. 2001. 'Towards multimodal corpora' in G. Aston and L. Burnard (eds) Corpora in the Description and Teaching of English: Papers from the Fifth ESSE Conference, pp. 87-102.
8. Bologna: Cooperativa Libraria Universitaria Editrice Bologna.
9. Baldry, A. and P.J. Thibault. 2006. Multimodal Transcription and Text Analysis: A Multimedia Toolkit and Coursebook. London: Equinox.
10. Bavelas, J.B. 1994. 'Gestures as part of speech: methodological implications', Research on Language and Social Interaction 27 (3), pp. 201-21.
11. Beattie, G. and H. Shovelton. 1999. 'Mapping the range of information contained in the iconic hand gestures that accompany speech', Journal of Language and Social Psychology 18, pp. 438-63.
12. Biber, D., S. Conrad and R. Reppen. 1998. Corpus Linguistics: Investigating Language Structure and Use. Cambridge: Cambridge University Press.
13. Bird, S. and M. Liberman. 2001. 'A formal framework for linguistic annotation', Speech Communication 33 (1-2), pp 23-60.
14. Birdwhistell, R.L. 1952. Introduction to Kinesics: An Annotated System for the Analysis of Body Motion and Gesture. Louisville, Kentucky: University of Louisville.
15. Brown, R. 1986. Social Psychology. (Second edition.) New York: Free Press.
16. Brugman, H. and A. Russel. 2004. 'Annotating multi-media / multi-modal resources with ELAN', LREC 2004, pp. 2065-8. Lisbon, Portugal.
17. Brundell, P. and D. Knight. 2005. 'Current research and tools to support data intensive analysis for digital records in e-social science', Unpublished NCeSS project report. University of Nottingham.
18. Burnard, L. 2004. 'Developing linguistic corpora: metadata for corpus work' in M. Wynne (ed.) Developing Linguistic Corpora: A Guide to Good Practice, pp. 30-46. Oxford: Oxbow Books.
19. Cameron, D. 2001. Working with Spoken Discourse. London: Sage.
20. Canepari, L. 2005. A Handbook of Phonetics. Munich: Lincom.
21. Carter, R. and S. Adolphs. 2005. 'Developing multimodal linguistic corpora', Unpublished NCeSS Node Meeting Presentation. University of Nottingham.
22. Cerrato, L. 2004. 'A coding scheme for the annotation of feedback phenomenon in everyday speech', LREC Workshop on Models of Human Behaviour for the Specification and Evaluation of Multimodal Input and Output Interfaces, pp. 25-8. Lisbon: Portugal.
23. Cerrato, L. and M. Skhiri. 2003. 'Analysis and measurement of head movements signalling feedback in face-to-face human dialogues', Proceedings of AVSP 2003, pp. 251-6. St Jorioz: France.
24. Chawla, P. and R.M. Krauss. 1994. 'Gesture and speech in spontaneous and rehearsed narratives', Journal of Experimental Social Psychology 30, pp. 580-601.
25. Church, R.B. and S. Goldin-Meadow. 1986. 'The mismatch between gesture and speech as an index of transitional knowledge', Cognition 23 (1), pp. 43-71.
26. Cook, G. 1995. 'Theoretical issues: transcribing the untranscribable' in G. Leech, G. Myers and J. Thomas (eds) Spoken English on Computer: Transcription, Mark-up and Applications, pp. 35-53.
27. London: Longman.
28. Davis, M. 1979. 'The state of the art: past and present trends in body movement research' in A. Wolfgang (ed.) Nonverbal Behaviour: Applications and Cultural Implications, pp. 51-66. London: Academic Press.
29. Du Bois, J.W. 2004. Representing Discourse. MS, University of California: Santa Barbara.
30. Du Bois, J.W., S. Cumming, S. Schuetze-Coburn and D. Paolino. 1992. 'Discourse transcription', Santa Barbara Papers in Linguistics 4.
31. Du Bois, J.W., S. Schuetze-Coburn, S. Cumming and D. Paolino. 1993. 'Outline of discourse transcription' in J.A. Edwards and M.A. Lampert (eds) Talking Data: Transcription and Coding in Discourse Research, pp. 45-89. Hillsdale, New Jersey: Erlbaum.
32. Dybkj r, L. and N. Ole Bernsen. 2004. 'Recommendations for natural interactivity and multimodal annotation schemes', Proceedings of the LREC 2004 Workshop on Multimodal Corpora, pp. 5-8. Lisbon, Portugal.
33. Efron, D. 1972 [1941]. Gesture, Race and Culture. The Hague: Mouton and Co.
34. Eggins, S. and D. Slade. 1997. Analysing Casual Conversation. London: Cassell.
35. Ekman, P. and W.V. Friesen. 1968. 'Nonverbal behavior in psychotherapy research' in J. Shlien (ed.) Research in Psychotherapy, Volume III, pp. 179-216. Washington, DC: American Psychological Association.
36. Ekman, P. and W.V. Friesen. 1969. 'The repertoire of non-verbal behavior: categories, origins, usage and coding', Semiotica 1 (1), pp. 49-98.
37. Ekman, P. and W.V. Friesen. 1978. The Facial Action Coding System. San Francisco, California: Consulting Psychologists Press Inc.
38. Evans, J.L., M.W. Alibali and N.M. McNeill. 2001. 'Divergence of verbal expression and embodied knowledge: evidence from speech and gesture in children with specific language impairment', Language and Cognitive Processes 16 (2-3), pp. 309-31.
39. French, A., C. Greenhalgh, A. Crabtree, M. Wright, P. Brundell, A. Hampshire and T. Rodden. 2006. 'Software replay tools for time-based social science data', Proceedings of the Second Annual International e-Social Science Conference. Manchester, England: University of Manchester.
40. Frey, S., H.P. Hirsbrunner, A. Florin, W. Daw and R. Crawford. 1983. 'A unified approach to the investigation of nonverbal and verbal behaviour in communication research' in W. Doise and S. Moscovici (eds) Current Issues in European Social Psychology, pp. 143-99. Cambridge: Cambridge University Press.
41. Gibbon, D., I. Mertins and R.K. Moore (eds). 1997. Handbook of Standards and Resources for Spoken Language Systems. Berlin: Mouton de Gruyter.
42. Goldin-Meadow, S. 1999. 'The role of gesture in communication and thinking', Trends in Cognitive Sciences 3 (11): pp. 419-29.
43. Goodwin, C. 2000. 'Action and embodiment within situated human interaction', Journal of Pragmatics 32 (10), pp. 1489-522.
44. Greenhalgh, C., A. French, P. Tennant, J. Humble and A. Crabtree. 2007. 'From ReplayTool to Digital Replay System', Online Proceedings of the Third International Conference on e-Social Science, ESRC / NSF, 7-9 October. Ann Arbor: Michigan.
45. Griffin, Z.M. 2004. 'The eyes are right when the mouth is wrong', Psychological Science 15 (12), pp. 814-21.
46. Griffin, Z.M. and K. Bock. 2000. 'What the eyes say about speaking', Psychological Science 11 (4), pp. 274-9.
47. Gripsrud, J. 2002. Understanding Media Culture. London: Arnold.
48. Gu, Y. 2006. 'Multimodal text analysis: a corpus linguistic approach to situated discourse', Text and Talk 26 (2), pp. 127-67.
49. Hadar, U. 1997. 'Interpreting at the surface', Clinical Studies 3, pp. 83-104.
50. H llgren, A. and B. Lyberg. 1998. 'Visual speech synthesis with concatenative speech', Proceedings of International Conference on Auditory-Visual Speech Processing (AVSP'98), pp. 181-3. Terrigal: Australia.
51. Hill, D.R. 2000. 'Give us the tools: a personal view of multimodal computer-human dialogue' in M.M. Taylor, F. Ne'el and G.G. Bouwhuis (eds) The Structure of Multimodal Dialogue II, pp. 25-62. Amsterdam: John Benjamins.
52. Holler, J. and G.W. Beattie. 2002. 'A micro-analytic investigation of how iconic gestures and speech represent core semantic features in talk', Semiotica 142 (1-4), pp. 31-69.
53. Holler, J. and G.W. Beattie. 2003. 'How iconic gestures and speech interact in the representation of meaning: are both aspects really integral to the process?', Semiotica 146 (1-4), pp. 81-116.
54. Holler, J. and G.W. Beattie. 2004. 'The interaction of iconic gesture and speech', Fifth International GestureWorkshop. Genova, Italy. Selected Revised Papers. Heidelberg: Springer Verlag.
55. Ide, N. 1998. 'Corpus encoding standard: SGML guidelines for encoding linguistic corpora', First International Language Resources and Evaluation (LREC) Conference. Granada, Spain.
56. Kapoor, A. and R.W. Picard. 2001. 'A real-time head nod and shake detector', ACM International Conference Proceedings Series, pp. 1-5.
57. Kawato, S. and J. Ohya. 2000. 'Real-time detection of nodding and head shaking by directly detecting and tracking the between eyes ', Proceedings of IEEE International Conference on Automatic Face and Gesture Recognition, pp. 40-5.
58. Kendon, A. 1972. 'Some relationships between body motion and speech' in A. Seigman and B. Pope (eds) Studies in Dyadic Communication, pp. 177-216. Elmsford, New York: Pergamon Press.
59. Kendon, A. 1980. 'Gesticulation and speech: two aspects of the process of utterance' in M.R. Key (ed.) The Relationship of Verbal and Nonverbal Communication, pp. 207-27. The Hague; New York: Mouton.
60. Kendon, A. 1982. 'The organisation of behaviour in face-to-face interaction: observations on the development of a methodology' in K.R. Scherer and P. Ekman (eds) Handbook of Methods in Nonverbal Behaviour Research, pp. 440-505. Cambridge: Cambridge University Press.
61. Kendon, A. 1987. 'On gesture: its complementary relationship with speech' in A.W. Siegman and S. Feldstein (eds) Nonverbal Behavior and Communication, pp. 65-97. London: Lawrence Erlbaum Associates.
62. Kendon, A. 1994. 'Do gestures communicate?. A review', Research on Language and Social Interaction 27 (3), pp. 175-200.
63. Kendon, A. 1997. 'Gesture', Annual Review of Anthropology 26, pp. 109-28.
64. Kipp, M. 2001. 'Anvil - a generic annotation tool for multimodal dialogue', Proceedings of the Seventh European Conference on Speech Communication and Technology (Eurospeech), pp. 1367-70. Aalborg, Denmark.
65. Kita, S. 2000. 'How representational gesture helps speaking' in D. McNeil (ed.) Gesture and Language. Cambridge: Cambridge University Press.
66. Kita, S. (ed.). 2003. Pointing: Where Language, Culture and CognitionMeet. Mahwah, New Jersey: Lawrence Erlbaum.
67. Kita, S., I. van Gijn and H. van der Hulst. 1997. 'Movement phase in signs and co-speech gestures and their transcriptions by human coders', Gesture Workshop, pp. 23-35.
68. Knight, D. 2006. 'Corpora: the next generation', Part of the AHRC funded online Introduction to Corpus Investigative Techniques, The University of Birmingham. Accessed 26 February 2006, at: http://www.humcorp.bham.ac.uk/
69. Knight, D. and S. Adolphs. 2008 'Multi-modal corpus pragmatics: the case of active listenership' in J. Romeo (ed.) Corpus and Pragmatics, pp. 175-90. Berlinand New York: Mouton de Gruyter.
70. Knight, D. and S. Adolphs. 2006. Text, Talk and Corpus Analysis. University of Nottingham Academic Module MA.
71. Knight, D., S. Bayoumi, S. Mills, A. Crabtree, S. Adolphs, T. Pridmore and R. Carter. 2006. 'Beyond the text: construction and analysis of multimodal linguistic corpora', Proceedings of the Second International Conference on e-Social Science, 28-30 June. Manchester.
72. Knudsen, M.W., J.C. Martin, L. Dybkj r, M.J.M.N. Ayuso, N.O. Bernsen, J. Carletta, S. Kita, U. Heid, J. Llisterri, C. Pelachaud, I. Poggi, N. Reithinger, G. van ElsWijk and P. Wittenburg. 2002. 'Survey of multimodal annotation schemes and best practice', ISLE Deliverable D9.1, 2002.
73. Kress, G.R. 2001. Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Arnold.
74. Kress, G.R. and T. van Leeuwen. 1996. Reading Images. London: Routledge.
75. Labov, W. 1972. Sociolinguistic Patterns. Philadelphia: University of Pennsylvania Press.
76. Lapadat, J.C. and A.C. Lindsay. 1999. 'Transcription in research and practice: from standardisation of technique to interpretative positioning', Qualitative Inquiry 5 (1), pp. 64-86.
77. Laver, J. 1994. Principles of Phonetics. Cambridge: Cambridge University Press.
78. Leech, G. 1997. 'Introducing corpus annotation' in R. Garside, G. Leech and T.McEnery (eds) Corpus Annotation: Linguistic Information from Computer Text Corpora, pp. 1-18. London: Longman.
79. Leech, G. 2004. 'Adding linguistic annotation' in M. Wynne (ed.) Developing Linguistic Corpora: A Guide to Good Practice, pp. 1-16. Oxford: Oxbow Books. Accessed 26 February 2006, at: http://ahds.ac.uk/linguistic-corpora/
80. Leech, G., G. Myers and J. Thomas (eds). 1995. Spoken English on Computer: Transcription, Mark-up and Application. London: Longman.
81. Markee, N. 2000. Conversation Analysis. Mahwah, New Jersey: Lawrence Erlbaum Associates.
82. Martinec, R. 1998. 'Cohesion in action', Semiotica 120 (1/2), pp. 161-80.
83. Martinec, R. 2001. 'Interpersonal resources in action', Semiotica 135 (1/4), pp. 117-45.
84. McClave, E.Z. 2000. 'Linguistic functions of head movements in the context of speech', Journal of Pragmatics 32 (7), pp. 855-78.
85. McEnery, A. and Z. Xiao. 2004. 'The Lancaster Corpus of Mandarin Chinese: a corpus for monolingual and contrastive language study' in M. Lino, M. Xavier, F. Ferreire, R. Costa and R. Silva (eds) Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC) 2004, pp. 1175-8. Lisbon, Portugal.
86. McNeill, D. 1979. The Conceptual Basis of Language. Hillsdale, New Jersey: Erlbaum.
87. McNeill, D. 1985. 'So you think gestures are nonverbal?', Psychological Review 92 (3), pp. 350-71.
88. McNeill, D. 1992. Hand and Mind: What Gestures Reveal about Thought. Chicago: University of Chicago Press.
89. McNeill, D., J. Cassell and K.E. McCullough. 1994. 'Communicative effects of speech-mismatches gestures', Research on Language and Social Interaction 27 (3), pp. 223-37.
90. Meyer, C.F. 2002. English Corpus Linguistics: An Introduction. Cambridge: Cambridge University Press.
91. Nobe, S. 1996. 'Cognitive rhythms gestures and acoustic aspects of speech'. PhD thesis, University of Chicago.
92. Noller, P. 1984. Nonverbal Communication and Marital Interaction. Oxford: Pergamon Press.
93. O'Connell, D.C. and S. Kowal. 1999. 'Transcription and the issue of standardisation', Journal of Psycholinguistic Research 28 (2), pp. 103-20.
94. Ochs, E. 1979. 'Transcription as theory' in E. Ochs and B.B. Schieffelin (eds) Developmental Pragmatics. New York: Academic Press.
95. O'Keeffe, A. and S. Adolphs. 2008. 'Response tokens in British and Irish discourse: corpus, context and variational pragmatics' in A. Barron and K. Schneider (eds) Variational Pragmatics. Amsterdam: John Benjamins.
96. Olsher, D. 2004. 'Talk and gesture: the embodied completion of sequential actions in spoken interaction' in R. Wagner and J. Gardner (eds) Second Language Conversations, pp. 221-45. London: Continuum.
97. Pea, R., M. Mills, J. Rosen, K. Dauber, W. Effelsberg and E. Hoffert. 2004. 'The diver project: interactive digital video repurposing', IEEE MultiMedia 11 (1), pp. 54-61.
98. Prillwitz, S., R. Leven, H. Zienert, T. Hanke and J. Henning. 1989.
99. HamNoSys. Version 2.0, Hamburg Notation System for Sign Language -An Introductory Guide. Hamburg: Signum.
100. Psathas, G and T. Anderson. 1990. 'The practices of transcription in conversation analysis', Semiotica 781 (2), pp. 75-99.
101. Rayson, P. 2003. 'Matrix: a statistical method and software tool for linguistic analysis through corpus comparison'. PhD thesis, Lancaster University.
102. Reppen, R. and R. Simpson. 2002. 'Corpus linguistics' in N. Schmitt (ed.) An Introduction to Applied Linguistics, pp. 92-111. London: Arnold.
103. Richmond, V.P., J.C. McCroskey and S.K. Payne. 1991. Nonverbal Behaviour in Interpersonal Relations. New Jersey: Prentice Hall.
104. Rim , B. and L. Schiarartura. 1991. 'Gesture and speech' in R.S. Feldman and D. Rim (eds) Fundamentals of Nonverbal Behaviour, pp. 239-84. New York: Cambridge University Press.
105. Roberts, C. 2006. 'Part one: issues in transcribing spoken discourse', from the academic course qualitative research methods and transcription, Kings College London. Available online at: http://www.kcl.ac.uk/ content/1/c6/01/81/04/part1.pdf
106. Saferstein, B. 2004. 'Digital technology and methodological adaptation: text on video as a resource for analytical reflexivity', Journal of Applied Linguistics 1 (2), pp. 197-223.
107. Schegloff, E.A. 1984. 'On some gestures' relation to talk' in J.M. Atkinson and E.J. Heritage (eds) Structures of Social Action: Studies in Conversation Analysis, pp. 266-96. Cambridge: Cambridge University Press.
108. Scholfield, P. 1995. Quantifying Language. Clevedon: Multilingual Matters Ltd.
109. Scollon, R. 1998. Mediated Discourse as Social Interaction. London: Longman.
110. Streeck, J. 1993. 'Gesture as communication 1: its coordination with gaze and speech', Communication Monographs 60, pp. 275-99.
111. Streeck, J. 1994. 'Gestures as communication 2: the audience as co-author', Research on Language and Social Interaction 27 (3), pp. 239-67.
112. Taylor, M.M., F. Ne'eland and G.G. Bouwhuis (eds). 2000. The Structure of Multimodal Dialogue II. Amsterdam: John Benjamins.
113. ten Have, P. 2007. Doing Conversational Analysis: A Practical Guide. (Second edition.) London: Sage.
114. Thompson, L.A. and D.W. Massaro. 1986. 'Evaluation and integration of speech and pointing gestures during referential understanding', Journal of Experimental Child Psychology 42 (1), pp. 144-68.
115. Thompson, P. 2004. 'Spoken language corpora' in M. Wynne (ed.) Developing Linguistic Corpora: A Guide to Good Practice, pp. 59-70. Oxford: Oxbow Books.
116. Tian, Y., T. Kanade and J.F. Cohn. 2000. 'Dual-state parametric eye tracking', Proceedings of IEEE International Conference on Automatic Face and Gesture Recognition.
117. Wilcox, S. 2004. 'Language from gesture', Behavioral and Brain Sciences 27 (4), pp. 524-5.
118. Wittenburg, P., D. Broeder and B. Sloman. 2000. 'Meta-description for language resources', EAGLES/ISLE White Paper. Accessed 2 October 2006, at: http://www.mpi.nl/world/ISLE/documents/papers/white_ paper_11.pdf
119. Yngve, V. 1970. 'On getting a word in edgewise', Papers from the Sixth Regional Meeting of the Chicago Linguistic Society, pp. 567-77. Chicago: Chicago Linguistic Society.