Analysis of inter-transcriber consistency in the Cat-ToBI prosodic labeling system

Speech Communication

Volumn 54, Issue 4, 2012, Pages 566-582

  Escudero, David ^a   Aguilar, Lourdes ^b   Vanrell, Maria Del Mar ^b,d   Prieto, Pilar ^c  

^a UNIVERSITY OF VALLADOLID   (Spain)

^b UNIVERSITAT AUTÒNOMA DE BARCELONA   (Spain)

^c INSTITUCIÓ CATALANA DE RECERCA I ESTUDIS AVANÇATS ICREA   (Spain)

^d UNIVERSITAT POMPEU FABRA   (Spain)

Author keywords

Inter transcriber consistency; Prosodic labeling; Prosody; ToBI

Indexed keywords

COMPUTER SIMULATION;

DISTANCE MATRICES; INTER-TRANSCRIBER CONSISTENCY; MULTI-DIMENSIONAL SCALING; PROSODIC LABELING; PROSODY; RELIABILITY TEST; TOBI;

COMPUTER APPLICATIONS;

EID: 84856620618     PISSN: 01676393     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.specom.2011.12.002     Document Type: Article

References (51)

1. Aguilar, L.; de la Mota, C.; Prieto, P. (coords.), 2009-2011. Cat-ToBI Training Materials URL .
2. S. Ananthakrishnan, and S. Narayanan Automatic Prosodic Event Detection Using Acoustic, Lexical, and Syntactic Evidence IEEE Trans. Audio, Speech, Lang. Process. 16 1 2008 216 228
3. A. Arvaniti, and M. Baltazani Intonational analysis and prosodic annotation of Greek spoken corpora S.-A. Jun, Prosodic Typology: The Phonology of Intonation and Phrasing 2005 Oxford University Press Oxford 84 117
4. M. Beckman Intonation across Spanish in the tones and break indices framework Probus 14 2002 9 36
5. M. Beckman, and S. Jun K-ToBI (Korean ToBI) labeling conventions: version 3 The Korean J. Speech Sci. 7 1 2000 143 169
6. Beckman, M.E.; Campos, M.D.; McGregory, J.T.; Morgan, T.A. 2000. Intonation across Spanish, in the tones and break indices framework. Technical Report, University of Ohio, http://www.ling.ohio-state.edu/tobi/sp-tobi/.
7. M. Beckman, J. Hirschberg, and S. Shattuck-Hufnagel The original ToBI system and the evolution of the ToBI framework S.-A. Jun, Prosodic Typology: The Phonology of Intonation and Phrasing 2005 Oxford University Press New York 9 54
8. Boersma, P.; Weenink, D. 2011. Praat, doing phonetics by computer [Computer program]. Version 5.2.26, (accessed 24.05.11).
9. Bonafonte, A.; Adell, J.; Esquerra, I.; Gallego, S.; Moreno, A.; Pérez, J. 2008. Corpus and Voices for Catalan Speech Synthesis. In: Proceedings of LREC, Marrakech, pp. 3325-3329.
10. I. Borg, and P. Groenen Modern Multimensional Scaling: theory and applications 2005 Springer-Verlag New York
11. Brugos, A.; Veilleux, N.; Breen, M.; Shattuck-Hufnagel, S. 2008. The Alternatives (Alt) Tier for ToBI: Advantages of Capturing Prosodic Ambiguity, in: Proceedings of Speech Prosody 2008, pp. 273-276.
12. Buhmann, J.; Caspers, J.; van Heuven, V.; Hoekstra, H.; Martens, J.; Swerts, M. 2002. Annotation of prominent words, prosodic boundaries and segmental lengthening by non-expert transcribers in the spoken Dutch corpus. In: Proceedings of LREC, pp. 779-785.
13. J. Cohen A coefficient for agreement for nominal scales Edu. Psychol. Measure. 20 1960 37 46
14. E. Estebas, and P. Prieto La notación prosódica del español. Una revisión del Sp-ToBI Estud. de Fonética Experiment. XVIII 2009 263 283
15. Estebas Vilaplana, E.; Prieto, P. 2010. Castilian Spanish Intonation. In: P. Prieto, P. Roseano (Eds.), Transcription of Intonation of the Spanish Language, Lincom Europa, München, pp. 17-48.
16. J. Fleiss Measuring nominal scale agreement among many raters Psychol. Bull. 76 5 1971 378 382
17. Godfrey, J.J.; Holliman, E.C.; McDaniel, J. 1992. SWITCHBOARD: telephone speech corpus for research and development. In: Proceedings of ICASSP, Vol. 1, pp. 517-520.
18. González, C.; Vivaracho, C.; Escudero, D.; Cardeñoso, V. 2010. On the automatic ToBI accent type identification from data. In: Proceedings of Interspeech 2010, pp. 142-145.
19. M. Grice, and R. Benzmüller Transcription of German using ToBI Tones- The Saarbrücken System Phonus 1 1995 33 51
20. Grice, M.; Reyelt, M.; Benzmüller, R.; Mayer, J.; Batliner, A. 1996. Consistency in Transcription and Labelling of German Intonation with GToBI. In: Proceedings ICSLP, pp. 1716-1719.
21. K.L. Gwet Computing inter-rater reliability and its variance in the presence of high agreement British J. Math. Stat. Psychol. 61 2008 26 48
22. K.L. Gwet Handbook of Inter-Rater Reliability second ed. Advanced Analytics 2010 LLC
23. M. Hasegawa-Johnson, K. Chen, J. Cole, S. Borys, S.-S. Kim, A. Cohen, T. Zhang, J.-Y. Choi, H. Kim, T. Yoon, and S. Chavarría Simultaneous recognition of words and prosody in the Boston university radio speech corpus Speech Commun. 46 2005 418 439 (Pubitemid 40952526)
24. R. Herman, and J. McGory The conceptual similarity of intonational tones and its effects on intertranscriber reliability Lang. Speech 45 2002 1 36
25. D. Hirst Form and function in the representation of speech prosody Speech Commun. 46 2005 334 347 (Pubitemid 40952520)
26. R. Ihaka, and R. Gentleman R: A language for data analysis and graphics J. Comput. Graph. Stat. 5 3 1996 299 314
27. Jun, S.; Lee, S.; Kim, K.; Lee, Y. 2000. Labeler agreement in transcribing Korean intonation with K-ToBI. In: Proceedings of ICSLP, Vol. 3, pp. 211-214.
28. Kruskal, J.; Wish, M. 1978. Multidimensonal Scaling. Sage University Paper series on Quantitative Application in the Social Sciences, Beverly Hills.
29. J. Landis, and G. Koch The measurement of observer agreement for categorical data Biometrics 33 1977 159 174 (Pubitemid 8054849)
30. Mayo, C.; Aylett, M.; Ladd, D. 1997. Prosodic Transcription of Glasgow English: An Evaluation Study of GlaToBI. In: Botinis, A.; Kouroupetroglou, G.; Carayiannis G. (Eds.), Proceedings of ESCA Workshop: Theory, Models and Applications, pp. 231-234.
31. Ohio-State-University, 2006. What is ToBI? URL http://www.ling.ohio- state.edu/tobi/.
32. Peña, D. 1999. Estadística. Modelos y Métodos, Alianza, Madrid.
33. Pierrehumbert, J.B. 1980. The phonology and phonetics of English intonation, Ph.D. thesis, MIT.
34. Pitrelli, J.F.; Beckman, M.E.; Hirschberg, J. 1994. Evaluation of prosodic transcription labeling reliability in the ToBI framework. In: Proceedings of ICSLP, pp. 123-126.
35. M. Pitt, K. Johnson, E. Hume, S. Kiesling, and W. Raymon The Buckeye corpus of conversational speech: labeling conventions and a test of transcriber reliability Speech Commun. 45 2005 89 95 (Pubitemid 40034711)
36. Prieto, P.; Cabré, T. (coords.), 2007-2010. The Interactive Atlas of Catalan Intonation, URL .
37. P. Prieto, L. Aguilar, I. Mascaró, F.J. Torres-Tamarit, and M.M. Vanrell L'etiquetatge prosdic Cat-ToBI Estud. de Fonética Experiment. XVIII 2009 287 309
38. Prieto, P.; 2012. The intonational phonology of Catalan. In: Jun, S. (Ed.), Prosodic Typology 2, The Phonology of Intonation and Phrasing. Oxford University Press, Oxford, in press.
39. S. Prom-on, Y. Xu, and B. Thipakorn Modeling tone and intonation in Mandarin and English as a process of target approximation J. Acoust. Soc. Amer. 125 2009 405 424
40. V.K. Rangarajan Sridhar, S. Bangalore, and S.S. Narayanan Exploiting acoustic and syntactic features for automatic prosody labeling in a maximum entropy framework IEEE Trans. Audio, Speech, Lang. Process. 16 4 2008 797 811
41. Rosenberg, A. 2009. Automatic Detection and Classification of Prosodic Events, Ph.D. thesis, Columbia University, USA.
42. Rosenberg, A. 2010. Classification of Prosodic Events using Quantized Contour Modeling. In: HLT-NAACL, pp. 721-724.
43. W. Scott Reliability of content analysis: The case of nominal scale coding Public Opinion Quart. 17 1955 321 325
44. Silverman, K.; Beckman, M.; Pitrelli, J.; Ostendorf, M.; Wightman, C.; Price, P.; Pierrehumbert, J.; Hirschberg, J. 1992. ToBI: A standard for labelling English prosody. In: Proceedings of ICSLP, pp. 867-870.
45. J. Sim, and C. Wright The kappa statistic in reliability studies: use, interpretation, and sample size requirements Phys. Therapy 85 3 2005 257 268 (Pubitemid 40333545)
46. Syrdal, A.; McGory, J. 2000. Inter-transcriber reliability of ToBI prosodic labeling. In: Proceedings of ICSLP, Vol. 3, pp. 235-238.
47. A.K. Syrdal, J. Hirshberg, J. McGory, and M. Beckman Automatic ToBI prediction and alignment to speech manual labeling of prosody Speech Commun. 33 2001 135 151 (Pubitemid 32029456)
48. J. Uebersax Diversity of decision making models and the measurement of interrater agreement Psychol. Bull. 101 1987 140 146
49. J.J. Venditti The J-ToBI model of Japanese intonation S.-A. Jun, Prosodic Typology: The Phonology of Intonation and Phrasing 2005 Oxford University Press New York 172 200
50. Wightman, C.W. 2002. ToBI or not ToBI. In: B. Bel, I. Marlien (Eds.), Proceedings of Speech Prosody.
51. Yoon, T.; Chavarría, S.; Cole, J.; Hasegawa-Johnson, M. 2004. Intertranscriber reliability of prosodic labeling on telephone conversation using ToBI, in: Proceedings of Speech Interspeech, Jeju, pp. 2729-2732.