Estimation of discourse segmentation labels from crowd data

Conference Proceedings - EMNLP 2015: Conference on Empirical Methods in Natural Language Processing

Volumn , Issue , 2015, Pages 2190-2200

  Huang, Ziheng ^a   Zhong, Jialu ^b   Passonneau, Rebecca J ^a  

^a Och Spine at New York Presbyterian Hospitals   (United States)

^b NONE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

NATURAL LANGUAGE PROCESSING SYSTEMS;

DISCOURSE SEGMENTATION; GROUND TRUTH; INDEPENDENCE ASSUMPTION; MAJORITY VOTE; PROBABILISTIC MODELS; SEQUENTIAL DATA;

HIDDEN MARKOV MODELS;

EID: 84959922810     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.18653/v1/d15-1261     Document Type: Conference Paper

References (37)

1. Paul S. Albert and Lori E. Dodd. 2008. On estimating diagnostic accuracy from studies with multiple raters and partial gold standard evaluation. Journal of the American Statistical Association, 103(481):61-73.
2. Andre Berchtold. 1999. The Double Chain Markov Model. Communications in Statistics: Theory and Methods, 28(348):2569-2589.
3. Mohammad Hadi Bokaei, Hossein Sameti, and Yang Liu. 2015. Linear discourse segmentation of multiparty meetings based on local and global information. IEEE Transactions on Audio, Speech and Language Processing, 23(11):1879-1891.
4. Wauter Bosma. 2005. Extending answers using discourse structure. In RANLP Workshop on Crossing Barriers in Text Summarization Research.
5. Glenn Branch. 2014. Whence lumpers and splitters? http://ncse.com/blog/2014/11/whence-lumpers-splitters-0 0160 04, December.
6. Rebecca F. Bruce and Janyce M. Wiebe. 1999. Recognizing subjectivity: a case study of manual tagging. Natural Language Engineering, 1(1):1-16.
7. A. P. Dawid and A. M. Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics), 28(1):20-28.
8. Han de Vries. 1998. Finding a dominance order most consistent with a linear hierarchy: a new procedure and review. Animal Behavior, 55:827-843.
9. Alfred Dielmann and Steve Renals. 2005. Multistream dynamic Bayesian network for meeting segmentation. In Samy Bengio and Herv Bourlard, editors, Machine Learning for Multimodal Interaction, volume 3361 of Lecture Notes in Computer Science, pages 76-86. Springer Berlin Heidelberg.
10. Dmitriy Dligach, Rodney D. Nielsen, and Martha Palmer. 2010. To annotate more accurately or to annotate more. In Proceedings of the Fourth Linguistic Annotation Workshop (LAW TV), pages 64-72.
11. Raquel Fernandez, Matthew Frampton, Patrick Ehlen, Matthew Purver, and Stanley Peters. 2008. Modelling and detecting decisions in multi-party dialogue. In Proceedings of the 9th SIGdial Workshop on Discourse and Dialogue.
12. Tim Finin, Will Murnane, Anand Karandikar, Nicholas Keller, Justin Martineau, and Mark Dredze. 2010. Annotating named entities in twitter data with crowdsourcing. In Proceedings of the NAACL HIT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, CSLDAMT'10, pages 80-88, Stroudsburg, PA, USA. Association for Computational Linguistics.
13. Michel Galley, Kathleen R. McKeown, Eric Fosler-Lussier, and Hongyan Jing. 2003. Discourse segmentation of multi-party conversation. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 562-569, Sapporo, Japan, July. Association for Computational Linguistics.
14. Sebastian Germesin and Theresa Wilson. 2009. Agreement detection in multiparty conversation. In Proceedings of the 2009 International Conference on Multimodal Interfaces, ICMI-MLMI'09, pages 7-14, New York, NY, USA. ACM.
15. Dustin Hillard, Mart Ostendorf, and Elizabeth Shriberg. 2003. Detection of agreement vs. disagreement in meetings: Training with unlabeled data. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology: Companion Volume of the Proceedings of HLT-NAACL 2003-short Papers - Volume 2, NAACL-Short'03, pages 34-36, Stroudsburg, PA, USA. Association for Computational Linguistics.
16. Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning whom to trust with MACE. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 1120-1130, Atlanta, Georgia, June. Association for Computational Linguistics.
17. Klaus Krippendorff. 1980. Content analysis: An introduction to its methodology. Sage Publications, Beverly Hills, CA.
18. David Leiva, Antonio Solanas, and Lluis Salafranca. 2008. Testing reciprocity in social interactions: A comparison between the directional consistency and skew-symmetry statistics. Behavior Research Methods, 40(2):626-634.
19. Miriam Martinez and L. Enrique Sucar. 2008. Learning dynamic Naive Bayesian classifiers. In Proceedings of the Twenty-First International FLAIRS Conference, pages 655-659.
20. Kevin Murphy. 2001. Bayes Net toolbox for Matlab. Computing Science and Statistics, 33(2):1024-1034.
21. John Niekrasz and Johanna Moore. 2009. Participant subjectivity and involvement as a basis for discourse segmentation. In Proceedings of the SIGDIAL 2009 Conference, pages 54-61, London, UK, September. Association for Computational Linguistics.
22. Rebecca J. Passonneau and Bob Carpenter. 2014. The benefits of a model of annotation. Transactions of the Association for Computational Linguistics, 2:311326.
23. Rebecca J. Passonneau and Diane Litman. 1997. Discourse segmentation by human and automated means. Computational Linguistics, 23. 1:103-139. Special Issue on Empirical Studies in Discourse Interpretation and Generation.
24. Rebecca J. Passonneau, Irene Alvarado, Phil Crone, and Simon Jerome. 2011. PARADISE-style evaluation of a human-human library corpus. In Proceedings of the SIGDIAL 2011 Conference, pages 325-331, Portland, Oregon, June. Association for Computational Linguistics.
25. Rebecca J. Passonneau, Boxuan Guan, Cho Ho Yeung, Yuan Du, and Emma Conner. 2014. Aspectual properties of conversational activities. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 228-237, Philadelphia, PA, U. S. A., June. Association for Computational Linguistics.
26. Matthew Purver, John Dowding, John Niekrasz, Patrick Ehlen, Sharareh Noorbaloochi, and Stanley Peters. 2007. Detecting and summarizing action items in multi-party dialogue. In Proceedings of the 8th SIGdial Workshop on Discourse and Dialogue.
27. Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. 2010. Learning from crowds. Journal of Machine Learning Research, 11:1297-1322.
28. Simon Rogers, Mark Girolami, and Tamara Polajnar. 2010. Semi-parametric analysis of multi-rater data. Statistical Computing, 20:317-334.
29. Andrey Rzhetsky, Hagit Shatkay, and W. John Wilbur. 2009. How to get the most out of your curation effort. PLoS Computational Biology., 5(5):1-13.
30. Victor S. Sheng, Foster Provost, and Panagiotis G. Ipeirotis. 2008. Get another label? Improving data quality and data mining using multiple, noisy labelers. In Proceedings of the Fourteenth ACM International Conference on Knowledge Discovery and Data Mining (KDD).
31. Padhraic Smyth, Usama Fayyad, Michael Burl, Pietro Perona, and Pierre Baldi. 1995. Inferring ground truth from subjectively-labeled images of Venus. In Advances in Neural Information Processing Systems 7, pages 1085-1092. MIT Press.
32. Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast - but is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of Empirical Methods in Natural Language Processing (EMNLP), pages 254-263, Honolulu.
33. Swapna Somasundaran, Josef Ruppenhofer, and Janyce Wiebe. 2007. Detecting arguing and sentiment in meetings. In Proceedings of the 8th SIGdial Workshop on Discourse and Dialogue (SIGdial 07), page 2634.
34. Mingyu Sun and Joyce Y Chai. 2007. Discourse processing for context question answering based on linguistic knowledge. Know.-Based Syst, 20(6):511-526, August.
35. Lu Wang and Claire Cardie. 2012. Focused meeting summarization via unsupervised relation extraction. In Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue, SIGDIAL'12, pages 304-313, Stroudsburg, PA, USA. Association for Computational Linguistics.
36. Nigel G. Ward, Steven D. Werner, Fernando Garcia, and Emilio Sanchis. 2015. A prosody-based vectorspace model of dialog activity for information retrieval. Speech Communication, 28:85-96.
37. Jacob Whitehill, Paul Ruvolo, Tingfan Wu, Jacob Bergsma, and Javier Movellan. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Proceedings of the 24th Annual Conference on Advances in Neural Information Processing Systems.