Evaluating dependency parsing: Robust and heuristics-free cross-annotation evaluation

EMNLP 2011 - Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference

Volumn , Issue , 2011, Pages 385-396

  Tsarfaty, Reut ^a   Nivre, Joakim ^a   Andersson, Evelina ^a  

^a UPPSALA UNIVERSITY   (Sweden)

Author keywords

[No Author keywords available]

Indexed keywords

COMMON GROUND; DEPENDENCY PARSING; GOLD STANDARDS; HIGHLY SENSITIVE; ROBUST PROCEDURES; TREE EDIT DISTANCE; TREEBANKS;

COMPUTATIONAL LINGUISTICS;

NATURAL LANGUAGE PROCESSING SYSTEMS;

EID: 80053262124     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (35)

1. Philip Bille. 2005. A survey on tree edit distance and related. problems. Theoretical Computer Science, 337:217-239.
2. Ezra Black, Steven P. Abney, D. Flickenger, Claudia Gdaniec, Ralph Grishman, P. Harrison, Donald Hindle, Robert Ingria, Frederick Jelinek, Judith L. Klavans, Mark Liberman, Mitchell P. Marcus, Salim Roukos, Beatrice Santorini, and Tomek Strzalkowski. 1991. Procedure for quantitatively comparing the syntactic coverage of English grammars. In E. Black, editor, Proceedings of the workshop on Speech and Natural Language, HLT, pages 306-311. Association for Computational Linguistics.
3. Ted Briscoe, John Carroll, Jonathan Graham, and Ann Copestake. 2002. Relational evaluation schemes. In Proceedings of LREC Workshop "Beyond Parseval - Towards improved evaluation measures for parsing systems".
4. Sabine Buchholz and Erwin Marsi. 2006. CoNLL-X shared task on multilingual dependency parsing. In Proceedings of CoNLL-X, pages 149-164.
5. Ekaterina Buyko and Udo Hahn. 2010. Evaluating the impact of alternative dependency graph encodings on solving event extraction tasks. In Proceedings of EMNLP, pages 982-992.
6. John Carroll, Ted Briscoe, and Antonio Sanfilippo. 1998. Parser evaluation: a survey and a new proposal. In Proceedings of LREC, pages 447-454.
7. Daniel Cer, Marie-Catherine de Marneffe, Daniel Jurafsky, and Christopher D. Manning. 2010. Parsing to stanford dependencies: Trade-offs between speed and accuracy. In Proceedings of LREC.
8. Jinho D. Choi and Martha Palmer. 2010. Robust constituent-to-dependency conversion for English. In Proceedings of TLT.
9. Paul Cohen. 1995. Empirical Methods for Artificial Intelligence. The MIT Press.
10. Marie-Catherine de Marneffe, Bill MacCartney, and Christopher D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In Proceedings of LREC, pages 449-454.
11. Martin Emms. 2008. Tree-distance and some other variants of evalb. In Proceedings of LREC.
12. Erhard Hinrichs, Sandra Kübler, Karin Naumann, Heike Telljohan, and Julia Trushkina. 2004. Recent development in linguistic annotations of the TüBa-D/Z Tree-bank. In Proceedings of TLT.
13. Richard Johansson and Pierre Nugues. 2007. Extended constituent-to- dependency conversion for English. In Proceedings of NODALIDA.
14. Sandra Kübler and Heike Telljohann. 2002. Towards a dependency-oriented evaluation for partial parsing. In Proceedings of LREC Workshop "Beyond Parseval - Towards improved evaluation measures for parsing systems".
15. Sandra Kübler, Ryan McDonald, and Joakim Nivre. 2009. Dependency Parsing. Number 2 in Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers.
16. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19:313-330.
17. Ryan McDonald and Fernando Pereira. 2006. Online learning of approximate dependency parsing algorithms. In Proceedings of EACL, pages 81-88.
18. Ryan McDonald, Koby Crammer, and Fernando Pereira. 2005. Online large-margin training of dependency parsers. In Proceedings of ACL, pages 91-98.
19. Igor Mel'čuk. 1988. Dependency Syntax: Theory and Practice. State University of New York Press.
20. Yusuke Miyao, Rune Sætre, Kenji Sagae, Takuya Matsuzaki, and Jun'ichi Tsujii. 2008. Task-oriented evaluation of syntactic parsers and their representations. In Proceedings of ACL, pages 46-54.
21. Joakim Nivre and Beata Megyesi. 2007. Bootstrapping a Swedish Treebank using cross-corpus harmonization and annotation projection. In Proceedings of TLT.
22. Joakim Nivre and Jens Nilsson. 2005. Pseudo projective dependency parsing. In Proceeding of ACL, pages 99-106.
23. Joakim Nivre and Mario Scholz. 2004. Deterministic dependency parsing of English text. In Proceedings of COLING, pages 64-70.
24. Joakim Nivre, Johan Hall, Sandra Kübler, Ryan McDonald, Jens Nilsson, Sebastian Riedel, and Deniz Yuret. 2007a. The CoNLL 2007 shared task on dependency parsing. In Proceedings of the CoNLL Shared Task Session of EMNLP-CoNLL 2007, pages 915-932.
25. Joakim Nivre, Jens Nilsson, Johan Hall, Atanas Chanev, Gülsen Eryigit, Sandra Kübler, Svetoslav Marinov, and Erwin Marsi. 2007b. Maltparser: A language-independent system for data-driven dependency parsing. Natural Language Engineering, 13(1):1-41.
26. Joakim Nivre, Laura Rimell, Ryan McDonald, and Carlos Gómez-Rodríguez. 2010. Evaluation of dependency parsers on unbounded dependencies. pages 813-821.
27. Owen Rambow. 2010. The Simple Truth about Dependency and Phrase Structure Representations: An Opinion Piece. In Proceedings of HLT-ACL, pages 337-340.
28. Ines Rehbein and Josef van Genabith. 2007. Why is it so difficult to compare treebanks? Tiger and TüBa-D/Z revisited. In Proceedings of TLT, pages 115-126.
29. Brian Roark. 2002. Evaluating parser accuracy using edit distance. In Proceedings of LREC Work-shop "Beyond Parseval - Towards improved evaluation measures for parsing systems".
30. Roy Schwartz, Omri Abend, Roi Reichart, and Ari Rappoport. 2011. Neutralizing linguistically problematic annotations in unsupervised dependency parsing evaluation. In Proceedings of ACL, pages 663-672.
31. Stuart M. Shieber. 1986. An Introduction to Unification-Based Grammars. Center for the Study of Language and Information.
32. Wojciech Skut, Brigitte Krenn, Thorsten Brants, and Hans Uszkoreit. 1997. An annotation scheme for free word-order languages. In Proceedings of the fifth conference on Applied natural language processing, pages 88-95.
33. Hiroyasu Yamada and Yuji Matsumoto. 2003. Statistical dependency analysis with support vector machines. In Proceeding of IWPT, pages 195-206.
34. Kaizhong Zhang and Dennis Shasha. 1989. Simple fast algorithms for the editing distance between trees and related problems. In SIAM Journal of Computing, volume 18, pages 1245-1262.
35. Arnold M. Zwicky. 1993. Heads, bases, and functors. In G.G. Corbett, N. Fraser, and S. McGlashan, editors, Heads in Grammatical Theory, pages 292-315. Cambridge University Press.