An empirical investigation of statistical significance in NLP

EMNLP-CoNLL 2012 - 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, Proceedings of the Conference

Volumn , Issue , 2012, Pages 995-1005

  Berg Kirkpatrick, Taylor ^a   Burkett, David ^a   Klein, Dan ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EMPIRICAL INVESTIGATION; HOLD UP; NLP SYSTEMS; SIGNIFICANCE LEVELS; SIGNIFICANCE TEST; STATISTICAL SIGNIFICANCE; TEST SETS;

NATURAL LANGUAGE PROCESSING SYSTEMS;

EID: 84883315086     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (28)

1. D.M. Bikel. 2004. Intricacies of collins' parsing model. Computational Linguistics.
2. M. Bisani and H. Ney. 2004. Bootstrap estimates for confidence intervals in asr performance evaluation. In Proc. of ICASSP.
3. C. Callison-Burch, P. Koehn, C. Monz, K. Peterson, M. Przybocki, and O.F. Zaidan. 2010. Findings of the 2010 joint workshop on statistical machine translation and metrics for machine translation. In Proc. of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR.
4. M. Collins. 1999. Head-driven statistical models for natural language parsing. Ph.D. thesis, University of Pennsylvania.
5. H.T. Dang and K. Owczarzak. 2008. Overview of the tac 2008 update summarization task. In Proc. of Text Analysis Conference.
6. B. Efron and R. Tibshirani. 1993. An introduction to the bootstrap. Chapman & Hall/CRC.
7. L. Gillick and S.J. Cox. 1989. Some statistical issues in the comparison of speech recognition algorithms. In Proc. of ICASSP.
8. A. Haghighi, J. Blitzer, J. DeNero, and D. Klein. 2009. Better word alignments with supervised ITG models. In Proc. of ACL.
9. D. Klein and C.D. Manning. 2003. Accurate unlexical-ized parsing. In Proc. of ACL.
10. P. Koehn, H. Hoang, A. Birch, C. Callison-Burch, M. Federico, N. Bertoldi, B. Cowan, W. Shen, C. Moran, R. Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proc. of ACL.
11. P. Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proc. of EMNLP.
12. P. Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In MT summit.
13. Z. Li, C. Callison-Burch, C. Dyer, J. Ganitkevitch, S. Khudanpur, L. Schwartz, W.N.G. Thornton, J. Weese, and O.F. Zaidan. 2009. Joshua: An open source toolkit for parsing-based machine translation. In Proc. of the Fourth Workshop on Statistical Machine Translation.
14. P. Liang, B. Taskar, and D. Klein. 2006. Alignment by agreement. In Proc. of NAACL.
15. C.Y. Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Proc. of the Workshop on Text Summarization.
16. M.P. Marcus, M.A. Marcinkiewicz, and B. Santorini. 1993. Building a large annotated corpus of english: The penn treebank. Computational linguistics.
17. R. McDonald, F. Pereira, K. Ribarov, and J. Hajič. 2005. Non-projective dependency parsing using spanning tree algorithms. In Proc. of EMNLP.
18. J. Nivre, J. Hall, and J. Nilsson. 2006. Maltparser: A data-driven parser-generator for dependency parsing. In Proc. of LREC.
19. J. Nivre, J. Hall, S. Kübler, R. McDonald, J. Nilsson, S. Riedel, and D. Yuret. 2007. The conll 2007 shared task on dependency parsing. In Proc. of the CoNLL Shared Task Session of EMNLP-CoNLL 2007.
20. E.W. Noreen. 1989. Computer Intensive Methods for Hypothesis Testing: An Introduction. Wiley, New York.
21. F.J. Och and H. Ney. 2003. A systematic comparison of various statistical alignment models. Computational linguistics.
22. F.J. Och. 2003. Minimum error rate training in statistical machine translation. In Proc. of ACL.
23. K. Papineni, S. Roukos, T. Ward, and W.J. Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proc. of ACL.
24. S. Petrov, L. Barrett, R. Thibaux, and D. Klein. 2006. Learning accurate, compact, and interpretable tree annotation. In Proc. of ACL.
25. S. Riezler and J.T. Maxwell. 2005. On some pitfalls in automatic evaluation and significance testing for mt. In Proc. of ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization.
26. M. Surdeanu and C.D. Manning. 2010. Ensemble models for dependency parsing: cheap and good? In Proc. of NAACL.
27. A. Yeh. 2000. More accurate tests for the statistical significance of result differences. In Proc. of ACL.
28. Y. Zhang, S. Vogel, and A. Waibel. 2004. Interpreting bleu/nist scores: How much improvement do we need to have a better system. In Proc. of LREC.