Treebanks gone bad: Parser evaluation and retraining using a treebank of ungrammatical sentences

International Journal on Document Analysis and Recognition

Volumn 10, Issue 3-4, 2007, Pages 129-145

  Foster, Jennifer ^a  

^a DUBLIN CITY UNIVERSITY   (Ireland)

Author keywords

Parser evaluation; Robust parsing; Treebanks; Ungrammatical language

Indexed keywords

COMPUTATIONAL GRAMMARS; DATA REDUCTION; ERROR ANALYSIS; STATISTICAL METHODS;

GRAMMATICAL ERRORS; PARSER EVALUATION; ROBUST PARSING; TREEBANK; UNGRAMMATICAL LANGUAGE; UNGRAMMATICAL SENTENCES;

DATA PROCESSING;

EID: 37249036496     PISSN: 14332833     EISSN: 14332825     Source Type: Journal    
DOI: 10.1007/s10032-007-0059-8     Document Type: Conference Paper

References (43)

1. Becker, M., Bredenkamp, A., Crysmann, B., Klein, J.: Annotation of error types for german news corpus. In: Proceedings of the ATALA Workshop on Treebanks. Paris (1999)
2. Bender, E.M., Flickinger, D., Oepen, S., Baldwin, T.: Arboretum: Using a precision grammar for grammar checking in CALL. In: Proceedings of the InSTIL/ICALL Symposium: NLP and Speech Technologies in Advanced Language Learning Systems. Venice (2004)
3. Bigert, J.: Probabilistic detection of context-sensitive spelling errors. In: Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC-04), Vol. 5, pp. 1633-1636. Lisbon (2004)
4. Bigert, J., Sjölbergh, J., Knutsson, O., Sahlgren, M.: Unsupervised evaluation of parser robustness. In: Proceedings of the Sixth International Conference on Intelligent Text Processing and Computational Linguistics (CICling-05), pp. 142-154. Mexico City (2005)
5. Bikel, D.: On the parameter space of generative lexicalized parsing models. PhD Thesis, University of Pennslyvania (2004)
6. Black, E., Abney, S., Flickinger, D., Gdaniec, C., Grishman, R., Harrison, P., Hindle, D., Ingria, R., Jelinek, F., Klavans, J., Liberman, M., Marcus, M., Roukos, S., Santorini, B., Strzalkowski, T.: A procedure for quantitatively comparing the syntactic coverage of English grammars. In: Proceedings of the 1991 DARPA Speech and Natural Language Workshop, pp. 306-311 (1991)
7. Burnard, L.: User reference guide for the British National Corpus. Tech. rep., Oxford University Computing Services (2000)
8. Charniak, E.: A maximum-entropy-inspired parser. In: Proceedings of the Annual Meeting of the North American Association for Computational Linguistics (NAACL-00), pp. 132-139. Seattle (2000)
9. Charniak, E., Johnson, M.: Course-to-fine n-best-parsing and maxent discriminative reranking. In: Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL-05), pp. 173-180. Ann Arbor (2005)
10. Choudhury, M., Saraf, R., Jain, V., Sarkar, S., Basu, A.: Investigation and modeling of the structure of texting language. In: Proceedings of the Workshop on Analytics for Noisy Unstructured Data (AND-07) at the 20th International Joint Conference on Artificial Intelligence (IJCAI-07), pp. 63-70. Hyderabad (2007)
11. Collins, M.: Discriminative reranking for natural language parsing. In: Machine Learning: Proceedings of the 17th International Conference (ICML 2000), pp. 175-182. Stanford (2000)
12. Collins M. (2003). Head-driven statistical models for natural language parsing. Comput. Linguist. 29(4): 499-637
13. Creswell, C., Schwartzmyer, N., Srihari, R.: Information extraction for multi-participant, task-oriented, synchronous, computer- mediated communication: A corpus study of chat data. In: Proceedings of the Workshop on Analytics for Noisy Unstructured Data (AND-07) at the 20th International Joint Conference on Artificial Intelligence (IJCAI-07), pp. 131-138. Hyderabad (2007)
14. Crystal, D. (ed.): Language and the Internet. Cambridge University Press, Cambridge (2001)
15. Douglas, S., Dale, R.: Towards robust PATR. In: Proceedings of the 15th International Conference on Computational Linguistics (COLING-92), pp. 468-474. Nantes (1992)
16. Emi, I., Uchimoto, K., Isahara, H.: The overview of the SST speech corpus of Japanese Learner English and evaluation through the experiment on automatic detection of learners' errors. In: Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC-04), Vol. 4, pp. 1435-1439. Lisbon (2004)
17. Foster, J.: Parsing ungrammatical input: An evaluation procedure. In: Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC-04), Vol. 6, pp. 2039-2042. Lisbon (2004)
18. Foster, J.: Good reasons for noting bad grammar: Empirical investigations into the parsing of ungrammatical written English. PhD Thesis, University of Dublin, Trinity College (2005)
19. Foster, J., Vogel, C.: Good reasons for noting bad grammar: Constructing a corpus of ungrammatical language. In: S. Kepser, M. Reis (eds.) Pre-Proceedings of the International Conference on Linguistic Evidence: Empirical, Theoretical and Computational Perspectives, pp. 151-152. Organized by the Sonderforschungsbereich 441 "Linguistic Data Structures", University of Tübingen, Germany, Tübingen (2004)
20. Foster J. and Vogel C. (2004). Parsing ill-formed text using an error grammar. Artif. Intell. Rev. Special AICS2003 Issue 21: 269-291
21. Fouvry, F.: Robust processing for constraint-based grammar formalisms. PhD Thesis, Department of Language and Linguistics, University of Essex (2003)
22. Granger, S.: International corpus of Learner English. In: Aarts, J., de Haan, P., Oostdijk, N. (eds.) English Language Corpora: Design, Analysis and Exploitation, pp. 57-71. Rodopi, Amsterdam (1993)
23. Hashemi, S.S.: Ambiguity resolution by reordering rules in text containing errors. In: Proceedings of the 10th International Conference on Parsing Technologies, pp. 69-79. Prague (2007)
24. James, C.: Errors in Language Learning and Use: Exploring Error Analysis. Addison Wesley Longman, Reading (1998)
25. Jensen K., Heidorn G., Miller L. and Ravin Y. (1983). Parse fitting and prose fixing: Getting a hold on ill-formedness. Am. J. Comput. Linguist. 9(3-4): 147-160
26. Johansson, R., Nugues, P.: Extended constituent-to-dependency conversion for English. In: Nivre, J., Kaalep, H.J., Muischnek, K., Koit, M. (eds.) Proceedings of NODALIDA 2007, pp. 105-112. Tartu (2007)
27. Johnson, M., Geman, S., Canon, S., Chi, Z., Riezler, S.: Estimators for stochastic "unification-based" grammars. In: Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics (ACL-99), pp. 535-541. San Francisco (1999)
28. Kepser, S., Steiner, I., Sternefeld, W.: Annotating and querying a treebank of suboptimal structures. In: Proceedings of the 3rd Workshop on Treebanks and Linguistic Theories (TLT2004), pp. 63-74. Tuebingen (2004)
29. Lee, K.J., Kweon, C.J., Seo, J., Kim, G.C.: A robust parser based on syntactic information. In: Proceedings of the Seventh Conference of the European Association for Computational Linguistics (EACL-95), pp. 223-228 (1995)
30. Lopresti, D.: Performance evaluation for text processing of noisy inputs. In: Proceedings of the 20th Annual ACM Symposium on Applied Computing (Document Engineering Track), pp. 759-763. Sante Fe (2005)
31. Marcus, M., Kim, G., Marcinkiewicz, M.A., MacIntyre, R., Bies, A., Ferguson, M., Katz, K., Schasberger, B.: The Penn Treebank: Annotating predicate argument structure. In: Proceedings of the 1994 ARPA Speech and Natural Language Workshop, pp. 114-119. Princeton (1994)
32. Marcus M.P., Santorini B. and Marcinkiewicz M.A. (1993). Building a large annotated corpus of English: The Penn treebank. Comput. Linguist. 19(2): 313-330
33. Menzel, W.: Robust processing of natural language. In: Proceedings of the 19th Annual German Conference on Artificial Intelligence, Lecture Notes in Computer Science, Vol. 981, pp. 19-34. Bielefeld (1995)
34. Nicholls, D.: The Cambridge Learner Corpus - error coding and analysis. In: Summer Workshop on Learner Corpora. Tokyo (1999)
35. Okanohara, D., Tsujii, J.: A discriminative language model with pseudo-negative examples. In: Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pp. 73-80. Prague (2007)
36. Penstein Rosé, C., Lavie, A.: An efficient distribution of labor in a two stage robust interpretation process. In: Proceedings of 1997 Conference on Empirical Methods in Natural Language Processing (EMNLP-97), pp. 1129-1135 (1997)
37. Sampson, G., Babarczy, A.: A test of the Leaf-Ancestor metric for parse accuracy. In: Carroll, J., Frank, A., Lin, D., Prescher, D., Uszkoreit H. (eds.) Proceedings of the "Beyond Parseval - Towards Improved Evaluation Measures for Parsing Systems" Workshop at the 3rd International Conference on Linguistic Resources and Evaluation (LREC). Las Palmas (2002)
38. Schneider, D., McCoy, K.: Recognizing syntactic errors in the writing of second language learners. In: Proceedings of the Thirty-Sixth Annual Meeting of the Association for Computational Linguistics and the Seventeenth International Conference on Computational Linguistics (ACL/ COLING-98), Vol. 2, pp. 1198-1204. Montreal (1998)
39. Smith, N.A., Eisner, J.: Contrastive Estimation: Training log- linear models on unlabeled data. In: Proceedings of the 43rd Annual Meeting of the Association of Computational Linguistics, pp. 354-362. Ann Arbor (2005)
40. Smith, N.A., Eisner, J.: Guiding unsupervised grammar induction using contrastive estimation. In: Proceedings of the IJCAI Workshop on Grammatical Inference Applications, pp. 73-82. Edinburgh (2005)
41. Stemberger J. (1982). Syntactic errors in speech. J. Psycholinguist. Res. 11(4): 313-45
42. Vogel, C., Cooper, R.: Robust chart parsing with mildly inconsistent feature structures. In: A. Schöter, C. Vogel (eds.) Nonclassical Feature Systems, pp. 197-216. Centre for Cognitive Science, University of Edinburgh, Working Papers in Cognitive Science, Vol. 10 (1995)
43. Weng, F.: Handling syntactic extra-grammaticality. In: Proceedings of the Third International Workshop on Parsing Technologies (IWPT-93), pp. 319-332. Tilburg (the Netherlands), Durbuy (Belgium) (1993)