Rating computer-generated questions with Mechanical Turk

Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, Mturk 2010 at the 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2010 - Proceedings

Volumn , Issue , 2010, Pages 35-40

  Heilman, Michael ^a   Smith, Noah A ^a  

^a Carnegie Mellon University ^*  (United States)

Author keywords

[No Author keywords available]

Indexed keywords

AMAZON'S MECHANICAL TURKS; APPLICATION SPECIFIC; COMPUTER GENERATED; MECHANICAL TURKS; NATURAL LANGUAGES; RATING SCHEMES; READING COMPREHENSION; WIKIPEDIA ARTICLES;

EID: 85074723130     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (11)

1. C. Callison-Burch. 2009. Fast, cheap, and creative: Evaluating translation quality using Amazon's Mechanical Turk. In Proc. of EMNLP.
2. G. Carroll and E. Charniak. 1992. Two experiments on learning probabilistic dependency grammars from corpora. Technical report, Brown University.
3. M. Heilman and N. A. Smith. 2009. Question generation via overgenerating transformations and ranking. Technical Report CMU-LTI-09-013, Language Technologies Institute, Carnegie Mellon University.
4. M. Heilman and N. A. Smith. 2010. Good question! statistical ranking for question generation. In Proc. of NAACL-HLT.
5. J. R. Landis and G. G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33.
6. I. Langkilde and Kevin Knight. 1998. Generation that exploits corpus-based statistical knowledge. In Proc. of ACL.
7. C. Lin. 2004. ROUGE: a package for automatic evaluation of summaries. In Proc. of Workshop on Text Summarization.
8. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proc. of ACL.
9. R. Snow, B. O'Connor, D. Jurafsky, and A. Ng. 2008. Cheap and fast - but is it good? evaluating non-expert annotations for natural language tasks. In Proc. of EMNLP.
10. M. A. Walker, O. Rambow, and M. Rogati. 2001. Spot: a trainable sentence planner. In Proc. of NAACL.
11. D. Yarowsky. 1995. Unsupervised word sense disambiguation rivaling supervised methods. In Proc. of ACL.