Automatic evaluation of spoken summaries: The case of language assessment

Proceedings of the Annual Meeting of the Association for Computational Linguistics

Volumn , Issue , 2014, Pages 68-78

  Loukina, Anastassia ^a   Zechner, Klaus ^a   Chen, Lei ^a  

^a EDUCATIONAL TESTING SERVICE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

AUTOMATION; NATURAL LANGUAGE PROCESSING SYSTEMS;

AUTOMATED SPEECH RECOGNITION; AUTOMATIC EVALUATION; LANGUAGE ASSESSMENT; MULTIPLE REFERENCES; NON-NATIVE SPEAKERS; PERFORMANCE; SPEARMAN CORRELATION; SPEECH RECOGNIZER; WORD FREQUENCIES;

SPEECH RECOGNITION;

EID: 85120315138     PISSN: 0736587X     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (30)

1. Yigal Attali and Jill Burstein. 2006. Automated essay scoring with e-rater V. 2. The Journal of Technology, Learning and Assessment, 4(3):1-30.
2. Thomas Baguley. 2012. Serious Stats: A guide to advanced statistics for the behavioral sciences. Pal-grave Macmillan.
3. Douglas Biber, Susan M. Conrad, Randi Reppen, Pat Byrd, Marie Helt, Victoria Clark, Viviana Cortes, Eniko Csomay, and Alfredo Urzua. 2004. Representing language use in the university: analysis of TOEFL 2000 Spoken and Written academic language corpus. Educational Testing Service, Princeton.
4. Miao Chen and Klaus Zechner. 2012. Using an ontology for improved automated content scoring of spontaneous non-native speech. In Proceedings of the 7th Workshop on Building Educational Applications Using NLP, pages 86-94, Stroudsburg, PA. Association for Computational Linguistics.
5. Lei Chen, Klaus Zechner, and Xiaoming Xi. 2009. Improved pronunciation features for construct-driven assessment of non-native spontaneous speech. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, NAACL'09, pages 442-449, Stroudsburg, PA, USA. Association for Computational Linguistics.
6. Lei Chen. 2013. Applying unsupervised learning to support vector space model based speaking assessment. Proceedings of the 8th Workshop on Innovative Use of NLP for Building Educational Applications, Atlanta, Georgia, pages 58-62.
7. Anthony C. Davison and David V. Hinkley. 1997. Bootstrap Methods and their Application (Cambridge Series in Statistical and Probabilistic Mathematics). Cambridge University Press.
8. Keelan Evanini and Xinhao Wang. 2013. Automated speech scoring for non-native middle school students with multiple task types. Proceedings of Interspeech 2013, Lyon, France, pages 2435-2439.
9. Peter W. Foltz, Darrell Laham, and Thomas K. Landauer. 1999. Automated essay scoring: applications to educational technology. In B. Collis and R. Oliver, editors, Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 1999, pages 939-944.
10. George Giannakopoulos and Vangelis Karkaletsis. 2011. AutoSummENG and MeMoG in Evaluating Guided Summaries. In TAC 2011 Workshop, Gaithersburg, MD, USA. NIST.
11. Derrick Higgins, Xiaoming Xi, Klaus Zechner, and David Williamson. 2011. A three-stage approach to the automated scoring of spontaneous spoken responses. Computer Speech & Language, 25(2):282-306, April.
12. Makoto Hirohata, Yousuke Shinnaka, Koji Iwano, and Sadaoke Furui. 2005. Sentence extraction-based presentation summarization techniques and evaluation metrics. In Acoustics, Speech, and Signal Processing, 2005. Proceedings. (ICASSP'05). IEEE International Conference on, volume 1, pages 1065-1068.
13. Max Kuhn and Kjell Johnson. 2013. Applied Predictive Modeling. Springer.
14. Chin-Yew Lin and Marina Rey. 2004. ROUGE: A package for automatic evaluation of summaries. In Stan Szpakowicz Marie-Francine Moens, editor, Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
15. Chin-Yew Lin. 2004. Looking for a few good metrics: Automatic summarization evaluation - how many samples are enough. In Proceedings of the NTCIR Workshop, pages 1765-1776, Tokyo.
16. Annie Louis and A Nenkova. 2013. Automatically assessing machine summary content without a gold standard. Computational Linguistics, 39(2):267-300.
17. Nitin Madnani, Jill Burstein, John Sabatini, and Tenaha O'Reilly. 2013. Automated scoring of a summary-writing task designed to measure reading comprehension. In Proceedings of the 8th Workshop on Innovative Use of NLP for Building Educational Applications, pages 163-168, Atlanta, Georgia. Association for Computational Linguistics.
18. Ani Nenkova and Annie Louis. 2008. Can you summarize this? Identifying correlates of input difficulty for generic multi-document summarization. In Proceedings of the ACL-08: HLT, pages 825-833. Association for Computational Linguistics.
19. Ani Nenkova and Kathleen McKeown. 2011. Automatic summarization. Foundations and Trends in Information Retrieval, 5(2-3):103-233.
20. Karolina Owczarzak, John M. Conroy, Hoa Trang Dang, and Ani Nenkova. 2012. An assessment of the accuracy of automatic evaluation in summarization. In Proceedings of workshop on evaluation metrics and system comparison for automatic summarization., pages 1-9, Stroudsburg, PA. Association for Computational Linguistics.
21. Art B. Owen. 2007. The pigeonhole bootstrap. The Annals of Applied Statistics, 1(2):386-411.
22. Kishore Papineni, Salim Roukos, Todd Ward, and Weijing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the ACL, pages 311-318, Philadelphia, PA. Association for Computational Linguistics.
23. Gerald Penn and Xiaodan Zhu. 2008. A Critical Reassessment of Evaluation Baselines for Speech Summarization. In in Proceedings of RANLP workshop on Crossing Barriers in Text Summarization Research, number June, pages 470-478, Columbus, Ohio, June. Association for Computational Linguistics.
24. Peter A. Rankel, John. M. Conroy, Hoa Trang Dang, and Ani Nenkova. 2013. A decade of automatic content evaluation of news summaries: reassessing the state of the art. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, Sofia, Bulgaria, August 4-9, 2013, pages 131-136, Sofia. Association for Computational Linguistics.
25. Rand R. Wilcox. 2009. Comparing Pearson correlations: dealing with heteroscedasticity and nonnormality. Communications in Statistics - Simulation and Computation, 38(10):2220-2234.
26. David M. Williamson, Xiaoming Xi, and F. Jay Breyer. 2012. A framework for evaluation and use of automated scoring. Educational measurement: issues and practice, 31(1):2-13.
27. Shasha Xie, Keelan Evanini, and Klaus Zechner. 2012. Exploring content features for automated speech scoring. In NAACL HLT'12 Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 103-111.
28. Wenting Xiong, Keelan Evanini, Klaus Zechner, and Lei Chen. 2013. Automated content scoring of spoken responses containing multiple parts with factual information. In Pierre Badin, Thomas Hueber, Gérard Bailly, Didier Demolin, and Françoise Raby, editors, Proceedings of SLaTE 2013, Grenoble, France, pages 137-142, Grenoble.
29. Klaus Zechner, Derrick Higgins, Xiaoming Xi, and David M. Williamson. 2009. Automatic scoring of non-native spontaneous speech in tests of spoken English. Speech Communication, 51(10):883-895.
30. Guang Yong Zou. 2007. Toward using confidence intervals to compare correlations. Psychological methods, 12(4):399-413.