Paraphrase acquisition via crowdsourcing and machine learning

ACM Transactions on Intelligent Systems and Technology

Volumn 4, Issue 3, 2013, Pages

  Burrows, Steven ^a   Potthast, Martin ^a   Stein, Benno ^a  

^a BAUHAUS UNIVERSITY WEIMAR   (Germany)

Author keywords

Corpus; Cost analysis; Mechanical Turk; Paraphrase generation; Plagiarism

Indexed keywords

CORPUS; COST ANALYSIS; MECHANICAL TURKS; PARAPHRASE GENERATION; PLAGIARISM;

COST ACCOUNTING; LEARNING SYSTEMS; QUALITY ASSURANCE; RESEARCH;

INTELLECTUAL PROPERTY;

EID: 84880207564     PISSN: 21576904     EISSN: 21576912     Source Type: Journal    
DOI: 10.1145/2483669.2483676     Document Type: Article

References (49)

1. ALONSO, O. AND MIZZARO, S. 2009. Can we get rid of trec assessors? Using mechanical turk for relevance assessment. In Proceedings of the 1st SIGIRWorkshop on the Future of IR Evaluation, S. Geva, J.Kamps, C. Peters, T. Sakai, A. Trotman, and E. Voorhees, Eds., IR Publications, Boston, MA, 15-16.
2. AMBATI, V., VOGEL, S., AND CARBONELL, J. 2010. Active learning and crowd-sourcing for machine translation. In Proceedings of the 17th Conference on International Language Resources and Evaluation (LREC'10), N. Calzolari, K. Choukri, B. Maegaard, J. Mariani, J. Odijk, S. Piperidis, M. Rosner, and D. Tapias, Eds., European Language Resources Association, 2169-2174.
3. ANDROUTSOPOULOS, I. AND MALAKASIOTIS, P. 2010. A survey of paraphrasing and textual entailment methods. J. Artif. Intell. Res. 38, 1, 135-187.
4. BARR, J. AND CABRERA, L. F. 2006. AI gets a brain. Queue 4, 4, 24-29.
5. BARZILAY, R. AND LEE, L. 2003. Learning to paraphrase: An unsupervised approach using multiple-sequence alignment. In Proceedings of the 3rd Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, E. Hovy, M. Hearst, and M. Ostendorf, Eds., Association for Computational Linguistics, 16-23.
6. BERNSTEIN, M. S., LITTLE, G., MILLER, R. C., HARTMANN, B., ACKERMAN, M. S., KARGER, D. R., CROWELL, D., AND PANOVICH, K. 2010. Soylent: A word processor with a crowd inside. In Proceedings of the 23rd Annual ACM Symposium on User Interface Software and Technology, K. Perlin, M. Czerwinski, and R. Miller, Eds., ACM, New York, 313-322.
7. BOONTHUM, C. 2004. iSTART: Paraphrase recognition. In Proceedings of the 5th ACL Workshop on Student Research, L. van der Beek, D. Genzel, and D. Midgley, Eds., Association for Computational Linguistics, 55-60.
8. BRABHAM, D. C. 2008a. Crowdsourcing as a model for problem solving. Convergence: The Int. J. Res. New Media Technol. 14, 1, 75-90.
9. BRABHAM, D. C. 2008b. Moving the crowd at istockphoto: The composition of the crowd and motivations for participation in a crowdsourcing application. First Monday 13, 6.
10. BROCKETT, C. AND DOLAN, W. B. 2005. Support vector machines for paraphrase identification and corpus construction. In Proceedings of the 3rd InternationalWorkshop on Paraphrasing, M. Dras and K. Yamamoto, Eds., 1-8.
11. BUZEK, O., RESNIK, P., AND BEDERSON, B. 2010. Error driven paraphrase annotation using mechanical turk. In Proceedings of the 1st NAACL HLT Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, C. Callison-Burch and M. Dredze, Eds., Association for Computational Linguistics, 217-221.
12. CALLISON-BURCH, C. 2009. Fast, cheap, and creative: Evaluating translation quality using amazon's mechanical turk. In Proceedings of the 14th Conference on Empirical Methods in Natural Language Processing, P. Koehn and R. Mihalcea, Eds., Association for Computational Linguistics. 286-295.
13. CALLISON-BURCH, C. AND DREDZE, M. 2010. Creating speech and language data with amazon's mechanical turk. In Proceedings of the 1st NAACL HLT Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, C. Callison-Burch and M. Dredze, Eds., Association for Computational Linguistics, 1-12.
14. CHEN, D. AND DOLAN,W. 2011. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, D. Lin, Y. Matsumoto, and R. Mihalcea, Eds., Association for Computational Linguistics, 190-200.
15. CHEVELU, J., LAVERGNE, T., LEPAGE, Y., AND MOUDENC, T. 2009. Introduction of a new paraphrase generation tool based on monte-carlo sampling. In Proceedings of the 47th Annual Meeting of the Association for Computational Linguistics and the 4th International Joint Conference on Natural Language Processing of the AFNLP, K.-Y. Su, J. Su, and J.Wiebe, Eds., The Association for Computer Linguistics, 249-252.
16. CLOUGH, P., GAIZAUSKAS, R., PIAO, S., AND WILKS, Y. 2002. METER: Measuring text reuse. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, P. Isabelle, Ed., Association for Computational Linguistics, 152-159.
17. COHN, T., CALLISON-BURCH, C., AND LAPATA, M. 2008. Constructing corpora for the development and evaluation of paraphrase systems. Comput. Linguist. 34, 4, 597-614.
18. CORDEIRO, J., DIAS, G., AND BRAZDIL, P. 2007a. A metric for paraphrase detection. In Proceedings of the 2nd International Multi-Conference on Computing in the Global Information Technology, M. Boicu, J. Costa- Requena, D. Thiebaut, C. Popoviciu, B. Tuy, and G. V. de Velde, Eds., IEEE Computer Society, 1-6.
19. CORDEIRO, J., DIAS, G., AND BRAZDIL, P. 2007b. New functions for unsupervised asymmetrical paraphrase detection. J. Softw. 2, 4, 12-23.
20. CORLEY, C. AND MIHALCEA, R. 2005. Measuring the semantic similarity of texts. In Proceedings of the 1st ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment, B. Dolan and I. Dagan, Eds., Association for Computational Linguistics, 13-18.
21. DAS, D. AND SMITH, N. A. 2009. Paraphrase identification as probabilistic quasi-synchronous recognition. In Proceedings of the Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, K.-Y. Su, J. Su, and J.Wiebe, Eds., Association for Computational Linguistics, 468-476.
22. DENKOWSKI, M., AL-HAJ, H., AND LAVIE, A. 2010. Turker-assisted paraphrasing for english-arabic machine translation. In Proceedings of the 1st NAACL HLT Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk, C. Callison-Burch and M. Dredze, Eds., Association for Computational Linguistics, 66-70.
23. DOLAN, W. B. AND BROCKETT, C. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the 3rd International Workshop on Paraphrasing, M. Dras and K. Yamamoto, Eds., 1-8.
24. DORDOY, A. 2002. Cheating and plagiarism: Student and staff perceptions at Northumbria. In Proceedings of the Northumbria Conference - Educating for the Future. 1-6.
25. FERNANDO, S. 2007. Paraphrase identification. M.S. Thesis, Department of Computer Science, University of Sheffield, Sheffield, UK.
26. FERNANDO, S. AND STEVENSON, M. 2008. A semantic approach to paraphrase identification. In Proceedings of the 11thAnnual Research Colloquium of the UK Special-Interest Group for Computational Lingusitics. 1-7.
27. FRANKLYN-STOKES, A. AND NEWSTEAD, S. E. 1995. Undergraduate cheating: Who does what and why? Studies Higher Educ. 20, 2, 159-172.
28. HAMILTON, M., TAHAGHOGHI, S. M. M., AND WALKER, C. 2004. Educating students about plagiarism avoidance - A computer science perspective. In Proceedings of the 12th International Conference on Computers in Education, E. McKay, Ed., 1275-1284.
29. HOLMES, G., DONKIN, A., AND WITTEN, I. 1994. Weka: A machine learning workbench. In Proceedings of the 2nd Australia and New Zealand Conference on Intelligent Information Systems, J. Sitte, Ed., IEEE Computer Society Press, 357-361.
30. KITTUR, A., CHI, E. H., AND SUH, B. 2008. Crowdsourcing user studies with mechanical turk. In Proceedings of the 26th Annual SIGCHI Conference on Human Factors in Computing Systems, M. Czerwinski, A. Lund, and D. Tan, Eds., ACM Press, New York, 453-456.
31. KNIGHT, K. AND MARCU, D. 2002. Summarization beyond sentence extraction: A probabilistic approach to sentence compression. Artif. Intell. 139, 1, 91-107.
32. KOZAREVA, Z. AND MONTOYO, A. 2006. Paraphrase identification on the basis of supervised machine learning techniques. In Proceedings of the 5th International Conference on Natural Language Processing, T. Salakoski, F. Ginter, S. Pyysalo, and T. Pahikkala, Eds., Springer, 524-533.
33. LEIMEISTER, J., HUBER, M., BRETSCHNEIDER, U., AND KRCMAR, H. 2009. Leveraging crowdsourcing: Activationsupporting components for it-based ideas competition. J. Manag. Inf. Syst. 26, 1, 197-224.
34. MADNANI, N. AND DORR, B. J. 2010. Generating phrasal and sentential paraphrases: A survey of data-driven methods. Comput. Linguist. 36, 3, 341-387.
35. MALAKASIOTIS, P. 2009. Paraphrase recognition using machine learning to combine similarity measures. In Proceedings of the 1st ACL-IJCNLP Student Research Workshop, D. M. D. Dimalen, J. R. Finkel, and B. Thomson, Eds., Association for Computational Linguistics, 27-35.
36. MAURER, H., KAPPE, F., AND ZAKA, B. 2006. Plagiarism - A survey. J. Univer. Comput. Sci. 12, 8, 1050-1084.
37. MCCABE, D. L. 2005. Cheating among college and university students: A North American perspective. Int. J. Educ. Integr. 1, 1, 1-11.
38. PAPINENI, K., ROUKOS, S., WARD, T., AND ZHU, W.-J. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, P. Isabelle, Ed., Association for Computational Linguistics, 311-318.
39. PEDERSEN, T., PATWARDHAN, S., AND MICHELIZZI, J. 2004. WordNet: Similarity: Measuring the relatedness of concepts. In Proceedings of the Demonstration Papers at HLT-NAACL, D. Palmer, J. Polifroni, and D. Roy, Eds., Association for Computational Linguistics, 38-41.
40. POTTHAST, M., BARŔON-CEDE?NO, A., EISELT, A., STEIN, B., AND ROSSO, P. 2010a. Overview of the 2nd international competition on plagiarism detection. In Notebook Papers of CLEF 10 Labs and Workshops, M. Braschler and D. Harman, Eds.
41. POTTHAST, M., STEIN, B., BARŔON-CEDE?NO, A., AND ROSSO, P. 2010b. An evaluation framework for plagiarism detection. In Proceedings of the 23rd International Conference on Computational Linguistics (COLING'10). C.-R. Huang and D. Jurafsky, Eds., Association for Computational Linguistics, 997-1005.
42. QIU, L.,KAN, M.-Y., AND CHUA, T.-S. 2006. Paraphrase recognition via dissimilarity significance classification. In Proceedings of the 11th Conference on Empirical Methods in Natural Language Processing, D. Jurafsky and E. Gaussier, Eds., Association for Computational Linguistics, 18-26.
43. QUINN, A. J. AND BEDERSON, B. B. 2011. Human computation: A survey and taxonomy of a growing field. In Proceedings of the 29th Annual Conference on Human Factors in Computing Systems, D. Tan, G. Fitzpatrick, C. Gutwin, B. Begole, and W. A. Kellogg, Eds., ACM, 1403-1412.
44. SHEARD, J.,DICK, M.,MARKHAM, S.,MACDONALD, I., AND WALSH, M. 2002. Cheating and plagiarism: Perceptions and practices of first year it students. In Proceedings of the 7th Annual Conference on Innovation and Technology in Computer Science Education, M. E. Caspersen, D. Joyce, D. Goelman, and I. Utting, Eds., ACM, 183-187.
45. SNOW, R., O'CONNOR, B., JURAFSKY, D., AND NG, A. Y. 2008. Cheap and fast - But is it good?: Evaluating non-expert annotations for natural language tasks. In Proceedings of the 13th Conference on Empirical Methods in Natural Language Processing, M. Lapata and H. T. Ng, Eds., Association for Computational Linguistics, 254-263.
46. SU, J. 2007. Probability Estimation Trees in Weka? Wekalist mailing list. https://list.scms.waikato.ac.nz/pipermail/wekalist/2007-September/011343.html.
47. TAX, D. 2001. One-class classification. Ph.D. Thesis, Delft University of Technology, Delft, The Netherlands.
48. WAN, S., DRAS, M.,DALE, R., AND PARIS, C. 2006. Using dependency-based features to take the "para-farce" out of paraphrase. In Proceedings of the 4th Australasian Language Technology Workshop, L. Cavedon and I. Zukerman, Eds., Australasian Language Technology Association, 131-138.
49. ZHAO, S.,LAN, X., LIU, T., AND LI, S. 2009. Application-driven statistical paraphrase generation. In Proceedings of the 47th Annual Meeting of the Association for Computational Linguistics and the 4th International Joint Conference on Natural Language Processing of the AFNLP, K.-Y. Su, J. Su, and J. Wiebe, Eds., Association for Computer Linguistics, 834-842.