Detecting text similarity over chinese research papers using MapReduce

Proceedings - 2011 12th ACIS International Conference on Software Engineering, Artificial Intelligence Networking and Parallel Distributed Computing, SNPD 2011

Volumn , Issue , 2011, Pages 197-202

  Xu, Fan ^a   Zhu, Qiaoming ^a   Li, Peifeng ^a  

^a SOOCHOW UNIVERSITY   (China)

Author keywords

Chinese Research Papers; Copy Detection; MapReduce; Parallel Algorithm; Similarity

Indexed keywords

2-TUPLE; COPY DETECTION; KEY-PHRASE; MAP-REDUCE; RESEARCH PAPERS; RUNNING TIME; SIMILARITY; SIMILARITY COEFFICIENTS; STATE-OF-THE-ART METHODS; TEXT SIMILARITY;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; PARALLEL ALGORITHMS; RESEARCH; SOFTWARE ENGINEERING;

DISTRIBUTED COMPUTER SYSTEMS;

EID: 81255143639     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/SNPD.2011.29     Document Type: Conference Paper

References (21)

1. Sahami M. and Heilman T. D., "A Web-based Kernel Function for Measuring the Similarity of Short Text Snippets," in proceedings of WWW,2006.
2. Hoad T. C. and Zobel. J., "Methods for Identifying Versioned and Plagiarized Documents," in JASIST,vol. 54,2003,pp. 203-215.
3. Spertus E.,Sahami M. and Buyukkokten O., "Evaluating Similarity Measures: A Large-scale Study in the Orkut Social Network," in proceedings of KDD,2005.
4. Kent C.W. and Salim N., "Features Based Text Similarity Detection," in Journal of Computing,vol. 2,2010,pp. 53-57.
5. Kumar J.P. and Govindarajulu P., "Duplicate and Near Duplicate Documents Detection:A Review," in European Journal of Scientific Research,vol. 32, 2009,pp. 514-527.
6. Chowdhury A., "Duplicate data detection," retrived from http://ir.iit.edu/~abdur/Research/Duplicate.html,2004.
7. Lyon C., Malcolm J. and Dickerson B., "Detecting Short Passages of Similar Text in Large Document Collections," in processing of EMNLP,2001.
8. Matveeva I., "Document Representation and Multilevel Measures of Document Similarity," in proceedings of ACL-HLT, 2006.
9. Hatzivassiloglou V.,Klavans J.L. and Eskin E., "Detecting Text Similarity over Short Passages: Exploring Linguistic Feature Cominations via Machine Learning," in proceedings of SIGDAT,1999.
10. Yih W., "Learning Term-weighting Functions for Similarity Measures," in proceedings of EMNLP,2009.
11. Broder A. Z., "On the Resemblance and Containment of Documents," in proceedings of SEQUENCES,1997.
12. Broder A. Z., "Identifying and Filtering Near Duplicate Documents," in processing of COM,2000.
13. Jaccard P. E., "Comparative de la Distribution Florale Dans Une Portion Des Alpes et Des Jura," in Bulletin del la Socit Vaudoise des Sciences Naturelles,vol. 37,1901,pp. 547-579.
14. Qi Zhang,Yue Zhang,Hao. Yu and Xuan. Huang, "Efficient Partial-Duplicate Detection Based on Sequence Matching," in proceedings of SIGIR,2010.
15. Vernica R.,Carey M. J., and Li C., "Efficient Parallel Set-similarity Joins Using MapReduce," in proceedings of SIGMOD, 2010.
16. Pei. Yuan,Chao. Sha,Xiao. Wang,Bin Yang,Ao. Zhou and Su Yang, "XML Structural Similarity Search Using MapReduce,"in proceedings of WAIM,2010.
17. Gianmarco D. F.,Claudio L. and Ranieri B., "Scaling Out All Pairs Similarity Search with MapReduce," in proceedings of LSDSIR, 2010.
18. Elsayed t.,Lin J. and Oard D.W., "Pairwise Document Similarity in Large Collections with MapReduce," in proceedings of ACL-HLT,2008.
19. Jacob B.,Benjamin C., "Calculating the Jaccard Similarity Coefficient with Map Reduce for Entity Pairs in Wikipedia,"http://www. infosci.cornell.edu/weblab/papers/Bank2008.pdf, 2008.
20. Dean,J. and Ghemawat,S., "MapReduce:Simplified Data Processing on Large Clusters," in Commun. ACM,vol. 51,2008,pp 107-113.
21. Hadoop website, http://hadoop.apache.org/, 2011.