Duplicate detection in documents and webpages using improved longest common subsequence and documents syntactical structures

ICCIT 2009 - 4th International Conference on Computer Sciences and Convergence Information Technology

Volumn , Issue , 2009, Pages 679-684

  Elhadi, Mohamed ^a   Al Tobi, Amjad ^a  

^a SULTAN QABOOS UNIVERSITY   (Oman)

Author keywords

Component: part of speech; Duplication filtering; Longest common subsequence; Syntactical structure

Indexed keywords

DUPLICATE DETECTION; LONGEST COMMON SUBSEQUENCES; PART OF SPEECH; SEARCH ENGINE RESULTS; SYNTACTICAL STRUCTURES; WEB PAGE;

COMPUTER SCIENCE; SEARCH ENGINES;

INFORMATION RETRIEVAL SYSTEMS;

EID: 77749301855     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICCIT.2009.235     Document Type: Conference Paper

References (38)

1. rd Inter. Conf. of Computational Linguistics and Intelligent Text Processing (Mexico City, Mexico, 2002, pp. 415-424.
2. S. Brin, J. Davis and H. Garcia-Molina, "Copy Detection Mechanisms for Digital Documents", in the ACM SIGMOD International Conference on Management of Data (San Jose, California, USA, May 22-25 1995), 1995, pp. 398-409.
3. A. Al-Tobi, "Combined Syntactical Structures and Sequence Alignment Approach to Document Similarity Calculation for Copy Detection", M.Sc. thesis, Department of Computer Science, Collage of Science, Sultan Qaboos University, Muscat, Oman, 2008.
4. A. Zaslavsky, A. Bia, and K. Monostori, "Using copy-detection and text comparison algorithms for cross-referencing multiple editions of literary works", in Proceedings of the 5th European Conference on Research and Advanced Technology for Digital Libraries (Darmstadt, Germany, September 4-9 2001), 2001, pp 103-114,.
5. A. Si, H.V. Leong and R.W.H. Lau, "CHECK: A document plagiarism detection system", in Proceedings of ACM Symposium for Applied Computing, ACM (San Jose, California, USA, February 28 - March 1 1997), 1997, pp. 70-77.
6. N. Shivakumar and H. Garcia-Molina, "SCAM: A Copy Detection Mechanism for Digital Documents", in Proceedings of 2nd International Conference in Theory and Practice of Digital Libraries (Austin, Texas, USA, June 11-13 1995), 1995.
7. REUTERS, Reuters Corpus (Volume 1: English Language, 1996-08-20 to 1997-08-19), Released date: November 2000, NIST, 2000.
8. Y. Liu and L. Liang, "A Dual-method Model for Copy Detection", in Proceedings of the IEEE/WIC/ACM international conference on Web Intelligence and Intelligent Agent Technology, IEEE (Hong Kong Convention and Exhibition Centre, Hong Kong, December 18-22 2006), 2006, pp. 634-637.
9. G. Forman, K. Eshghi and S. Chiocchetti, "Finding Similar Files in Large Document Repositories", in Proceedings of the 11th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Chicago, Illinois, USA, August 2005, pp 394-400.
10. M. Elhadi and A. Al-Tobi, "Use of Text Syntactical Structures in Detection of Document Duplicates", in Third IEEE International Conference on Digital Information Management (University of East London, London. UK, November 13-16 2008), 2008.
11. A. Singhal, "Modern Information Retrieval: A Brief Overview", IEEE Data Engin. Bulletin, Vol. 24, No. 4, pp. 35-43, March 2001.
12. S.F. Altschul, W. Gish, W. Miller, E.W. Myers and D.J. Lipman, "Basic Local Alignment Search Tool", Journal of Molecular Biology, Vol. 215, No. 3, pp. 403-410, October 1990.
13. "Local Alignment: Smith-Waterman algorithm", class notes for CSE 591: Computational Molecular Biology, Depar. of Computer Science & Engineering, Arizona State University, Spring 2003.
14. A. M. Lesk, Introduction to Bioinformatics, Second Edition, Oxford University Press Inc., New York, USA, 2005.
15. S. Schleimer, D. S. Wilkerson and A. Aiken, "Winnowing: Local Algorithms for Document Fingerprinting", in Proceedings of the 2003 ACM SIGMOD Inter. Conf. on Management of Data, San Diego,California, USA, June 2003, pp. 76-85.
16. Lexicon and Textcorpora Group, Institut für Maschinelle Sprachverarbeitung, Universität Stuttgart, Germany, "TreeTagger - a language independent part-of-speech tagger", 2003, http://www.ims.uni- stuttgart.de/projekte/corplex/TreeTagger.
17. X.R. Hu and E. Atwell, "A survey of machine learning approaches to analysis of large corpora", in Proceedings of the Workshop on Shallow Processing of Large Corpora (SProLaC), (Lancaster University, UK, March 28 - 31 2003), 2003, pp. 45-52.
18. M.S. Waterman, "General Methods of Sequence Comparison", Bulletin of Math. Biology, Vol. 46, No. 4, pp. 473-500, 1984.
19. ELC Courses, English Language Centre, University of Victoria, "Parts of Speech", 1997, http://web2.uvcs.uvic.ca/elc/StudyZone/330/grammar/ parts.htm.
20. H. Schmid, "Probabilistic Part-of-Speech Tagging Using Decision Trees", in Proceedings of International Conference on New Methods in Language Processing, (Manchester, UK, September 1994), 1994, pp. 44-49.
21. H. Schmid, "First refinments in Part-of-Speech Tagging With an Application To German", in Proceedings of the 14th International Conference on Computational Linguistics, (Kyoto, Japan, March 1995), 1995, pp. 172-176.
22. A.G. Maguitman, F. Menczer, H. Roinestad and A. Vespignani, "Algorithmic Detection of Semantic Similarity", in Proceedings of the 14th international conference on World Wide Web, (Chiba, Japan, May 10-14 2005), 2005, pp. 107-116.
23. R. Mihalcea, C. Corley and C. Strapparava, "Corpus-based and Knowledge-based Measures of Text Semantic Similarity", in Proceedings of The Twenty-First National Conference on Artificial Intelligence and the Eighteenth Innovative Applications of Artificial Intelligence Conference, (Boston, Massachusetts, USA, July 16-20, 2006), 2006.
24. K. Monostori, R. Finkel, A. Zaslavsky, G. Hodasz and M. Pataki, "Comparison of Overlap Detection Techniques", International Conference on Computational Science, (Amsterdam, The Netherlands, April 21-24 2002), 2002, pp 51-60.
25. N. Kang, A. Gelbukh and S. Han, "PPChecker: Plagiarism Pattern Checker in Document Copy Detection", in Proceedings of Text, Speech and Dialogue 9th International Conference, (Brno, Czech Republic, September 11-15 2006), 2006, pp 661-667.
26. N. Heintze, "Scalable Document Fingerprinting", In Proceedings of the Second USENIX Workshop on Electronic Commerce, (Oakland, California, November 18-21 1996), 1996.
27. J. Cussens, "Part-of-Speech Tagging Using Progol", in Proceedings of the 7th International Workshop (Inductive Logic Programming), (Prague, Czech Republic, September 17-20 1997), 1997, pp. 93-108.
28. H. MacFadyen, University of Ottawa, "The Parts of Speech", 2007, http://www.arts.uottawa.ca/writcent/hypergrammar/partsp.html, [Accessed: 25th Sep 2008].
29. Wikipedia® , Wikimedia Foundation, Inc., "Viterbi algorithm", 8th Sept 2008, http://en.wikipedia.org/wiki/Viterbi-algorithm..
30. W. Daelemans, J. Zavrel, P. Berck and S. Gillis, "MBT: A Memory-Based Part of Speech Tagger Generator", in Proceedings of Fourth Workshop on Very Large Corpora (WVLC), (University of Copenhagen, Copenhagen, Denmark, August 5-9 1996), 1996, pp. 14-27.
31. C. J. van RIJSBERGEN, INFORMATION RETRIEVAL, Department of Computing Science, University of Glasgow, London: Butterworths, 1979.
32. P. Ingwersen, Information Retrieval Interaction, Taylor Graham Publishing, London, United Kingdom, 1992.
33. G. Salton, "Automatic Content Analysis in Information Retrieval", Department of Computing Science, Cornell University, Ithaca, N.Y., 1968.
34. Yonghong Mao, Natural Language Processing Module (Part of Speech Tagging and Sentence Parsing), Cognitive Science in Context Laboratory, Cornell University, New York, U.S., 1997.
35. Wikipedia® , Wikimedia Foundation, Inc., "diff", 25th Sep 2007, http://en.wikipedia.org/wiki/Diff.
36. Algorithmist, GNU Free Documentation License, "Longest Common Subsequence", 23rd Oct 2006, http://www.algorithmist.com/index.php/Longest- Common-Subsequence.
37. C. Charras and T. Lecroq, "Longest common subsequences", February 1998, .
38. Wikipedia® , Wikimedia Foundation, Inc., "Part-of-speech tagging", 8th Sept 2008, http://en.wikipedia.org/wiki/Part-ofspeech- tagging.