Partial duplicate detection for large book collections

International Conference on Information and Knowledge Management, Proceedings

Volumn , Issue , 2011, Pages 469-474

  Yalniz, Ismet Zeki ^a   Can, Ethem F ^a   Manmatha, R ^a  

^a Biologically Inspired Neural and Dynamical Systems Laboratory   (United States)

Author keywords

partial duplicate detection; sequence matching; unique words

Indexed keywords

COMPACT REPRESENTATION; DATA SETS; DUPLICATE DETECTION; LONGEST COMMON SUBSEQUENCES; SEQUENCE MATCHING; UNIQUE WORD;

OPTICAL CHARACTER RECOGNITION;

KNOWLEDGE MANAGEMENT;

EID: 83055181761     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2063576.2063647     Document Type: Conference Paper

References (26)

1. Internet Archive. http://www.archive.org, 2010.
2. Project Gutenberg. http://www.gutenberg.org, 2010.
3. Y. Bernstein and J. Zobel. A scalable system for identifying co-derivative documents. In SPIRE, pages 55-67, 2004.
4. S. Brin, J. Davis, and H. Garcia-Molina. Copy detection mechanisms for digital documents. In ACM SIGMOD, pages 398-409, 1995.
5. A. Z. Broder, S. C. Glassman, M. S. Manasse, and G. Zweig. Syntactic clustering of the web. Computer Networks, 29(8-13):1157-1166, 1997.
6. M. S. Charikar. Similarity estimation techniques from rounding algorithms. In 34th Ann. ACM Symp. on Theory of computing, pages 380-388, 2002.
7. A. Chowdhury, O. Frieder, D. A. Grossman, and M. C. McCabe. Collection statistics for fast duplicate document detection. ACM Trans. Inf. Syst., 20(2):171-191, 2002. (Pubitemid 44642301)
8. P. Clough. Old and new challenges in automatic plagiarism detection. National UK Plagiarism Advisory Service, http://www.ir.shef.ac.uk/cloughie/ papers/pas plagiarism.pdf, 2003.
9. J. Cooper, A. Coden, and E. Brown. Detecting similar documents using salient terms. In CIKM, pages 245-251, 2002.
10. S. Deorowicz. Solving longest common subsequence and related problems on graphical processing units. Softw. Pract. Exper., 40:673-700, July 2010.
11. M. Errami, Z. Sun, A. C. George, T. C. Long, M. A. Skinner, J. D. Wren, and H. R. Garner. Identifying duplicate content using statistically improbable phrases. Bioinformatics, 26(11):1453-1457, 2010.
12. S. Feng and R. Manmatha. A hierarchical, HMM-based automatic evaluation of OCR accuracy for a digital library of books. In JCDL, pages 109-118, 2006.
13. H. Hajishirzi, W. tau Yih, and A. Kolcz. Adaptive near-duplicate detection via similarity learning. In SIGIR'10, pages 419-426, 2010.
14. N. Heintze. Scalable document ngerprinting. In USENIX Workshop on Electronic Commerce, 1996.
15. M. Henzinger. Finding near-duplicate web pages: a large-scale evaluation of algorithms. In ACM SIGIR, pages 284-291, 2006.
16. T. C. Hoad and J. Zobel. Methods for identifying versioned and plagiarized documents. JASIST, 54(3):203-215, 2003.
17. J. W. Hunt and M. D. McIlroy. An algorithm for differential file comparison. Technical Report CSTR 41, Bell Laboratories, Murray Hill, NJ, 1976.
18. J. W. Hunt and T. G. Szymanski. A fast algorithm for computing longest common subsequences. Commun. ACM, 20:350-353, May 1977.
19. D. Lin. An information-theoretic definition of similarity. In ICML '98, pages 296-304, 1998.
20. U. Manber. Finding similar files in a large file system. In USENIX Winter 1994 Tech. Conf, pages 1-10, 1994.
21. D. Mimno, G. Crane, and A. Jones. Hierarchical catalog records: Implementing a FRBR catalog. In D-Lib Magazine, http://www.dlib.org/dlib/ october05/crane/10crane.html, volume 11, Oct 2005.
22. S. Schleimer, D. Wilkerson, and A. Aiken. Winnowing: local algorithms for document fingerprinting. In ACM SIGMOD conference, pages 76-85, 2003.
23. J. Seo and W. B. Croft. Local text reuse detection. In ACM SIGIR, pages 571-578, 2008.
24. N. Shivakumar and H. Garcia-Molina. Scam: A copy detection mechanism for digital documents. In Ann. Conf. on the Theory and Practice of Digital Libraries, 1995.
25. N. Shivakumar and H. Garcia-Molina. Finding near-replicas of documents on the web. In Intl. Workshop on the World Wide Web and Databases, 1999.
26. G. Stewart, G. Crane, and A. Babeu. A new generation of textual corpora: mining corpora from very large collections. In JCDL, pages 356-365, 2007.