SpotSigs: Robust and efficient near duplicate detection in large web collections

ACM SIGIR 2008 - 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Proceedings

Volumn , Issue , 2008, Pages 563-570

  Theobald, Martin ^a   Siddharth, Jonathan ^a   Paepcke, Andreas ^a  

^a Stanford University   (United States)

Author keywords

High dimensional similarity search; Inverted index pruning; Optimal partitioning; Stopword signatures

Indexed keywords

INFORMATION RETRIEVAL; INFORMATION RETRIEVAL SYSTEMS; INFORMATION SERVICES; PATTERN MATCHING; RESEARCH AND DEVELOPMENT MANAGEMENT; WORLD WIDE WEB;

DOCUMENT SIGNATURES; DUPLICATE DETECTIONS; EXECUTION TIMES; HIGH-DIMENSIONAL SIMILARITY SEARCH; INVERTED INDEX PRUNING; INVERTED INDICES; MATCHING ALGORITHMS; NEW ALGORITHMS; NEWS ARTICLES; OPTIMAL PARTITIONING; PRECISION AND RECALLS; SELF-TUNING; SENSITIVE HASHING; SIMILARITY SEARCHES; STOPWORD SIGNATURES; WEB ARCHIVES; WEB COLLECTIONS; WEB CRAWLS; WEB PAGES;

DATABASE SYSTEMS;

EID: 57349131623     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1390334.1390431     Document Type: Conference Paper

References (30)

1. A. Andoni and P. Indyk. Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions. In FOCS, p. 459-468, 2006.
2. R. A. Baeza-Yates and B. Ribeiro-Neto. Modern Information Retrieval. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA, 1999.
3. M. Bawa, T. Condie, and P. Ganesan. LSH forest: self-tuning indexes for similarity search. In WWW, p. 651-660, 2005.
4. S. Brin, J. Davis, and H. García-Molina. Copy detection mechanisms for digital documents. In SIGMOD, p. 398-409, 1995.
5. A. Z. Broder. Identifying and filtering near-duplicate documents. In COM, p. 1-10, 2000.
6. A. Z. Broder, M. Charikar, and M. Mitzenmacher. A derandomization using min-wise independent permutations. J. Discrete Algorithms, 1(1):11-20, 2003.
7. A. Z. Broder, S. C. Glassman, M. S. Manasse, and G. Zweig. Syntactic clustering of the Web. Computer Networks, 29(8-13):1157-1166, 1997.
8. C. Buckley, G. Salton, and J. Allan. Automatic retrieval with locality information using SMART. In TREC, p. 59-72, 1992.
9. S. Büttcher and C L. A. Clarke. A document-centric approach to static index pruning in text retrieval systems. In CIKM, p. 182-189, 2006.
10. M. S. Charikar. Similarity estimation techniques from rounding algorithms. In STOC, p. 380-388, 2002.
11. J. Cho, H. Garcia-Molina, T. Haveliwala, W. Lam, A. Paepcke, S. Raghavan, and G. Wesley. Stanford WebBase components and applications. ACM Trans. Inter. Tech., 6(2):153-186, 2006.
12. A. Chowdhury, O. Frieder, D. Grossman, and M. C. McCabe. Collection statistics for fast duplicate document detection. ACM Trans. Inf. Syst., 20(2):171-191, 2002.
13. E. Cohen, M. Datar, S. Fujiwara, A. Gionis, P. Indyk, R. Motwani, J. D. Ullman, and C. Yang. Finding interesting associations without support pruning. Knowledge and Data Engineering, 13(1):64-78, 2001.
14. J. G. Conrad, X. S. Guo, and C. P. Schriber. Online duplicate document detection: signature reliability in a dynamic retrieval environment. In CIKM, p. 443-452, 2003.
15. A. Gionis, P. Indyk, and R. Motwani. Similarity search in high dimensions via hashing. In VLDB, p. 518-529, 1999.
16. M. Henzinger. Finding near-duplicate Web pages: a large-scale evaluation of algorithms. In SIGIR, p. 284-291, 2006.
17. T. C. Hoad and J. Zobel. Methods for identifying versioned and plagiarized documents. JASIST, 54(3):203-215, 2003.
18. P. Indyk. A small approximately min-wise independent family of hash functions. J. Algorithms, 38(1):84-90, 2001.
19. P. Indyk. Nearest neighbors in high-dimensional spaces. In Handbook of Discrete and Computational Geometry. CRC Press, 2004.
20. P. Indyk and R. Motwani. Approximate nearest neighbors: towards removing the curse of dimensionality. In STOC, p. 604-613, 1998.
21. D. Klein, S. D. Kamvar, and C. D. Manning. From instance-level constraints to space-level constraints: Making the most of prior knowledge in data clustering. In ICML, p. 307-314, 2002.
22. A. Kolcz, A. Chowdhury, and J. Alspector. Improved robustness of signature-based near-replica detection via lexicon randomization. In KDD, p. 605-610, 2004.
23. Q. Lv, W. Josephson, Z. Wang, M. Charikar, and K. Li. Multi-probe LSH: Efficient indexing for high-dimensional similarity search. In VLDB, p. 950-961, 2007.
24. U. Manber. Finding similar files in a large file system. In WTEC, p. 2, 1994.
25. G. S. Manku, A. Jain, and A. D. Sarma. Detecting near-duplicates for Web crawling. In WWW, p. 141-150, 2007.
26. Web Sociologist's Workbench: http://dbpubs.stanford.edu/~testbed/doc2/ WabBase/SGERHighlight.pdf
27. N. Shivakumar and H. García-Molina. SCAM: A copy detection mechanism for digital documents. In DL, 1995.
28. N. Shivakumar and H. Garcia-Molina. Finding near-replicas of documents and servers on the Web. In WebDB, p. 204-212, 1998.
29. K. Wagstaff and C. Cardie. Clustering with instance-level constraints. In AAAI/IAAI, p. 1097, 2000.
30. H. Yang and J. P. Callan. Near-duplicate detection by instance-level constrained clustering. In SIGIR, p. 421-428, 2006.