Adaptive near-duplicate detection via similarity learning

SIGIR 2010 Proceedings - 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn , Issue , 2010, Pages 419-426

  Hajishirzi, Hannaneh ^a   Yih, Wen Tau ^b   Kołcz, Aleksander ^b  

^a UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

^b MICROSOFT RESEARCH   (United States)

Author keywords

Near duplicate detection; Similarity learning; Spam detection

Indexed keywords

COMMONLY USED; COSINE SIMILARITY; DUPLICATE DOCUMENT DETECTION; EMAIL MESSAGES; EXISTING METHOD; JACCARD COEFFICIENTS; LOCALITY SENSITIVE HASHING; NEAR-DUPLICATE DETECTION; NEWS ARTICLES; SIMILARITY COMPUTATION; SIMILARITY FUNCTIONS; SIMILARITY LEARNING; SIMILARITY MEASURE; SPAM DETECTION; TARGET DOMAIN;

BUILDING MATERIALS; INFORMATION RETRIEVAL;

INTERNET;

EID: 77956039068     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1835449.1835520     Document Type: Conference Paper

References (21)

1. A. Andoni and P. Indyk. Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions. Communications of the ACM, 51(1):117-122, 2008.
2. A. Z. Broder. Identifying and filtering near-duplicate documents. In COM '00, pages 1-10. Springer-Verlag, 2000.
3. A. Z. Broder, M. Charikar, A. M. Frieze, and M. Mitzenmacher. Min-wise independent permutations. Journal of Computer and System Sciences, 60:630-659, 2000.
4. A. Z. Broder, S. C. Glassman, M. S. Manasse, and G. Zweig. Syntactic clustering of the web. Comput. Netw. ISDN Syst., 29(8-13):1157-1166, 1997.
5. M. S. Charikar. Similarity estimation techniques from rounding algorithms. In Proceedings of the 34th Annual ACM Symposium on Theory of Computing, 2002.
6. A. Chowdhury, O. Frieder, D. Grossman, and M. C. McCabe. Collection statistics for fast duplicate document detection. ACM Trans. Inf. Syst., 20(2):171-191, 2002.
7. D. Fetterly, M. Manasse, and M. Najork. On the evolution of clusters of near-duplicate web pages. In LA-WEB '03, 2003.
8. A. Gionis, P. Indyk, and R. Motwani. Similarity search in high dimensions via hashing. In VLDB '99, 1999.
9. R. J. Hall. A countermeasure to duplicate-detecting anti-spam techniques. Technical report, AT&T, 1999.
10. M. Henzinger. Finding near-duplicate web pages: a large-scale evaluation of algorithms. In SIGIR '06, pages 284-291, New York, NY, USA, 2006. ACM.
11. T. C. Hoad and J. Zobel. Methods for identifying versioned and plagiarized documents. Journal of the American Society for Information Science and Technology, 54(3):203-215, 2003.
12. P. Indyk and R. Motwani. Approximate nearest neighbors: towards removing the curse of dimensionality. In Proceedings of 30th STOC, pages 604-613, 1998.
13. A. Kolcz and A. Chowdhury. Hardening fingerprinting by context. In CEAS '07, 2007.
14. A. Kolcz and A. Chowdhury. Lexicon randomization for near-duplicate detection with I-match. Journal of Supercomputing, 45(3):255-276, 2008.
15. A. Kolcz, A. Chowdhury, and J. Alspector. Improved robustness of signature-based near-replica detection via lexicon randomization. In KDD '04, 2004.
16. D. Lowd and C. Meek. Good word attacks on statistical spam filters. In CEAS'05, 2005.
17. G. S. Manku, A. Jain, and A. Das Sarma. Detecting near-duplicates for web crawling. In WWW '07, 2007.
18. V. V. Prakash and A. O'Donnell. Fighting spam with reputation systems. Queue, 3(9):36-41, 2005.
19. M. Rabin. Fingerprinting by random polynomials. Report TR-1581, Harvard University, 1981.
20. M. Theobald, J. Siddharth, and A. Paepcke. Spotsigs: robust and efficient near duplicate detection in large web collections. In SIGIR '08, pages 563-570, 2008.
21. W. Yih. Learning term-weighting functions for similarity measures. In Proc. of EMNLP-09, 2009.