A systematic study on parameter correlations in large-scale duplicate document detection

Knowledge and Information Systems

Volumn 14, Issue 2, 2008, Pages 217-232

  Ye, Shaozhi ^a   Wen, Ji Rong ^b   Ma, Wei Ying ^b  

^a University of California   (United States)

^b MICROSOFT RESEARCH ASIA   (China)

Author keywords

Clustering; Duplicate document detection; Sampling; Shingling

Indexed keywords

SAMPLING; SCALABILITY;

ADAPTIVE SAMPLING STRATEGIES; CLUSTERING; DUPLICATE DOCUMENT DETECTION; EMPIRICAL ANALYSIS; PARAMETER CORRELATION; PERFORMANCE AND SCALABILITIES; SHINGLING; SIMILARITY THRESHOLD;

IMPORTANCE SAMPLING;

EID: 38649124934     PISSN: 02191377     EISSN: 02193116     Source Type: Journal    
DOI: 10.1007/s10115-007-0071-9     Document Type: Conference Paper

References (26)

1. Bellare M, Kohno T (2004) Hash function balance and its impact on birthday aItacks. In: EUROCRYPT 2004: international conference on the theory and applications of cryptographic techniques, pp 401-418
2. Bharat K, Broder AZ (1999) Mirror, mirror on the Web: A study of host pairs with replicated content. In: Proceedings of the 8th international World Wide Web (WWW) conference, pp 501-512
3. Bharat K, Broder AZ, Dean J, Henzinger MR (2000) A comparison of techniques to find mirrored hosts on the WWW. J Am Soc Inf Sci (JASIS) 51(12):1114-1122
4. Brin S, Davis J, Garcia-Molina H (1995) Copy detection mechanisms for digital documents. In: Proceedings of the 1995 ACM international conference on management of data (SIGMOD), pp 398-409
5. Broder AZ, Glassman SC, Manasse MS, Zweig G (1997) Syntactic clustering of the Web. In: Proceedings of the sixth international World Wide Web (WWW) conference, pp 1157-1166
6. Cho J, Shivakumar N, Garcia-Molina H (2000) Finding replicated Web collections. In: Proceedings of the 2000 ACM international conference on management of data (SIGMOD), pp 355-366
7. Chowdhury A, Frieder O, Grossman D, McCabe MC (2002) Collection statistics for fast duplicate document detection. ACM Trans Inf Syst 20(2):171-191
8. Conrad JG, Guo XS, Schriber CP (2003) Online duplicate document detection: Signature reliability in a dynamic retrieval environment. In: Proceedings of the 12th international conference on information and knowledge management (CIKM), pp 443-452
9. Cooper JW, Coden A, Brown EW (2002) Detecting similar documents using salient terms. In: Proceedings of the 11th ACM international conference on information and knowledge management (CIKM), pp 245-251
10. Crovella ME, Taqqu MS, Bestavros A (1998) Heavy-tailed probability distributions in the World Wide Web. In: Adler R, Feldman R, Taqqu M (eds.) A practical guide to heavy tails: Statistical techniques and applications. Birkhauser, Boston, pp 3-25
11. Dean J, Henzinger MR (1999) Finding related pages in the World Wide Web. In: Proceeding of the 8th international World Wide Web conference (WWW), pp 1467-1479
12. Fetterly D, Manasse M, Najork M (2003) On the evolution of clusters of near-duplicate Web pages. In: Proceedings of the 1st Latin American Web Congress (LA-Web), pp 37-45
13. Fetterly D, Manasse M, Najork M (2004) Spam, damn spam, and statistics: using statistical analysis to locate spam Web pages. In: Proceedings of the 7th international workshop on the Web and databases (WebDB), pp 1-6
14. Fetterly D, Manasse M, Najork M, Wiener J (2003) A large-scale study of the evolution of Web pages. In: Proceedings of the 12th international World Wide Web (WWW) conference, pp 669-678
15. Gyongyi Z, Garcia-Molina H (2005) Web spam taxonomy, Technical report, Stanford University
16. Heintze N (1996) Scalable document fingerprinting. In: Proceedings of the 2nd USENIX electronic commerce workshop, pp 191-200
17. Li Z, Ng WK, Sun A (2005) Web data extraction based on structural similarity. Knowl Inf Syst 8(4):438-461
18. Mukherjea1 S (2004) Discovering and analyzing World Wide Web collections. Knowl Inf Syst 6(2):230-241
19. Rabin M (1981) Fingerprinting by random polynomials, Technical report tr-15-81, Center for Research in Computing Technology, Harvard University
20. Shivakumar N, Garcia-Molina H (1998) Finding near-replicas of documents and servers on the Web. In: Proceedings of the 1st international workshop on World Wide Web and Databases (WebDB), pp 204-212
21. Soboroff I (2002) Do TREC Web collections look like the Web? SIGIR Forum 36(2):23-31
22. Wang Y, Kitsuregawa M (2002) Evaluating contents-link coupled Web page clustering for Web search results. In: Proceedings of the 11th ACM international conference on information and knowledge management (CIKM), pp 499-506
23. Ye S, Song R, Wen J-R, Ma W-Y (2004) A query-dependent duplicate detection approach for large scale search engines. In: Proceedings of the 6th Asia-Pacific Web conference (APWeb), pp 48-58
24. Yi L, Liu B, Li X (2003) Eliminating noisy information in Web pages for data mining. In: Proceedings of the 9th ACM international conference on knowledge discovery and data mining (SIGKDD), pp 296-305
25. Zamir O, Etzioni O (1998) Web document clustering: A feasibility demonstration. In: Proceedings of the 21st annual international ACM conference on research and development in information retrieval (SIGIR), pp 46-54
26. Zhong S, Ghosh J (2005) Generative model-based document clustering: A comparative study. Knowl Inf Syst 8(3):374-384