Compact features for detection of near-duplicates in distributed retrieval

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Volumn 4209 LNCS, Issue , 2006, Pages 110-121

  Bernstein, Yaniv ^a   Shokouhi, Milad ^a   Zobel, Justin ^a  

^a RMIT UNIVERSITY   (Australia)

Author keywords

[No Author keywords available]

Indexed keywords

BANDWIDTH; COMPUTATIONAL METHODS; DISTRIBUTED COMPUTER SYSTEMS; INDEXING (OF INFORMATION); INFORMATION RETRIEVAL; QUERY LANGUAGES;

DISTRIBUTED INFORMATION RETRIEVAL SYSTEMS; DOCUMENTS; HASH VECTORS;

INFORMATION RETRIEVAL SYSTEMS;

EID: 33750314850     PISSN: 03029743     EISSN: 16113349     Source Type: Book Series    
DOI: 10.1007/11880561_10     Document Type: Conference Paper

References (31)

1. J. Allan et al. Challenges in information retrieval and language modeling: report of a workshop held at the center for intelligent information retrieval, University of Massachusetts Amherst, September 2002. SIGIR Forum, 37(1):31-47, 2003.
2. Y. Bernstein and J. Zobel. A scalable system for identifying co-derivative documents. In Proc. String Processing and Information Retrieval Symposium, pages 55-67, Padova, Italy, 2004.
3. Y. Bernstein and J. Zobel. Redundant documents and search effectiveness. In Proc. ACM CIKM Conf., pages 736-743, Bremen, Germany, 2005.
4. S. Brin, J. Davis, and H. García-Molina. Copy detection mechanisms for digital documents. In Proc. ACM SIGMOD international conference on Management of Data, pages 398-409, San Jose, California, 1995.
5. A. Z. Broder, S. C. Glassman, M. S. Manasse, and G. Zweig. Syntactic clustering of the web. Computer Networks and ISDN Systems, 29(8-13):1157-1166, 1997.
6. A. Z. Broder, M. Charikar, A. M. Frieze, and M. Mitzenmacher. Min-wise independent permutations (extended abstract). In Proc. ACM symposium on Theory of computing (STOC), pages 327-336, New York, NY, USA, 1998. ACM Press. ISBN 0-89791-962-9.
7. J. Callan and M. Connell. Query-based sampling of text databases. ACM Transactions on Information Systems, 19(2):97-130, 2001.
8. J. Callan, Z. Lu, and W. B. Croft. Searching distributed collections with inference networks. In Proc. Int. ACM-SIGIR Conf., pages 21-28, Seattle, Washington, 1995.
9. A. Chowdhury, O. Frieder, D. Grossman, and M. C. McCabe. Collection statistics for fast duplicate document detection. ACM Transactions on Information Systems, 20 (2):171-191, 2002.
10. J. G. Conrad, X. S. Guo, and C. P. Schriber. Online duplicate document detection: Signature reliability in a dynamic retrieval environment. In Proc. ACM-CIKM Conf., pages 443-452, New Orleans, Louisiana, 2003.
11. J. W. Cooper, A. R. Coden, and E. W. Brown. Detecting similar documents using salient terms. In Proc. ACM-CIKM Conf., pages 245-251, McLean, Virginia, 2002.
12. D. Fetterly, M. Manasse, and M. Najork. On the evolution of clusters of near-duplicate web pages. In Proc. first Latin American Web Congress, pages 37-45. IEEE, 2003.
13. S. Gauch, G. Wang, and M. Gomez. ProFusion: Intelligent fusion from multiple, distributed search engines. J. Universal Computer Science, 2(9):637-649, 1996.
14. L. Gravano, C. K. Chang, H. Garcia-Molina, and A. Paepcke. STARTS: Stanford proposal for Internet meta-searching. In Proc. ACM SIGMOD international conference on Management of Data, pages 207-218, Tucson, Arizona, 1997.
15. D. Harman. Overview of the first TREC conference. In Proc. ACM-SIGIR Conf., pages 36-47, Pittsburgh, Pennsylvania, 1993.
16. T. Hernandez and S. Kambhampati. Improving text collection selection with coverage and overlap statistics. In Proc. Int. Conf. on World Wide Web, pages 1128-1129, Chiba, Japan, 2005.
17. T. C. Hoad and J. Zobel. Methods for identifying versioned and plagiarised documents. J. the American Society for Information Science and Technology, 54(3):203-215, 2003.
18. S. Ilyinski, M. Kuzmin, A. Melkov, and I. Segalovich. An efficient method to detect duplicates of web documents with the use of inverted index. In Proc, Int. Conf. on World Wide Web, Honolulu, Hawaii, 2002.
19. A. Kolcz, A. Chowdhury, and J. Alspector. Improved robustness of signature-based near-replica detection via lexicon randomization. In Proc. ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining, pages 605-610, Seattle, WA, 2004.
20. C. Lyon, J. Malcolm, and B. Dickerson. Detecting short passages of similar text in large document collections. In Proc. Conf. on Empirical Methods in Natural Language Processing, Philadelphia, Pennsylvania, 2001.
21. U. Manber. Finding similar files in a large file system. In Proc. USENIX Winter Technical Conf., pages 1-10, San Fransisco, CA, 17-21 1994.
22. W. Meng, C. Yu, and K. Liu. Building efficient and effective metasearch engines. ACM Computing Surveys, 34(1):48-89, 2002.
23. H. Nottelmann and N. Fuhr. Evaluating different methods of estimating retrieval quality for resource selection. In Proc. Int. ACM-SIGIR Conf., pages 290-297, Toronto, Canada, 2003.
24. A. L. Powell and J. French. Comparing the performance of collection selection algorithms. ACM Transactions on Information Systems, 21(4):412-456, 2003.
25. W. Pugh and M. H. Henzinger. Detecting duplicate and near-duplicate files (United States Patent 6,658,423), 2003.
26. E. Selberg and O. Etzioni. The MetaCrawler architecture for resource aggregation on the Web. IEEE Expert, (January-February): 11-14, 1997.
27. L. Si and J. Callan. Unified utility maximization framework for resource selection. In Proc. ACM-CIKM Conf., pages 32-41, Washington, D.C., 2004.
28. L. Si and J. Callan. Relevant document distribution estimation method for resource selection. In Proc. ACM-SIGIR Conf., pages 298-305, Toronto, Canada, 2003.
29. H. S. Warren, Jr. Hacker's Delight. Addison Wesley, 2002.
30. O. Zamir and O. Etzioni. Grouper: a dynamic clustering interface to web search results. In Proc. Int. Conf. on World Wide Web, pages 1361-1374, Toronto, Canada, 1999.
31. J. Zobel and Y. Bernstein. The case of the duplicate documents: Measurement, search, and science. In Proc. Asia-Pacific Web Con}., pages 26-39, Harbin, China, 2006.