Accurate discovery of co-derivative documents via duplicate text detection

Information Systems

Volumn 31, Issue 7, 2006, Pages 595-609

  Bernstein, Yaniv ^a   Zobel, Justin ^a  

^a RMIT UNIVERSITY   (Australia)

Author keywords

Duplicate detection; Fingerprinting; Hashing

Indexed keywords

ALGORITHMS; DATA ACQUISITION; FEATURE EXTRACTION; INDEXING (OF INFORMATION); PATTERN RECOGNITION;

DUPLICATE DETECTION; FINGERPRINTING; HASHING;

INFORMATION RETRIEVAL SYSTEMS;

EID: 33747171448     PISSN: 03064379     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.is.2005.11.006     Document Type: Article

References (24)

1. A.Z. Broder, On the resemblance and containment of documents, in: Compression and Complexity of Sequences (SEQUENCES'97)', 1997, pp. 21-29.
2. M. Sanderson, Duplicate detection in the Reuters collection, Technical Report TR-1997-5, University of Glasgow, 1997.
3. N. Shivakumar, H. García-Molina, SCAM: a copy detection mechanism for digital documents, in: Proceedings of the Second Annual Conference on the Theory and Practice of Digital Libraries, 1995.
4. Hoad T.C., and Zobel J. Methods for identifying versioned and plagiarised documents. Journal of the American Society for Information Science and Technology 54 3 (2003) 203-215
5. U. Manber, Finding similar files in a large file system, in: Proceedings of the USENIX Winter 1994 Technical Conference, 1994, pp. 1-10.
6. S. Brin, J. Davis, H. García-Molina, Copy detection mechanisms for digital documents, in: Proceedings of the ACM SIGMOD Annual Conference, 1995, pp. 398-409.
7. N. Heintze, Scalable document fingerprinting, in: 1996 USENIX Workshop on Electronic Commerce, 1996.
8. Broder A.Z., Glassman S.C., Manasse M.S., and Zweig G. Syntactic clustering of the Web. Computer Networks and ISDN Systems 29 8-13 (1997) 1157-1166
9. Witten I.H., Moffat A., and Bell T.C. Managing Gigabytes: Compressing and Indexing Documents and Images (1999), Morgan Kauffman, Los Altos, CA
10. Schleimer S., Wilkerson D.S., and Aiken A. Winnowing. local algorithms for document fingerprinting. Proceedings of ACM SIGMOD conference. ACM Press (2003) 76-85
11. Fang M., Shivakumar N., Garcia-Molina H., Motwani R., and Ullman J.D. Computing iceberg queries efficiently. Proceedings of the 24th International Conference on Very Large Data Bases. Morgan Kaufmann, Los Altos, CA (1998) 299-310
12. N. Shivakumar, H. García-Molina, Finding near-replicas of documents on the web, in: WEBDB, International Workshop on the World Wide Web and Databases, WebDB, Springer, Berlin, 1999.
13. Nevill-Manning C.G., and Witten I.H. Compression and explanation using hierarchical grammars. The Computer Journal 40 2/3 (1997) 103-116
14. Larsson N.J., and Moffat A. Offline dictionary-based compression. Proc. IEEE 88 11 (2000) 1722-1732
15. Nevill-Manning C.G., Witten I.H., and Paynter G.W. Browsing in digital libraries: a phrase-based approach. Proceedings of the Second ACM International Conference on Digital Libraries (1997), ACM Press 230-236
16. Moffat A., and Wan R. Re-Store: a system for compressing, browsing, and searching large documents. Proceedings of the International Symposium on String Processing and Information Retrieval (2001), IEEE Computer Society 162-174
17. Gusfield D. Algorithms on Strings, Trees, and Sequences: Computer Science and Computational Biology (1997), Cambridge University Press, Cambridge
18. Agrawal R., and Srikant R. Fast algorithms for mining association rules in large databases. Proceedings of the 20th Conference on Very Large Data Bases. Morgan Kaufmann, Los Altos, CA (1994) 487-499
19. Witten I.H., and Bell T.C. Source models for natural language text. Int. J. Man Machine Studies 32 (1990) 545-579
20. R. Rivest, The MD5 Message-Digest Algorithm, RFC 1321, 1992.
21. Harman D. Overview of the second text retrieval conference (TREC-2). Information Processing and Management 31 3 (1995) 271-289
22. Baeza-Yates R., and Ribeiro-Neto B. Modern Information Retrieval (1999), Addison-Wesley, Longman, Reading, MA, New York
23. Chowdhury A., Frieder O., Grossman D., and McCabe M.C. Collection statistics for fast duplicate document detection. ACM Transactions on Information Systems (TOIS) 20 2 (2002) 171-191
24. Cooper J.W., Coden A.R., and Brown E.W. Detecting similar documents using salient terms. Proceedings of the Eleventh International Conference on Information and Knowledge Management (2002), ACM Press 245-251