Efficient similarity joins for near-duplicate detection

ACM Transactions on Database Systems

Volumn 36, Issue 3, 2011, Pages

  Xiao, Chuan ^a   Wang, Wei ^a   Lin, Xuemin ^a   Yu, Jeffrey Xu ^b   Wang, Guoren ^c  

^a UNIVERSITY OF NEW SOUTH WALES   (Australia)

^b CHINESE UNIVERSITY OF HONG KONG   (Hong Kong)

^c NORTHEASTERN UNIVERSITY   (China)

Author keywords

Near duplicate detection; Similarity join

Indexed keywords

EFFICIENT ALGORITHM; FILTERING TECHNIQUE; MULTIPLE DATA SOURCES; NEAR-DUPLICATE DETECTION; REAL DATA SETS; SIMILARITY JOIN;

ALGORITHMS;

EID: 80052344031     PISSN: 03625915     EISSN: 15574644     Source Type: Journal    
DOI: 10.1145/2000824.2000825     Document Type: Article

References (53)

1. AGRAWAL, S., CHAUDHURI, S., DAS, G., AND GIONIS, A. 2003. Automated ranking of database query results. In Proceedings of the Conference on Innovative Data Systems Research (CIDR'03).
2. ARASU, A., CHAUDHURI, S., AND KAUSHIK, R. 2008. Transformation-Based framework for record matching. In Proceedings of the International Conference on Data Engineering (ICDE'08). 40-49.
3. ARASU, A., GANTI, V., AND KAUSHIK, R. 2006. Efficient exact set-similarity joins. In Proceedings of the International Conference on Very Large Databases (VLDB'06).
4. BAEZA-YATES, R. AND RIBEIRO-NETO, B. 1999. Modern Information Retrieval 1st Ed. Addison Wesley.
5. BAYARDO, R. J., MA, Y., AND SRIKANT, R. 2007. Scaling up all pairs similarity search. In Proceedings of the International World Wide Web Conference (WWW'07).
6. BEHM, A., JI, S., LI, C., AND LU, J. 2009. Space-Constrained gram-based indexing for efficient approximate string search. In Proceedings of the International Conference on Data Engineering (ICDE'09). 604-615.
7. BILENKO, M., MOONEY, R. J., COHEN, W. W., RAVIKUMAR, P., AND FIENBERG, S. E. 2003. Adaptive name matching in information integration. IEEE Intell. Syst. 18, 5, 16-23.
8. BRIN, S., DAVIS, J., AND GARCIA-MOLINA, H. 1995. Copy detection mechanisms for digital documents. In Proceedings of the ACM SIGMOD International Conference on Management of Data. 398-409.
9. BRODER, A. Z. 1997. On the resemblance and containment of documents. In Proceedings of the SEQS Conference.
10. BRODER, A. Z., GLASSMAN, S. C., MANASSE, M. S., AND ZWEIG, G. 1997. Syntactic clustering of the web. Comput. Netw. 29, 8-13, 1157-1166.
11. BURKHARDT, S. AND KARKKAINEN, J. 2002. One-Gapped q-gram filtersfor levenshtein distance. In Proceedings of the Annual Symposium on Combinatorial Pattern Matching (CPM'02). 225-234.
12. CHANG, K. C.-C. AND WON HWANG, S. 2002. Minimal probing: Supporting expensive predicates for top-k queries. In Proceedings of the ACM SIGMOD International Conference on Management of Data. 346-357. (Pubitemid 34985560)
13. CHARIKAR, M. 2002. Similarity estimation techniques from rounding algorithms. In Proceedings of the Annual ACM Symposium on Theory of Computing (STOC'02).
14. CHAUDHURI, S., GANJAM, K., GANTI, V., AND MOTWANI, R. 2003. Robust and efficient fuzzy match for online data cleaning. In Proceedings of the ACM SIGMOD International Conference on Management of Data. 313-324.
15. CHAUDHURI, S., GANTI, V., AND KAUSHIK, R. 2006. A primitive operator for similarity joins in data cleaning. In Proceedings of the International Conference on Data Engineering (ICDE'06).
16. CHO, J., SHIVAKUMAR, N., AND GARCIA-MOLINA, H. 2000. Finding replicated web collections. In Proceedings of the ACM SIGMOD International Conference on Management of Data.
17. CHOWDHURY, A., FRIEDER, O., GROSSMAN, D. A., AND MCCABE, M. C. 2002. Collection statistics for fast duplicate document detection. ACM Trans. Inf. Syst. 20, 2, 171-191. (Pubitemid 44642301)
18. CONRAD, J. G., GUO, X. S., AND SCHRIBER, C. P. 2003. Online duplicate document detection: Signature reliability in a dynamic retrieval environment. In Proceedings of the ACM International Conference on Information and Knowledge Management (CIKM'03).
19. ELMAGARMID, A. K., IPEIROTIS, P. G., AND VERYKIOS, V. S. 2007. Duplicate record detection: A survey. Trans. Knowl. Data Engin.19, 1, 1-16. (Pubitemid 44955773)
20. FAGIN, R. 1999. Combining fuzzy information from multiple systems. J. Comput. Syst. Sci. 58, 1, 83-99.
21. FAGIN, R., KUMAR, R., AND SIVAKUMAR, D. 2003a. Efficient similarity search and classification via rank aggregation. In Proceedings of the ACM SIGMOD International Conference on Management of Data.
22. FAGIN, R., LOTEM, A., AND NAOR, M. 2003b. Optimal aggregation algorithms for middleware. J. Comput. Syst. Sci. 66, 4, 614-656.
23. FETTERLY, D., MANASSE, M., AND NAJORK, M. 2003. On the evolution of clusters of near-duplicate web pages. In Proceedings of the Latin American Web Congress (LAWEB'03).
24. GIBSON, D., KUMAR, R., AND TOMKINS, A. 2005. Discovering large dense subgraphs in massive graphs. In Proceedings of the International Conference on Very Large Databases (VLDB'05).
25. GIONIS, A., INDYK, P., AND MOTWANI, R. 1999. Similarity search in high dimensions via hashing. In Proceedings of the International Conference on Very Large Databases (VLDB'99).
26. GRAVANO, L., IPEIROTIS, P. G., JAGADISH, H. V., KOUDAS, N., MUTHUKRISHNAN, S., AND SRIVASTAVA, D. 2001. Approximate string joins in a database (almost) for free. In Proceedings of the International Conference on Very Large Databases (VLDB'01).
27. GUSFIELD, D. 1997. Algorithms on Strings, Trees, and Sequences. Computer Science and Computational Biology. Cambridge University Press.
28. HADJIELEFTHERIOU, M., CHANDEL, A., KOUDAS, N., AND SRIVASTAVA, D. 2008. Fast indexes and algorithms for set similarity selection queries. In Proceedings of the International Conference on Data Engineering (ICDE'08). 267-276.
29. HAMID, O. A., BEHZADI, B., CHRISTOPH, S., AND HENZINGER, M. R. 2009. Detecting the origin of text segments efficiently. In Proceedings of the International World Wide Web Conference (WWW'09). 61-70.
30. HENZINGER, M. R. 2006. Finding near-duplicate web pages: A large-scale evaluation of algorithms. In Proceedings of the Annual ACM SIGIR Conference on Research and Development in Information Retrieval.
31. HERNANDEZ, M. A. AND STOLFO, S. J. 1998. Real-World data is dirty: Data cleansing and the merge/purge problem. Data Min. Knowl. Discov. 2, 1, 9-37. (Pubitemid 128696797)
32. HOAD, T. C. AND ZOBEL, J. 2003. Methods for identifying versioned and plagiarized documents. J. Amer. Soc. Inf. Sci. Technol. 54, 3, 203-215.
33. INDYK, P. AND MOTWANI, R. 1998. Approximate nearest neighbors: Towards removing the curse of dimensionality. In Proceedings of the Annual ACM Symposium on Theory of Computing (STOC'98).
34. LI, C., LU, J., AND LU, Y. 2008. Efficient merging and filtering algorithms for approximate string searches. In Proceedings of the International Conference on Data Engineering (ICDE'08). 257-266.
35. LI, C., WANG, B., AND YANG, X. 2007. VGRAM: Improving performance of approximate queries on string collections using variable-length grams. In Proceedings of the International Conference on Very Large Databases (VLDB'07).
36. LIEBERMAN, M. D., SANKARANARAYANAN, J., AND SAMET, H. 2008. A fast similarity join algorithm using graphics processing units. In Proceedings of the International Conference on Data Engineering (ICDE'08). 1111-1120.
37. LUO, Y., LIN, X., WANG, W., AND ZHOU, X. 2007. SPARK: Top-k keyword query in relational databases. In Proceedings of the ACM SIGMOD International Conference on Management of Data. 115-126. (Pubitemid 47630801)
38. MANBER, U. 1994. Finding similar files in a large file system. In Proceedings of theUSENIXWinter Conference. 1-10.
39. MANKU, G. S., JAIN, A., AND SARMA, A. D. 2007. Detecting near-duplicates for web crawling. In Proceedings of the International World Wide Web Conference (WWW'07). 141-150. (Pubitemid 47582246)
40. NAVARRO, G. 2001. A guided tour to approximate string matching. ACM Comput. Surv. 33, 1, 31-88. (Pubitemid 33768480)
41. RUSSELL, R. C. 1918. Index. U.S. patent 1, 261, 167.
42. SARAWAGI, S. AND BHAMIDIPATY, A. 2002. Interactive deduplication using active learning. In Proceedings of the International SIGKDD Conference on Knowledge Discovery and Data Mining (KDD'02).
43. SARAWAGI, S. AND KIRPAL, A. 2004. Efficient set joins on similarity predicates. In Proceedings of the ACM SIGMOD International Conference on Management of Data.
44. SCHLEIMER, S., WILKERSON, D. S., AND AIKEN, A. 2003.Winnowing: Local algorithms for document fingerprinting. In Proceedings of the ACM SIGMOD International Conference on Management of Data. 76-85.
45. SEO, J. AND CROFT, W. B. 2008. Local text reuse detection. In Proceedings of the Annual ACM SIGIR Conference on Research and Development in Information Retrieval. 571-578.
46. SPERTUS, E., SAHAMI, M., AND BUYUKKOKTEN, O. 2005. Evaluating similarity measures: A large-scale study in the orkut social network. In Proceedings of the International SIGKDD Conference on Knowledge Discovery and Data Mining (KDD'05).
47. THEOBALD, M., SIDDHARTH, J., AND PAEPCKE, A. 2008. Spotsigs: Robust and efficient near duplicate detection in large web collections. In Proceedings of the Annual ACMSIGIR Conference on Research and Development in Information Retrieval. 563-570.
48. UKKONEN, E. 1983. On approximate string matching. In Proceedings of the International Symposium on Foundations of Computation Theory (FCT'83).
49. WINKLER, W. E. 1999. The state of record linkage and current research problems. Tech. rep., U.S. Bureau of the Census.
50. XIAO, C., WANG, W., AND LIN, X. 2008a. Ed-Join: An efficient algorithm for similarity joins with edit distance constraints. Proc. VLDB 1, 1, 933-944.
51. XIAO, C., WANG, W., LIN, X., AND SHANG, H. 2009. Top-k set similarity joins. In Proceedings of the International Conference on Data Engineering (ICDE'09). 916-927.
52. XIAO, C., WANG, W., LIN, X., AND YU, J. X. 2008b. Efficient similarity joins for near duplicate detection. In Proceedings of the International World Wide Web Conference (WWW'08).
53. YANG, X., WANG, B., AND LI, C. 2008. Cost-Based variable-length-gram selection for string collections to support approximate queries efficiently. In Proceedings of the ACM SIGMOD International Conference on Management of Data. 353-364.