An incremental clustering scheme for data de-duplication

Data Mining and Knowledge Discovery

Volumn 20, Issue 1, 2010, Pages 152-187

  Costa, Gianni ^a   Manco, Giuseppe ^a   Ortale, Riccardo ^a  

^a ICAR CNR   (Italy)

Author keywords

Approximated similarity measures; Clustering mining methods and algorithms; De duplication; Indexing methods and structures; Locality sensitive hashing; Min wise independent permutations; Record classification

Indexed keywords

INDEXING METHODS; LOCALITY SENSITIVE HASHING; MIN-WISE INDEPENDENT PERMUTATIONS; MINING METHODS AND ALGORITHMS; SIMILARITY MEASURE;

CONTENT BASED RETRIEVAL; MINING;

INDEXING (OF INFORMATION);

EID: 76749114248     PISSN: 13845810     EISSN: None     Source Type: Journal    
DOI: 10.1007/s10618-009-0155-0     Document Type: Article

References (47)

1. Agichtein E, Ganti V (2004) Mining reference tables for automatic text segmentation. In: Proceedings of the ACM SIGKDD international conference on knowledge discovery and data mining, pp 20-29
2. Ananthakrishna R, Chaudhuri S, Ganti V (2002) Eliminating fuzzy duplicates in data warehouses. In: Proceedings of the international conference on very large databases, pp 586-597
3. Arasu A, Ganti V, Kaushik R (2006) Efficient exact set-similarity joins. In: Proceedings of the international conference on very large databases, pp 918-929
4. Bawa M, Tyson S, Condie, Ganesan P (2005) LSH forest: self-tuning indexes for similarity search. In: Proceedings of the international conference on world wide web, pp 651-660
5. Bayardo RJ, Ma Y, Srikant R (2007) Scaling up all pairs similarity search. In: Proceedings of the international conference on world wide web, pp 131-140
6. Bhattacharya I, Getoor L (2004) Iterative record linkage for cleaning and integration. In: Proceedings of the SIGMOD workshop on research issues on data mining and knowledge discovery, pp 11-18
7. Bilenko M, Mooney RJ (2003a) Adaptive duplicate detection using learnable string similarity measures. In: Proceedings of the ACM SIGKDD international conference on knowledge discovery and data mining, pp 39-48
8. Bilenko M, Mooney RJ (2003b) On evaluation and training-set construction for duplicate detection. In: Proceedings of the KDD workshop on data cleaning, record linkage, and object consolidation, pp 7-12
9. Broder A, Glassman S, Manasse M, Zweig G (1997) Syntactic clustering on the Web. In: Proceedings of the international conference on World Wide Web, pp 1157-1166
10. Broder A, Charikar M, Frieze AM, Mitzenmacher M (1998) Minwise independent permutations. In: Proceedings of the ACM symposium on theory of computing, pp 327-336
11. Cesario E, Folino F, Manco G, Pontieri L (2005) An incremental clustering scheme for duplicate detection in large databases. In: Proceedings of the international conference databases and applications symposium, pp 89-95
12. E Cesario F Folino A Locane G Manco R Ortale 2008 Boosting text segmentation via progressive classification J Knowl Inf Syst 15 3 285 320 10.1007/s10115-007-0085-3
13. Chaudhuri S, Ganjam K, Ganti V, Motwani R (2003) Robust and efficient fuzzy match for online data cleaning. In: Proceedings of the ACM SIGMOD conference on management of data, pp 313-324
14. Chaudhuri S, Ganti V, Motwani R (2005) Robust identification of fuzzy duplicates. In: Proceedings of the international conference on data engineering, pp 865-876
15. E Chavez G Navarro R Baeza-Yates J Luis Marroquin 2001 Searching in metric spaces ACM Comput Surv 33 3 273 321 10.1145/502807.502808
16. Ciaccia P, Patella M, Zezula P (1997) M-Tree: an efficient access method for similarity search in metric spaces. In: Proceedings of the international conference on very large databases, pp 426-435
17. Cochinwala M, Dalal S, Elmagarmid AK, Verykios VS (2005) Record matching: past, present and future
18. Cohen W, Richman J (2001) Learning to match and cluster entity names. In: Proceedings of the ACM SIGIR workshop on mathematical/formal methods in information retrieval, pp 13-18
19. Cohen WW, Richman J (2002) Learning to match and cluster large high-dimensional data sets for data integration. In: Proceedings of the ACM SIGKDD international conference on knowledge discovery and data mining, pp 475-480
20. Cohen WW, Ravikumar P, Fienberg SE (2003) A comparison of string distance metrics for name-matching tasks. In: Proceedings of the IJCAI workshop on information integration on the web, pp 73-78
21. Ester M, Kriegel HP, Sander J, Xu X (1996) A density-based algorithm for discovering clusters in large spatial databases with noise. In: Proceedings of the international conference on knowledge discovery and data mining, pp 226-231
22. IP Fellegi AB Sunter 1969 A theory for record linkage J Am Stat Assoc 64 1183 1210 10.2307/2286061
23. Ganti V et al (1999) Clustering large datasets in arbitrary metric spaces. In: Proceedings of the international conference on data engineering, pp 502-511
24. Gionis A, Indyk P, Motwani R (1999) Similarity search in high dimensions via hashing. In: Proceedings of the international conference on very large databases, pp 518-529
25. Gravano L, Ipeirotis PG, Jagadish HV, Koudas N, Muthukrishnan S, Srivastava D (2001) Approximate string joins in a database (Almost) for free. In: Proceedings of the international conference on very large databases, pp 491-500
26. Gu L, Baxter RA, Vickers D, Rainsford C (2003) Record linkage: current practice and future directions. Technical Report, number 03/83. CSIRO Mathematical and Information Sciences
27. Guha S, Rastogi R, Shim K (1998) CURE: an efficient clustering algorithm for large databases. In: Proceedings of the ACM SIGMOD international conference on management of data, pp 73-84
28. S Guha R Rastogi K Shim 2001 ROCK: a robust clustering algorithm for categorical attributes Inf Syst 25 5 345 366 10.1016/S0306-4379(00)00022-3
29. Gunsfield D (1997) Algorithms on strings, trees and sequences. Cambridge University Press, Cambridge
30. Hernández MA, Stolfo SJ (1995) The Merge/Purge problem for large databases. In: Proceedings of the ACM SIGMOD international conference on management of data, pp 127-138
31. GR Hjatason H Samet 2003 Index-driven similarity search in metric spaces ACM Trans Database Syst 28 4 517 518 10.1145/958942.958948
32. Indyk P, Motwani R (1998) Approximate nearest neighbor-towards removing the curse of dimensionality. In: Proceedings of symposium on theory of computing, pp 604-613
33. PG Ipeirotis VS Verykios AK Elmagarmid 2007 Duplicate record detection: a review IEEE Trans Knowl Data Eng 18 1 1 16
34. AK Jain MN Murty PJ Flynn 1999 Data clustering: a review ACM Comput Surv 31 3 264 323 10.1145/331499.331504
35. Kalashnikov DV, Mehrotra S, Chen Z (2005) Exploiting relationships for domain independent data cleaning. In: Proceedings of the SIAM conference on data mining, pp 262-273
36. McCallum AK, Nigam K, Ungar L (2000) Efficient clustering of high-dimensional data sets with application to reference matching. In: Proceedings of the ACM SIGKDD international conference on knowledge discovery and data mining, pp 169-178
37. Monge AE, Elkan CP (1996) The field matching problem: algorithms and applications. In: Proceedings of the international conference on knowledge discovery and data mining, pp 267-270
38. Monge AE, Elkan CP (1997) An efficient domain-independent algorithm for detecting approximately duplicate database records. In: Proceedings of the SIGMOD workshop on research issues on data mining and knowledge discovery, pp 23-29
39. Monge AE, Elkan CP (2001) Automatic segmentation of text into structured records. In: Proceedings of the ACM SIGMOD conference on management of data
40. Neiling M, Jurk S (2003) The object identification framework. In: Proceedings of the KDD workshop on data cleaning, record linkage, and object consolidation, pp 37-39
41. Sarawagi S, Bhamidipaty A (2002) Interactive deduplication using active learning. In: Proceedings of the 8th ACM SIGKDD international conference on knowledge discovery and data mining, pp 269-278
42. Sarawagi S, Kirpal A (2004) Efficient exact set-similarity joins. In: Proceedings of the SIGMOD international conference on management of data, pp 743-754
43. Tejada S, Knoblock CA, Minton S (2002) Learning domain-independent string transformation weights for high accuracy object identification. In: Proceedings of the ACM SIGKDD international conference on knowledge discovery and data mining, pp 350-359
44. E Ukkonen 1982 Approximate string matching using q-grams and maximal matches Theor Comput Sci 92 1 191 211 10.1016/0304-3975(92)90143-4 1143139
45. Weber R, Schek HJ, Blott S (1998) A quantitative analsysis and performance study for similarity search in high-dimensional spaces. In: Proceedings of the international conference on very large databases, pp 194-205
46. Winkler WE (1990) String comparator metrics and enhanced decision rules in the Fellegi-Sunter model of record linkage. In: Proceedings of the section on survey research methods, American Statistical Association, pp 354-359
47. Winkler WE (1999) The state of record linkage and current research problems. Technical Report. Statistical Research Division, US Census Bureau