An automatic blocking mechanism for large-scale de-duplication tasks

ACM International Conference Proceeding Series

Volumn , Issue , 2012, Pages 1055-1064

  Das Sarma, Anish ^a   Jain, Ankur ^b   Machanavajjhala, Ashwin ^c   Bohannon, Philip ^d  

^a GOOGLE INC   (United States)

^b YAHOO RESEARCH   (United States)

^c DUKE UNIVERSITY   (United States)

^d FACEBOOK   (United States)

Author keywords

blocking; canopy formation; de duplication

Indexed keywords

ARCHITECTURAL CONSTRAINTS; AUTOMATICALLY GENERATED; BLOCKING; BLOCKING MECHANISMS; BLOCKING PROCESS; CANOPY FORMATION; CENSUS DATA; DATA INTEGRATION; DATA SETS; DE-DUPLICATION; DESIGN PROBLEMS; DISJOINTNESS; GRID ENVIRONMENTS; HIERARCHICAL TREE; LARGE DATASETS; MAP-REDUCE; MEDICAL RECORD; MEMORY REQUIREMENTS; NEW DIMENSIONS; OBJECTIVE FUNCTIONS; PAIR-WISE COMPARISON; POST PROCESSING; PROBLEM SIZE; REAL-WORLD ENTITIES; STRUCTURED DATA;

COMMERCE; KNOWLEDGE MANAGEMENT; POPULATION STATISTICS; SEARCH ENGINES; TREES (MATHEMATICS);

HASH FUNCTIONS;

EID: 84871075183     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2396761.2398403     Document Type: Conference Paper

References (29)

1. DBPedia. http://dbpedia.org/.
2. IMDb: The Internet Movie Database. http://www.imdb.com.
3. MAHOUT. http://cwiki.apache.org/MAHOUT/canopy-clustering.html.
4. R. Baxter, P. Christen, and T. Churches. A comparison of fast blocking methods for record linkage. In KDD, 2003.
5. O. Benjelloun, H. Garcia-Molina, D. Menestrina, Q. Su, S. E. Whang, and J. Widom. Swoosh: a generic approach to entity resolution. VLDB Journal, 18(1), 2009.
6. O. Benjelloun, H. Molina, H. Gong, H. Kawai, T. E. Larson, D. Menestrina, and S. Thavisomboon. D-Swoosh: A Family of Algorithms for Generic, Distributed Entity Resolution. Technical Report, Stanford University, 2007.
7. M. Bilenko, B. Kamath, and R. J. Mooney. Adaptive blocking: Learning to scale up record linkage and clustering. In ICDM, 2006.
8. P. Christen. A survey of indexing techniques for scalable record linkage and deduplication. TKDE, 2011.
9. P. Christen and T. C. Febrl. Freely extensible biomedical record linkage manual. http://datamining.anu.edu.au/linkage.html, 2, 2003.
10. G. B. Dantzig. Discrete-variable extremum problems. Operations Research, 5(2), 1957.
11. A. K. Elmagarmid, P. G. Ipeirotis, and V. S. Verykios. Duplicate record detection: A survey. IEEE TKDE, 2007.
12. I. P. Fellegi and A. B. Sunter. A theory for record linkage. Journal of the American Statistical Society, 64(2283), 1969.
13. M. R. Garey and D. S. Johnson. Computers and Intractability. W. H. Freeman and Company, 1979.
14. M. A. Hernandez and S. J. Stolfo. The merge/purge problem for large databases. In SIGMOD, 1995.
15. Y. E. Ioannidis. The history of histograms (abridged). In VLDB, 2003.
16. M. A. Jaro. Advances in record linkage methodology as applied to matching the 1985 census of tampa. JASA, 84, 1989.
17. L. Jin, C. Li, and S. Mehrotra. Efficient record linkage in large data sets. In DASFAA, 2003.
18. R. P. Kelley. Advances in record linkage methodology: a method for determining the best blocking strategy. Record Linkage Techniques, 1985.
19. N. Koudas, S. Sarawagi, and D. Srivastava. Record linkage: Similarity measures and algorithms. Tutorial at SIGMOD, 2006.
20. A. McCallum, K. Nigam, and L. H. Ungar. Efficient clustering of high-dimensional data sets with application to reference matching. In KDD, 2000.
21. M. Michelson and C. A. Knoblock. Learning blocking schemes for record linkage. In AAAI, 2006.
22. A. Okcan and M. Riedewald. Processing theta-joins using mapreduce. In SIGMOD, 2011.
23. S. Rendle and L. Schmidt-Thieme. Scaling record linkage to non-uniform distributed class sizes. In PAKDD, 2008.
24. S. Sarawagi and A. Bhamidipaty. Interactive deduplication using active learning. In SIGKDD, 2002.
25. H. sik Kim and D. Lee. Parallel linkage. In CIKM, 2007.
26. R. Vernica, M. J. Carey, and C. Li. Efficient parallel set-similarity joins using mapreduce. In SIGMOD, 2010.
27. S. E. F. W. Cohen, P. Ravikumar. A comparison of string distance metrics for name-matching tasks. In Proc. of IJCAI, 2003.
28. S. E. Whang, D. Menestrina, G. Koutrika, M. Theobald, and H. Garcia-Molina. Entity resolution with iterative blocking. In SIGMOD, 2009.
29. W. Winkler. Overview of record linkage and current research directions. Statistical Research Division, U.S. Bureau of the Census, Tech. Report, 2006.