On choosing thresholds for duplicate detection

Proceedings of the 18th International Conference on Information Quality, ICIQ 2013

Volumn , Issue , 2013, Pages

  Draisbach, Uwe ^a   Naumann, Felix ^a  

^a UNIVERSITY OF POTSDAM   (Germany)

Author keywords

Data quality; Duplicate detection; Entity resolution

Indexed keywords

INFORMATION ANALYSIS;

DATA QUALITY; DUPLICATE DETECTION; ENTITY RESOLUTIONS; MULTIPLE INPUTS; REAL-WORLD ENTITIES; SIMILARITY MEASURE; TRIAL AND ERROR;

STATISTICAL TESTS;

EID: 85084162292     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (19)

1. M. Bilenko, B. Kamath, and R. J. Mooney. Adaptive blocking: Learning to scale up record linkage. In Proceedings of the International Conference on Data Mining (ICDM), pages 87–96, 2006.
2. M. Bilenko and R. J. Mooney. Adaptive duplicate detection using learnable string similarity measures. In Proceedings of the ACM SIGKDD International Conference of Knowledge Discovery and Data Mining, pages 39–48, 2003.
3. P. Christen. Probabilistic data generation for deduplication and data linkage. In International Conference on Intelligent Data Engineering and Automated Learning (IDEAL), pages 109–116. Springer LNCS, 2005.
4. P. Christen. A survey of indexing techniques for scalable record linkage and deduplication. IEEE Transactions on Knowledge and Data Engineering (TKDE), 2011.
5. X. Dong, A. Halevy, and J. Madhavan. Reference reconciliation in complex information spaces. In Proceedings of the ACM International Conference on Management of Data (SIGMOD). ACM, 2005.
6. U. Draisbach and F. Naumann. A comparison and generalization of blocking and windowing algorithms for duplicate detection. In Proceedings of the International Workshop on Quality in Databases (QDB), Lyon, France, 2009.
7. U. Draisbach and F. Naumann. DuDe: The duplicate detection toolkit. In Proceedings of the International Workshop on Quality in Databases (QDB), 2010.
8. A. K. Elmagarmid, P. G. Ipeirotis, and V. S. Verykios. Duplicate record detection: A survey. IEEE Transactions on Knowledge and Data Engineering (TKDE), 19(1):1–16, 2007.
9. O. Hassanzadeh, F. Chiang, R. J. Miller, and H. C. Lee. Framework for evaluating clustering algorithms in duplicate detection. Proceedings of the VLDB Endowment, 2(1):1282–1293, 2009.
10. M. A. Hernández and S. J. Stolfo. Real-world data is dirty: Data cleansing and the merge/purge problem. Data Mining and Knowledge Discovery, 2(1):9–37, 1998.
11. M. Herschel, F. Naumann, S. Szott, and M. Taubert. Scalable iterative graph duplicate detection. IEEE Transactions on Knowledge and Data Engineering (TKDE), 24(11), 2012.
12. B. Kenig and A. Gal. MFIBlocks: An effective blocking algorithm for entity resolution. Information Systems, 38(6):908–926, 2013.
13. H. Köpcke, A. Thor, and E. Rahm. Evaluation of entity resolution approaches on real-world match problems. Proceedings of the VLDB Endowment, 3(1):484–493, 2010.
14. V. I. Levenshtein. Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady, 10(8):707–710, 1966.
15. M. Michelson and C. A. Knoblock. Learning blocking schemes for record linkage. In Proceedings of the National Conference on Artificial Intelligence (AAAI), pages 440–445, 2006.
16. F. Naumann and M. Herschel. An Introduction to Duplicate Detection (Synthesis Lectures on Data Management). Morgan and Claypool Publishers, 2010.
17. T. Vogel and F. Naumann. Instance-based “one-to-some” assignment of similarity measures to attributes. In Proceedings of the International Conference on Cooperative Information Systems (CoopIS), pages 412–420, 2011.
18. T. Vogel and F. Naumann. Automatic blocking key selection for duplicate detection based on unigram combinations. In Proceedings of the International Workshop on Quality in Databases (QDB), 2012.
19. W. E. Winkler and Y. Thibaudeau. An application of the Fellegi-Sunter model of record linkage to the 1990 U.S. decennial census. In U.S. Decennial Census. Technical report, US Bureau of the Census, pages 11–13, 1991.