A load-balanced mapreduce algorithm for blocking-based entity-resolution with multiple keys

Conferences in Research and Practice in Information Technology Series

Volumn 152, Issue , 2014, Pages 3-9

  Hsueh, Sue Chen ^a   Lin, Ming Yen ^b   Chiu, Yi Chun ^b  

^a CHAOYANG UNIVERSITY OF TECHNOLOGY   (Taiwan)

^b FENG CHIA UNIVERSITY   (Taiwan)

Author keywords

Cloud computing; Entity resolution; Load balance; MapReduce

Indexed keywords

CLOUD COMPUTING; DIGITAL LIBRARIES;

DATA COLLECTION; DATABASE MANAGEMENT; ENTITY RESOLUTIONS; LOAD-BALANCE; MAP-REDUCE; MAPREDUCE FRAMEWORKS; MULTIPLE KEYS; SIMILARITY COMPUTATION;

ALGORITHMS;

EID: 84907001000     PISSN: 14451336     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (22)

1. Baraglia, R., De Francisci Morales, G., and Lucchese, C. (2010): Document similarity self-join with MapReduce. Proc. IEEE International Conference on Data Mining:731-736.
2. Brizan, D. G. and Tansel A. U. (2006): A survey of entity resolution and record linkage methodologies. Communications of the International Information Management Association 6(3): 41-50.
3. Chaudhuri, S., Ganti, V., and Kaushik, R. (2006): A primitive operator for similarity joins in data cleaning. Proc. International Conference on Data Engineering:1-5.
4. Dean, J. and Ghemawat, S. (2004): MapReduce:simplified data processing on large clusters. Proc. Symposium on Operating System Design and Implementation: 137-150.
5. Elsayed, T., Lin, J., and Oard, D. W. (2008): Pairwise document similarity in large collections with MapReduce. Proc. Association for Computational Linguistics on Human Language Technologies:265-268.
6. Hernández, M. A. and Stolfo, S. J. (1995): The merge/purge problem for large databases. Proc. ACM SIGMOD International Conference on Management of data, 127-138.
7. Kiefer, T., Volk, P. B., and Lehner, W. (2010): Pairwise element computation with MapReduce. Proc. High Performance Distributed Computing: 826-833.
8. Kim, Y. and Shim, K. (2012): Parallel top-k similarity join algorithms using MapReduce. Proc. International Conference on Data Engineering: 510-521.
9. Kolb, L. and Rahm, E. (2013): Parallel Entity Resolution with Dedoop. Datenbank-Spektrum 13(1): 1-10.
10. Kolb, L., Thor, A., and Rahm, E. (2013): Don't match twice: redundancy-free similarity computation with MapReduce. Proc. Workshop on Data Analytics in the Cloud: 1-5.
11. Kolb, L., Thor, A., and Rahm, E. (2012a): Dedoop:efficient deduplication with Hadoop. Proc. VLDB Endowment 5(12):1878-1881.
12. Kolb, L., Thor, A., and Rahm, E. (2012b): Load balancing for MapReduce-based entity resolution. Proc. International Conference on Data Engineering:618-629.
13. Kolb, L., Thor, A., and Rahm, E. (2011a): Multi-pass sorted neighborhood blocking with MapReduce. Computer Science - Research and Development 27(1):45-63.
14. Kolb, L., Thor, A., and Rahm, E. (2011b): Parallel sorted neighborhood blocking with MapReduce. Proc. Database Systems for Business, Technology, and Web:45-64.
15. Lin, J. (2009): Brute force and indexed approaches to pairwise document similarity comparisons with MapReduce. Proc. ACM SIGIR Conference on Research and Development in Information Retrieval:155-162.
16. Lu, W., Shen, Y., Chen, S. and Ooi, B. C. (2012): Efficient processing of k nearest neighbor joins using MapReduce. Proceedings VLDB Endowment 5(10): 1016-1027.
17. Metwally, A. and Faloutsos, C. (2012): V-SMART-Join: a scalable MapReduce framework for all-pair similarity joins of multisets and vectors. Proc. VLDB Endowment 5(8): 704-715.
18. Pasula, H., Marthi, B., Milch, B., Russell, S. and Shpitser, I. (2002): Identity uncertainty and citation matching. Proc. Neural Information Processing Systems:1401-1408.
19. Vernica, R., Carey, M. J., and Li, C. (2010): Efficient parallel set-similarity joins using MapReduce. Proc. ACM International Conference on Management of Data:495-506.
20. Zhang, Q., Zhang, Y., Yu, H., and Huang, X. (2010):Efficient partial-duplicate detection based on sequence matching. Proc. ACM SIGIR Conference on Research and Development in Information Retrieval: 675-682.
21. CiteSeerX dataset URL. http://asterix.ics.uci.edu/data/csx.raw.txt.gz (Accessed:07/01/2012).
22. Hadoop URL. http://hadoop.apache.org/ (Accessed: 08/24/2012).