MRO-MPI: MapReduce overlapping using MPI and an optimized data exchange policy

Parallel Computing

Volumn 39, Issue 12, 2013, Pages 851-866

  Mohamed, Hisham ^a   Marchand Maillet, Stéphane ^a  

^a UNIVERSITY OF GENEVA   (Switzerland)

Author keywords

Big data; Large scale data processing; MapReduce overlapping; MPI MapReduce; Parallel MapReduce

Indexed keywords

BIG DATUM; EFFICIENT COMMUNICATIONS; LARGE-SCALE DATA PROCESSING; MAP-REDUCE; MAP-REDUCE PROGRAMMING; MESSAGE PASSING INTERFACE; MPI-MAPREDUCE; PROGRAMMING MODELS;

DISTRIBUTED COMPUTER SYSTEMS; ELECTRONIC DATA INTERCHANGE;

DATA PROCESSING;

EID: 84890130572     PISSN: 01678191     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.parco.2013.08.010     Document Type: Article

References (34)

1. J. Dean, and S. Ghemawat Mapreduce: simplified data processing on large clusters Proceedings of the 6th Conference on Symposium on Opearting Systems Design & Implementation vol. 6 2004 USENIX Association Berkeley, CA, USA 10
2. R. McCreadie, C. Macdonald, and I. Ounis Mapreduce indexing strategies: studying scalability and efficiency Information Processing and Management 2011
3. D. Gillick, A. Faria, J. Denero, Mapreduce: distributed computing for machine learning, 2006.
4. T. White first ed. Hadoop: The Definitive Guide 2009 O'Reilly
5. Mpich2, .
6. E. Gabriel, G.E. Fagg, G. Bosilca, T. Angskun, J.J. Dongarra, J.M. Squyres, V. Sahay, P. Kambadur, B. Barrett, A. Lumsdaine, R.H. Castain, D.J. Daniel, R.L. Graham, T.S. Woodall, Open MPI: goals, concept, and design of a next generation MPI implementation, in: Proceedings of 11th European PVM/MPI Users' Group Meeting, pp. 97-104.
7. Message passing interface, .
8. R. Rajasekaran, and J. Reif Handbook of Parallel Computing: Models Algorithms and Applications 2007 CRC Press
9. V. Kumar second ed. Introduction to Parallel Computing 2002 Addison-Wesley Longman Publishing Co., Inc. Boston, MA, USA
10. S.J. Plimpton, and K.D. Devine Mapreduce in mpi for large-scale graph algorithms Parallel Computing 37 9 2011 610 632
11. T. Hoefler, A. Lumsdaine, and J. Dongarra Towards efficient mapreduce using mpi M. Ropo, J. Westerholm, J. Dongarra, PVM/MPI, ser. Lecture Notes in Computer Science vol. 5759 2009 Springer 240 249
12. F. Ahmad, S. Lee, M. Thottethodi, T.N. Vijaykumar, Mapreduce with communication overlap, Technical Report, 2007.
13. T. Condie, N. Conway, P. Alvaro, J.M. Hellerstein, K. Elmeleegy, and R. Sears Mapreduce online Proceedings of the 7th USENIX Conference on Networked Systems Design and Implementation, ser. NSDI'10 2010 USENIX Association Berkeley, CA, USA 21
14. J. Ekanayake, H. Li, B. Zhang, T. Gunarathne, S.-H. Bae, J. Qiu, G. Fox, Twister: a runtime for iterative mapreduce, in: Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, ser. HPDC '10. ACM, pp. 810-818.
15. M. Elteir, H. Lin, W. chun Feng, Enhancing mapreduce via asynchronous data processing, in: IEEE 16th International Conference on Parallel and Distributed Systems (ICPADS), December 2010, pp. 397 -405.
16. X. Lu, B. Wang, L. Zha, Z. Xu, Can mpi benefit hadoop and mapreduce applications? in: 40th International Conference on Parallel Processing Workshops (ICPPW), 2011, pp. 371-379.
17. I.H. Witten, A. Moffat, and T.C. Bell second ed. Managing Gigabytes: Compressing and Indexing Documents and Images 1999 Morgan Kaufmann San Francisco, CA
18. G. Csurka, C.R. Dance, L. Fan, J. Willamowski, C. Bray, Visual categorization with bags of keypoints, in: Workshop on Statistical Learning in Computer Vision, ECCV, pp. 1-22.
19. E. Bruno, and S. Marchand-Maillet Multimodal preference aggregation for multimedia information retrieval Journal of Multimedia 4 5 2009 321 329
20. M. von Wyl, H. Mohamed, E. Bruno, S. Marchand-Maillet, A parallel cross-modal search engine over large-scale multimedia collections with interactive relevance feedback, in: Proceedings of the 1st ACM International Conference on Multimedia Retrieval, pp. 73:1-73:2.
21. H.V. Jagadish, A.O. Mendelzon, and T. Milo Similarity-based queries Proceedings of the fourteenth ACM SIGACT-SIGMOD-SIGART symposium on Principles of database systems, ser. PODS '95 1995 ACM New York, NY, USA 36 45
22. P. Zezula, G. Amato, V. Dohnal, and M. Batko Similarity Search: The Metric Space Approach Advances in Database Systems vol. 32 2006 Springer
23. P. Indyk, and R. Motwani Approximate nearest neighbors: towards removing the curse of dimensionality Proceedings of the thirtieth annual ACM symposium on Theory of computing, ser. STOC '98 1998 ACM New York, NY, USA 604 613
24. M. Patella, and P. Ciaccia Approximate similarity search: a multi-faceted problem Journal of Discrete Algorithms 7 1 2009 36 48
25. E. Gonzalez, K. Figueroa, and G. Navarro Effective proximity retrieval by ordering permutations IEEE Transactions on Pattern Analysis and Machine Intelligence 30 9 2008 1647 1658
26. G. Amato and P. Savino, Approximate similarity search in metric spaces using inverted files, in: Proceedings of the 3rd international conference on Scalable information systems, ser. InfoScale '08. ICST, Brussels, Belgium, Belgium: ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering), 2008, pp. 28:1-28:10.
27. H. Mohamed and S. Marchand-Maillet, Metric suffix array for large-scale similarity search, in: ACM WSDM 2013 Workshop on Large Scale and Distributed Systems for Information Retrieval, Rome, IT, February 2013. [Online]. Available: .
28. Permutation-based pruning for approximate k-nn search, in: Database and Expert Systems Applications, 24th International Conference, DEXA 2013, Prague, Czech Republic, August 26-29, 2013, Proceedings, Part I, ser. Lecture Notes in Computer Science, vol. 8055. Springer, 2013.
29. Quantized ranking for permutation-based indexing, in: Similarity Search and Applications, 6th International Conference, SISAP 2013, Corua, Spain, October 2-4, 2013, ser. Lecture Notes in Computer Science, P. Zezula and N.R. Brisaboa, Eds. Springer, 2013.
30. W. Gropp, E. Lusk, and A. Skjellum Using MPI: Portable Parallel Programming with the Message-Passing Interface 1994 MIT Press Cambridge, MA
31. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei, ImageNet: a large-scale hierarchical image database, in: IEEE Computer Vision and Pattern Recognition (CVPR), 2009.
32. Project gutenberg, .
33. C. Stanfill Partitioned posting files: a parallel inverted file structure for information retrieval Proceedings of the 13th annual international ACM SIGIR conference on Research and development in information retrieval, ser. SIGIR '90 1990 ACM New York, NY, USA 413 428
34. H. Mohamed, and S. Marchand-Maillet Parallel approaches to permutation-based indexing using inverted files Similarity Search and Applications G. Navarro, V. Pestov, Lecture Notes in Computer Science vol. 7404 2012 Springer Berlin Heidelberg 148 161