An overlay-based data mining architecture tolerant to physical network disruptions

IEEE Transactions on Emerging Topics in Computing

Volumn 2, Issue 3, 2014, Pages 292-301

  Suto, Katsuya ^a   Nishiyama, Hiroki ^a   Kato, Nei ^a   Mizutani, Kimihiro ^b   Akashi, Osamu ^b   Takahara, Atsushi ^b  

^a TOHOKU UNIVERSITY   (Japan)

^b NTT CORPORATION   (Japan)

Author keywords

Big data mining; neighbor selection; overlay network; physical network disruption; service availability; task allocation

Indexed keywords

BIG DATA; DATA HANDLING; INFORMATION MANAGEMENT; NETWORK ARCHITECTURE; OVERLAY NETWORKS; PARALLEL ARCHITECTURES;

DISTRIBUTED DATA MANAGEMENTS; DISTRIBUTED TASK ALLOCATION; NEIGHBOR SELECTION; NETWORK CONSTRUCTION; NETWORK TECHNOLOGIES; PHYSICAL NETWORK; SERVICE AVAILABILITY; TASK ALLOCATION;

DATA MINING;

EID: 84908574117     PISSN: None     EISSN: 21686750     Source Type: Journal    
DOI: 10.1109/TETC.2014.2330517     Document Type: Article

References (26)

1. N. Elmqvist and P. Irani, "Ubiquitous analytics: Interacting with big data anywhere, anytime, " IEEE Computer Magazine, vol. 46, no. 4, pp. 86-89, Apr. 2013.
2. F. Gebara, H. Hofstee, J. Hayes, and A. Hylick, "Big data text-oriented benchmark creation for hadoop, " IBM Journal of Research and Development, vol. 57, no. 3/4, pp. 10:1-10:6, May-Jul. 2013.
3. J. Dean and S. Ghemawat, "MapReduce: Simplified data processing on large clusters, " in Proc. of 6th Symposium on Operating Systems Design and Implementation, San Francisco, USA, Dec. 2004, pp. 137-150.
4. "Hadoop [Online]." Available: http://hadoop.apache.org/.
5. K. McKusick and S. Quinlan, "GFS: Evolution on fast-forward, " ACM Queue Magazine, vol. 7, no. 7, pp. 1-11, Aug. 2009.
6. K. Suto, H. Nishiyama, X. S. Shen, and N. Kato, "Designing P2P networks tolerant to attacks and faults based on bimodal degree distribution, " Journal of Communications, vol. 7, no. 8, pp. 587-595, Aug. 2012.
7. S. Ghemawat, H. Gobioff, and S.-T. Leung, "The google file system, " in Proc. of 19th Symposium on Operating Systems Principles, New York, USA, Oct. 2003, pp. 29-43.
8. A. Verma, L. Cherkasova, and R.H. Campbell, "Orchestrating an ensemble of MapReduce jobs for minimizing their makespan, "IEEE Transactions on Dependable and Secure Computing, vol. 10, no. 5, pp. 314-327, Sept.-Oct. 2013.
9. M. Cardosa, A. Singh, H. Pucha, and A. Chandra, "Exploiting spatio-temporal tradeoffs for energy-aware MapReduce in the cloud, " IEEE Transactions on Computers, vol. 61, no. 12, pp. 1737-1751, Dec. 2012.
10. Y. Zhang, Q. Gao, L. Gao, and C. Wang, "Priter: A distributed framework for prioritizing iterative computations, " IEEE Transactions on Parallel and Distributed Systems, vol. 24, no. 9, pp. 1884-1893, Sept. 2013.
11. D. Warneke and O. Kao, "Exploiting dynamic resource allocation for efficient parallel data processing in the cloud, " IEEE Transactions on Parallel and Distributed Systems, vol. 22, no. 6, pp. 985-997, Jun. 2011.
12. M. Asahara, S. Nakadai, and T. Araki, "LoadAtomizer: A locality and I/O load aware task scheduler for MapReduce, " in Proc. of IEEE 4th International Conference on Cloud Computing Technology and Science, Taipei, Dec. 2012, pp. 317-324.
13. A. Rabkin and R. H. Katz, "How hadoop clusters break, " IEEE Software Magazine, vol. 30, no. 4, pp. 88-94, Jul.-Aug. 2013.
14. H. Jin, X. Yang, X.-H. Sun, and I. Raicu, "Large-scale distributed systems at google: Current systems and future directions, " in Keynote Speech at the 3rd ACM SIGOPS International Workshop Large Scale Distributed Systems and Middleware, Montana, USA, OCT. 2009.
15. "HDFS Fereration [Online]." Available: http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html.
16. Q. Zheng, "Improving MapReduce fault tolerance in the cloud, " in Proc. of IEEE International Symposium on Parallel & Distributed Processing, Workshop and Phd Forum, Atlanta, USA, Apr. 2010, pp. 1-6.
17. J. Dean, "Adapt: Availability-aware MapReduce data placement for non-dedicated distributed computing, " in Proc. of 32nd IEEE International Conference on Distributed Computing Systems, Macau, Jun. 2012, pp. 516-525.
18. H.-C. Hsiao, H.-Y. Chung, H. Shen, and Y.-C. Chao, "Load rebalancing for distributed file systems in clouds, " IEEE Transactions on Parallel and Distributed Systems, vol. 24, no. 5, pp. 951-962, May 2013.
19. F. Azzedin, "Towards a scalable HDFS architecture, " in Proc. of International Conference on Collaboration Technologies and Systems, San Diego, USA, May 2013, pp. 155-161.
20. J. Zhang, G. Wu, X. Hu, and X. Wu, "A distributed cache for hadoop distributed file system in real-time cloud services, " in Proc. of ACM/IEEE 13th International Conference on Grid Computing, Beijing, China, Sept. 2012, pp. 12-21.
21. Z. Yao, X. Wang, D. Leonard, and D.Loguinov, "Node isolation model and age-based neighbor selection in unstructured P2P networks, " IEEE/ACM Transactions on Networking, vol. 17, no. 1, pp. 144-157, Feb. 2009.
22. W. Xiao, M. He, and H. Liang, "Cayleyccc: A robust P2P overlay network with simple routing and small-world, " Academy Publisher Journal of Networks, vol. 6, no. 9, pp. 1247-1253, Sept. 2011.
23. P. Flocchini, A. Nayak, S. Member, and M. Xie, "Enhancing peer-to-peer systems through redundancy, " IEEE Journal on Selected Areas in Communications, vol. 25, no. 1, pp. 15-24, Jan. 2007.
24. K. Suto, H. Nishiyama, N. Kato, T. Nakachi, T. Fujii, and A. Takahara, "THUP: A P2P network robust to churn and dos attack based on bimodal degree distribution, " IEEE Journal on Selected Areas in Communications, vol. 31, no. 9, pp. 247-256, Sept. 2013.
25. T. Tanizawa, G. Paul, R. Cohen, S. Havlin, and H. E. Stanley, "Optimization of network robustness to waves of targeted and random attacks, " Physical Review E, vol. 71, no. 4, Apr. 2005.
26. M. E. J. Newman, "Assortative mixing in networks, " Physical Review Letters, vol. 89, no. 208701, May 2002.