Optimizing multiple machine learning jobs on MapReduce

Proceedings - 2011 3rd IEEE International Conference on Cloud Computing Technology and Science, CloudCom 2011

Volumn , Issue , 2011, Pages 59-66

  Tamano, Hiroshi ^a   Nakadai, Shinji ^a   Araki, Takuya ^a  

^a NEC CORPORATION   (Japan)

Author keywords

Job Scheduling; Machine Learning; MapReduce

Indexed keywords

COST MODELS; EXECUTION TIME; JOB ASSIGNMENTS; JOB SCHEDULING; LOGISTIC REGRESSIONS; MACHINE-LEARNING; MAP-REDUCE; MESSAGE PASSING INTERFACE; MULTIPLE MACHINE; OPTIMAL ASSIGNMENT; RUNTIMES;

CLOUD COMPUTING; LEARNING SYSTEMS; LOGISTICS; OPTIMIZATION;

LEARNING ALGORITHMS;

EID: 84857148705     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CloudCom.2011.18     Document Type: Conference Paper

References (15)

1. J. Dean and S. Ghemawat, "Mapreduce: Simplified data processing on large clusters," in OSDI, 2004, pp. 137-150.
2. H. Li, Y. Wang, D. Zhang, M. Zhang, and E. Y. Chang, "Pfp: parallel fp-growth for query recommendation," in RecSys, P. Pu, D. G. Bridge, B. Mobasher, and F. Ricci, Eds. ACM, 2008, pp. 107-114.
3. B. Panda, J. Herbach, S. Basu, and R. J. Bayardo, "Planet: Massively parallel learning of tree ensembles with mapreduce," PVLDB, vol. 2, no. 2, pp. 1426-1437, 2009.
4. Y. Wang, H. Bai, M. Stanton, W.-Y. Chen, and E. Y. Chang, "Plda: Parallel latent dirichlet allocation for large-scale applications," in AAIM, ser. Lecture Notes in Computer Science, A. V. Goldberg and Y. Zhou, Eds., vol. 5564. Springer, 2009, pp. 301-314.
5. C.-T. Chu, S. K. Kim, Y.-A. Lin, Y. Yu, G. R. Bradski, A. Y. Ng, and K. Olukotun, "Map-reduce for machine learning on multicore," in NIPS, B. Schölkopf, J. C. Platt, and T. Hoffman, Eds. MIT Press, 2006, pp. 281-288.
6. Mahout, http://mahout.apache.org/, 2011.
7. Hadoop, http://hadoop.apache.org/, 2011.
8. J. Ekanayake, H. Li, B. Zhang, T. Gunarathne, S.-H. Bae, J. Qiu, and G. Fox, "Twister: a runtime for iterative mapreduce," in HPDC, S. Hariri and K. Keahey, Eds. ACM, 2010, pp. 810-818.
9. M. Zaharia, M. Chowdhury, M. Franklin, S. Shenker, and I. Stoica, "Spark: Cluster computing with working sets," in Proceedings of the 2nd USENIX conference on Hot topics in cloud computing. USENIX Association, 2010, pp. 10-10.
10. Y. Fukumoto and M. Onizuka, "Optimization for Multiple Analysis Jobs on MapReduce," in DEIM, 2011, in Japanese.
11. G. Schwarz, "Estimating the dimension of a model," The annals of statistics, pp. 461-464, 1978.
12. U. Kang, C. E. Tsourakakis, and C. Faloutsos, "Pegasus: A peta-scale graph mining system," in ICDM, W. W. 0010, H. Kargupta, S. Ranka, P. S. Yu, and X. Wu, Eds. IEEE Computer Society, 2009, pp. 229-238.
13. K. Shirahata, H. Sato, and S. Matsuoka, "Hybrid map task scheduling for gpu-based heterogeneous clusters," in 2nd IEEE International Conference on Cloud Computing Technology and Science. IEEE, 2010, pp. 733-740.
14. A. Aboulnaga, Z. Wang, and Z. Zhang, "Packing the most onto your cloud," in Proceeding of the first international workshop on Cloud data management. ACM, 2009, pp. 25-28.
15. K. Kc and K. Anyanwu, "Scheduling hadoop jobs to meet deadlines," in 2nd IEEE International Conference on Cloud Computing Technology and Science. IEEE, 2010, pp. 388-392.