SparkNet: Training deep networks in spark

4th International Conference on Learning Representations, ICLR 2016 - Conference Track Proceedings

Volumn , Issue , 2016, Pages

  Moritz, Philipp ^a   Nishihara, Robert ^a   Stoica, Ion ^a   Jordan, Michael I ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BATCH DATA PROCESSING; BENCHMARKING; DEEP LEARNING; GRADIENT METHODS; OBJECT RECOGNITION; STOCHASTIC SYSTEMS;

CLUSTER SIZES; COMMUNICATION FREQUENCY; COMMUNICATION OVERHEADS; COMPUTATIONAL FRAMEWORK; LEARNING FRAMEWORKS; PARALLELIZATIONS; PARAMETER-TUNING; STOCHASTIC GRADIENT DESCENT;

DISTRIBUTED COMPUTER SYSTEMS;

EID: 85083952081     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (25)

1. Abadi, Martín, Agarwal, Ashish, Barham, Paul, et al. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL http://tensorflow.org/. Software available from tensorflow.org.
2. Abuzaid, Firas, Hadjis, Stefan, Zhang, Ce, and Ré, Christopher. Caffe con Troll: Shallow ideas to speed up deep learning. arXiv preprint arXiv:1504.04343, 2015.
3. Armbrust, Michael, Xin, Reynold S, Lian, Cheng, Huai, Yin, Liu, Davies, Bradley, Joseph K, Meng, Xiangrui, Kaftan, Tomer, Franklin, Michael J, Ghodsi, Ali, et al. Spark SQL: Relational data processing in Spark. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, pp. 1383–1394. ACM, 2015.
4. Chilimbi, Trishul, Suzue, Yutaka, Apacible, Johnson, and Kalyanaraman, Karthik. Project Adam: Building an efficient and scalable deep learning training system. In 11th USENIX Symposium on Operating Systems Design and Implementation, pp. 571–582, 2014.
5. Coates, Adam, Huval, Brody, Wang, Tao, Wu, David, Catanzaro, Bryan, and Andrew, Ng. Deep learning with cots hpc systems. In Proceedings of the 30th International Conference on Machine Learning, pp. 1337–1345, 2013.
6. Dean, Jeffrey and Ghemawat, Sanjay. MapReduce: simplified data processing on large clusters. Communications of the ACM, 51(1):107–113, 2008.
7. Dean, Jeffrey, Corrado, Greg, Monga, Rajat, Chen, Kai, Devin, Matthieu, Mao, Mark, Ranzato, Marc’Aurelio, Senior, Andrew, Tucker, Paul, Yang, Ke, Le, Quoc V., and Ng, Andrew Y. Large scale distributed deep networks. In Advances in Neural Information Processing Systems, pp. 1223–1231, 2012.
8. Gonzalez, Joseph E, Xin, Reynold S, Dave, Ankur, Crankshaw, Daniel, Franklin, Michael J, and Stoica, Ion. Graphx: Graph processing in a distributed dataflow framework. In Proceedings of OSDI, pp. 599–613, 2014.
9. Ho, Qirong, Cipar, James, Cui, Henggang, Lee, Seunghak, Kim, Jin Kyu, Gibbons, Phillip B, Gibson, Garth A, Ganger, Greg, and Xing, Eric P. More effective distributed ML via a stale synchronous parallel parameter server. In Advances in Neural Information Processing Systems, pp. 1223–1231, 2013.
10. Iandola, Forrest N, Ashraf, Khalid, Moskewicz, Mattthew W, and Keutzer, Kurt. FireCaffe: near-linear acceleration of deep neural network training on compute clusters. arXiv preprint arXiv:1511.00175, 2015.
11. Isard, Michael, Budiu, Mihai, Yu, Yuan, Birrell, Andrew, and Fetterly, Dennis. Dryad: Distributed data-parallel programs from sequential building blocks. In Proceedings of the 2nd ACM SIGOP-S/EuroSys European Conference on Computer Systems, pp. 59–72, 2007.
12. Jia, Yangqing, Shelhamer, Evan, Donahue, Jeff, Karayev, Sergey, Long, Jonathan, Girshick, Ross, Guadarrama, Sergio, and Darrell, Trevor. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the ACM International Conference on Multimedia, pp. 675–678. ACM, 2014.
13. Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with deep convo-lutional neural networks. In Advances in Neural Information Processing Systems, pp. 1097–1105, 2012.
14. Li, Mu, Andersen, David G, Park, Jun Woo, Smola, Alexander J, Ahmed, Amr, Josifovski, Vanja, Long, James, Shekita, Eugene J, and Su, Bor-Yiing. Scaling distributed machine learning with the parameter server. In 11th USENIX Symposium on Operating Systems Design and Implementation, pp. 583–598, 2014.
15. Meng, Xiangrui, Bradley, Joseph, Yavuz, Burak, Sparks, Evan, Venkataraman, Shivaram, Liu, Davies, Freeman, Jeremy, Tsai, DB, Amde, Manish, Owen, Sean, et al. MLlib: Machine learning in Apache Spark. arXiv preprint arXiv:1505.06807, 2015.
16. Murray, Derek G, McSherry, Frank, Isaacs, Rebecca, Isard, Michael, Barham, Paul, and Abadi, Martín. Naiad: a timely dataflow system. In Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles, pp. 439–455. ACM, 2013.
17. Noel, Cyprien, Shi, Jun, and Feng, Andy. Large scale distributed deep learning on Hadoop clusters, 2015. URL http://yahoohadoop.tumblr.com/post/129872361846/large-scale-distributed-deep-learning-on-hadoop.
18. Russakovsky, Olga, Deng, Jia, Su, Hao, Krause, Jonathan, Satheesh, Sanjeev, Ma, Sean, Huang, Zhiheng, Karpathy, Andrej, Khosla, Aditya, Bernstein, Michael, Berg, Alexander C., and Fei-Fei, Li. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, pp. 1–42, 2015.
19. Sparks, Evan R., Venkataraman, Shivaram, Kaftan, Tomer, Franklin, Michael, and Recht, Benjamin. KeystoneML: End-to-end machine learning pipelines at scale. 2015.
20. Szegedy, Christian, Liu, Wei, Jia, Yangqing, Sermanet, Pierre, Reed, Scott, Anguelov, Dragomir, Erhan, Dumitru, Vanhoucke, Vincent, and Rabinovich, Andrew. Going deeper with convolutions. In Computer Vision and Pattern Recognition, 2015.
21. Zaharia, Matei, Chowdhury, Mosharaf, Franklin, Michael J, Shenker, Scott, and Stoica, Ion. Spark: cluster computing with working sets. In Proceedings of the 2nd USENIX conference on Hot topics in cloud computing, volume 10, pp. 10, 2010.
22. Zaharia, Matei, Das, Tathagata, Li, Haoyuan, Hunter, Timothy, Shenker, Scott, and Stoica, Ion. Discretized streams: Fault-tolerant streaming computation at scale. In Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles, pp. 423–438. ACM, 2013.
23. Zhang, Sixin, Choromanska, Anna E, and LeCun, Yann. Deep learning with elastic averaging SGD. In Advances in Neural Information Processing Systems, pp. 685–693, 2015.
24. Zhang, Yuchen and Jordan, Michael I. Splash: User-friendly programming interface for parallelizing stochastic algorithms. arXiv preprint arXiv:1506.07552, 2015.
25. Zinkevich, Martin, Weimer, Markus, Li, Lihong, and Smola, Alex J. Parallelized stochastic gradient descent. In Advances in Neural Information Processing Systems, pp. 2595–2603, 2010.