ESGD: Commutation efficient distributed deep learning on the edge

USENIX Workshop on Hot Topics in Edge Computing, HotEdge 2018, co-located with USENIX ATC 2018

Volumn , Issue , 2018, Pages

  Tao, Zeyi ^a   Li, Qun ^a  

^a Dept of Economics and Thomas Jefferson Program in Public Policy at the College of William and Mary   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EDGE COMPUTING; GRADIENT METHODS; INTERNET OF THINGS; LEARNING ALGORITHMS; NEURAL NETWORKS; STOCHASTIC SYSTEMS;

COMMUNICATION COST; CONVOLUTIONAL NEURAL NETWORK; GRADIENT-BASED OPTIMIZATION; LEARNING FRAMEWORKS; NETWORK COMMUNICATIONS; OBJECTIVE FUNCTIONS; PERFORMANCE GUARANTEES; STOCHASTIC GRADIENT DESCENT;

DEEP LEARNING;

EID: 85084164409     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (21)

1. Martin Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, and Jeffrey Dean. “Tensorflow: Large-scale machine learning on heterogeneous distributed systems”. In: arXiv preprint:1603.04467 (2016).
2. Alham Fikri Aji and Kenneth Heafield. “Sparse communication for distributed gradient descent”. In: Empirical Methods in Natural Language Processing (EMNLP) (2017).
3. Xinghao Pan Jianmin Chen Rajat Monga Samy Bengio and Rafal Jozefowicz. “Revisiting distributed synchronous sgd”. In: arXiv preprint:1702.05800 (2017).
4. Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang. “Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems”. In: arXiv preprint:1512.01274 (2015).
5. Trishul M Chilimbi, Yutaka Suzue, Johnson Apacible, and Karthik Kalyanaraman. “Project adam: Buildingan efficient and scalable deep learning training system”. In: USENIX Symposium on Operating Systems Design and Implementation 14 (2014), pp. 571–582.
6. Collaborative machine learning without centralized training data. URL: https://research.googleblog.com/2017/04/federated-learning-collaborative.html (visited on 05/02/2018).
7. Elias De Coninck, Tim Verbelen, Bert Vankeirsbilck, Steven Bohez, Sam Leroux, and Pieter Simoens. “DIANNE: Distributed Artificial Neural Networks for the Internet of Things”. In: (2015), pp. 19–24. URL: http://doi.acm.org/10.1145/2836127.2836130.
8. Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc'aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, Quoc V. Le, and Andrew Y. Ng. “Large scale distributed deep networks”. In: Advances in Neural Information Processing Systems (2012), pp. 1223–1231.
9. Frank Seide Hao Fu Jasha Droppo Gang Li and Dong Yu. “1-Bit Stochastic Gradient Descent and its Application to Data-Parallel Distributed Training of Speech DNNs”. In: INTERSPEECH (2014).
10. Martin Zinkevich Markus Weimer Lihong Li and Alex J Smola. “Parallelized stochastic gradient descent”. In: In Advances in neural information processing systems (2010), pp. 2595–2603.
11. Yujun Lin, Song Han, Huizi Mao, Yu Wang, and William J. Dally. “Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training”. In: arXiv:1712.01887 (2018).
12. Nikoli Dryden Sam Ade Jacobs Tim Moon and Brian Van Essen. “Communication quantization for data parallel training of deep neural networks”. In: In Proceedings of the Workshop on Machine Learning in High Performance Computing Environments (2016).
13. Philipp Moritz, Robert Nishihara, Ion Stoica, and Michael I Jordan. “Sparknet: Training deep networks in spark”. In: arXiv preprint:1511.06051 (2015).
14. Resource efficient ML for Edge and Endpoint IoT Devices. URL: https://www.microsoft.com/en-us/research/project/resource-efficient-ml-for-the-edge-and-endpoint-iot-devices/ (visited on 05/02/2018).
15. Mu Li David G Andersen Jun Woo Park Alexander J Smola Amr Ahmed Vanja Josifovski James-Long Eugene J Shekita and Bor-Yiing Su. “Scaling distributed machine learning with the parameter server”. In: OSDI (2014), pp. 583–598.
16. Nikko Strom. “Scalable Distributed DNN Training Using Commodity GPU Cloud Computing”. In: INTERSPEECH (2015).
17. Dan Alistarh Jerry Li Ryota Tomioka and Milan Vojnovic. “Qsgd: Randomized quantization for communication-optimal stochastic gradient descent”. In: arXiv preprint arXiv:1610.02132 (2016).
18. Shuchang Zhou Yuxin Wu Zekun Ni Xinyu Zhou He Wen and Yuheng Zou. “Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients”. In: arXiv preprint arXiv:1606.06160 (2016).
19. Wei Wen, Cong Xu, Feng Yan, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. “TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning”. In: arXiv print:1705.07878 (2017).
20. Benjamin Recht Christopher Re Stephen Wright and Feng Niu Hogwild. “A lock free approach to parallelizing stochastic gradient descent”. In: In Advances in Neural Information Processing Systems (2011), pp. 693–701.
21. Chia-Yu Chen Jungwook Choi Daniel Brand Ankur Agrawal Wei Zhang and Kailash Gopalakrishnan. “Adacomp: Adaptive residual gradient compression for data-parallel distributed training”. In: arXiv preprint arXiv:1712.02679 (2017).