Large-scale distributed L-BFGS

Journal of Big Data

Volumn 4, Issue 1, 2017, Pages

  Najafabadi, Maryam M ^a   Khoshgoftaar, Taghi M ^a   Villanustre, Flavio ^b   Holt, John ^b  

^a FLORIDA ATLANTIC UNIVERSITY   (United States)

^b LexisNexis Risk Solutions   (United States)

Author keywords

HPCC systems; Large scale L BFGS implementation; Parallel and distributed processing

Indexed keywords

BIG DATA; CLUSTER COMPUTING; DATA ANALYTICS; DATA MINING; LEARNING ALGORITHMS; LEARNING SYSTEMS; MACHINE LEARNING; OPEN SYSTEMS;

BROYDEN FLETCHER GOLDFARB SHANNO ALGORITHMS; DISTRIBUTED SYSTEMS; HPCC SYSTEM; LARGE MODELS; LARGE-SCALE LIMITED-MEMORY BROYDEN FLETCHER GOLDFARB SHANNO IMPLEMENTATION; LARGE-SCALES; LIMITED MEMORY BROYDEN-FLETCHER-GOLDFARB-SHANNO; MACHINE-LEARNING; NUMERIC OPTIMIZATION; PARALLEL AND DISTRIBUTED PROCESSING;

PARAMETER ESTIMATION;

EID: 85024831970     PISSN: None     EISSN: 21961115     Source Type: Journal    
DOI: 10.1186/s40537-017-0084-5     Document Type: Article

References (32)

1. Bennett KP, Parrado-Hernández E. The interplay of optimization and machine learning research. J Mach Learn Res. 2006;7:1265–81.
2. Najafabadi MM, Villanustre F, Khoshgoftaar TM, Seliya N, Wald R, Muharemagic E. Deep learning applications and challenges in big data analytics. J Big Data. 2015;2(1):1–21.
3. Xing EP, Ho Q, Xie P, Wei D. Strategies and principles of distributed machine learning on big data. Engineering. 2016;2(2):179–95.
4. Dean J, Corrado G, Monga R, Chen K, Devin M, Mao M, Senior A, Tucker P, Yang K, Le QV, et al. Large scale distributed deep networks. In: Advances in neural information processing systems. Lake Tahoe, Nevada: Curran Associates Inc.; 2012. p. 1223–31.
5. Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks. In: Pereira F, Burges CJC, Bottou L, Weinberger KQ, editors. Advances in neural information processing systems. Lake Tahoe, Nevada: Curran Associates, Inc.; 2012. p. 1097–05.
6. Dong L, Lin Z, Liang Y, He L, Zhang N, Chen Q, Cao X, Izquierdo E. A hierarchical distributed processing framework for big image data. IEEE Trans Big Data. 2016;2(4):297–309.
7. Sliwinski TS, Kang SL. Applying parallel computing techniques to analyze terabyte atmospheric boundary layer model outputs. Big Data Res. 2017;7:31–41.
8. Shalev-Shwartz S, Singer Y, Srebro N. Pegasos: primal estimated sub-gradient solver for svm. In: Proceedings of the 24th international conference on machine learning. New York: ACM; 2007. p. 807–14.
9. Zinkevich M, Weimer M, Li L, Smola AJ. Parallelized stochastic gradient descent. In: Lafferty JD, Williams CKI, Shawe-Taylor J, Zemel RS, Culotta A, editors. Advances in neural information processing systems. Vancouver, British Columbia, Canada: Curran Associates Inc.; 2010. p. 2595–03.
10. Nocedal J, Wright SJ. Numerical optimization. 2nd ed. New York: Springer; 2006.
11. Ngiam J, Coates A, Lahiri A, Prochnow B, Le QV, Ng AY. On optimization methods for deep learning. In: Proceedings of the 28th international conference on machine learning (ICML-11). 2011. p. 265–72.
12. Schraudolph NN, Yu J, Günter S, et al. A stochastic quasi-newton method for online convex optimization. Artif Intell Stat Conf. 2007;7:436–43.
13. Daumé III, H.: Notes on cg and lm-bfgs optimization of logistic regression. http://www.umiacs.umd.edu/~hal/docs/daume04cg-bfgs, implementation http://www.umiacs.umd.edu/~hal/megam/. 2004; 198: 282.
14. Middleton A, Solutions P. Hpcc systems: introduction to hpcc (high-performance computing cluster). White paper, LexisNexis Risk Solutions. 2011. http://cdn.hpccsystems.com/whitepapers/wp_introduction_HPCC.pdf.
15. Dean J, Ghemawat S. Mapreduce: simplified data processing on large clusters. Commun ACM. 2008;51(1):107–13.
16. White T. Hadoop: the definitive guide. 3rd ed. 2012.
17. Datasets, R.D.: A faulttolerant abstraction for inmemory cluster computing Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy Mccauley, Michael J. Franklin, Scott Shenker, Ion Stoica University of California: Berkeley.
18. Agarwal A, Chapelle O, Dudík M, Langford J. A reliable effective terascale linear learning system. J Mach Learn Res. 2014;15(1):1111–33.
19. Sra S, Nowozin S, Wright SJ. Optimization for machine learning. Cambridge: The MIT Press; 2011.
20. Dekel O, Gilad-Bachrach R, Shamir O, Xiao L. Optimal distributed online prediction using mini-batches. J Mach Learn Res. 2012;13:165–202.
21. Teo CH, Smola A, Vishwanathan S, Le QV. A scalable modular convex solver for regularized risk minimization. In: Proceedings of the 13th ACM SIGKDD international conference on knowledge discovery and data mining. New York: ACM; 2007. p. 727–36.
22. Recht B, Re C, Wright S, Niu F. Hogwild: a lock-free approach to parallelizing stochastic gradient descent. In: Shawe-Taylor J, Zemel RS, Bartlett PL, Pereira F, Weinberger KQ, editors. Advances in neural information processing systems. Granada, Spain: Curran Associates, Inc.; 2011. p. 693–701.
23. Dean J, Ghemawat S. Mapreduce: a flexible data processing tool. Commun ACM. 2010;53(1):72–7.
24. Chaiken R, Jenkins B, Larson P-Å, Ramsey B, Shakib D, Weaver S, Zhou J. Scope: easy and efficient parallel processing of massive data sets. Proc VLDB Endow. 2008;1(2):1265–76.
25. Stonebraker M, Abadi D, DeWitt DJ, Madden S, Paulson E, Pavlo A, Rasin A. Mapreduce and parallel dbmss: friends or foes? Commun ACM. 2010;53(1):64–71.
26. Pike R, Dorward S, Griesemer R, Quinlan S. Interpreting the data: parallel analysis with sawzall. Sci Program. 2005;13(4):277–98.
27. Gates AF, Natkovich O, Chopra S, Kamath P, Narayanamurthy SM, Olston C, Reed B, Srinivasan S, Srivastava U. Building a high-level dataflow system on top of map-reduce: the pig experience. Proc VLDB Endow. 2009;2(2):1414–25.
28. Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado GS, Davis A, Dean J, Devin M, et al. Tensorflow: large-scale machine learning on heterogeneous distributed systems. arXiv preprint. 2016. arXiv:1603.04467.
29. Wolfe P. Convergence conditions for ascent methods. SIAM Rev. 1969;11(2):226–35.
30. Team BRD. Ecl language reference. White paper, LexisNexis Risk Solutions. 2015. http://cdn.hpccsystems.com/install/docs/3_4_0_1/ECLLanguageReference.pdf.
31. Bishop CM. Pattern recognition and machine learning (information science and statistics). Secaucus: Springer; 2006.
32. Schmidt M. minFunc: unconstrained differentiable multivariate optimization in Matlab. 2005. http://www.cs.ubc.ca/~schmidtm/Software/minFunc.html.