A decentralized training algorithm for Echo State Networks in distributed big data applications

Neural Networks

Volumn 78, Issue , 2016, Pages 65-74

  Scardapane, Simone ^a   Wang, Dianhui ^b   Panella, Massimo ^a  

^a SAPIENZA UNIVERSITY OF ROME   (Italy)

^b LA TROBE UNIVERSITY   (Australia)

Author keywords

Alternating Direction Method of Multipliers; Big data; Distributed learning; Echo State Network; Recurrent neural network

Indexed keywords

ALGORITHMS; RECURRENT NEURAL NETWORKS;

ALTERNATING DIRECTION METHOD OF MULTIPLIERS; BIG DATA APPLICATIONS; DISTRIBUTED LEARNING; ECHO STATE NETWORKS; GENERALIZATION ACCURACY; OPTIMIZATION PROCEDURES; RECURRENT NEURAL NETWORK (RNNS); TECHNOLOGICAL CONSTRAINTS;

BIG DATA;

ALGORITHM; ARTICLE; ARTIFICIAL NEURAL NETWORK; CONTROLLED STUDY; DECENTRALIZED TRAINING ALGORITHM; ECHO STATE NETWORK; INFORMATION; PRIORITY JOURNAL; RECURRENT NEURAL NETWORK; STATISTICAL ANALYSIS; STATISTICAL PARAMETERS; TECHNOLOGY; FACTUAL DATABASE; HUMAN; INFORMATION PROCESSING; STATISTICS AND NUMERICAL DATA; TRENDS;

ALGORITHMS; DATABASES, FACTUAL; DATASETS AS TOPIC; HUMANS; NEURAL NETWORKS (COMPUTER);

EID: 84940676201     PISSN: 08936080     EISSN: 18792782     Source Type: Journal    
DOI: 10.1016/j.neunet.2015.07.006     Document Type: Article

References (55)

1. Bakir G. Predicting structured data 2007, MIT press.
2. Barbarossa S., Sardellitti S., Di Lorenzo P. Distributed detection and estimation in wireless sensor networks. E-Reference signal processing 2013, 329-408. Elsevier. R. Chellapa, S. Theodoridis (Eds.).
3. ® in Machine Learning 2011, 3(1):1-122.
4. Butcher J.B., Verstraeten D., Schrauwen B., Day C.R., Haycock P.W. Reservoir computing and extreme learning machines for non-linear time-series data analysis. Neural Networks 2013, 38:76-89.
5. Campolucci P., Uncini A., Piazza F., Rao B.D. On-line learning algorithms for locally recurrent neural networks. IEEE Transactions on Neural Networks 1999, 10(2):253-271.
6. Cattivelli F.S., Lopes C.G., Sayed A.H. Diffusion recursive least-squares for distributed estimation over adaptive networks. IEEE Transactions on Signal Processing 2008, 56(5):1865-1877.
7. Cevher V., Becker S., Schmidt M. Convex optimization for big data: scalable, randomized, and parallel algorithms for big data analytics. IEEE Signal Processing Magazine 2014, 31(5):32-43.
8. Chu C.-T., Kim S.K., Lin Y.A., Yu Y.Y. Map-reduce for machine learning on multicore. Advances in neural information processing systems 2007, 281-288.
9. Dean J., Corrado G., Monga R., Chen K., Devin M., Mao M., et al. Large scale distributed deep networks. Advances in neural information processing systems 2012, 1223-1231.
10. Di Lorenzo P., Sayed A.H. Sparse distributed learning based on diffusion adaptation. IEEE Transactions on Signal Processing 2013, 61(6):1419-1433.
11. Forero P.A., Cano A., Giannakis G.B. Consensus-based distributed support vector machines. The Journal of Machine Learning Research 2010, 11:1663-1707.
12. Georgopoulos L., Hasler M. Distributed machine learning in networks by consensus. Neurocomputing 2014, 124:2-12.
13. Hermans M., Schrauwen B. Training and analysing deep recurrent neural networks. Advances in neural information processing systems 2013, 190-198.
14. Honeine P., Richard C., Bermudez J.C.M., Snoussi H. Distributed prediction of time series data with kernels and adaptive filtering techniques in sensor networks. Proceedings of the 42nd Asilomar conference on signals, systems and computers 2008, 246-250. IEEE.
15. Igelnik B., Pao Y.-H. Stochastic choice of basis functions in adaptive function approximation and the functional-link net. IEEE Transactions on Neural Networks 1995, 6(6):1320-1329.
16. Jaeger H. Adaptive nonlinear system identification with echo state networks. Advances in neural information processing systems 2002, 593-600.
17. Jaeger H., Haas H. Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication. Science 2004, 304(5667):78-80.
18. Li D., Han M., Wang J. Chaotic time series prediction based on a novel robust echo state network. IEEE Transactions on Neural Networks and Learning Systems 2012, 23(5):787-799.
19. Lin X., Yang Z., Song Y. Short-term stock price prediction based on echo state networks. Expert Systems with Applications 2009, 36(3):7313-7317.
20. Lu Y., Roychowdhury V., Vandenberghe L. Distributed parallel support vector machines in strongly connected networks. IEEE Transactions on Neural Networks 2008, 19(7):1167-1178.
21. Luitel B., Venayagamoorthy G.K. Decentralized asynchronous learning in cellular neural networks. IEEE Transactions on Neural Networks and Learning Systems 2012, 23(11):1755-1766.
22. Lukoševičius M., Jaeger H. Reservoir computing approaches to recurrent neural network training. Computer Science Review 2009, 3(3):127-149.
23. Martens, J., & Sutskever, I. (2011). Learning recurrent neural networks with hessian-free optimization. In Proceedings of the 28th international conference on machine learning, ICML (pp. 1033-1040).
24. Mateos G., Bazerque J.A., Giannakis G.B. Distributed sparse linear regression. IEEE Transactions on Signal Processing 2010, 58(10):5262-5276.
25. Monner D., Reggia J.A. A generalized LSTM-like training algorithm for second-order recurrent neural networks. Neural Networks 2012, 25:70-83.
26. Navia-Vázquez A., Gutierrez-Gonzalez D., Parrado-Hernández E., Navarro-Abellan J.J. Distributed support vector machines. IEEE Transactions on Neural Networks 2006, 17(4):1091-1097.
27. Obst O. Distributed fault detection in sensor networks using a recurrent neural network. Neural Processing Letters 2014, 40(3):261-273.
28. Olfati-Saber R., Fax J.A., Murray R.M. Consensus and cooperation in networked multi-agent systems. Proceedings of the IEEE 2007, 95(1):215-233.
29. Pao Y.-H., Takefuji Y. Functional-link net computing: theory, system architecture, and functionalities. Computer 1992, 25(5):76-79.
30. Pascanu, R., Mikolov, T., & Bengio, Y. (2013). On the difficulty of training recurrent neural networks. In Proceedings of the 30th international conference on machine learning.
31. Pearlmutter B.A. Gradient calculations for dynamic recurrent neural networks: A survey. IEEE Transactions on Neural Networks 1995, 6(5):1212-1228.
32. Predd J.B., Kulkarni S.R., Poor H.V. Distributed learning in wireless sensor networks. IEEE Signal Processing Magazine 2007, 56-69.
33. Predd J.B., Kulkarni S.R., Poor H.V. A collaborative training algorithm for distributed learning. IEEE Transactions on Information Theory 2009, 55(4):1856-1871.
34. Sak, H., Senior, A., & Beaufays, F. (2014). Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In Proceedings of the annual conference of international speech communication association, INTERSPEECH.
35. Sardellitti S., Giona M., Barbarossa S. Fast distributed average consensus algorithms based on advection-diffusion processes. IEEE Transactions on Signal Processing 2010, 58(2):826-842.
36. Scardapane S., Nocco G., Comminiello D., Scarpiniti M., Uncini A. An effective criterion for pruning reservoir's connections in echo state networks. 2014 International joint conference on neural networks (IJCNN) 2014, 1205-1212. INNS/IEEE.
37. Scardapane S., Wang D., Panella M., Uncini A. Distributed learning with random vector functional-link networks. Information Sciences 2015, 301:271-284.
38. Scardapane S., Wang D., Panella M., Uncini A. Distributed music classification using random vector functional-link nets. 2015 International joint conference on neural networks (IJCNN) 2015, INNS/IEEE.
39. Schliebs S., Mohemmed A., Kasabov N. Are probabilistic spiking neural networks suitable for reservoir computing?. 2011 International joint conference on neural networks (IJCNN) 2011, 3156-3163. INNS/IEEE.
40. Shutin, D., & Kubin, G. (2008). Echo state wireless sensor networks. In 2008 IEEE workshop on machine learning for signal processing (pp. 151-156).
41. Skowronski M.D., Harris J.G. Automatic speech recognition using a predictive echo state network classifier. Neural Networks 2007, 20(3):414-423.
42. Steil J.J. Online reservoir adaptation by intrinsic plasticity for backpropagation-decorrelation and echo state learning. Neural Networks 2007, 20(3):353-364.
43. Sutskever I., Vinyals O., Le Q.V.V. Sequence to sequence learning with neural networks. Advances in neural information processing systems 2014, 3104-3112.
44. Tong M.H., Bickett A.D., Christiansen E.M., Cottrell G.W. Learning grammatical structure with echo state networks. Neural Networks 2007, 20(3):424-432.
45. Triefenbach F., Jalalvand A., Demuynck K., Martens J.-P. Acoustic modeling with hierarchical reservoirs. IEEE Transactions on Audio, Speech and Language Processing 2013, 21(11):2439-2450.
46. Vandoorne K., Dambre J., Verstraeten D., Schrauwen B., Bienstman P. Parallel reservoir computing using optical amplifiers. IEEE Transactions on Neural Networks 2011, 22(9):1469-1481.
47. Vandoorne K., Mechet P., Van Vaerenbergh T., Fiers M., Morthier G., Verstraeten D., et al. Experimental demonstration of reservoir computing on a silicon photonics chip. Nature Communications 2014, 5.
48. Verstraeten D., Schrauwen B., d'Haene M., Stroobandt D. An experimental unification of reservoir computing methods. Neural Networks 2007, 20(3):391-403.
49. Verykios V.S., Bertino E., Fovino I.N., Provenza L.P., Saygin Y., Theodoridis Y. State-of-the-art in privacy preserving data mining. ACM SIGMOD Record 2004, 33(1):50-57.
50. Wu X., Zhu X., Wu G.-Q., Ding W. Data mining with big data. The IEEE Transactions on Knowledge and Data Engineering 2014, 26(1):97-107.
51. Xiao L., Boyd S., Lall S. A scheme for robust distributed sensor fusion based on average consensus. Fourth international symposium on information processing in sensor networks, 2005 2005, 63-70. IEEE.
52. Xue Y., Yang L., Haykin S. Decoupled echo state networks with lateral inhibition. Neural Networks 2007, 20(3):365-376.
53. Yildiz I.B., Jaeger H., Kiebel S.J. Re-visiting the echo state property. Neural Networks 2012, 35:1-9.
54. Zhang B., Miller D.J., Wang Y. Nonlinear system modeling with random matrices: echo state networks revisited. IEEE Transactions on Neural Networks and Learning Systems 2012, 23(1):175-182.
55. Zinkevich M., Weimer M., Li L., Smola A.J. Parallelized stochastic gradient descent. Advances in neural information processing systems 2010, 2595-2603.