Fast, simple and accurate handwritten digit classification by training shallow neural network classifiers with the 'Extreme learning machine' algorithm

PLoS ONE

Volumn 10, Issue 8, 2015, Pages

  McDonnell, Mark D ^a   Tissera, Migel D ^a   Vladusich, Tony ^a   Van Schaik, André ^b   Tapson, Jonathan ^b   Schwenker, Friedhelm ^c  

^a UNIVERSITY OF SOUTH AUSTRALIA   (Australia)

^b WESTERN SYDNEY UNIVERSITY   (Australia)

^c UNIVERSITY OF ULM   (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

ACCURACY; ALGORITHM; ANALYTICAL ERROR; ARTICLE; ARTIFICIAL NEURAL NETWORK; AUTOMATED PATTERN RECOGNITION; CLASSIFIER; DATA BASE; DATA PROCESSING; EXTREME MACHINE LEARNING; HANDWRITING; IMAGE PROCESSING; MACHINE LEARNING; PROCESS OPTIMIZATION; RECEPTIVE FIELD; THEORETICAL MODEL;

ALGORITHMS; MODELS, THEORETICAL;

EID: 84942903272     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0134254     Document Type: Article

References (37)

1. LeCun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proceedings of the IEEE. 1998; 86:2278-2324. doi:10.1109/5.726791
2. Hinton GE, Osindero S, Teh YW A fast learning algorithm for deep belief nets. Neural computation. 2006; 18:1527-1554. doi: 10.1162/neco.2006.18.7.1527 PMID: 16764513
3. LeCun Y, Cortes C, Burges CJC. The MNIST database of handwritten digits; Accessed August 2014. http://yann.lecun.com/exdb/mnist/
4. Cireşan D., Meier U, Gambardella L.M., Schmidhuber J. Deep, big, simple neural nets for handwritten digit recognition. Neural Computation. 2010; 22:3207-3220. doi: 10.1162/NECOa00052 PMID: 20858131
5. Cireşan D., Meier U, Schmidhuber J. Multi-column deep neural networks for image classification. In: Proc. CVPR; 2012. p. 3642-3649.
6. Wan L, Zeiler M, Zhang S., LeCun Y, Fergus R. Regularization of neural networks using Drop Connect. In: Proceedings of the 30th International Conference on Machine Learning, Atlanta, Georgia, USA; JMLR: W & CP Volume 28; 2013.
7. Zelier MD, Fergus R. Stochastic pooling for regularization of deep convolutional neural networks. In: Proc. International Conference on Learning Representations, Scottsdale, USA, 2013; 2013.
8. Goodfellow IJ, Warde-Farley D, Mirza M, Courville A, Bengio Y. Maxout networks. In: Proceedings of the 30th International Conference on Machine Learning, Atlanta, Georgia, USA; JMLR: W & CP Volume 28; 2013.
9. Lee CY, Xie S, Gallagher P., Zhang Z, Tu Z. Deeply-supervised nets. In: Deep Learning and Representation Learning Workshop, NIPS; 2014.
10. Huang GB, Zhu QY, Siew CK. Extreme learning machine: Theory and applications. Neurocomputing. 2006; 70:489-501. doi:10.1016/j.neucom.2005.12.126
11. Schmidt PF, Kraaijveld MA, Duin RPW. Feedforward neural networks with random weights. In: Proc. 11th IAPR Int. Conf. on Pattern Recognition, Volume II, Conf. B: Pattern Recognition Methodology and Systems (ICPR11, The Hague, Aug. 30 - Sep. 3), IEEE Computer Society Press, Los Alamitos, CA, 1992, 1-4; 1992.
12. Chen CLP. A rapid supervised learning neural networkforfunction interpolation and approximation. IEEE Transactions on Neural Networks. 1996; 7:1220-1230. doi: 10.1109/72.536316 PMID: 18263516
13. Eliasmith C, Anderson CH Developing and applying a toolkit from a general neurocomputational framework. Neurocomputing. 1999; 26:1013-1018. doi:10.1016/S0925-2312(99)00098-3
14. Eliasmith C, Anderson CH Neural Engineering: Computation, Representation, and Dynamics in Neuro-biological Systems. MIT Press, Cambridge, MA; 2003.
15. Tapson J, deChazal P, van Schaik A. Explicit computation of input weights in extreme learning machines. In: Proc. ELM2014 conference, Accepted; 2014. p. arXiv:1406.2889.
16. Nair V, Hinton GE. Rectified linear units improve restricted Boltzmann machines. In: Proceedings of the 27th International Conference on Machine Learning (ICML), Haifa, Israel; 2010.
17. Marquardt DW, Snee RD Ridge Regression in Practice. The American Statistician. 1975; 29:3-20. doi:10.2307/2683673
18. Press WH, Teukolsky SA, Vetterling W.T., Flannery BP. Numerical Recipes in C: The Art of Scientific Computing. 2nd ed. Cambridge University Press, Cambridge, UK; 1992.
19. Huang GB, Zhou H, Ding X, Zhang R. Extreme learning machine for regression and multiclass classification. IEEE Transactions on Systems, Man, and Cybernetics - PartB: Cybernetics. 2012; 42:513-529. doi:10.1109/TSMCB.2011.2168604
20. Kasun LLC, Zhou H, Huang G.B., Vong CM. Representational learning with extreme learning machine for big data. IEEE Intelligent Systems. 2013; 28:31-34.
21. Zhu W, Miao J, Qing L. Constrained extreme learning machines: A study on classification cases; 2015. ArXiv:1501.06115.
22. Coates A, Lee H, Ng AY. An analysis of single-layer networks in unsupervised feature learning. In: Proc.14th International Conference on Artificial Intelligence and Statistics (AISTATS), 2011, Fort Lauderdale, FL, USA. Volume 15 of JMLR: W & CP 15; 2011.
23. Yu D, Ding L. Efficient and effective algorithms for training single-hidden-layer neural networks. Pattern Recognition Letters. 2012; 33:554-558. doi:10.1016/j.patrec.2011.12.002
24. Rumelhart DE, Hinton GE, Williams RJ Learning representations by back-propagating errors. Nature. 1986; 323:533-536. doi:10.1038/323533a0
25. Huang GB. An insight into extreme learning machines: Random neurons, random features and kernels. Cognitive Computation. 2014; 6:376-390. doi:10.1007/s12559-014-9255-2
26. Widrow B, Greenblatt A, Kim Y., Park D. The No-Prop algorithm: A new learning algorithm for multilayer neural networks. Neural Networks. 2013; 37:182-188. doi: 10.1016/j.neunet.2012.09.020 PMID: 23140797
27. Tapson J, van Schaik A. Learning the pseudoinverse solution to network weights. Neural Networks. 2013; 45:94-100. doi: 10.1016/j.neunet.2013.02.008 PMID: 23541926
28. Simard PY, Steinkraus D, Platt JC Best practices for convolutional neural networks applied to visual document analysis. In: Proceedings of the Seventh International Conference on Document Analysis and Recognition (ICDAR 2003); 2003.
29. LeCun Y, Huang FJ, Bottou L. Learning Methods for Generic Object Recognition with Invariance to Pose and Lighting. In: Proceedings IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR). Vol. 2; 2004. p. 97-104.
30. Cireşan D.C.D., Meier U, Masci J., Gambardella LM, Schmidhuber J. Flexible, high performance convolutional neural networks for image classification. In: Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence; 2011. p. 1237-1242.
31. Jarrett K, Kavukcuoglu K, Ranzato M., LeCun Y. What is the best multi-stage architecture for object recognition? In: Proc. IEEE 12th International Conference on Computer Vision; 2009.
32. Widrow B. Reply to the comments on the "No-Prop" algorithm. Neural Networks. 2013; 48:204. doi: 10.1016/j.neunet.2013.11.003 PMID: 24314855
33. Liang NY, Huang GB, Saratchandran P, Sundararajan N. A fast and accurate online sequential learning algorithm for feedforward networks. IEEE Transactions on Neural Networks. 2006; 17:1411-1423. doi: 10.1109/TNN.2006.880583 PMID: 17131657
34. van Schaik A, Tapson J. Online and adaptive pseudoinverse solutions for ELM weights. Neurocomputing. 2015; 149:233-238. doi:10.1016/j.neucom.2014.01.071
35. Lim MH. Comments on the "No-Prop" algorithm. Neural Networks. 2013; 48:59-60. doi: 10.1016/j. neunet.2013.07.004 PMID: 23945268
36. Eliasmith C, Stewart TC, Choo X, Bekolay T, De Wolf T, Tang C, et al. A Large-Scale Model of the Functioning Brain. Science. 2012; 338:1202-1205. doi: 10.1126/science.1225266 PMID: 23197532
37. Stewart TC, Eliasmith C. Large-scale synthesis of functional spiking neural circuits. Proceedings of the IEEE. 2014; 102:881-898. doi:10.1109/JPROC.2014.2306061