Large-scale minimal complexity machines using explicit feature maps

IEEE Transactions on Systems, Man, and Cybernetics: Systems

Volumn 47, Issue 10, 2017, Pages 2653-2662

  Sharma, Mayank ^a   Jayadeva, ^a   Soman, Sumit ^a   Pant, Himanshu ^a  

^a INDIAN INSTITUTE OF TECHNOLOGY DELHI   (India)

Author keywords

Fastfood; minimal complexity machines (MCMs); random Fourier features; stochastic gradient descent; Vapnik Chervonenkis (VC) dimension

Indexed keywords

APPROXIMATION ALGORITHMS; STOCHASTIC SYSTEMS; SUPPORT VECTOR MACHINES;

HIGHER DIMENSIONAL FEATURES; KERNEL MATRICES; LARGE DATASETS; LARGE-SCALE MACHINE LEARNING; SPARSE SOLUTIONS; SUBGRADIENT DESCENT; SUPPORT VECTOR; VAPNIK-CHERVONENKIS DIMENSIONS;

LEARNING SYSTEMS;

EID: 85020478484     PISSN: 21682216     EISSN: 21682232     Source Type: Journal    
DOI: 10.1109/TSMC.2017.2694321     Document Type: Article

References (63)

1. Jayadeva, "Learning a hyperplane classifier by minimizing an exact bound on the VC dimension," Neurocomputing, vol. 149, pp. 683-689, Feb. 2015. [Online]. Available: http://www.sciencedirect.com/science/ article/pii/S0925231214010194
2. Jayadeva, S. Chandra, S. S. Batra, and S. Sabharwal, "Learning a hyperplane regressor through a tight bound on the VC dimension," Neurocomputing, vol. 171, pp. 1610-1616, Jan. 2016. [Online]. Available: http://www.sciencedirect.com/science/article/pii/ S092523121500939X
3. Jayadeva, S. Soman, and A. Bhaya, "The MC-ELM: Learning an ELMlike network with minimum VC dimension," in Proc. Int. Joint Conf. Neural Netw. (IJCNN), Jul. 2015, pp. 1-7
4. Y.-L. Hou, X. Hao, and H. Chen, "A cognitively motivated method for classification of occluded traffic signs," IEEE Trans. Syst., Man, Cybern., Syst., vol. 47, no. 2, pp. 255-262, Feb. 2017
5. W. Yang, X. Sun, and Q. Liao, "Cascaded elastically progressive model for accurate face alignment," IEEE Trans. Syst., Man, Cybern., Syst., to be published, doi: 10.1109/TSMC.2016.2523930
6. L. Ladický and P. H. S. Torr, "Locally linear support vector machines," in Proc. 28th Int. Conf. Mach. Learn. (ICML), 2011, pp. 985-992
7. D. D. Wu, L. Zheng, and D. L. Olson, "A decision support approach for online stock forum sentiment analysis," IEEE Trans. Syst., Man, Cybern., Syst., vol. 44, no. 8, pp. 1077-1087, Aug. 2014
8. Q. Wu, Z. Wang, F. Deng, Z. Chi, and D. D. Feng, "Realistic human action recognition with multimodal feature selection and fusion," IEEE Trans. Syst., Man, Cybern., Syst., vol. 43, no. 4, pp. 875-885, Jul. 2013
9. J. Sun, H. Li, and H. Adeli, "Concept drift-oriented adaptive and dynamic support vector machine ensemble with time window in corporate financial risk prediction," IEEE Trans. Syst., Man, Cybern., Syst., vol. 43, no. 4, pp. 801-813, Jul. 2013
10. B. Zhang, W. Li, P. Qing, and D. Zhang, "Palm-print classification by global features," IEEE Trans. Syst., Man, Cybern., Syst., vol. 43, no. 2, pp. 370-378, Mar. 2013
11. H. Kashima, T. Idé, T, Kato, and M. Sugiyama, "Recent advances and trends in large-scale kernel methods," IEICE Trans. Inf. Syst., vol. 92-D, no. 7, pp. 1338-1353, 2009
12. L. Bottou, O. Chapelle, D. DeCoste, and J. Weston, Large-Scale Kernel Machines (Neural Information Processing). Cambridge, MA, USA: MIT Press, 2007
13. D. Decoste and B. Schölkopf, "Training invariant support vector machines," Mach. Learn., vol. 46, no. 1, pp. 161-190, 2002. [Online]. Available: http://dx.doi.org/10.1023/A:1012454411458
14. A. J. Smola and B. Schökopf, "Sparse greedy matrix approximation for machine learning," in Proc. ICML, 2000, pp. 911-918. [Online]. Available: http://dl.acm.org/citation.cfm?id=645529.657980
15. C. K. I. Williams and M. Seeger, "Using the nyström method to speed up kernel machines," in Proc. 13th Int. Conf. Neural Inf. Process. Syst. (NIPS), Denver, CO, USA, 2000, pp. 661-667. [Online]. Available: http://dl.acm.org/citation.cfm?id=3008751.3008847
16. S. Fine and K. Scheinberg, "Efficient SVM training using low-rank kernel representations," J. Mach. Learn. Res., vol. 2, pp. 243-264, Mar. 2002. [Online]. Available: http://dl.acm.org/citation.cfm?id=944790.944812
17. C. Cortes, M. Mohri, and A. Talwalkar, "On the impact of kernel approximation on learning accuracy," in Proc. 13th Int. Conf. Artif. Intell. Stat., vol. 9. Sardinia, Italy, 2010, pp. 113-120. [Online]. Available: http://proceedings.mlr.press/v9/cortes10a.html
18. J. C. Platt, Advances in Kernel Methods. Cambridge, MA, USA: MIT Press, 1999, pp. 185-208. [Online]. Available: http://dl.acm.org/citation.cfm?id=299094.299105
19. S. S. Keerthi, S. K. Shevade, C. Bhattacharyya, and K. R. K. Murthy, "Improvements to Platt's SMO Algorithm for SVM classifier design," Neural Comput., vol. 13, no. 3, pp. 637-649, 2001
20. J. Zhai, C. Li, and T. Li, "Sample selection based on K-L divergence for effectively training SVM," in Proc. IEEE Int. Conf. Syst. Man Cybern., Oct. 2013, pp. 4837-4842
21. S. Joshi, Jayadeva, G. Ramakrishnan, and S. Chandra, "Using sequential unconstrained minimization techniques to simplify SVM solvers," Neurocomputing, vol. 77, no. 1, pp. 253-260, 2012. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0925231211004188
22. T. Joachims, Advances in Kernel Methods. Cambridge, MA, USA: MIT Press, 1999. [Online]. Available: http://dl.acm.org/citation.cfm?id=299094.299104
23. I. W. Tsang, J. T. Kwok, and P.-M. Cheung, "Core vector machines: Fast SVM training on very large data sets," J. Mach. Learn. Res., vol. 6, pp. 363-392, Dec. 2005. [Online]. Available: http://dl.acm.org/citation.cfm?id=1046920.1058114
24. I. W. Tsang, A. Kocsor, and J. T. Kwok, "Simpler core vector machines with enclosing balls," in Proc. 24th Int. Conf. Mach. Learn. (ICML), Corvallis, OR, USA, 2007, pp. 911-918. [Online]. Available: http://doi.acm.org/10.1145/1273496.1273611
25. A. Bordes, S. Ertekin, J. Weston, and L. Bottou, "Fast kernel classifiers with online and active learning," J. Mach. Learn. Res., vol. 6, pp. 1579-1619, Dec. 2005. [Online]. Available: http://dl.acm.org/citation.cfm?id=1046920.1194898
26. B. Catanzaro, N. Sundaram, and K. Keutzer, "Fast support vector machine training and classification on graphics processors," in Proc. 25th Int. Conf. Mach. Learn. (ICML), Helsinki, Finland, 2008, pp. 104-111. [Online]. Available: http://doi.acm.org/10.1145/ 1390156.1390170
27. C.-C. Chang and C.-J. Lin, "LIBSVM: A library for support vector machines," ACM Trans. Intell. Syst. Technol., vol. 2, no. 3, pp. 1-27, Apr. 2011. [Online]. Available: http://doi.acm.org/10.1145/ 1961189.1961199
28. L. J. Cao et al., "Parallel sequential minimal optimization for the training of support vector machines," IEEE Trans. Neural Netw., vol. 17, no. 4, pp. 1039-1049, Jul. 2006. [Online]. Available: http://dx.doi.org/10.1109/TNN.2006.875989
29. G. Cauwenberghs and T. Poggio, "Incremental and decremental support vector machine learning," in Proc. NIPS, 2000, pp. 388-394. [Online]. Available: http://dl.acm.org/citation.cfm?id=3008751.3008808
30. L. Bottou and C.-J. Lin, "Support vector machine solvers," in Large Scale Kernel Machines. Cambridge, MA, USA: MIT Press, 2007, pp. 301-320
31. O. Chapelle, "Training a support vector machine in the primal," Neural Comput., vol. 19, no. 5, pp. 1155-1178, May 2007. [Online]. Available: http://dx.doi.org/10.1162/neco.2007.19.5.1155
32. Y. Mu, G. Hua, W. Fan, and S.-F. Chang, "Hash-SVM: Scalable kernel machines for large-scale visual classification," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Columbus, OH, USA, Jun. 2014, pp. 979-986
33. G.-B. Huang, "An insight into extreme learning machines: Random neurons, random features and kernels," Cogn. Comput., vol. 6, no. 3, pp. 376-390, 2014. [Online]. Available: http://dx.doi.org/10.1007/ s12559-014-9255-2
34. S. Zhou et al., "Random Fourier extreme learning machine with l2,1-norm regularization," Neurocomputing, vol. 174, pp. 143-153, Jan. 2016. [Online]. Available: http://dx.doi.org/10.1016/ j.neucom.2015.03.113
35. C.-J. Hsieh, S. Si, and I. S. Dhillon, "Fast prediction for large-scale kernel machines," in Proc. 27th Int. Conf. Neural Inf. Process. Syst. (NIPS), Montreal, QC, Canada, 2014, pp. 3689-3697. [Online]. Available: http://dl.acm.org/citation.cfm?id= 2969033.2969238
36. P. Rebentrost, M. Mohseni, and S. Lloyd, "Quantum support vector machine for big data classification," Phys. Rev. Lett., vol. 113, no. 13, Sep. 2014, Art. no. 130503. [Online]. Available: https://link.aps.org/doi/10.1103/PhysRevLett.113.130503
37. C.-J. Hsieh, K.-W. Chang, C.-J. Lin, S. S. Keerthi, and S. Sundararajan, "A dual coordinate descent method for largescale linear svm," in Proc. 25th Int. Conf. Mach. Learn. (ICML), Helsinki, Finland, 2008, pp. 408-415. [Online]. Available: http://doi.acm.org/10.1145/1390156.1390208
38. S. Shalev-Shwartz, Y. Singer, and N. Srebro, "Pegasos: Primal estimated sub-gradient solver for SVM," in Proc. ICML, Corvallis, OR, USA, 2007, pp. 807-814. [Online]. Available: http://doi.acm.org/10.1145/1273496.1273598
39. T. Joachims, "Training linear SVMs in linear time," in Proc. 12th ACM SIGKDD Int. Conf. Knowl. Discov. Data Min. (KDD), Philadelphia, PA, USA, 2006, pp. 217-226. [Online]. Available: http://doi.acm.org/10.1145/1150402.1150429
40. V. Sindhwani and S. S. Keerthi, "Large scale semi-supervised linear SVMs," in Proc. 29th Annu. Int. ACM SIGIR Conf. Res. Develop. Inf. Retrieval (SIGIR), Seattle, WA, USA, 2006, pp. 477-484. [Online]. Available: http://doi.acm.org/10.1145/1148170.1148253
41. K.-W. Chang, C.-J. Hsieh, and C.-J. Lin, "Coordinate descent method for large-scale l2-loss linear support vector machines," J. Mach. Learn. Res., vol. 9, pp. 1369-1398, Jun. 2008. [Online]. Available: http://dl.acm.org/citation.cfm?id=1390681.1442778
42. R.-E. Fan, K.-W. Chang, C.-J. Hsieh, X.-R. Wang, and C.-J. Lin, "Liblinear: A library for large linear classification," J. Mach. Learn. Res., vol. 9, pp. 1871-1874, Jun. 2008. [Online]. Available: http://dl.acm.org/citation.cfm?id=1390681.1442794
43. S. Shu, L. Ren, Y. Ding, K. Hao, and R. Jiang, "SVM optimization algorithm based on dynamic clustering and ensemble learning for large scale dataset," in Proc. IEEE Int. Conf. Syst. Man Cybern. (SMC), San Diego, CA, USA, 2014, pp. 2278-2283
44. C.-J. Hsieh, S. Si, and I. S. Dhillon, "A divide-and-conquer solver for kernel support vector machines," in Proc. ICML, Beijing, China, 2014, pp. 566-574
45. T. Sun, H. Wang, Y. Shen, and J. Wu, "Accelerating support vector machine learning with GPU-based mapreduce," in Proc. IEEE Int. Conf. Syst. Man Cybern., Hong Kong, Oct. 2015, pp. 876-881
46. H. Liu, F. Sun, D. Guo, B. Fang, and Z. Peng, "Structured output-associated dictionary learning for haptic understanding," IEEE Trans. Syst., Man, Cybern., Syst., to be published, doi: 10.1109/TSMC.2016.2635141
47. H. Liu, J. Qin, F. Sun, and D. Guo, "Extreme kernel sparse learning for tactile object recognition," IEEE Trans. Cybern., to be published, doi: 10.1109/TCYB.2016.2614809
48. A. Rahimi and B. Recht, "Random features for large-scale kernel machines," in Proc. 20th Int. Conf. Neural Inf. Process. Syst. (NIPS), Vancouver, BC, Canada, 2007, pp. 1177-1184. [Online]. Available: http://dl.acm.org/citation.cfm?id=2981562.2981710
49. Q. Le, T. Sarlos, and A. Smola, "Fastfood-Computing Hilbert space expansions in loglinear time," in Proc. 30th Int. Conf. Mach. Learn., vol. 28. Jun. 2013, pp. 244-252. [Online]. Available: http://proceedings.mlr.press/v28/le13.html
50. B. Dai et al., "Scalable kernel methods via doubly stochastic gradients," in Proc. 27th Int. Conf. Neural Inf. Process. Syst. (NIPS), 2014, pp. 3041-3049. [Online]. Available: http://dl.acm.org/ citation.cfm?id=2969033.2969166
51. V. N. Vapnik, Statistical Learning Theory, 1st ed. New York, NY, USA: Wiley, Sep. 1998. [Online]. Available: http://www.amazon.com/exec/obidos/redirect?tag=citeulike07-20&path=ASIN/0471030031
52. C. J. Burges, "A tutorial on support vector machines for pattern recognition," Data Min. Knowl. Discov., vol. 2, no. 2, pp. 121-167, 1998. [Online]. Available: http://dx.doi.org/10.1023/A:1009715923555
53. B. Schölkopf, R. Herbrich, and A. J. Smola, "A generalized representer theorem," in Proc. 14th Annu. Conf. Comput. Learn. Theory 5th Eur. Conf. Comput. Learn. Theory (COLT), London, U.K., 2001, pp. 416-426. [Online]. Available: http://dl.acm.org/citation.cfm?id= 648300.755324
54. F. Dinuzzo and B. Schölkopf, "The representer theorem for Hilbert spaces: A necessary and sufficient condition," in Proc. 25th Int. Conf. Neural Inf. Process. Syst. (NIPS), 2012, pp. 189-196. [Online]. Available: http://dl.acm.org/citation.cfm?id=2999134.2999156
55. R. E. Edwards, "Fourier analysis on groups," J. London Math. Soc., vol. 1-39, pp. 187-188, 1964, doi: 10.1112/jlms/s1-39.1.187
56. A. Rudi, R. Camoriano, and L. Rosasco, "Generalization properties of learning with random features," unpublished paper, 2017. [Online]. Available: https://arxiv.org/abs/1602.04474
57. D. J. Sutherland and J. Schneider, "On the error of random Fourier features," in Proc. UAI, 2015, pp. 862-871. [Online]. Available: http://dl.acm.org/citation.cfm?id=3020847.3020936
58. T. Yang, Y.-F. Li, M. Mahdavi, R. Jin, and Z.-H. Zhou, "Nyström method vs random Fourier features: A theoretical and empirical comparison," in Proc. 25th Int. Conf. Neural Inf. Process. Syst. (NIPS), 2012, pp. 476-484. [Online]. Available: http://dl.acm.org/citation.cfm?id=2999134.2999188
59. F. Pedregosa et al., "Scikit-learn: Machine learning in python," J. Mach. Learn. Res., vol. 12, pp. 2825-2830, Nov. 2011. [Online]. Available: http://dl.acm.org/citation.cfm?id=1953048.2078195
60. C. M. Bishop, Pattern Recognition and Machine Learning (Information Science and Statistics). New York, NY, USA: Springer-Verlag, 2006
61. K. Crammer and Y. Singer, "On the algorithmic implementation of multiclass kernel-based vector machines," J. Mach. Learn. Res., vol. 2, pp. 265-292, Mar. 2002. [Online]. Available: http://dl.acm.org/citation.cfm?id=944790.944813
62. H. B. Mann and D. R. Whitney, "On a test of whether one of two random variables is stochastically larger than the other," Ann. Math. Stat., vol. 18, no. 1, pp. 50-60, 1947
63. M. P. Fay and M. A. Proschan, "Wilcoxon-mann-whitney or t-test? on assumptions for hypothesis tests and multiple interpretations of decision rules," Stat. Surv., vol. 4, pp. 1-39, 2010. [Online]. Available: http://projecteuclid.org/all/euclid.ssu