Linearithmic time sparse and convex maximum margin clustering

IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics

Volumn 42, Issue 6, 2012, Pages 1669-1692

  Zhang, Xiao Lei ^a   Wu, Ji ^a  

^a TSINGHUA UNIVERSITY   (China)

Author keywords

Clustering; maximum margin; multiple kernel learning (MKL); unsupervised learning; voice activity detection (VAD)

Indexed keywords

CLUSTERING; CLUSTERING METHODS; CONVEX OPTIMIZATION PROBLEMS; CUTTING PLANE ALGORITHMS; INTEGER MATRICES; KEYPOINTS; LINEAR TIME; LINEAR TIME COMPLEXITY; MAXIMUM MARGIN; MODEL TRAINING; MULTI-CLASS; MULTIPLE-KERNEL LEARNING (MKL); NONCONVEX; NONCONVEX OPTIMIZATION PROBLEM; NONLINEAR KERNELS; REAL WORLD DATA; TIME COMPLEXITY; VOICE ACTIVITY DETECTION;

ALGORITHMS; CONVEX OPTIMIZATION; SPEECH RECOGNITION; UNSUPERVISED LEARNING; VIRTUAL REALITY;

ITERATIVE METHODS;

EID: 84869505051     PISSN: 10834419     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSMCB.2012.2197824     Document Type: Article

References (102)

1. P. Maji, "Fuzzy-rough supervised attribute clustering algorithm and classification of microarray data," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 41, no. 1, pp. 222-233, Feb. 2011.
2. D. Wang, T. Li, S. Zhu, and Y. Gong, "Ihelp: An intelligent online helpdesk system," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 41, no. 1, pp. 173-182, Feb. 2011.
3. J. MacQueen, "Some methods for classification and analysis of multivariate observations," in Proc. 5th Berkeley Symp. Math. Stat. Prob., 1967, vol. 1, pp. 281-297.
4. J. A. Hartigan andM. A.Wong, "A k-means clustering algorithm," Appl. Stat., vol. 28, pp. 100-108, 1979.
5. C. Ding and X. He, "K-means clustering via principal component analysis," in Proc. 21st Int. Conf. Mach. Learn., 2004, pp. 225-232. (Pubitemid 40290812)
6. G. J. McLachlan and D. Peel, Finite Mixture Models. Hoboken, NJ: Wiley-Interscience, 2000.
7. X. L. Xie and G. Beni, "A validity measure for fuzzy clustering," IEEE Trans. Pattern Anal. Mach. Intell., vol. 13, no. 8, pp. 841-847, Aug. 1991. (Pubitemid 21704193)
8. I. Gath and A. B. Geva, "Unsupervised optimal fuzzy clustering," IEEE Trans. Pattern Anal. Mach. Intell., vol. 11, no. 7, pp. 773-780, Jul. 1989.
9. J. Shi and J. Malik, "Normalized cuts and image segmentation," IEEE Trans. Pattern Anal. Mach. Intell., vol. 22, no. 8, pp. 888-905, Aug. 2000.
10. A. Ng,M. Jordan, and Y.Weiss, "On spectral clustering: Analysis and an algorithm," in Proc. Adv. Neural Inf. Process. Syst., 2001, pp. 849-856.
11. R. Kannan, S. Vempala, and A. Vetta, "On clusterings: Good, bad and spectral," J. ACM, vol. 51, no. 3, pp. 497-515, May 2004.
12. W. Hu, W. Hu, N. Xie, and S. Maybank, "Unsupervised active learning based on hierarchical graph-theoretic clustering," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 39, no. 5, pp. 1147-1161, Oct. 2009.
13. K. Fukunaga and L. Hostetler, "The estimation of the gradient of a density function, with applications in pattern recognition," IEEE Trans. Inf. Theory, vol. IT-21, no. 1, pp. 32-40, Jan. 1975.
14. X. T. Yuan and S. Z. Li, "Stochastic gradient kernel density modeseeking," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2009, pp. 1926-1931.
15. X. T. Yuan, B. G. Hu, and R. He, "Agglomerative mean-shift clustering," IEEE Trans. Knowl. Data Eng., vol. 24, no. 2, pp. 209-219, Feb. 2012.
16. L. Xu, J. Neufeld, B. Larson, and D. Schuurmans, "Maximum margin clustering," in Proc. Adv. Neural Inf. Process. Syst., 2005, vol. 17, pp. 1537-1544.
17. C. Cortes and V. Vapnik, "Support-vector networks," Mach. Learn., vol. 20, no. 3, pp. 273-297, Sep. 1995.
18. A. Ben-Hur, D. Horn, H. Siegelmann, and V. Vapnik, "A support vector clustering method," in Proc. 15th Int. Conf. Pattern Recog., 2000, vol. 2, pp. 724-727.
19. A. Ben-Hur, D. Horn, H. T. Siegelmann, and V. Vapnik, "Support vector clustering," J. Mach. Learn. Res., vol. 2, pp. 125-137, Dec. 2001.
20. W. Jeen-Shing and C. Jen-Chieh, "A cluster validity measure with outlier detection for support vector clustering," IEEE Trans. Syst.,Man, Cybern. B, Cybern., vol. 38, no. 1, pp. 78-89, Feb. 2008.
21. H. Valizadegan and R. Jin, "Generalized maximum margin clustering and unsupervised kernel learning," in Proc. Adv. Neural Inf. Process. Syst., 2007, vol. 19, pp. 1417-1425.
22. L. Xu and D. Schuurmans, "Unsupervised and semi-supervised multiclass support vector machines," in Proc. 20th Nat. Conf. Artif. Intell., 2005, vol. 2, pp. 904-910. (Pubitemid 43006723)
23. F. Gieseke, T. Pahikkala, and O. Kramer, "Fast evolutionary maximum margin clustering," in Proc. 26th Int. Conf. Mach. Learn., 2009, pp. 361-368.
24. K. Zhang, I. W. Tsang, and J. T. Kwok, "Maximum margin clustering made practical," in Proc. 24th Int. Conf. Mach. Learn., 2007, pp. 1119-1126.
25. K. Zhang, I. W. Tsang, and J. T. Kwok, "Maximum margin clustering made practical," IEEE Trans. Neural Netw., vol. 20, no. 4, pp. 583-596, Apr. 2009.
26. B. Zhao, F. Wang, and C. Zhang, "Efficient maximum margin clustering via cutting plane algorithm," in Proc. 8th SIAM Int. Conf. Data Min., 2008, pp. 751-762.
27. F. Wang, B. Zhao, and C. S. Zhang, "Linear time maximum margin clustering," IEEE Trans. Neural Netw., vol. 21, no. 2, pp. 319-332, Feb. 2010.
28. Y. Hu, J. Wang, N. Yu, and X. S. Hua, "Maximum margin clustering with pairwise constraints," in Proc. 8th IEEE Int. Conf. Data Min., 2008, pp. 253-262.
29. J. Wu and X. L. Zhang, "Sparse kernel maximum margin clustering," Neural Netw. World, vol. 21, no. 6, pp. 551-574, 2011.
30. H. Zeng and Y. M. Cheung, "Semi-supervised maximum margin clustering with pairwise constraints," IEEE Trans. Knowl. Data Eng., vol. 24, no. 5, pp. 926-939, May 2012.
31. A. L. Yuille and A. Rangarajan, "The concave-convex procedure," Neural Comput., vol. 15, no. 4, pp. 915-936, Apr. 2003. (Pubitemid 37049812)
32. A. J. Smola, S. V. N. Vishwanathan, and T. Hofmann, "Kernel methods for missing variables," in Proc. 10th Int. Workshop Artif. Intell. Stat., 2005, pp. 325-332.
33. J. E. Kelley, "The cutting-plane method for solving convex programs," J. Soc. Ind. Appl. Math., vol. 8, no. 4, pp. 703-712, Dec. 1960.
34. T. Joachims, "Training linear SVMs in linear time," in Proc. 12th ACM Int. Conf. Knowl. Disc. Data Min., 2006, pp. 217-226. (Pubitemid 44535518)
35. B. Schölkopf, A. Smola, and K. R. Müller, "Kernel principal component analysis," in Proc. Artif. Neural Netw., 1997, pp. 583-588.
36. R. He, B. G. Hu, W. S. Zheng, and X. Kong, "Robust principal component analysis based on maximum correntropy criterion," IEEE Trans. Image Process., vol. 20, no. 6, pp. 1485-1494, Jun. 2011.
37. O. Chapelle and A. Zien, "Semi-supervised classification by low density separation," in Proc. 10th Int. Workshop Artif. Intell. Stat., 2005, pp. 1-8.
38. B. Schölkopf and A. J. Smola, Learning With Kernels. Cambridge, MA: MIT Press, 2002.
39. L. N. Trefethen and D. Bau, Numerical Linear Algebra. Philadelphia, PA: SIAM, 1997.
40. B. Zhao, J. T. Kwok, and C. S. Zhang, "Multiple kernel clustering," in Proc. 9th SIAM Int. Conf. Data Min., 2009, pp. 638-649.
41. Y. F. Li, I. W. Tsang, J. T. Kwok, and Z. H. Zhou, "Tighter and convex maximum margin clustering," in Proc. 12th Int. Conf. Artif. Intell. Stat., Clearwater Beach, FL, 2009, pp. 344-351.
42. S. P. Boyd and L. Vandenberghe, Convex Optimization. Cambridge, U.K.: Cambridge Univ. Press, 2004.
43. A. Rakotomamonjy, F. Bach, S. Canu, and Y. Grandvalet, "SimpleMKL," J. Mach. Learn. Res., vol. 9, pp. 2491-2521, Nov. 2008.
44. A. J. Smola and B. Schölkopf, "A tutorial on support vector regression," Stat. Comput., vol. 14, no. 3, pp. 199-222, Aug. 2004. (Pubitemid 39063488)
45. V. N. Vapnik, The Nature of Statistical Learning Theory. NewYork: Springer-Verlag, 2000.
46. C. H. Teo, A. Smola, S. V. N. Vishwanathan, and Q. V. Le, "A scalable modular convex solver for regularized risk minimization," in Proc. 13th ACM SIGKDD Int. Conf. Knowl. Disc. Data Min., 2007, pp. 727-736. (Pubitemid 350229258)
47. V. Franc and S. Sonnenburg, "Optimized cutting plane algorithm for support vector machines," in Proc. 25th Int. Conf. Mach. Learn., 2008, pp. 320-327.
48. C. J. C. Burges, "Simplified support vector decision rules," in Proc. Int. Conf. Mach. Learn., 1996, pp. 71-77.
49. B. Zhao, F. Wang, and C. Zhang, "Efficient multiclass maximum margin clustering," in Proc. 25th Int. Conf. Mach. Learn., 2008, pp. 1248-1255.
50. T. Joachims, "A support vector method for multivariate performance measures," in Proc. 22nd Int. Conf. Mach. Learn., 2005, pp. 377-384. (Pubitemid 43183356)
51. T. Joachims, T. Finley, and C. N. J. Yu, "Cutting-plane training of structural SVMs," Mach. Learn., vol. 77, no. 1, pp. 27-59, Oct. 2009.
52. G. R. G. Lanckriet, N. Cristianini, P. Bartlett, L. E. Ghaoui, and M. I. Jordan, "Learning the kernel matrix with semidefinite programming," J. Mach. Learn. Res., vol. 5, pp. 27-72, Dec. 2004.
53. F. R. Bach, G. R. G. Lanckriet, and M. I. Jordan, "Multiple kernel learning, conic duality, and the SMO algorithm," in Proc. 21th Int. Conf. Mach. Learn., 2004, pp. 6-13.
54. S. Sonnenburg, G. Rätsch, C. Schäfer, and B. Schölkopf, "Large scale multiple kernel learning," J. Mach. Learn. Res., vol. 7, pp. 1531-1565, Dec. 2006. (Pubitemid 44373694)
55. A. Zien and C. S. Ong, "Multiclass multiple kernel learning," in Proc. 24th Int. Conf. Mach. Learn., 2007, pp. 1191-1198.
56. Z. Xu, R. Jin, I. King, and M. R. Lyu, "An extended level method for efficient multiple kernel learning," in Proc. Adv. Neural Inf. Process. Syst., 2009, vol. 21, pp. 1825-1832.
57. H. Yang, Z. Xu, J. Ye, I. King, and M. R. Lyu, "Efficient sparse generalized multiple kernel learning," IEEE Trans. Neural Netw., vol. 22, no. 3, pp. 433-446, Mar. 2011.
58. M. Kloft, U. Brefeld, S. Sonnenburg, P. Laskov, K. R. Müller, and A. Zien, "Efficient and accurate LP-norm multiple kernel learning," in Proc. Adv. Neural Inf. Process. Syst., 2009, vol. 22, pp. 997-1005.
59. A. Rakotomamonjy, F. Bach, S. Canu, and Y. Grandvalet, "More efficiency in multiple kernel learning," in Proc. 24th Int. Conf. Mach. Learn., 2007, pp. 775-782.
60. C. Lemaréchal, A. Nemirovskii, and Y. Nesterov, "New variants of bundle methods," Math. Program., vol. 69, no. 1, pp. 111-147, Jul. 1995.
61. T. G. Dietterich and G. Bakiri, "Solving multiclass learning problems via error-correcting output codes," J. Aritif. Intell. Res., vol. 2, no. 1, pp. 263-286, Aug. 1994.
62. K. Cherkauer, "Human expert-level performance on a scientific image analysis task by a system using combined artificial neural networks," in Proc. Working Notes AAAI Workshop Integr. Multiple Learned Models, 1996, pp. 15-21.
63. L. Breiman, "Random forests," Mach. Learn., vol. 45, no. 1, pp. 5-32, Oct. 2001. (Pubitemid 32933532)
64. G. Martinez-Muoz, D. Hernández-Lobato, and A. Suárez, "An analysis of ensemble pruning techniques based on ordered aggregation," IEEE Trans. Pattern Anal.Mach. Intell., vol. 31, no. 2, pp. 245-259, Feb. 2009.
65. L. Chen and M. Kamel, "A generalized adaptive ensemble generation and aggregation approach for multiple classifier systems," Pattern Recognit., vol. 42, no. 5, pp. 629-644, May 2009.
66. L. Nanni and A. Lumini, "Fuzzy bagging: A novel ensemble of classifiers," Pattern Recognit., vol. 39, no. 3, pp. 488-490, Mar. 2006. (Pubitemid 41808629)
67. C. W. Hsu and C. J. Lin, "A comparison of methods for multiclass support vector machines," IEEE Trans. Neural Netw., vol. 13, no. 2, pp. 415-425, Mar. 2002.
68. E. L. Allwein, R. E. Schapire, and Y. Singer, "Reducing multiclass to binary: A unifying approach for margin classifiers," J. Mach. Learn. Res., vol. 1, pp. 113-141, Sep. 2001. (Pubitemid 33738778)
69. K. Crammer and Y. Singer, "On the learnability and design of output codes for multiclass problems," Mach. Learn., vol. 47, no. 2/3, pp. 201-233, May/Jun. 2002. (Pubitemid 34126110)
70. O. Pujol, P. Radeva, and J. Vitria, "Discriminant ECOC: A heuristic method for application dependent design of error correcting output codes," IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 6, pp. 1007-1012, Jun. 2006.
71. O. Pujol, S. Escalera, and P. Radeva, "An incremental node embedding technique for error correcting output codes," Pattern Recogn., vol. 41, no. 2, pp. 713-725, Feb. 2008. (Pubitemid 47516589)
72. S. Escalera, D. M. J. Tax, O. Pujol, P. Radeva, and R. P. W. Duin, "Subclass problem-dependent design for error-correcting output codes," IEEE Trans. Pattern Anal. Mach. Intell., vol. 30, no. 6, pp. 1041-1054, Jun. 2008. (Pubitemid 351650107)
73. S. Escalera, O. Pujol, and P. Radeva, "On the decoding process in ternary error-correcting output codes," IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 1, pp. 120-134, Jan. 2010.
74. G. Fung and O. Mangasarian, "Multicategory proximal support vector machine classifiers," Mach. Learn., vol. 59, no. 1/2, pp. 77-97, May 2005. (Pubitemid 40890413)
75. J. Weston and C. Watkins, "Support vector machines for multi-class pattern recognition," in Proc. 7th Eur. Symp. Artif. Neural Netw., 1999, vol. 4, no. 6, pp. 219-224.
76. Y. Guermeur, "Combining discriminant models with new multi-class SVMs," Pattern Anal. Appl., vol. 5, no. 2, pp. 168-179, 2002. (Pubitemid 40830130)
77. Y. Lee, Y. Lin, and G. Wahba, "Multicategory support vector machines: Theory and application to the classification of microarray data and satellite radiance data," J. Amer. Stat. Assoc., vol. 99, no. 465, pp. 67-81, Jan. 2004. (Pubitemid 38545195)
78. Y. Zhang and Z. H. Zhou, "Cost-sensitive face recognition," IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 10, pp. 1758-1769, Oct. 2010.
79. P. Chen, K. Y. Lee, T. J. Lee, Y. J. Lee, and S. Y. Huang, "Multiclass support vector classification via coding and regression," Neurocomputing, vol. 73, no. 7-9, pp. 1501-1512, Mar. 2010.
80. S. Ghorai, A. Mukherjee, and P. K. Dutta, "Discriminant analysis for fast multiclass data classification through regularized kernel function approximation," IEEE Trans. Neural Netw., vol. 21, no. 6, pp. 1020-1029, Jun. 2010.
81. R. Collobert and S. Bengio, "SVMTorch: Support vector machines for large-scale regression problems," J. Mach. Learn. Res., vol. 1, pp. 143-160, Sep. 2001. (Pubitemid 33738779)
82. W. F. Zhang, D. Q. Dai, and H. Yan, "Framelet kernels with applications to support vector regression and regularization networks," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 40, no. 4, pp. 1128-1144, Aug. 2010.
83. S. Kim and S. Boyd, "A minimax theorem with applications to machine learning, signal processing, and finance," in Proc. 46th IEEE Int. Conf. Decision, Control, 2007, pp. 751-758. (Pubitemid 351277243)
84. T. Joachims and C. N. J. Yu, "Sparse kernel SVMs via cutting-plane training," Mach. Learn., vol. 76, no. 2/3, pp. 179-193, Sep. 2009.
85. P. C. Chen, T. J. Lee, Y. J. Lee, and S. Y. Huang, Multiclass Support Vector Classification via Regression. [Online]. Available: http://www. stat.sinica.edu.tw/syhuang/papersdownload/MC-reg-121006.pdf
86. A. Benyassine, E. Shlomot, H. Y. Su, D. Massaloux, C. Lamblin, and J. P. Petit, "ITU-T Recommendation G. 729 Annex B: A silence compression scheme for use with G. 729 optimized for V. 70 digital simultaneous voice and data applications," IEEE Commun. Mag., vol. 35, no. 9, pp. 64-73, Sep. 1997. (Pubitemid 127557050)
87. Speech Processing, Transmission and Quality Aspects (STQ); Distributed Speech Recognition; Advanced Front-End Feature Extraction Algorithm; Compression Algorithms, ETSI ES 202 050, 2007.
88. C. W. Hsu, C. C. Chang, and C. J. Lin, A practical guide to support vector classification, Dept. Comput. Sci., Nat. Taiwan Univ., Taipei, Taiwan. [Online]. Available: http://www.csie.ntu.edu.tw/~cjlin/papers/guide/guide.pdf
89. D. Enqing, L. Guizhong, Z. Yatong, and Z. Xiaodi, "Applying support vector machines to voice activity detection," in Proc. Int. Conf. Signal Process., 2002, vol. 2, pp. 1124-1127.
90. J. Ramírez, P. Yélamos, J. M. Górriz, and J. C. Segura, "SVM-based speech endpoint detection using contextual speech features," Electron. Lett., vol. 42, no. 7, pp. 426-428, Mar. 2006.
91. D. Kim, K. W. Jang, and J. Chang, "A new statistical voice activity detection based on UMP test," IEEE Signal Process. Lett., vol. 14, no. 11, pp. 891-894, Nov. 2007. (Pubitemid 350138949)
92. S. I. Kang, Q. H. Jo, and J. H. Chang, "Discriminative weight training for a statistical model-based voice activity detection," IEEE Signal Process. Lett., vol. 15, pp. 170-173, 2008.
93. Q. H. Jo, J. H. Chang, J. W. Shin, and N. S. Kim, "Statistical modelbased voice activity detection using support vector machine," IET Signal Process., vol. 3, no. 3, pp. 205-210, May 2009.
94. J. W. Shin, J. H. Chang, and N. S. Kim, "Voice activity detection based on statistical models and machine learning approaches," Comput. Speech Lang., vol. 24, no. 3, pp. 515-530, Jul. 2010.
95. T. Yu and J. H. L. Hansen, "Discriminative training for multiple observation likelihood ratio based voice activity detection," IEEE Signal Process. Lett., vol. 17, no. 11, pp. 897-900, Nov. 2010.
96. J. Wu and X. L. Zhang, "Maximum margin clustering based statistical VAD with multiple observation compound feature," IEEE Signal Process. Lett., vol. 18, no. 5, pp. 283-286, May 2011.
97. J. Wu and X. L. Zhang, "An efficient voice activity detection algorithm by combining statistical model and energy detection," EURASIP J. Adv. Signal Process., vol. 2011, no. 1, pp. 18-27, Dec. 2011.
98. J.Wu and X. L. Zhang, "Efficient multiple kernel support vector machine based voice activity detection," IEEE Signal Process. Lett., vol. 18, no. 8, pp. 466-469, Aug. 2011.
99. D. Cournapeau, S. Watanabe, A. Nakamura, and T. Kawahara, "Online unsupervised classification with model comparison in the variational Bayes framework for voice activity detection," IEEE J. Sel. Topics Signal Process., vol. 4, no. 6, pp. 1071-1083, Dec. 2010.
100. D. Ying, Y. Yan, J. Dang, and F. Soong, "Voice activity detection based on an unsupervised learning framework," IEEE Trans. Audio, Speech, Lang. Process., vol. 19, no. 8, pp. 2624-2633, Nov. 2011.
101. D. Pearce and H. Hirsch, "The AURORA experimental framework for the performance evaluation of speech recognition systems under noisy conditions," in Proc. ICSLP, 2000, vol. 4, pp. 29-32.
102. S. M. Kay, Fundamentals of Statistical Signal Processing. Englewood Cliffs, NJ: Prentice-Hall, 1993.