Dimensionality-dependent generalization bounds for k-dimensional coding schemes

Neural Computation

Volumn 28, Issue 10, 2016, Pages 2213-2249

  Liu, Tongliang ^a   Tao, Dacheng ^a   Xu, Dong ^b  

^a UNIVERSITY OF TECHNOLOGY SYDNEY   (Australia)

^b UNIVERSITY OF SYDNEY   (Australia)

Author keywords

[No Author keywords available]

Indexed keywords

ERRORS; FACTORIZATION; MATRIX ALGEBRA;

DICTIONARY LEARNING; DIMENSIONAL VECTORS; GENERALIZATION BOUND; GENERALIZATION ERROR; HIGH-DIMENSIONAL FEATURE SPACE; K-MEANS CLUSTERING; NONNEGATIVE MATRIX FACTORIZATION; RECONSTRUCTION ERROR;

CODES (SYMBOLS);

EID: 84988624672     PISSN: 08997667     EISSN: 1530888X     Source Type: Journal    
DOI: 10.1162/NECO_a_00872     Document Type: Letter

References (56)

1. Abbott, L., & Dayan, P. (1999). The effect of correlated variability on the accuracy of a population code. Neural Computation, 11(1), 91-101.
2. Alexander, K. (1984). Probability inequalities for empirical processes and a law of the iterated logarithm. Annals of Probability, 12(4), 1041-1067.
3. Amiri, A., & Haykin, S. (2014). Improved sparse coding under the influence of perceptual attention. Neural Computation, 26(2), 377-420.
4. Anderberg, M. R. (1973). Cluster analysis for applications. New York: Academic Press.
5. Antos, A. (2005). Improved minimax bounds on the test and training distortion of empirically designed vector quantizers. IEEE Transactions on Information Theory, 51(11), 4022-4032.
6. Antos, A., Györfi, L., & György, A. (2005). Improved convergence rates in empirical vector quantizer design. IEEE Transactions on Information Theory, 51(11), 4013-4022.
7. Bartlett, P. L., Linder, T., & Lugosi, G. (1998). The minimax distortion redundancy in empirical quantizer design. IEEE Transactions on Information Theory, 44(5), 1802-1813.
8. Bartlett, P. L., & Mendelson, S. (2003). Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3, 463-482.
9. Biau, G., Devroye, L., & Lugosi, G. (2008).On the performance of clustering in Hilbert spaces. IEEE Transactions on Information Theory, 54(2), 781-790.
10. Boucheron, S., Lugosi, G., & Massart, P. (2013). Concentration inequalities: A nonasymptotic theory of independence. New York: Oxford University Press.
11. Chen, S. S., Donoho, D. L., & Saunders, M. A. (1999). Atomic decomposition by basis pursuit. SIAM Journal on Scientific Computing, 20(1), 33-61.
12. p(1/n). In Proceedings of the IEEE International Symposium on Information Theory. Piscataway, NJ: IEEE.
13. Cucker, F., & Smale, S. (2002). On the mathematical foundations of learning. Bulletin of the American Mathematical Society, 39(1), 1-49.
14. Devroye, L., Györfi, L., & Lugosi, G. (1996). A probabilistic theory of pattern recognition. New York: Springer.
15. Dhillon, I. S., Guan, Y., & Kulis, B. (2007).Weighted graph cuts without eigenvectors a multilevel approach. IEEE Transactions on Pattern Analysis andMachine Intelligence, 29(11), 1944-1957.
16. Ding, C., He, X., & Simon, H. D. (2005). On the equivalence of nonnegative matrix factorization and spectral clustering. In Proceedings of the SIAM International Conference on Data Mining. Philadelphia: SIAM.
17. Févotte, C., Bertin, N., & Durrieu, J.-L. (2009). Nonnegative matrix factorization with the Itakura-Saito divergence: With application to music analysis. Neural Computation, 21(3), 793-830.
18. Gillis, N., & Vavasis, S. A. (2014). Fast and robust recursive algorithms for separable nonnegative matrix factorization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(4), 698-714.
19. Gong, M., Zhang, K., Schölkopf, B., Tao, D., & Geiger, P. (2015).Discovering temporal causal relations from subsampled data. In Proceedings of the 32nd International Conference on Machine Learning (pp. 1898-1906). JMLR.
20. Gray, R. (1984). Vector quantization. IEEE Acoustics, Speech and Signal Processing Magazine, 1(2), 4-29.
21. Gribonval, R., Jenatton, R., Bach, F., Kleinsteuber, M., & Seibert, M. (2015). Sample complexity of dictionary learning and other matrix factorizations. IEEE Transactions on Information Theory, 61(6), 3469-3486.
22. Guan, N., Tao, D., Luo, Z., & Yuan, B. (2011). Manifold regularized discriminative nonnegative matrix factorization with fast gradient descent. IEEE Transactions on Image Processing, 20(7), 2030-2048.
23. Guan, N., Tao, D., Luo, Z., & Yuan, B. (2012). NeNMF: An optimal gradient method for nonnegative matrix factorization. IEEE Transactions on Signal Processing, 60(6), 2882-2898.
24. Gui, J., Sun, Z., Ji, S., Tao, D., & Tan, T. (2016). Feature selection based on structured sparsity: A comprehensive study. IEEE Transactions on Neural Networks and Learning Systems. doi:10.1109/TNNLS.2013.2551724
25. Hoeffding, W. (1963). Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58(301), 13-30.
26. Hunt, J. J., Ibbotson, M., & Goodhill, G. J. (2012). Sparse coding on the spot: Spontaneous retinal waves suffice for orientation selectivity. Neural Computation, 24(9), 2422-2433.
27. Ivana, T., & Pascal, F. (2011). Dictionary learning: What is the right representation for my signal? IEEE Signal Processing Magazine, 4(2), 27-38.
28. Kanungo, T., Mount, D.M., Netanyahu, N. S., Piatko, C.D., Silverman, R., & Wu, A. Y. (2002). An efficient k-means clustering algorithm: Analysis and implementation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(7), 881-892.
29. Lee, D. D., & Seung, H. S. (1999). Learning the parts of objects by nonnegative matrix factorization. Nature, 401(6755), 788-791.
30. Levrard, C. (2013). Fast rates for empirical vector quantization. Electronic Journal of Statistics, 7, 1716-1746.
31. Levrard, C. (2015). Nonasymptotic bounds for vector quantization in Hilbert spaces. Annals of Statistics, 43(2), 592-619.
32. Linder, T. (2000). On the training distortion of vector quantizers. IEEE Transactions on Information Theory, 46(4), 1617-1623.
33. Linder, T., Lugosi, G., & Zeger, K. (1994). Rates of convergence in the source coding theorem, in empirical quantizer design, and in universal lossy source coding. IEEE Transactions on Information Theory, 40(6), 1728-1740.
34. Liu, H., Liu, T., Wu, J., Tao, D., & Fu, Y. (2015). Spectral ensemble clustering. In Proceedings of the ACM SIGKKD International Conference on Knowledge Discovery and Data Mining (pp. 715-724). New York: ACM.
35. Liu, H., Shao, M., Li, S., & Fu, Y. (2016). Infinite ensemble for image clustering. In Proceedings of the ACM SIGKKD International Conference on Knowledge Discovery and Data Mining New York: ACM.
36. Liu, H., Wu, J., Tao, D., Zhang, Y., & Fu, Y. (2015). Dias: A disassemble-assemble framework for highly sparse text clustering. In Proceedings of the SIAM International Conference on Data Mining (pp. 766-774). Philadelphia: SIAM.
37. Liu, T., & Tao, D. (2015). On the performance of manhattan nonnegative matrix factorization. IEEE Transactions on Neural Networks and Learning Systems. doi:10.1109/TNNLS.2015.2458986
38. Liu, W., Tao, D., Cheng, J., & Tang, Y. (2014). Multiview Hessian discriminative sparse coding for image annotation. Computer Vision and Image Understanding, 118, 50-60.
39. MacQueen, J. (1967). Somemethods for classification and analysis of multivariate observations. In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability. Berkeley: University of California Press.
40. Mairal, J., Bach, F., & Ponce, J. (2012). Task-driven dictionary learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(4), 791-804.
41. Maurer, A., & Pontil, M. (2010). K-dimensional coding schemes in Hilbert spaces. IEEE Transactions on Information Theory, 56(11), 5839-5846.
42. Maurer, A., Pontil, M., & Romera-Paredes, B. (2013). Sparse coding for multitask and transfer learning. In Proceedings of the International Conference onMachine Learning. New York: ACM.
43. Mehta, N., & Gray, A. G. (2013). Sparsity-based generalization bounds for predictive sparse coding. In Proceedings of the International Conference on Machine Learning. New York: ACM.
44. Olshausen, B. A., & Field, D. J. (1996). Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381(6583), 607-609.
45. Pehlevan, C., Hu, T., & Chklovskii, D. B. (2015).AHebbian/anti-Hebbian neural network for linear subspace learning: A derivation from multidimensional scaling of streaming data. Neural Computation, 27(1), 1461-1495.
46. Pollard, D. (1982). A central limit theorem for k-means clustering. IEEE Transactions on Information Theory, 10(4), 912-926.
47. Quiroga, R. Q., Nadasdy, Z., & Ben-Shaul, Y. (2004). Unsupervised spike detection and sorting with wavelets and superparamagnetic clustering. NeuralComputation, 16(8), 1661-1687.
48. Schneider, P., Biehl, M., & Hammer, B. (2009a). Adaptive relevance matrices in learning vector quantization. Neural Computation, 21(12), 3532-3561.
49. Schneider, P., Biehl, M., & Hammer, B. (2009b). Distance learning in discriminative vector quantization. Neural Computation, 21(10), 2942-2969.
50. Talagrand, M. (1994). Sharper bounds for gaussian and empirical processes. Annals of Probability, 22(1), 28-76.
51. Vainsencher, D., Mannor, S., & Bruckstein, A. M. (2011). The sample complexity of dictionary learning. Journal of Machine Learning Research, 12, 3259-3281.
52. Vapnik, V. (2000). The nature of statistical learning theory. New York: Springer.
53. Wright, J., Yang, A. Y., Ganesh, A., Sastry, S. S., & Ma, Y. (2009). Robust face recognition via sparse representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(2), 210-227.
54. Xu, M., & Lafferty, J. D. (2012). Conditional sparse coding and groupedmultivariate regression. In Proceedings of the International Conference on Machine Learning. New York: ACM.
55. Zhang, C. (2013). Bennett type generalization bounds: Large deviation case and faster rate of convergence. In Proceedings of the Conference on Uncertainty in Artificial Intelligence. New York: ACM.
56. Zhang, T. (2002). Covering number bounds of certain regularized linear function classes. Journal of Machine Learning Research, 2, 527-550.