Large-scale approximate kernel canonical correlation analysis

4th International Conference on Learning Representations, ICLR 2016 - Conference Track Proceedings

Volumn , Issue , 2016, Pages

  Wang, Weiran ^a   Livescu, Karen ^a  

^a TOYOTA TECHNOLOGICAL INSTITUTE AT CHICAGO   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CLUSTERING ALGORITHMS; CORRELATION METHODS; EIGENVALUES AND EIGENFUNCTIONS; MATHEMATICAL TRANSFORMATIONS; OPTIMIZATION;

APPROXIMATION TECHNIQUES; CLOSED FORM SOLUTIONS; COMPUTATIONAL REQUIREMENTS; HIGH DIMENSIONAL DATA; KERNEL CANONICAL CORRELATION ANALYSIS; LARGE-SCALE PROBLEM; LEARNING TECHNIQUES; STOCHASTIC OPTIMIZATION ALGORITHM;

LEARNING SYSTEMS;

EID: 85083950557     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (84)

1. Akaho, Shotaro. A kernel method for canonical correlation analysis. In Proceedings of the International Meeting of the Psychometric Society (IMPS2001), 2001.
2. Anderson, T. W. An Introduction to Multivariate Statistical Analysis. third edition, 2003.
3. Andrew, Galen, Arora, Raman, Bilmes, Jeff, and Livescu, Karen. Deep canonical correlation analysis. In Proc. of the 30th Int. Conf. Machine Learning (ICML 2013), pp. 1247–1255, 2013.
4. Arora, Raman and Livescu, Karen. Kernel CCA for multi-view learning of acoustic features using articulatory measurements. In Symposium on Machine Learning in Speech and Language Processing (MLSLP), 2012.
5. Arora, Raman and Livescu, Karen. Multi-view CCA-based acoustic features for phonetic recognition across speakers and domains. In Proc. of the IEEE Int. Conf. Acoustics, Speech and Sig. Proc. (ICASSP’13), 2013.
6. Arora, Raman, Cotter, Andy, Livescu, Karen, and Srebro, Nati. Stochastic optimization for PCA and PLS. In 50th Annual Allerton Conference on Communication, Control, and Computing, pp. 861–868, 2012.
7. Arora, Raman, Cotter, Andy, and Srebro, Nati. Stochastic optimization of PCA with capped MSG. In Advances in Neural Information Processing Systems (NIPS), volume 26, pp. 1815–1823, 2013.
8. Avron, Haim, Boutsidis, Christos, Toledo, Sivan, and Zouzias, Anastasios. Efficient dimensionality reduction for canonical correlation analysis. In Proc. of the 30th Int. Conf. Machine Learning (ICML 2013), pp. 347–355, 2013.
9. Bach, Francis R. and Jordan, Michael I. Kernel independent component analysis. Journal of Machine Learning Research, 3:1–48, 2002.
10. Bach, Francis R. and Jordan, Michael I. A probabilistic interpretation of canonical correlation analysis. Technical Report 688, Dept. of Statistics, University of California, Berkeley, 2005.
11. Balsubramani, Akshay, Dasgupta, Sanjoy, and Freund, Yoav. The fast convergence of incremental PCA. In Advances in Neural Information Processing Systems (NIPS), volume 26, pp. 3174–3182, 2013.
12. Bie, Tijl De and Moor, Bart De. On the regularization of canonical correlation analysis. www.esat.kuleuven.ac.be/sista-cosic-docarch/, 2003. Int. Sympos. ICA and BSS.
13. Blaschkoa, Matthew B., Sheltonb, Jacquelyn A., Bartelsc, Andreas, Lamperte, Christoph H., and Gretton, Arthur. Semi-supervised kernel canonical correlation analysis with application to human fMRI. Pattern Recognition Letters, 32(11):1572–1583, 2011.
14. Borga, Magnus. Canonical correlation: A tutorial. 2001.
15. Bottou, Leon and Bousquet, Olivier. The tradeoffs of large scale learning. In Advances in Neural Information Processing Systems (NIPS), volume 20, pp. 161–168, 2008.
16. Chaudhuri, Kamalika, Kakade, Sham M., Livescu, Karen, and Sridharan, Karthik. Multi-view clustering via canonical correlation analysis. In Proc. of the 26th Int. Conf. Machine Learning (ICML’09), pp. 129–136, 2009.
17. Chechik, Gal, Globerson, Amir, Tishby, Naftali, and Weiss, Yair. Information bottleneck for Gaussian variables. Journal of Machine Learning Research, 6:165–188, 2005.
18. Cotter, Andrew, Keshet, Joseph, and Srebro, Nathan. Explicit approximations of the gaussian kernel. arXiv:1109.4603 [cs.AI], September 21 2011.
19. Dhillon, Paramveer, Foster, Dean, and Ungar, Lyle. Multi-view learning of word embeddings via CCA. In Advances in Neural Information Processing Systems (NIPS), volume 24, pp. 199–207, 2011.
20. Faruqui, Manaal and Dyer, Chris. Improving vector space word representations using multilingual correlation. In Proceedings of EACL, 2014.
21. Foster, Dean P., Johnson, Rie, Kakade, Sham M., and Zhang, Tong. Multi-view dimensionality reduction via canonical correlation analysis. 2009.
22. Fukumizu, Kenji, Bach, Francis R., and Gretton, Arthur. Statistical consistency of kernel canonical correlation analysis. Journal of Machine Learning Research, 8:361–383, 2007.
23. Gittens, Alex and Mahoney, Michael. Revisiting the nystrom method for improved large-scale machine learning. In Proc. of the 30th Int. Conf. Machine Learning (ICML 2013), pp. 567–575, 2013.
24. Golub, Gene H. and van Loan, Charles F. Matrix Computations. third edition, 1996.
25. Golub, Gene H. and Zha, Hongyuan. Linear Algebra for Signal Processing, volume 69 of The IMA Volumes in Mathematics and its Applications, chapter The Canonical Correlations of Matrix Pairs and their Numerical Computation, pp. 27–49. 1995.
26. Haghighi, Aria, Liang, Percy, Berg-Kirkpatrick, Taylor, and Klein, Dan. Learning bilingual lexicons from monolingual corpora. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL 2008), pp. 771–779, 2008.
27. Halko, Nathan, Martinsson, Per-Gunnar, Shkolnisky, Yoel, and Tygert, Mark. An algorithm for the principal component analysis of large data sets. SIAM J. Sci. Comput., 33(5):2580–2594, 2011.
28. Hamid, Raffay, Xiao, Ying, Gittens, Alex, and Decoste, Dennis. Compact random feature maps. In Proc. of the 31st Int. Conf. Machine Learning (ICML 2014), pp. 19–27, 2014.
29. Hardoon, David R., Szedmak, Sandor, and Shawe-Taylor, John. Canonical correlation analysis: An overview with application to learning methods. Neural Computation, 16(12):2639–2664, 2004.
30. Hardoon, David R., Mouräo-Miranda, Janaina, Brammer, Michael, and Shawe-Taylor, John. Unsupervised analysis of fMRI data using kernel canonical correlation. NeuroImage, 37(4):1250–1259, 2007.
31. Hermansky, Hynek, Ellis, Daniel P. W., and Sharma, Sangita. Tandem connectionist feature extraction for conventional HMM systems. In Proc. of the IEEE Int. Conf. Acoustics, Speech and Sig. Proc. (ICASSP’00), pp. 1635–1638, 2000.
32. Hodosh, Micah, Young, Peter, and Hockenmaier, Julia. Framing image description as a ranking task: Data, models and evaluation metrics. Journal of Artificial Intelligence Research, 47:853–899, 2013.
33. Hotelling, Harold. Relations between two sets of variates. Biometrika, 28(3/4):321–377, 1936.
34. Huang, Po-Sen, Avron, Haim, Sainath, Tara, Sindhwani, Vikas, and Ramabhadran, Bhuvana. Kernel methods match deep neural networks on TIMIT: Scalable learning in high-dimensional random Fourier spaces. In Proc. of the IEEE Int. Conf. Acoustics, Speech and Sig. Proc. (ICASSP’14), pp. 205–209, 2014.
35. Kakade, Sham M. and Foster, Dean P. Multi-view regression via canonical correlation analysis. In Proc. of the 20th Annual Conference on Learning Theory (COLT’07), number 4539 in Lecture Notes in Computer Science, pp. 82–96, 2007.
36. Kar, Purushottam and Karnick, Harish. Random feature maps for dot product kernels. In Proc. of the 15th Int. Workshop on Artificial Intelligence and Statistics (AISTATS 2012), pp. 583–591, 2012.
37. Kim, Tae-Kyun, Kittler, Josef, and Cipolla, Roberto. Discriminative learning and recognition of image set classes using canonical correlations. IEEE Trans. Pattern Analysis and Machine Intelligence, 29(6):1005–1018, 2007.
38. Krasulina, T. P. A method of stochastic approximation for the determination of the least eigenvalue of a symmetric matrix. USSR Computational Mathematics and Mathematical Physics, 9(6):189–195, 1969.
39. Kumar, Sanjiv, Mohri, Mehryar, and Talwalkar, Ameet. Sampling methods for the nyström method. Journal of Machine Learning Research, 13:981–1006, 2012.
40. Lai, P. L. and Fyfe, C. Kernel and nonlinear canonical correlation analysis. Int. J. Neural Syst., 10(5):365–377, 2000.
41. Le, Quoc, Sarlos, Tamas, and Smola, Alexander. Fastfood - computing Hilbert space expansions in loglinear time. In Proc. of the 30th Int. Conf. Machine Learning (ICML 2013), pp. 244–252, 2013.
42. Li, M., Kwok, J. T., and Lu, B. Making large-scale Nyström approximation possible. In Proc. of the 27th Int. Conf. Machine Learning (ICML 2010), pp. 631–638, 2010.
43. Li, Yaoyong and Shawe-Taylor, John. Using KCCA for japanese-english cross-language information retrieval and classification. Journal of Intelligent Information Systems, 27(2):117–133, 2005.
44. Loosli, Gaëlle, Canu, Stéphane, and Bottou, Léon. Training invariant support vector machines using selective sampling. In Large Scale Kernel Machines, pp. 301–320. 2007.
45. Lopez-Paz, David, Hennig, Philipp, and Schölkopf, Bernhard. The randomized dependence coefficient. In Advances in Neural Information Processing Systems (NIPS), volume 26, pp. 1–9, 2013.
46. Lopez-Paz, David, Sra, Suvrit, Smola, Alex, Ghahramani, Zoubin, and Schoelkopf, Bernhard. Randomized nonlinear component analysis. In Proc. of the 31st Int. Conf. Machine Learning (ICML 2014), pp. 1359–1367, 2014.
47. Lu, Ang, Wang, Weiran, Bansal, Mohit, Gimpel, Kevin, and Livescu, Karen. Deep multilingual correlation for improved word embeddings. In The 2015 Conference of the North American Chapter of the Association for Computational Linguistics - Human Language Technologies (NAACL-HLT 2015), 2015a.
48. Lu, Yichao and Foster, Dean P. Large scale canonical correlation analysis with iterative least squares. In Advances in Neural Information Processing Systems (NIPS), volume 27, pp. 91–99, 2014.
49. Lu, Zhiyun, May, Avner, Liu, Kuan, Garakani, Alireza Bagheri, Guo, Dong, Bellet, Aurélien, Fan, Linxi, Collins, Michael, Kingsbury, Brian, Picheny, Michael, and Sha, Fei. How to scale up kernel methods to be as good as deep neural nets. arXiv:1411.4000 [cs.LG], 2015b.
50. Ma, Zhuang, Lu, Yichao, and Foster, Dean. Finding linear structure in large datasets with scalable canonical correlation analysis. In Proc. of the 32st Int. Conf. Machine Learning (ICML 2015), pp. 169–178, 2015.
51. Makur, Anuran, Kozynski, Fabían, Huang, Shao-Lun, and Zheng, Lizhong. An efficient algorithm for information decomposition and extraction. In 53nd Annual Allerton Conference on Communication, Control and Computing, 2015.
52. Melzer, Thomas, Reiter, Michael, and Bischof, Horst. Nonlinear feature extraction using generalized canonical correlation analysis. In Proc. of the 11th Int. Conf. Artificial Neural Networks (ICANN’01), pp. 353–360, 2001.
53. Mitliagkas, Ioannis, Caramanis, Constantine, and Jain, Prateek. Memory limited, streaming PCA. In Advances in Neural Information Processing Systems (NIPS), volume 26, pp. 2886–2894, 2013.
54. Montanarella, Luca, Bassani, Maria Rosa, and Bréas, Olivier. Chemometric classification of some European wines using pyrolysis mass spectrometry. Rapid Communications in Mass Spectrometry, 9(15):1589–1593, 1995.
55. Nocedal, Jorge and Wright, Stephen J. Numerical Optimization. Springer Series in Operations Research and Financial Engineering. second edition, 2006.
56. Oja, Erkki and Karhunen, Juha. On stochastic approximation of the eigenvectors and eigenvalues of the expectation of a random matrix. J. Math. Anal. Appl., 106(1):69–84, 1985.
57. Pennington, Jeffrey, Yu, Felix, and Kumar, Sanjiv. Spherical random features for polynomial kernels. In Cortes, C., Lawrence, N. D., Lee, D. D., Sugiyama, M., and Garnett, R. (eds.), Advances in Neural Information Processing Systems (NIPS), volume 28, pp. 1837–1845. MIT Press, Cambridge, MA, 2015.
58. Povey, Daniel, Ghoshal, Arnab, Boulianne, Gilles, Burget, Lukas, Glembek, Ondrej, Goel, Nagendra, Han-nemann, Mirko, Motlicek, Petr, Qian, Yanmin, Schwarz, Petr, Silovsky, Jan, Stemmer, Georg, and Vesely, Karel. The Kaldi speech recognition toolkit. In Proc. of the 2011 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU 2011), 2011.
59. Rahimi, Ali and Recht, Ben. Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems (NIPS), volume 20, pp. 1177–1184, 2008.
60. Rahimi, Ali and Recht, Benjamin. Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. In Advances in Neural Information Processing Systems (NIPS), volume 21, pp. 1313–1320, 2009.
61. Roux, Nicolas Le, Schmidt, Mark, and Bach, Francis. A stochastic gradient method with an exponential convergence rate for finite training sets. In Advances in Neural Information Processing Systems (NIPS), volume 25, pp. 2672–2680, 2012.
62. Rudin, Walter. Fourier Analysis on Groups. 1994.
63. Rudzicz, Frank. Correcting errors in speech recognition with articulatory dynamics. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pp. 60–68. Association for Computational Linguistics, 2010.
64. Schölkopf, Bernhard and Smola, Alexander J. Learning with Kernels. Support Vector Machines, Regularization, Optimization, and Beyond. Adaptive Computation and Machine Learning Series. 2001.
65. Shamir, Ohad. A stochastic PCA and SVD algorithm with an exponential convergence rate. In Proc. of the 32st Int. Conf. Machine Learning (ICML 2015), pp. 144–152, 2015.
66. Socher, Richard and Li, Fei-Fei. Connecting modalities: Semi-supervised segmentation and annotation of images using unaligned text corpora. In Proc. of the 2010 IEEE Computer Society Conf. Computer Vision and Pattern Recognition (CVPR’10), pp. 966–973, 2010.
67. Sutskever, Ilya, Martens, James, Dahl, George, and Hinton, Geoffrey. On the importance of initialization and momentum in deep learning. In Proc. of the 30th Int. Conf. Machine Learning (ICML 2013), pp. 1139–1147, 2013.
68. Vinod, H. D. Canonical ridge and econometrics of joint production. Journal of Econometrics, 4(2):147–166, 1976.
69. Vinokourov, Alexei, Cristianini, Nello, and Shawe-Taylor, John. Inferring a semantic representation of text via cross-language correlation analysis. In Advances in Neural Information Processing Systems (NIPS), volume 15, pp. 1497–1504, 2003.
70. Wang, Weiran, Arora, Raman, Livescu, Karen, and Bilmes, Jeff. Unsupervised learning of acoustic features via deep canonical correlation analysis. In Proc. of the IEEE Int. Conf. Acoustics, Speech and Sig. Proc. (ICASSP’15), 2015a.
71. Wang, Weiran, Arora, Raman, Livescu, Karen, and Bilmes, Jeff. On deep multi-view representation learning. In Proc. of the 32st Int. Conf. Machine Learning (ICML 2015), pp. 1083–1092, 2015b.
72. Wang, Weiran, Arora, Raman, Srebro, Nati, and Livescu, Karen. Stochastic optimization for deep cca via nonlinear orthogonal iterations. In 53nd Annual Allerton Conference on Communication, Control and Computing, 2015c.
73. Warmuth, Manfred K. and Kuzmin, Dima. Randomized online PCA algorithms with regret bounds that are logarithmic in the dimension. Journal of Machine Learning Research, 9:2287–2320, 2008.
74. Westbury, John R. X-Ray Microbeam Speech Production Database User’s Handbook Version 1.0, 1994.
75. Williams, Christopher K. I. and Seeger, Matthias. Using the Nyström method to speed up kernel machines. In Advances in Neural Information Processing Systems (NIPS), volume 13, pp. 682–688, 2001.
76. Witten, Daniela M., Tibshirani, Robert, and Hastie, Trevor. A penalized matrix decomposition, with applications to sparse principal components and canonical correlation analysis. Biostatistics, 10(3):515–534, 2009.
77. Xie, Bo, Liang, Yingyu, and Song, Le. Scale up nonlinear component analysis with doubly stochastic gradients. In Cortes, C., Lawrence, N. D., Lee, D. D., Sugiyama, M., and Garnett, R. (eds.), Advances in Neural Information Processing Systems (NIPS), volume 28, pp. 2332–2340. MIT Press, Cambridge, MA, 2015.
78. Yamanishi, Yoshihiro, Vert, Jean-Philippe, and Kanehisa, Minoru. Kernel Methods in Computational Biology, chapter Heterogeneous Data Comparison and Gene Selection with Kernel Canonical Correlation Analysis, pp. 209–229. MIT Press, 2004.
79. Yang, Jiyan, Sindhwani, Vikas, Avron, Haim, and Mahoney, Michael. Quasi-Monte Carlo feature maps for shift-invariant kernels. In Proc. of the 31st Int. Conf. Machine Learning (ICML 2014), pp. 485–493, 2014.
80. Yang, Tianbao, Li, Yu-Feng, Mahdavi, Mehrdad, Jin, Rong, and Zhou, Zhi-Hua. Nyström method vs random Fourier features: A theoretical and empirical comparison. In Advances in Neural Information Processing Systems (NIPS), volume 25, pp. 476–484, 2012.
81. Yger, Florian, Berar, Maxime, Gasso, Gilles, and Rakotomamonjy, Alain. Adaptive canonical correlation analysis based on matrix manifolds. In Proc. of the 29th Int. Conf. Machine Learning (ICML 2012), pp. 1071–1078, 2012.
82. Young, Steve J., Kernshaw, Dan, Odell, Julian, Ollason, Dave, Valtchev, Valtcho, and Woodland, Phil. The HTK book version 2.2. Technical report, Entropic, Ltd., 1999.
83. Zhang, Kai and Kwok, James T. Density-weighted Nyström method for computing large kernel eigen-systems. Neural Computation, 21(1):121–146, 2009.
84. Zhang, Kai and Kwok, James T. Clustered Nyström method for large scale manifold learning and dimension reduction. IEEE Trans. Neural Networks, 21(10):1576–1587, 2010.