Incremental slow feature analysis: Adaptive low-complexity slow feature updating from high-dimensional input streams

Neural Computation

Volumn 24, Issue 11, 2012, Pages 2994-3024

  Kompella, Varun Raj ^a   Luciw, Matthew ^a   Schmidhuber, Jürgen ^a  

^a DALLE MOLLE INSTITUTE FOR ARTIFICIAL INTELLIGENCE IDSIA   (Switzerland)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84867687400     PISSN: 08997667     EISSN: 1530888X     Source Type: Journal    
DOI: 10.1162/NECO_a_00344     Document Type: Article

References (65)

1. Abut, H. (Ed.) (1990). Vector quantization. Piscataway, NJ: IEEE Press.
2. Amari, S. (1977). Neural theory of association and concept-formation. Biological Cybernetics, 26(3), 175-185.
3. Barlow, H. (2001). Redundancy reduction revisited. Network: Computation in Neural Systems, 12(3), 241-253.
4. Bergstra, J., & Bengio, Y. (2009). Slow, decorrelated features for pretraining complex cell-like networks. In D. Koller, D. Schuurmans, J. Lafferty, C. Williams, & A. Culotta (Eds.), Advances in neural information processing systems, 22 (pp. 99-107). Red Hook, NY: Curran.
5. Chatterjee, C., Kang, Z., & Roychowdhury, V. (2000). Algorithms for accelerated convergence of adaptive PCA. IEEE Transactions on Neural Networks, 11(2), 338-355.
6. Chen, T., Amari, S., & Lin, Q. (1998). A unified algorithm for principal and minor components extraction. Neural Networks, 11(3), 385-390.
7. Chen, T., Amari, S., & Murata, N. (2001). Sequential extraction of minor components. Neural Processing Letters, 13(3), 195-201.
8. Comon, P. (1994). Independent component analysis, A new concept? Signal Processing, 36, 287-314.
9. Dayan, P.,&Abbott, L. (2001). Theoretical neuroscience: Computational and mathematical modeling of neural systems. Cambridge, MA: MIT Press.
10. Doersch, C., Lee, T., Huang, G., & Miller, E. (2010). Temporal continuity learning for convolutional deep belief networks.
11. F̈oldíak, P. (1991). Learning invariance from transformation sequences. Neural Computation, 3(2), 194-200.
12. Forsythe, G., & Henrici, P. (1958). The cyclic Jacobi method for computing the principal values of a complex matrix. Palo Alto, CA: Applied Mathematics and Statistics Laboratories, Stanford University.
13. Franzius, M., Sprekeler, H., & Wiskott, L. (2007). Slowness and sparseness lead to place, head-direction, and spatial-view cells. PLoS Computational Biology, 3(8), e166.
14. Gisslen, L., Luciw, M., Graziano, V., & Schmidhuber, J. (2011). Sequential constant size compressors for reinforcement learning. In Fourth Conference on Artificial General Intelligence (AGI). Berlin: Springer.
15. Groetsch, C. W. (1998). Lanczos' generalized derivative. American Mathematical Monthly, 105(4), 320-326.
16. Grossberg, S. (1980). How does a brain build a cognitive code? Psychological Review, 87(1), 1-5.
17. Hafting, T., Fyhn, M.,Molden, S.,Moser,M., &Moser, E. (2005).Microstructure of a spatial map in the entorhinal cortex. Nature, 7052, 801-806. 3022
18. V. Kompella, M. Luciw, and J. Schmidhuber Hinton, G. (1989). Connectionist learning procedures. Artificial Intelligence, 40(1-3), 185-234.
19. Hinton, G. E. (2002). Training products of experts by minimizing contrastive divergence. Neural Comp., 14(8), 1771-1800.
20. Hyv̈arinen, A., Karhunen, J., & Oja, E. (2001). Independent component analysis. Hoboken, NJ:Wiley.
21. Jenkins, O., & Mataríc, M. (2004). A spatio-temporal extension to isomap nonlinear dimension reduction. In Proceedings of the Twenty-First International Conference on Machine Learning. New York: ACM.
22. Jolliffe, I. T. (1986). Principal component analysis. NewYork: Springer-Verlag.
23. Klapper-Rybicka, M., Schraudolph, N. N., & Schmidhuber, J. (2001). Unsupervised learning in LSTM recurrent neural networks. In Proc. Intl. Conf. on Artificial Neural Networks (pp. 684-691). New York: Springer.
24. Kohonen, T. (2001). Self-organizing maps (3rd ed.). New York: Springer-Verlag.
25. Kompella,V., Luciw, M.,&Schmidhuber, J. (2011). Incremental slow feature analysis. In International Joint Conference of Artificial Intelligence. Menlo Park, CA: AAAI Press.
26. Kompella, V. R., Pape, L., Masci, J., Frank, M., & Schmidhuber, J. (2011). Autoincsfa and vision-based developmental learning for humanoid robots. In IEEE-RAS International Conference on Humanoid Robots. Piscataway, NJ: IEEE.
27. Krasulina, T. (1970).Method of stochastic approximation in the determination of the largest eigenvalue of the mathematical expectation of random matrices. Automat.Remote Contr, 2, 215-221.
28. Kreyszig, E. (1988). Advanced engineering mathematics. Hoboken, NJ: Wiley.
29. Lee, D., & Seung, H. (1999). Learning the parts of objects by non-negative matrix factorization. Nature, 401(6755), 788-791.
30. Lee, H., Largman, Y., Pham, P., & Ng, A. (2010). Unsupervised feature learning for audio classification using convolutional deep belief networks. In D. Koller, D. Schuurmans, J. Lafferty, C. Williams, & A. Culotta (Eds.), Advances in neural information processing systems, 22 (pp. 1096-1104). Red Hook, NY: Curran.
31. Legenstein, R., Wilbert, N., & Wiskott, L. (2010). Reinforcement learning on slow features of high-dimensional input streams. PLoS Computational Biology, 6(8), e1000894.
32. Lindsẗadt, S. (1993). Comparison of two unsupervised neural network models for redundancy reduction. InM. C.Mozer, P. Smolensky, D. S. Touretzky, J. L. Elman, & A. S. Weigend (Eds.), Proc. of the 1993 Connectionist Models Summer School (pp. 308-315). Mahwah, NJ: Erlbaum.
33. Mahadevan, S., & Maggioni, M. (2007). Proto-value functions: A Laplacian framework for learning representation and control in markov decision processes. Journal of Machine Learning Research, 8, 2169-2231.
34. Mitchison, G. (1991). Removing time variation with the anti-Hebbian differential synapse. Neural Computation, 3(3), 312-320.
35. Oja, E. (1982). Simplified neuron model as a principal component analyzer. Journal of Mathematical Biology, 15(3), 267-273.
36. Oja, E. (1985).On stochastic approximation of the eigenvectors and eigenvalues of the expectation of a random matrix. Journal of Mathematical Analysis and Applications, 106, 69-84.
37. Oja, E. (1989). Neural networks, principal components, and subspaces. International Journal of Neural Systems, 1(1), 61-68.
38. Oja, E. (1992). Principal components, minor components, and linear neural networks. Neural Networks, 5(6), 927-935.
39. O'Keefe, J., & Dostrovsky, J. (1971). The hippocampus as a spatial map: Preliminary evidence from unit activity in the freely-moving rat. Brain Research, 34, 171-175.
40. O'Reilly, R., & Johnson, M. (1994). Object recognition and sensitive periods: A computational analysis of visual imprinting. Neural Computation, 6(3), 357-389.
41. Papoulis, A., Pillai, S., & Unnikrishna, S. (1965). Probability, random variables, and stochastic processes. New York: McGraw-Hill.
42. Park, M., & Choi, J. (2008). Novel incremental principal component analysis with improved performance. In Structural, syntactic, and statistical pattern recognition, (pp. 592-601). New York: Springer.
43. Peng, D., & Yi, Z. (2006). A new algorithm for sequential minor component analysis. International Journal of Computational Intelligence Research, 2(2), 207-215.
44. Peng,D., Yi, Z.,&Luo,W. (2007).Convergence analysis of a simple minor component analysis algorithm. Neural Networks, 20(7), 842-850.
45. Rolls, E. (1999). Spatial view cells and the representation of place in the primate hippocampus. Hippocampus, 9(4), 467-480.
46. Sanger, T. (1989).Optimal unsupervised learning in a single-layer linear feedforward neural network. Neural Networks, 2(6), 459-473.
47. Schmidhuber, J. (1992a). Learning complex, extended sequences using the principle of history compression. Neural Computation, 4(2), 234-242.
48. Schmidhuber, J. (1992b). Learning factorial codes by predictability minimization. Neural Computation, 4(6), 863-879.
49. Schmidhuber, J. (1992c). Learning unambiguous reduced sequence descriptions. In J. E. Moody, S. J. Hanson, & R. P. Lippman (Eds.), Advances in neural information processing systems (pp. 291-298). San Francisco: Morgan Kaufmann.
50. Schmidhuber, J. (1999). Neural predictors for detecting and removing redundant information. In H. Cruse, J. Dean,&H. Ritter (Eds.), Adaptive behavior and learning. New York: Kluwer.
51. Sprekeler, H. (2011). On the relation of slow feature analysis and Laplacian eigenmaps. Neural Computation, 23, 3287-3302.
52. Sprekeler, H., Michaelis, C., & Wiskott, L. (2007). Slowness: An objective for spiketiming- dependent plasticity? PLoS Computational Biology, 3(6), e112.
53. Sprekeler, H., Zito, T., & Wiskott, L. (2010). An extension of slow feature analysis for nonlinear blind source separation. Manuscript submitted for publication.
54. Taube, J., Muller, R., & Ranck, J. (1990). Head-direction cells recorded from the postsubiculum in freely moving rats. I. description and quantitative analysis. Journal of Neuroscience, 10(2), 420-435.
55. Tikhanoff, V., Cangelosi, A., Fitzpatrick, P., Metta, G., Natale, L., & Nori, F. (2008). An open-source simulator for cognitive robotics research: The prototype of the iCub humanoid robot simulator. In Proceedings of the 8thWorkshop on Performance Metrics for Intelligence Systems (pp. 57-61). New York: ACM.
56. Wallis, G., & Rolls, E. (1997). Invariant face and object recognition in the visual system. Progress in Neurobiology, 51(2), 167-194.
57. V. Kompella, M. Luciw, and J. Schmidhuber Wang, L., & Karhunen, J. (1996). A unified neural bigradient algorithm for robust PCA and MCA. International Journal of Neural Systems, 7(1), 53-67.
58. Weng, J., & Zhang, N. (2006). Optimal in-place learning and the lobe component analysis. In International Joint Conference on Neural Networks, 2006 (pp. 3887-3894). Piscataway, NJ: IEEE.
59. Weng, J., Zhang, Y., & Hwang, W. (2003). Candid covariance-free incremental principal component analysis. IEEE Trans. Pattern Analysis and Machine Intelligence, 25(8), 1034-1040.
60. Wiskott, L. (2003). Estimating driving forces of nonstationary time series with slow feature analysis. Arxiv preprint cond-mat/0312317.
61. Wiskott, L., Berkes, P., Franzius, M., Sprekeler, H., &Wilbert, N. (2011). Slow featureanalysis. Scholarpedia, 6(4), 52-82.
62. Wiskott, L., & Sejnowski, T. (2002). Slow feature analysis: Unsupervised learning of invariances. Neural Computation, 14(4), 715-770.
63. Xu, L., Oja, E., & Suen, C. (1992). Modified Hebbian learning for curve and surface fitting. Neural Networks, 5(3), 441-457.
64. Zhang, Y., &Weng, J. (2001). Convergence analysis of complementary candid incremental principal component analysis (Tech. Rep. MSU-CSE-01-23). East Lansing: Michigan State University.
65. Zito, N., Wilbert, L. W., & Berkes, P. (2008). Modular toolkit for data processing (MDP): A Python data processing framework. Frontiers in Neuroinformatics, 2.