The geometry of algorithms with orthogonality constraints

SIAM Journal on Matrix Analysis and Applications

Volumn 20, Issue 2, 1998, Pages 303-353

  Edelman, Alan ^a   Arias, TOMÁS A ^a   Smith, Steven T ^b  

^a MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^b MIT LINCOLN LABORATORY   (United States)

Author keywords

Conjugate gradient; Eigenvalue optimization; Eigenvalues and eigenvectors; Grassmann manifold; Invariant subspace; Newton's method; Orthogonality constraints; Rayleigh quotient iteration; Sequential quadratic programming; Stiefel manifold

Indexed keywords

EID: 0032216898     PISSN: 08954798     EISSN: None     Source Type: Journal    
DOI: 10.1137/S0895479895290954     Document Type: Article

References (97)

1. D. ABBOTT, ED., The Biographical Dictionary of Scientists; Mathematicians, P. Bedrick Books, New York, 1986.
2. B. M. ALSÉN, Multiple Step Gradient Iterative Methods for Computing Eigenvalues of Large Symmetric Matrices, Tech. report UMINF-15, University of Umeå, Sweden, 1971.
3. I. ANDERSSON, Experiments with the Conjugate Gradient Algorithm for the Determination of Eigenvalues of Symmetric Matrices, Tech. report UMINF-4.71, University of Umeå, Sweden, 1971.
4. T. A. ARIAS AND J. D. JOANNOPOULOS, Ab initio theory of dislocation interactions: From close-range spontaneous annihilation to the long-range continuum limit, Phys. Rev. Lett., 73 (1994), pp. 680-683.
5. T. A. ARIAS, M. C. PAYNE, AND J. D. JOANNOPOULOUS, Ab initio molecular dynamics: Analytically continued energy functionals and insights into iterative solutions, Phys. Rev. Lett., 71 (1992), pp. 1077-1080.
6. G. BIENVENU AND L. KOPP, Optimality of high resolution array processing using the eigensystem approach, IEEE Trans. Acoust., Speech, Signal Processing, ASSP-31 (1983), pp. 1235-1248.
7. P. T. BOGGS AND J. W. TOLLE, Sequential quadratic programming, Acta Numerica, (1995), pp. 199-242.
8. W. M. BOOTHBY, An Introduction to Differentiable Manifolds and Riemannian Geometry, 2nd ed., Academic Press, New York, 1986.
9. W. W. BRADBURY AND R. FLETCHER, New iterative methods for solutions of the eigenproblem, Numer. Math., 9 (1966), pp. 259-267.
10. R. W. BROCKETT, Dynamical systems that learn subspaces, in Mathematical Systems Theory: The Influence of R. E. Kalman, Springer-Verlag, Berlin, 1991.
11. K. BROMMER, M. NEEDELS, B. E. LARSON, AND J. D. JOANNOPOULOS, Ab initio theory of the Si(111)-(7×7) surface reconstruction: A challenge for massively parallel computation, Phys. Rev. Lett., 68 (1992), pp. 1355-1358.
12. F. BUDA, R. CAR, AND M. PARRINELLO, Thermal expansion of c-Si via ab initio molecular dynamics., Phys. Rev. B, 41 (1990), pp. 1680-1683.
13. R. CAR AND M. PARRINELLO, Unified approach for molecular dynamics and density-functional theory, Phys. Rev. Lett., 55 (1985), pp. 2471-2474.
14. B. CHAMPAGNE, Adaptive eigendecomposition of data covariance matrices based on first-order perturbations, IEEE Trans. Signal Processing, 42 (1994), pp. 2758-2770.
15. F. CHATELIN, Simultaneous newton's iteration for the eigenproblem, Comput. Suppl., 5 (1984), pp. 67-74.
16. F. CHATELIN, Eigenvalues of Matrices, John Wiley & Sons, New York, 1993.
17. F. CHATELIN AND V. FRAYSSÉ, Lectures on Finite Precision Computations, SIAM, Philadelphia, PA, 1996.
18. I. CHAVEL, Riemannian Geometry - A Modern Introduction, Cambridge University Press, London, 1993.
19. H. CHEN, T. K. SARKAR, S. A. DIANAT, AND J. D. BRULÉ, Adaptive spectral estimation by the conjugate gradient method, IEEE Trans. Acoust., Speech, Signal Processing, ASSP-34 (1986), pp. 272-284.
20. P. COMON AND G. H. GOLUB, Tracking a few extreme singular values and vectors in signal processing, Proc. IEEE, 78 (1990), pp. 1327-1343.
21. S. H. CRANDALL, Rotordynamic software, in Rotating Machinery Dynamics, Proceedings of the Third International Symposium on Transport Phenomena and Dynamics of Rotating Machinery (ISROMAC-3), J. H. Kim and W.-J. Wang, eds., Hemisphere Publishing, New York, 1992, pp. 3-21.
22. M. CROUZEIX, B. PHILIPPE, AND M. SADKANE, The Davidson method, SIAM J. Sci. Comput., 15 (1994), pp. 62-76.
23. J. K. CULLUM, The simultaneous computation of a few of the algebraically largest and smallest eigenvalues of a large, sparse, symmetric matrix, BIT, 18 (1978), pp. 265-275.
24. J. K. CULLUM AND R. A. WILLOUGHBY, Lanczos Algorithms for Large Symmetric Eigenvalue Computations, Vol. 1, Birkhäuser, Stuttgart, 1985.
25. B. DATTA, Numerical Linear Algebra and Applications, Brooks/Cole, Pacific Grove, CA, 1995.
26. E. R. DAVIDSON, The iterative calculation of a few of the lowest eigenvalues and corresponding eigenvectors of large real symmetric matrices, J. Comput. Phys., 17 (1975), pp. 87-94.
27. B. DE MOOR, The Riemannian singular value decomposition, in Proc. 3rd Internat. Workshop on SVD and Signal Processing, Vol. 3, M. Moonen and B. D. Moor, eds., Elsevier Science, 1995, pp. 61-78.
28. J. W. DEMMEL, Three methods for refining estimates of invariant subspaces, Computing, 38 (1987), pp. 43-57.
29. J. J. DONGARRA, C. B. MOLER, AND J. H. WILKINSON, Improving the accuracy of computed eigenvalues and eigenvectors, SIAM J. Numer. Anal., 20 (1983), pp. 23-45.
30. A. EDELMAN, Large dense numerical linear algebra in 1993: The parallel computing influence, J. Supercomputing Applications, 7 (1993), pp. 113-128.
31. A. EDELMAN AND S. T. SMITH, On conjugate gradient-like methods for eigen-like problems, BIT, 36 (1996), pp. 494-508. See also Proc. Linear and Nonlinear Conjugate Gradient-Related Methods, Loyce Adams and J. L. Nazareth, eds., SIAM, Philadelphia, PA, 1996.
32. A. EDELMAN AND S. T. SMITH, On conjugate gradient-like methods for eigen-like problems, BIT, 36 (1996), pp. 494-508. See also Proc. Linear and Nonlinear Conjugate Gradient-Related Methods, Loyce Adams and J. L. Nazareth, eds., SIAM, Philadelphia, PA, 1996.
33. L. ELDÉN, Algorithms for the regularization of ill-conditioned least-squares problems, BIT, 17 (1977), pp. 134-145.
34. R. L. Fox AND M. P. KAPOOR, A miminimization method for the solution of the eigenproblem arising in structural dynamics, in Proc. 2nd Conf. Matrix Methods in Structural Mechanics, L. Berke, R. M. Bader, W. J. Mykytow, J. S. Przemieniecki, and M. H. Shirk, eds., Wright-Patterson Air Force Base, OH, 1969, pp. 271-306.
35. I. FRIED, Gradient methods for finite element eigenproblems, AIAA J., 7 (1969), pp. 739-741.
36. I. FRIED, Optimal gradient minimization scheme for finite element eigenproblems, J. Sound Vibration, 20 (1972), pp. 333-342.
37. Z. FU AND E. M. DOWLING, Conjugate gradient eigenstructure tracking for adaptive spectral estimation, IEEE Trans. Signal Processing, 43 (1995), pp. 1151-1160.
38. D. R. FUHRMANN, An algorithm for subspace computation, with applications in signal processing, SIAM J. Matrix Anal. Appl., 9 (1988), pp. 213-220.
39. D. R. FUHRMANN AND B. LIU, An iterative algorithm for locating the minimal eigenvector of a symmetric matrix, in Proc. IEEE Int. Conf. Acoust., Speech, Signal Processing, 1984, pp. 45.8.1-4.
40. M. GERADIN, The computational efficiency of a new minimization algorithm for eigenvalue analysis, J. Sound Vibration, 19 (1971), pp. 319-331.
41. P. E. GILL AND W. MURRAY, The computation of Lagrange multipier estimates for constrained minimization, Math. Programming, 17 (1979), pp. 32-60.
42. P. E. GILL, W. MURRAY, AND M. H. WRIGHT, Practical Optimization, 2nd ed., Academic Press, New York, 1981.
43. M. J. GILLAN, Calculation of the vacancy formation energy in aluminium. J. Physics, Condensed Matter, 1 (1989), pp. 689-711.
44. S. GOEDECKER AND L. COLOMBO, Efficient linear scaling algorithm for tight-binding molecular dynamics, Phys. Rev. Lett., 73 (1994), pp. 122-125.
45. G. GOLUB AND D. O'LEARY, Some history of the conjugate gradient and Lanczos methods, SIAM Rev., 31 (1989), pp. 50-102.
46. G. H. GOLUB AND C. F. V. LOAN, Matrix Computations, 2nd ed., Johns Hopkins University Press, Baltimore, MD, 1989.
47. R. HAIMI-COHEN AND A. COHEN, Gradient-type algorithms for partial singular value decomposition, IEEE Trans. Pattern. Anal. Machine Intell., PAMI-9 (1987), pp. 137-142.
48. S. HELGASON, Differential Geometry, Lie Groups, and Symmetric Spaces, Academic Press, New York, 1978.
49. H. G. GRASSMANN, Die Ausdehnungslehre, Enslin, Berlin, 1862.
50. M. R. HESTENES AND E. STIEFEL, Methods of conjugate gradients for solving linear systems, J. Res. National Bureau of Standards, 49 (1952), pp. 409-436.
51. P. HOHENBERG AND W. KOHN, Inhomogeneous electron gas. Phys. Rev., 136 (1964), pp. B864-B871.
52. S. IHARA, S. L. HO, T. UDA, AND M. HIRAO, Ab initio molecular-dynamics study of defects on the reconstructed Si(001) surface, Phys. Rev. Lett., 65 (1990), pp. 1909-1912.
53. W. KAHAN AND J. DEMMEL, Personal communication, 1994-96.
54. J. KARHUNEN, Adaptive algorithms for estimating eigenvectors of correlation type matrices, in Proc. IEEE Int. Conf. Acoust., Speech, Signal Processing, 1984, pp. 14.6.1-4.
55. S. KOBAYASHI AND K. NOMIZU, Foundations of Differential Geometry, Vois. 1 and 2, Wiley, New York, 1969.
56. W. KOHN AND L. J. SHAM, Self-consistent equations including exchange and correlation effects, Phys. Rev., 140 (1965), pp. A1133-A1138.
57. R. A. LIPPERT, Personal communication (see also http://www.mit.edu/people/ripper/grass/ grassmann.html), 1995.
58. G. MATHEW, V. U. REDDY, AND S. DASGUPTA, Adaptive estimation of eigensubspace, IEEE Trans. Signal Processing, 43 (1995), pp. 401-411.
59. M. MOONEN, P. VAN DOOREN, AND J. VANDEWALLE, A singular value decomposition updating algorithm for subspace tracking, SIAM J. Matrix Anal. Appl., 13 (1992), pp. 1015-1038.
60. S. G. NASH AND A. SOFER, Linear and Nonlinear Programming, McGraw-Hill, New York, 1995.
61. M. NEEDELS, J. D. JOANNOPOULOS, Y. BAR-YAM, AND S. T. PANTELIDES, Oxygen complexes in silicon, Phys. Rev. B, 43 (1991), pp. 4208-4215.
62. M. NEEDELS, M. C. PAYNE, AND J. D. JOANNOPOULOS, High order reconstructions of the Ge(100) surface. Phys. Rev. B, 38 (1988), pp. 5543-5546.
63. B. O'NEILL, Semi-Riemannian Geometry with Applications to Relativity, Academic Press, New York, 1983.
64. M. L. OVERTON AND R. S. WOMERSLEY, Second derivatives for optimizing eigenvalues of symmetric matrices, SIAM J. Matrix Anal. Appl., 16 (1995), pp. 698-718.
65. N. L. OWSLEY, Adaptive data orthogonalization, in Proc. IEEE Internat. Conf. Acoust., Speech, Signal Processing, 1978, pp. 109-112.
66. B. PARLETT, The Symmetric Eigenvalue Problem, Prentice-Hall, Englewood Cliffs, NJ, 1980.
67. M. C. PAYNE, M. P. TETER, D. C. ALLAN, T. A. ARIAS, AND J. D. JOANNOPOULOS, Iterative minimization techniques for ab initio total-energy calculations: Molecular dynamics and conjugate gradients, Rev. Mod. Phys, 64 (1992), pp. 1045-1097.
68. A. PERDON AND G. GAMBOLATI, Extreme eigenvalues of large sparse matrices by Rayleigh quotient and modified conjugate gradients, Comput. Methods Appl. Mech. Engrg., 56 (1986), pp. 251-264.
69. R. ROY AND T. KAILATH, ESPRIT - estimation of signal parameters via rotational invariance techniques, IEEE Trans. Acoust., Speech, Signal Processing, 37 (1989), pp. 984-995.
70. A. RUHE, Iterative eigenvalue algorithms for large symmetric matrices, in Numerische Behandlung von Eigenwertaaufgaben Oberwolfach 1972, Internat. Series Numerical Math., Vol. 24, 1974, pp. 97-115.
71. A. H. SAMEH AND J. A. WISNIEWSKI, A trace minimization algorithm for the generalized eigenvalue problem, SIAM J. Numer. Anal., 19 (1982), pp. 1243-1259.
72. T. P. SARKAR AND N. SANGRUJI, An adaptive nulling system for a narrow-band signal with a look direction constraint utilizing the conjugate gradient method, IEEE Trans. Antennas, Propagation, 37 (1989), pp. 940-944.
73. R. O. SCHMIDT, Multiple emitter location and signal parameter estimation, IEEE Trans. Antennas, Propagation, AP-34 (1986), pp. 276-280. Reprinted from Proc. RADC Spectrum Estimation Workshop, Griffiss Air Force Base, NY, 1979.
74. R. O. SCHMIDT, Multiple emitter location and signal parameter estimation, IEEE Trans. Antennas, Propagation, AP-34 (1986), pp. 276-280. Reprinted from Proc. RADC Spectrum Estimation Workshop, Griffiss Air Force Base, NY, 1979.
75. R. SCHREIBER, Implementation of adaptive array algorithms, IEEE Trans. Acoust., Speech, Signal Processing, ASSP-34 (1986), pp. 1038-1045.
76. G. L. G. SLEIJPEN AND H. A. VAN DER VORST, A Jacobi-Davidson iteration method for linear eigenvalue problems, SIAM J. Matrix Anal. Appl., 17 (1996), pp. 401-425.
77. S. T. SMITH, Geometric Optimization Methods for Adaptive Filtering, Ph.D. thesis, Harvard University, Cambridge, MA, 1993.
78. S. T. SMITH, Optimization techniques on Riemannian manifolds, in Fields Institute Communications, Vol. 3, AMS, Providence, RI, 1994, pp. 113-146.
79. S. T. SMITH, Space-Time clutter covariance matrix computation and interference subspace tracking, in Proc. 29th Asilomar Conf. Signals, Systems, Computers, Vol. 2, 1995, pp. 1193-1197.
80. S. T. SMITH, Subspace tracking with full rank updates, 31st Asilomar Conf. Signals, Systems, Computers, Vol. 1, 1997, pp. 793-797.
81. M. SPIVAK, A Comprehensive Introduction to Differential Geometry, Vols. 1-3, 2nd ed., Publish or Perish, Houston, TX, 1979.
82. G. W. STEWART, Error and perturbation bounds for subspaces associated with certain eigenvalue problems, SIAM Rev., 15 (1973), pp. 727-764.
83. G. W. STEWART, An updating algorithm for subspace tracking, IEEE Trans. Signal Processing, 40 (1992), pp. 1535-1541.
84. E. STIEFEL, Richtungsfelder und fernparallelismus in n-dimensionalem mannig faltigkeiten, Commentarii Math. Helvetici, 8 (1935-1936), pp. 305-353.
85. M. P. TETER, M. C. PAYNE, AND D. C. ALLAN, Solution of Schrödinger's equation for large systems, Phys. Rev. B, 40 (1989), pp. 12255-12263.
86. C. G. VAN DE WALLE, Y. BAR-YAM, AND S. T. PANTELIDES, Theory of hydrogen diffusion and reactions in crystalline silicon, Phys. Rev. Lett., 60 (1988), pp. 2761-2764.
87. J. WARD, Space-Time adaptive processing for airborne radar, Tech. report 1015, DTIC No. ESC-TR-94-109, MIT Lincoln Laboratory, December 1994. See also Proc. IEEE Internat. Conf. Acoust., Speech, Signal Processing, Detroit, MI, May 1995, Vol. 5, pp. 2809-2812.
88. J. WARD, Space-Time adaptive processing for airborne radar, Tech. report 1015, DTIC No. ESC-TR-94-109, MIT Lincoln Laboratory, December 1994. See also Proc. IEEE Internat. Conf. Acoust., Speech, Signal Processing, Detroit, MI, May 1995, Vol. 5, pp. 2809-2812.
89. R. C. WARD AND L. J. GRAY, Eigensystem computation for skew-symmetric matrices and a class of symmetric matrices, ACM Trans. Math. Software, 4 (1978), pp. 278-285.
90. F. W. WARNER, Foundations of Differentiable Manifolds and Lie Groups, Springer-Verlag, New York, 1983.
91. J. H. WILKINSON, The Algebraic Eigenvalue Problem, Clarendon Press, Oxford, England, 1965.
92. Y.-C. WONG, Differential geometry of Grassmann manifolds, Proc. Nat. Acad. Sci. U.S.A., 57 (1967), pp. 589-594.
93. G. XU AND T. KAILATH, Fast estimation of principal eigenspace using Lanczos algorithm, SIAM J. Matrix Anal. Appl., 15 (1994), pp. 974-994.
94. G. XU AND T. KAILATH, Fast subspace decomposition, IEEE Trans. Signal Processing, 42 (1994), pp. 539-551.
95. J.-F. YANG AND M. KAVEH, Adaptive eigenspace algorithms for direction or frequency estimation and tracking, IEEE Trans. Acoust., Speech, Signal Processing, 36 (1988), pp. 241-251.
96. X. YANG, T. P. SARKAR, AND E. ARVAS, A survey of conjugate gradient algorithms for solution of extreme eigen-problems of a symmetric matrix, IEEE Trans. Acoust., Speech, Signal Processing, 37 (1989), pp. 1550-1556.
97. K.-B. YU, Recursive updating the eigenvalue decomposition of a covariance matrix, IEEE Trans. Signal Processing, 39 (1991), pp. 1136-1145.