A mathematical view of automatic differentiation

Acta Numerica

Volumn 12, Issue , 2003, Pages 321-398

  Griewank, Andreas ^a  

^a DRESDEN UNIVERSITY OF TECHNOLOGY   (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 24644437042     PISSN: 09624929     EISSN: None     Source Type: Journal    
DOI: 10.1017/S0962492902000132     Document Type: Review

References (99)

1. A. Aho, J. Hopcroft and J. Ullman (1974), The Design and Analysis of Computer Algorithms, Addison-Wesley, Reading, MA.
2. R. Anderssen and P. Bloomfield (1974), 'Numerical differentiation proceedings for non-exact data', Numer. Math. 22, 157-182.
3. W. Ball (1969), Material and Energy Balance Computations, Wiley, pp. 560-566.
4. R. Becker and R. Rannacher (2001), An optimal control approach to error control and mesh adaptation in finite element methods, in Acta Numerica, Vol. 10, Cambridge University Press, pp. 1-102.
5. C. Bennett (1973), 'Logical reversibility of computation', IBM J. Research Development 17, 525-532.
6. M. Berz, C. Bischof, G. Corliss and A. Griewank, eds (1996), Computational Differentiation: Techniques, Applications, and Tools, SIAM, Philadelphia, PA.
7. C. Bischof, G. Corliss and A. Griewank (1993), 'Structured second- and higher-order derivatives through univariate Taylor series', Optim. Methods Software 2, 211-232.
8. C. Bischof, A. Carle, P. Khademi and A. Mauer (1996), 'The ADIFOR 2.0 system for the automatic differentiation of Fortran 77 programs', IEEE Comput. Sci. Engr. 3, 18-32.
9. T. Braconnier and P. Langlois (2001), From rounding error estimation to automatic correction with AD, in Corliss et al. (2001), Chapter 42, pp. 333-339.
10. D. Cacuci, C. Weber, E. Oblow and J. Marable (1980), 'Sensitivity theory for general systems of nonlinear equations', Nucl. Sci. Engr. 88, 88-110.
11. J.-B. Caillau and J. Noailles (2001), Optimal control sensitivity analysis with AD, in Corliss et al. (2001), Chapter 11, pp. 105-111.
12. S. Campbell and R. Hollenbeck (1996), Automatic differentiation and implicit differential equations, in Berz et al. (1996), pp. 215-227.
13. S. Campbell, E. Moore and Y. Zhong (1994), 'Utilization of automatic differentiation in control algorithms', IEEE Trans. Automatic Control 39, 1047-1052.
14. B. Cappelaere, D. Elizondo and C. Faure (2001), Odyssée versus hand differentiation of a terrain modelling application, in Corliss et al. (2001), Chapter 7, pp. 71-78.
15. A. Carle and M. Fagan (1996), Improving derivative performance for CFD by using simplified recurrences, in Berz et al. (1996), pp. 343-351.
16. B. Christianson (1992), 'Automatic Hessians by reverse accumulation', IMA J. Numer. Anal 12, 135-150.
17. B. Christianson (1994), 'Reverse accumulation and attractive fixed points', Optim. Methods Software 3, 311-326.
18. B. Christianson (1998), 'Reverse accumulation and implicit functions', Optim. Methods Software 9, 307-322.
19. B. Christianson (1999), 'Cheap Newton steps for optimal control problems: Automatic differentiation and Pantoja's algorithm', Optim. Methods Software 10, 729-743.
20. B. Christianson, L. Dixon and S. Brown (1997), Automatic differentiation of computer programs in a parallel computing environment, in Applications of High Performance Computing in Engineering V (H. Power and J. C. Long, eds), Computational Mechanics Publications, Southampton, pp. 169-178.
21. T. Coleman and J. Moré (1984), 'Estimation of sparse Jacobian matrices and graph coloring problems', SIAM J. Numer. Anal. 20, 187-209.
22. T. Coleman and A. Verma (1996), Structure and efficient Jacobian calculation, in Berz et al. (1996), pp. 149-159.
23. A. Conn, N. Gould and P. Toint (1992), LANCELOT, a Fortran Package for Large-0 Scale Nonlinear Optimization (Release A), Vol. 17 of Computational Mathematics, Springer, Berlin.
24. G. Corliss (1991), Automatic differentiation bibliography, in Griewank and Corliss (1991), pp. 331-353.
25. G. Corliss, C. Faure, A. Griewank, L. Hascoët and U. Naumann, eds (2001), Automatic Differentiation: From Simulation to Optimization, series in Computer and Information Science, Springer, New York.
26. F. Courty, A. Dervieux, B. Koobus and L. Hascoët (2002), Reverse automatic differentiation for optimum design: From adjoint state assembly to gradient computation, Technical report, INRIA Sophia-Antipolis.
27. A. Curtis, M. Powell and J. Reid (1974), 'On the estimation of sparse Jacobian matrices', J. Inst. Math. Appl. 13, 117-119.
28. L. C. W. Dixon (1991), Use of automatic differentiation for calculating Hessians and Newton steps, in Griewank and Corliss (1991), pp. 114-125.
29. J. Dunn and D. Bertsekas (1989), 'Efficient dynamic programming implementations of Newton's method for unconstrained optimal control problems', J. Optim. Theory Appl. 63, 23-38.
30. Faa di Bruno (1856), 'Note sur une nouvelle formule de calcule differentiel', Quar. J. Math. 1, 359-360.
31. C. Faure and U. Naumann (2001), Minimizing the tape size, in Corliss et al. (2001), Chapter 34, pp. 279-284.
32. S. A. Forth and T. Evans (2001), Aerofoil optimisation via AD of a multigrid cell-vertex Euler flow solver, in Corliss et al. (2001), Chapter 17, pp. 149-156.
33. Y. M. V. G. M. Ostrovskii and W. W. Borisov (1971), 'Über die Berechnung von Ableitungen', Wissenschaftliche Zeitschrift der Technischen Hochschule für Chemie 13, 382-384.
34. R. Giering and T. Kaminski (1998), 'Recipes for adjoint code construction', ACM Trans. Math. Software 24, 437-474.
35. R. Giering and T. Kaminski (2000), On the performance of derivative code generated by TAMC. Manuscript, FastOpt, Hamburg, Germany. Submitted to Optim. Methods Software See www.FastOpt.de/papers/perftamc.ps.gz.
36. R. Giering and T. Kaminski (2001 a), 'Automatic sparsity detection'. Draft.
37. R. Giering and T. Kaminski (2001b), Generating recomputations in reverse mode AD, in Corliss et al. (2001), Chapter 33, pp. 271-278.
38. J. C. Gilbert (1992), 'Automatic differentiation and iterative processes', Optim. Methods Software 1, 13-21.
39. M. Giles (2001), On the iterative solution of adjoint equations, in Corliss et al. (2001), pp. 145-151.
40. M. B. Giles and N. A. Pierce (2001), 'An introduction to the adjoint approach to design', Flow, Turbulence and Combustion 65, 393-415.
41. M. B Giles and E. Süli (2002), Adjoint methods for PDEs: a posteriori error analysis and postprocessing by duality, in Acta Numerica, Vol. 11, Cambridge University Press, pp. 145-236.
42. M. Gockenbach, M. J. Petro and W. Symes (1999), 'C++ classes for linking optimization with complex simulations', ACM Trans. Math. Software 25, 191-212.
43. A. Griewank (2000), Evaluating Derivatives, Principles and Techniques of Algorithmic Differentiation, Vol. 19 of Frontiers in Applied Mathematics, SIAM, Philadelphia.
44. A. Griewank and G. Corliss, eds (1991), Automatic Differentiation of Algorithms: Theory, Implementation, and Application, SIAM, Philadelphia, PA.
45. A. Griewank and C. Faure (2002), 'Reduced functions, gradients and Hessians from fixed point iteration for state equations', Numer. Alg. 30, 113-139.
46. A. Griewank and C. Mitev (2002), 'Detecting Jacobian sparsity patterns by Bayesian probing', Math. Prog. 93, 1-25.
47. A. Griewank and U. Naumann (2003a), Accumulating Jacobians by vertex, edge or face elimination, in Proceedings of the 6th African Conference on Research in Computer Science (L. Andriamampianina, B. Philippe, E. Kamgnia and M. Tchuente, eds), Imprimerie Saint Paul, Camerun and INRIA, France, pp. 375-383.
48. A. Griewank and U. Naumann (2003b), 'Accumulating Jacobians as chained sparse matrix products', Math. Prog. 95, 555-571.
49. A. Griewank and A. Verma (2003), 'On Newsam-Ramsdell-seeds for calculating sparse Jacobians'. In preparation.
50. A. Griewank and A. Walther (2000), 'Revolve: An implementation of checkpointing for the reverse or adjoint mode of computational differentiation', Trans. Math. Software 26, 19-45.
51. A. Griewank, C. Bischof, G. Corliss, A. Carle and K. Williamson (1993), 'Derivative convergence for iterative equation solvers', Optim. Methods Software 2, 321-355.
52. A. Griewank, J. Utke and A. Walther (2000), 'Evaluating higher derivative tensors by forward propagation of univariate Taylor series', Math. Comp. 69, 1117-1130.
53. J. Grimm, L. Pottier and N. Rostaing-Schmidt (1996), Optimal time and minimum space-time product for reversing a certain class of programs, in Berz et al. (1996), pp. 95-106.
54. S. Hague and U. Naumann (2001), Present and future scientific computation environments, in Corliss et al. (2001), Chapter 5, pp. 55-62.
55. L. Hascoët (2001), The data-dependence graph of adjoint programs, Research Report 4167, INRIA, Sophia-Antipolis, France. See www.inria.fr/rrrt/ rr-4167.html.
56. L. Hascoët, S. Fidanova and C. Held (2001), Adjoining independent computations, in Corliss et al. (2001), Chapter 35, pp. 285-290.
57. M. Hinze and T. Slawig (2002), Adjoint gradients compared to gradients from algorithmic differentiation in instantaneous control of the Navier-Stokes equations, Preprint, TU Dresden.
58. M. Hinze and A. Walther (2002), An optimal memory-reduced procedure for calculating adjoints of the instationary Navier-Stokes equations, Preprint MATH-NM-06-2002, TU Dresden.
59. J. Horwedel (1991), GRESS: A preprocessor for sensitivity studies on Fortran programs, in Griewank and Corliss (1991), pp. 243-250.
60. A. S. Hossain and T. Steihaug (1998), 'Computing a sparse Jacobian matrix by rows and columns', Optim. Methods Software 10, 33-48.
61. S. Hossain and T. Steihaug (2002), Sparsity issues in the computation of Jacobian matrices, Technical Report 223, Department of Computer Science, University of Bergen, Norway.
62. M. Iri, T. Tsuchiya and M. Hoshi (1988), 'Automatic computation of partial derivatives and rounding error estimates with applications to large-scale systems of nonlinear equations', Comput. Appl. Math. 24, 365-392.
63. Original Japanese version appeared in J. Information Processing 26 (1985), 1411-1420.
64. L. Kantorovich (1957), 'Ob odnoj matematicheskoj simvolike, udobnoj pri provedenii vychislenij na mashinakh', Doklady Akademii Nauk. SSSR 113, 738-741.
65. G. Kedem (1980), 'Automatic differentiation of computer programs', ACM Trans. Math. Software 6, 150-165.
66. D. Keyes, P. Hovland, L. McInnes and W. Samyono (2001), Using automatic differentiation for second-order matrix-free methods in PDE-constrained optimization, in Corliss et al. (2001), Chapter 3, pp. 33-48.
67. D. Knuth (1973), The Art of Computer Programming, Vol. 1 of Computer Science and Information Processing, Addison-Wesley, MI.
68. S. Linnainmaa (1976), 'Taylor expansion of the accumulated rounding error', BIT (Nordisk Tidskrift for Informationsbehandling) 16, 146-160.
69. S. Linnainmaa (1983), 'Error linearization as an effective tool for experimental analysis of the numerical stability of algorithms', BIT 23, 346-359.
70. R. Lohner (1992), Verified computing and programs in PASCAL-XSC, with examples, Habilitationsschrift, Institute for Applied Mathematics, University of Karlsruhe.
71. M. Mancini (2001), A parallel hierarchical approach for automatic differentiation, in Corliss et al. (2001), Chapter 27, pp. 225-230.
72. G. Marcuk (1996), Adjoint Equations and Perturbation Algorithms in Nonlinear Problems, CRC Press, Boca Raton, FL.
73. H. Maurer and D. Augustin (2001), Sensitivity analysis and real-time control of parametric optimal control problems using boundary value methods, in Online Optimization of Large Scale Systems (M. Grötschel, S. Krumke and J. Rambau, eds), Springer, Berlin/Heidelberg/New York, Chapter I, pp. 17-55.
74. B. Mohammadi and O. Pironneau (2001), Applied Shape Optimization for Fluids, series in Numerical Mathematics and Scientific Computation, Clarendon Press, Oxford.
75. R. Moore (1979), Methods and Applications of Interval Analysis, SIAM, Philadelphia, PA.
76. U. Naumann (1999), Efficient calculation of Jacobian matrices by optimized application of the chain rule to computational graphs, PhD thesis, Institute of Scientific Computing, Germany.
77. U. Naumann (2001), Elimination techniques for cheap Jacobians, in Corliss et al. (2001), Chapter 29, pp. 241-246.
78. R. Neidinger (200x), Directions for computing multivariate Taylor series. Technical report, Davidson College, Davidson. To appear in Math. Comp.
79. P. Newman, G.-W. Hou, H. Jones, A. Taylor and V. Korivi (1992), Observations on computational methodologies for use in large-scale, gradient-based, multi-disciplinary design incorporating advanced CFD codes, Technical Memorandum 104206, NASA Langley Research Center. AVSCOM Technical Report 92-B-007.
80. G. Newsam and J. Ramsdell (1983), 'Estimation of sparse Jacobian matrices', SIAM J. Algebraic Discrete Methods 4, 404-417.
81. J. Ortega and W. Rheinboldt (1970), Iterative Solution of Nonlinear Equations in Several Variables, Academic Press, New York.
82. C. Pantelides (1988), 'The consistent initialization of differential-algebraic systems', SIAM J. Sci. Statist. Comput. 9, 213-231.
83. J. D. O. Pantoja (1988), 'Differential dynamic programming and Newton's method', Internat. J. Control 47, 1539-1553.
84. J. Pryce (1998), 'Solving high-index DAEs by Taylor series', Numer. Alg. 19, 195-211.
85. L. Rail (1983), Differentiation and generation of Taylor coefficients in Pascal-SC, in A New Approach to Scientific Computation (U. Kulisch and W. L. Miranker, eds), Academic Press, New York, pp. 291-309.
86. J. Restrepo, G. Leaf and A. Griewank (1998), 'Circumventing storage limitations in variational data assimilation studies', SIAM J. Sci. Comput. 19, 1586-1605.
87. K. Röbenack and K. Reinschke (2000), 'Graph-theoretic characterization of structural controllability for singular DAE', Z. Angew. Math. Mech. 8 Suppl. 1, 849-850.
88. K. Röbenack and K. Reinschke (2001), Nonlinear observer design using automatic differentiation, in Corliss et al. (2001), Chapter 15, pp. 133-138.
89. D. Rose and R. Tarjan (1978), 'Algorithmic aspects of vertex elimination on directed graphs', SIAM J. Appl. Math. 34, 177-197.
90. J. Sternberg (2002), Adaptive Umkehrschemata für Schrittfolgen mit nichtuniformen Kosten, Diploma Thesis, Institute of Scientific Computing, Germany.
91. F. Stummel (1981), 'Optimal error estimates for Gaussian elimination in floating-point arithmetic', GAMM, Numerical Analysis 62, 355-357.
92. M. Tadjouddine, S. A. Forth and J. Pryce (2001), AD tools and prospects for optimal AD in CFD flux Jacobian calculations, in Corliss et al. (2001), Chapter 30, pp. 247-252.
93. R. Tarjan (1983), 'Data structures and network algorithms', CBMS-NSF Reg. Conf. Ser. Appl. Math. 44, 131.
94. J. van der Snepscheut (1993), What Computing Is All About, Suppl. 2 of Texts and Monographs in Computer Science, Springer, Berlin.
95. D. Venditti and D. Darmofal (2000), 'Adjoint error estimation and grid adaptation for functional outputs: application to quasi-one-dimensional flow', J. Comput. Phys. 104, 204-227.
96. Y. Volin and G. Ostrovskii (1985), 'Automatic computation of derivatives with the use of the multilevel differentiating technique, I: Algorithmic basis', Comput. Math. Appl. 11, 1099-1114.
97. A. Walther (1999), Program reversal schedules for single- and multi-processor machines, PhD thesis, Institute of Scientific Computing, Germany.
98. A. Walther (2002), 'Adjoint based truncated Newton methods for equality constrained optimization'.
99. X. Zou, F. Vandenberghe, M. Pondeca and Y.-H. Kuo (1997), Introduction to adjoint techniques and the MM5 adjoint modeling system, NCAR Technical Note NCAR/TN-435-STR, Mesocale and Microscale Meteorology Division, National Center for Atmospheric Research, Boulder, Colorado.