A New Multi-Step Quasi-Newton Method for Unconstrained Optimization

ABSTRACT
Previous work on so-called “fixed-point” multi-step quasi-Newton methods for unconstrained optimization employed the latest iterate as the “base point” for calculating the parameter values needed to carry out the requisite interpolations in the variable- and gradient-spaces at each iteration. In this paper, we consider the use of an alternative base point and derive the formulae required to implement the new method in a computationally efficient manner. Numerical experiments are carried out to assess the usefulness of the new approach, by comparison with existing algorithms. On the basis of the experimental results, some conclusions are drawn.
2. The Secant Equation The matrix Bk is used in quasi-Newton methods to approximate the Hessian G(xk), as outlined above. It is reasonable to require that the approximation must be revised at each iteration to reflect the change that would occur in the Hessian. In order to update the approximation, it is necessary to derive a condition ensuring that the approximation is a suitable one; otherwise it would be likely that the desirable aspects of Newton’s method would be forfeited. This condition is known as the Secant (or quasi-Newton) equation, and is derived by Ford and Moghrabi (1994) as an approximation to the Newton equation,
1. Introduction We consider algorithms for the solution of the problem “minimise f(x)”, where x has dimension n and f is a given function possessing continuous first partial derivatives. In Newton’s method, it is assumed, further, that second partial derivatives exist and the search direction needed to compute the next iterate, xk+1, from the current one, xk, is calculated from G(xk) pk = -g(xk), where G(xk) is the Hessian and g(xk) is the gradient of the function evaluated at xk, and pk is the search direction at iteration k. Typically, a line search is then used to find (at least approximately) the minimum of the function along the chosen search direction, and in this way xk+1 is computed from xk. For a function of dimension n, the Hessian is a matrix of dimension (n x n), and so a set of linear equations must be solved at each iteration. This represents a substantial amount of computation during each step of the algorithm, and can be expected to have a serious impact on the time required to compute the minimum. Quasi-Newton
3பைடு நூலகம்
which (in turn) is obtained from an application of the Chain Rule to the vector function g(x(τ)):
∗ d x(τ ∗ ) d g( x(τ )) = G( x(τ )) . dτ dτ ∗
Here, x(τ) is any differentiable curve in the variable-space. If, specifically, we take {x(τ)} to be the line which interpolates the last two iterates xi+1 and xi : x(τ ) ≡ x i + τ s i where s i = x i +1 − x i and τ* is any specific value of τ. Then x(0) = xi, x(1) = xi+1 and dx = si . dτ It is not generally possible to compute an exact value for dg / dτ as this calculation requires the Hessian, which is not available to us. Therefore we estimate it using the values of g already known (since further values are assumed to be computationally expensive to calculate in practical problems). These values are g(x(0)) and g(x(1)), i.e. g(xi) and g(xi+1), and we obtain: d g( x(τ ∗ )) dτ ≅ g ( x i +1 ) − g ( x i ) = y .
methods were developed in an attempt to remove the disadvantages imposed by the use of the Hessian matrix in Newton’s method, whilst retaining (as far as possible) the convergence properties. The overheads involved with maintaining the Hessian are overcome by the use of an approximation to it, which is updated at each iteration to reflect the change which would occur in the actual Hessian. The search direction in quasi-Newton methods is computed thus (by analogy with the process described above for Newton’s Method):
i
We saw that xi+1 = x(1). Thus, we take τ* = 1, and the Newton equation then becomes: G( x i +1 ) s i ≅ y i . The Secant equation requires that this relationship is satisfied by the Hessian approximation as an equality, i.e. Bi +1 s i = y i ,
2
Bk pk = -g(xk), where Bk is an (n x n) matrix which approximates G(xk). (Of course, the direction pk resulting from solution of this linear system will differ, in general, from that arising from Newton’s Method.) The use of an approximation to the Hessian has removed the need for the user to derive and code the Hessian definition into a program that implements the Newton method, but this new definition of the search direction still involves solving an identicallysized system of linear equations at each iteration. This disadvantage is overcome by working with the inverse of the approximation. Defining Hk = Bk-1 gives the following equation to compute pk: pk = - Hk g(xk). The search direction is now computed by a matrix-vector multiplication, a computation of order n2, rather than having to solve a full linear system of equations, which is a computation of order n3.