Fdtxjr

In numerical analysis, Broyden's method is a quasi-Newton method for finding roots in k variables. It was originally described by C. G. Broyden in 1965.^[1]

Newton's method for solving f(x) = 0 uses the Jacobian matrix, J, at every iteration. However, computing this Jacobian is a difficult and expensive operation. The idea behind Broyden's method is to compute the whole Jacobian only at the first iteration and to do rank-one updates at other iterations.

In 1979 Gay proved that when Broyden's method is applied to a linear system of size n × n, it
terminates in 2 n steps,^[2] although like all quasi-Newton methods, it may not converge for nonlinear systems.

Description of the method

Solving single-variable equation

In the secant method, we replace the first derivative f′ at x_n with the finite-difference approximation:

f′(xn)≃f(xn)−f(xn−1)xn−xn−1,displaystyle f'(x_n)simeq frac f(x_n)-f(x_n-1)x_n-x_n-1,

and proceed similar to Newton's method:

xn+1=xn−f(xn)f′(xn),displaystyle x_n+1=x_n-frac f(x_n)f'(x_n),

where n is the iteration index.

Solving a system of nonlinear equations

Consider a system of k nonlinear equations

f(x)=0,displaystyle mathbf f (mathbf x )=mathbf 0 ,

where f is a vector-valued function of vector x:

x=(x1,x2,x3,…,xk),displaystyle mathbf x =(x_1,x_2,x_3,dotsc ,x_k),

f(x)=(f1(x1,x2,…,xk),f2(x1,x2,…,xk),…,fk(x1,x2,…,xk)).displaystyle mathbf f (mathbf x )=big (f_1(x_1,x_2,dotsc ,x_k),f_2(x_1,x_2,dotsc ,x_k),dotsc ,f_k(x_1,x_2,dotsc ,x_k)big ).

For such problems, Broyden gives a generalization of the one-dimensional Newton's method, replacing the derivative with the Jacobian J. The Jacobian matrix is determined iteratively, based on the secant equation in the finite-difference approximation:

Jn(xn−xn−1)≃f(xn)−f(xn−1),displaystyle mathbf J _n(mathbf x _n-mathbf x _n-1)simeq mathbf f (mathbf x _n)-mathbf f (mathbf x _n-1),

where n is the iteration index. For clarity, let us define:

fn=f(xn),displaystyle mathbf f _n=mathbf f (mathbf x _n),

Δxn=xn−xn−1,displaystyle Delta mathbf x _n=mathbf x _n-mathbf x _n-1,

Δfn=fn−fn−1,displaystyle Delta mathbf f _n=mathbf f _n-mathbf f _n-1,

so the above may be rewritten as

JnΔxn≃Δfn.displaystyle mathbf J _nDelta mathbf x _nsimeq Delta mathbf f _n.

The above equation is underdetermined when k is greater than one. Broyden suggests using the current estimate of the Jacobian matrix J_n−1 and improving upon it by taking the solution to the secant equation that is a minimal modification to J_n−1:

Jn=Jn−1+Δfn−Jn−1Δxn‖Δxn‖2ΔxnT.displaystyle mathbf J _n=mathbf J _n-1+frac Delta mathbf f _n-mathbf J _n-1Delta mathbf x _nDelta mathbf x _nDelta mathbf x _n^mathrm T .

This minimizes the following Frobenius norm:

‖Jn−Jn−1‖F.displaystyle

We may then proceed in the Newton direction:

xn+1=xn−Jn−1f(xn).displaystyle mathbf x _n+1=mathbf x _n-mathbf J _n^-1mathbf f (mathbf x _n).

Broyden also suggested using the Sherman–Morrison formula to update directly the inverse of the Jacobian matrix:

Jn−1=Jn−1−1+Δxn−Jn−1−1ΔfnΔxnTJn−1−1ΔfnΔxnTJn−1−1.displaystyle mathbf J _n^-1=mathbf J _n-1^-1+frac Delta mathbf x _n-mathbf J _n-1^-1Delta mathbf f _nDelta mathbf x _n^mathrm T mathbf J _n-1^-1Delta mathbf f _nDelta mathbf x _n^mathrm T mathbf J _n-1^-1.

This first method is commonly known as the "good Broyden's method".

A similar technique can be derived by using a slightly different modification to J_n−1. This yields a second method, the so-called "bad Broyden's method" (but see^[3]):

Jn−1=Jn−1−1+Δxn−Jn−1−1Δfn‖Δfn‖2ΔfnT.displaystyle mathbf J _n^-1=mathbf J _n-1^-1+frac Delta mathbf x _n-mathbf J _n-1^-1Delta mathbf f _nDelta mathbf f _n^mathrm T .

This minimizes a different Frobenius norm:

‖Jn−1−Jn−1−1‖F.mathbf J _n^-1-mathbf J _n-1^-1

Many other quasi-Newton schemes have been suggested in optimization, where one seeks a maximum or minimum by finding the root of the first derivative (gradient in multiple dimensions). The Jacobian of the gradient is called Hessian and is symmetric, adding further constraints to its update.

Other members of the Broyden class

Broyden has defined not only two methods, but a whole class of methods. Other members of this class have been added by other authors.

• The Davidon–Fletcher–Powell update is the only member of this class being published before the two members defined by Broyden.^[4]


• Schubert's or sparse Broyden algorithm – a modification for sparse Jacobian matrices.^[5]


• Klement (2014) – uses fewer iterations to solve many equation systems.^[6][7]

See also

• Secant method


• Newton's method


• Quasi-Newton method


• Newton's method in optimization


• Davidon-Fletcher-Powell formula


• Broyden-Fletcher-Goldfarb-Shanno (BFGS) method

References

External links

• Simple basic explanation: The story of the blind archer