Optimization by Solving Equations

Before discussing specific optimization algorithms, we first we show that the problems of unconstrained minimization and equation solving are equivalent as they can be converted to each other.

• To minimize an objective function #tex2html_wrap_inline7080#, we can set its gradient vector #math355##tex2html_wrap_inline7082# to zero and solve the resulting equation system:
  #math356#   #tex2html_wrap_indisplay18659#
  (4)
  The solution #tex2html_wrap_inline7084# of this equation system, called a <#78#>critical, stationary, or stable point<#78#> of #tex2html_wrap_inline7086#, is also the solution to the optimization problem, if it is not a saddle point, that either maximizes or minimizes #tex2html_wrap_inline7088#.

  This equation system can be solved by any of the methods discussed in the previous chapter, such as the Newton-Raphson method, which finds the root of a general function #math357##tex2html_wrap_inline7090# iteratively: #math358##tex2html_wrap_inline7092#. Here, specifically, to solve the equation #math359##tex2html_wrap_inline7094#, we first get the Jacobian #math360##tex2html_wrap_inline7096# of the gradient #math361##tex2html_wrap_inline7098# of #tex2html_wrap_inline7100#, which is the Hessian of #tex2html_wrap_inline7102#, and then carry out the iteration below to eventually find #tex2html_wrap_inline7104# that minimizes #tex2html_wrap_inline7106#:

  #math362#   #tex2html_wrap_indisplay18673#
  (5)
  The second equality is due to the fact that the Jacobian of the gradient #tex2html_wrap_inline7108# of function #math363##tex2html_wrap_inline7110# is the Hessian of the function, i.e., #math364##tex2html_wrap_inline7112#.

• To solve an equation system #math365##tex2html_wrap_inline7114#, we can minimize an objective function defined as
  #math366#   #tex2html_wrap_indisplay18679#
  (6)
  The solution that minimizes #tex2html_wrap_inline7116# is also the solution of the original equation. To see this, we again set the gradient of #tex2html_wrap_inline7118# zero to get as above:
  #math367#   #tex2html_wrap_indisplay18683#
  (7)
  As in general the Jacobian #math368##tex2html_wrap_inline7120# is a full rank matrix, the homogeneous equation above only has a zero solution #math369##tex2html_wrap_inline7122#, which is indeed the original equation.